Prerequisites

• Install git. Running the script requires cloning the tap repository using git. To install git, refer to these instructions.

• A local cloned docs repository. This is required to run the schema script. If you don’t have the repository cloned locally, refer to the prerequisites first.

• Install Clojure. Clojure is required to run the schema script. If you have Homebrew installed, run the following to install Clojure:

  brew install clojure
  

Step 1: Clone the tap repository

1. Open your Terminal and navigate to the desktop:

   cd ~/Desktop
   

2. Clone the tap repository, replacing <TAP-NAME> with the name of the tap. For example: bigcommerce or facebook-ads:

   git clone https://github.com/singer-io/tap-<TAP-NAME>.git
   

Step 2: Navigate to your local docs repository

1. In the Terminal, navigate to the local docs repository:

   cd /some/directory/location/docs
   

2. Navigate to scripts/tap-generate-docs:

   cd scripts/tap-generate-docs
   

Step 3: Run the script

In the scripts/tap-generate-docs folder, run the script. Replace some/directory/location with the location of the local tap repository.

For example: If you cloned the tap-bigcommerce repository to your desktop, the command might look like this:

clojure -Sdeps "$(< deps.edn)" -m tap-generate-docs /Users/erincochran/Desktop/tap-bigcommerce

The script will process the JSON schema files in the tap repository:

talendadmin’s-MacBook-Pro:tap-generate-docs erincochran$ clojure -Sdeps "$(< deps.edn)" -m tap-generate-docs /Users/erincochran/Desktop/tap-bigcommerce
Processing file: /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/coupons.json
Writing converted /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/coupons.json to coupons.md
Processing file: /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/orders.json
Writing converted /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/orders.json to orders.md
Processing file: /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/customers.json
Writing converted /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/customers.json to customers.md
Processing file: /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/products.json
Writing converted /Users/erincochran/Desktop/tap-bigcommerce/tap_bigcommerce/schemas/products.json to products.md

The script will save the processed schema files to the scripts/tap-generate-docs folder. The image below demonstrates this, with the exception of the deps.edn file:

Step 4: Move the schema files to the integration-schemas collection

Next, you’ll move the processed schema files to the integration’s folder in the integration-schemas collection.

Every integration has its own folder in the integration-schemas collection. In every integration’s folder is a set of subfolders specific to a version of the integration.

1. If the integration doesn’t already have a folder in the integration-schemas collection, create one. The folder should be named for the integration, in all lowercase. For example: bigcommerce
2. In the integration’s folder, create a subfolder named for the version the schema files pertain to. The folder should be named v<version>. For example: v1 for version 1, v2 for version 2, etc.

3. Move the schema files to the integration’s version folder in integration-schemas. For example:

   /integration-schemas
      - /bigcommerce
         - /v1
            - coupons.md
            - customers.md
            - orders.md
            - products.md
   

Before committing the changes, verify that there aren’t any schema files remaining in the scripts/tap-generate-docs folder.

Step 5: Fill in the schema files

Fill in the schema files for the integration. Refer to the Add SaaS integration tables guide and SaaS integration table schema template reference for more info.

Troubleshooting

• Invalid JSON file format

Invalid JSON file format

The following error usually indicates an issue with how the JSON schema file is formatted:

/some/file/location/tap-<name>/tap_<name>/schemas/<schema>.json is not a valid schema file

The schema script requires JSON schema files to be formatted in a specific way. The top-level object in the file must contain a properties object and a type to be considered valid. For example: The example on the left is considered valid for the script, while the example on the right isn’t.

{
  "type": "object",
  "additionalProperties": false,
  "inclusion": "available",
  "properties": {
    "id": {
      "type": "integer",
      "inclusion": "automatic"
    },
    "createdAt": {
      "type": "string",
      "format": "date-time",
      "inclusion": "automatic"
    },
    "updatedAt": {
      "type": "string",
      "format": "date-time",
      "inclusion": "automatic"
    },
    "active": {
      "type": ["boolean", "null"],
      "inclusion": "available"
    },
    "description": {
      "type": ["string", "null"],
      "inclusion": "available"
    },
    "name": {
      "type": "string",
      "inclusion": "available"
    },
    "programId": {
      "type": ["integer", "null"],
      "inclusion": "available"
    },
    "programName": {
      "type": ["string", "null"],
      "inclusion": "available"
    },
    "type": {
      "type": "string",
      "inclusion": "available"
    },
    "workspaceName": {
      "type": ["string", "null"],
      "inclusion": "available"
    }
  }
}

{
  "tap_stream_id": "campaigns",
  "type": "object",
  "stream": "campaigns",
  "key_properties": ["id"],
  "schema": {
    "type": "object",
    "additionalProperties": false,
    "inclusion": "available",
    "properties": {
      "id": {
        "type": "integer",
        "inclusion": "automatic"
      },
      "createdAt": {
        "type": "string",
        "format": "date-time",
        "inclusion": "automatic"
      },
      "updatedAt": {
        "type": "string",
        "format": "date-time",
        "inclusion": "automatic"
      },
      "active": {
        "type": ["boolean", "null"],
        "inclusion": "available"
      },
      "description": {
        "type": ["string", "null"],
        "inclusion": "available"
      },
      "name": {
        "type": "string",
        "inclusion": "available"
      },
      "programId": {
        "type": ["integer", "null"],
        "inclusion": "available"
      },
      "programName": {
        "type": ["string", "null"],
        "inclusion": "available"
      },
      "type": {
        "type": "string",
        "inclusion": "available"
      },
      "workspaceName": {
        "type": ["string", "null"],
        "inclusion": "available"
      }
    }
  }
}

To get the script to run successfully, edit the local JSON files to make them valid for the script. For example: To make the Invalid formatting example above work with the script, you could remove the top-level object (everything between "tap_stream_id" and "schema": {, and the closing } at the bottom).

To double-check your work, use a JSON validator like JSONLint to make sure the JSON is valid.

Last updated: 22 February 2022