📘 Connector Status: Alpha
Consult Dadosfera support to determine whether this connector can collect data from your API. As an Alpha connector, it works only in specific cases.
Features
| Feature | Supported | Notes |
|---|---|---|
| Header authentication | Yes | |
| Pagination | Yes | |
| Full Load collection | Yes | |
| Incremental collection | Yes |
Quick Start
To create a pipeline, open the Collect module, go to "Pipelines", and click "New Pipeline".
Choose the Data Source
Use an existing source or register a new one
Connection Parameters (Source Registration)
Authentication uses headers, which are required for private APIs. Public APIs do not require an authentication header.
| Field name | Description |
|---|---|
| API | Define whether the API to collect is private or public |
| Headers (optional) | Key-value pairs containing sensitive authentication data, configured according to your API settings. You can register any number of headers; all values entered here are stored as secrets. |
**Exemplo de Header**
Chave: "Basic"
Valor: "YWlyZmxvdzptaJLgbG96"
Pipeline Information
Assign a name and a brief description to your pipeline.
Pipeline Configuration Parameters
Step 1: URL
| Field name | Description | Example |
|---|---|---|
| API URL | API base URL or endpoint | https://my-api.com |
Step 2 - Replication Method
Select full table collection (all data is collected again every time the pipeline runs) or incremental collection. For incremental replication, the Primary key field (entered in step 5) is used to determine the record from which each new collection retrieves data.
Step 3 - Additional Headers (Optional)
| Field name | Description |
|---|---|
| Additional Headers (optional) | Any number of additional key-value headers can be registered. They will be merged with the Stream headers. |
Step 4 - Pagination (Optional)
| Field name | Description |
|---|---|
| Page size (optional) | Numeric field greater than or equal to 1 |
Example of Supported Pagination:
REST APIs support many pagination models. This connector currently supports the model with the following control fields:
- offset: indicates the current page in the data query process.
- limit: indicates the page size, i.e., the number of records each page returns.
- total: the total number of records to query and paginate.
A response containing the following model is required for pagination to work correctly:
{
"pagination": {
"total": 120,
"limit": 20,
"offset": 20
}
}
The pagination indicator groups the pagination control fields explained above.
In the Dadosfera connector configuration, you only need to enter the page size, i.e., the limit.
If your pagination requirements differ from the supported models, contact support to discuss a collection solution.
Step 5 - Stream
List of objects containing the configuration for each Stream. Only 1 Stream can be registered per pipeline.
| Field name | Description | Example |
|---|---|---|
| Name (required) | Stream name | meu_stream |
| Primary key (required) | More than one key can be registered per Stream | stream_id |
| Path (optional) | Path to append to the API URL | /api/v1/stream |
| Response path (optional) | Path within request responses that contains the records to process | $.stream[*] |
| Number of records for schema inference | Number of records used to infer the Stream schema. Defaults to 50. | 50 |
| Parameters (optional) | Parameters for a request.get method. Stream-level parameters are merged with top-level parameters, with Stream-level parameters overriding top-level parameters that have the same key. |
Click "Save and Continue".
Frequency
- Finally, configure how often your pipeline should run. Choose one of the available options or enter a custom frequency using a cron expression. For more information, see the guide.
📘
The default time zone for scheduling is UTC.
All frequency methods define when extractions begin. They do not control how long replication runs or when the data actually reaches its destination.
Done! Wait for collection to run on the scheduled day and time.
To run the pipeline immediately, you can start it manually within 30 seconds after creating it. Go to "Pipelines", "List", and "Synchronize Pipeline".
After a few minutes, your pipeline will be cataloged as a Data Asset in the Explore tab.
You can also open "View pipeline" in the pipeline list to review the summary, collected entities and columns, execution history, and micro-transformation history.