Features
| Feature | Supported | Notes |
|---|---|---|
| Set pipeline frequency | Yes | For more details, see the guide. |
| Full load synchronization | Yes | |
| Incremental - Append synchronization | No | |
| Select collected tables | Yes | |
| VPN connection | Yes | Connections are only possible through a VPN. |
Quick Start
To create a pipeline, open the Collect module, go to "Pipelines", and click "New Pipeline".
Choose the Data Source
Use an existing source or register a new one
Connection Parameters (Source Registration)
| Field name | Description | Example |
|---|---|---|
| Endpoint | Public address for connecting to your database, in a valid IPv4 or DNS format | 32.142.63.4 |
| Port number | Database port | 27017 |
| Connection type | Only VPN connections are supported, to access private databases | |
| User | Database user | admin |
| Password | Database user's password | |
| Authentication database | Database where the user is defined | my-db |
Pipeline Information
Assign a name and a brief description to your pipeline.
Pipeline Configuration Parameters
| Field name | Description | Example |
|---|---|---|
| Database | Database where the databases will be synchronized | my-db |
| Enable SSL (optional) | Enable SSL for encrypted collection |
Specify Tables
Enter the names of the database tables to collect.
📘
A separate dataset is created in the catalog for each imported table.
Scheduling
- Finally, configure how often your pipeline should run. Choose one of the available options or enter a custom frequency using a cron expression. For more information, see the guide.
📘
The default time zone for scheduling is UTC.
All frequency methods define when extractions begin. They do not control how long replication runs or when the data actually reaches its destination.
Done! Wait for collection to run on the scheduled day and time.
To run the pipeline immediately, you can start it manually within 30 seconds after creating it. Go to "Pipelines", "List", and "Synchronize Pipeline".
After a few minutes, your pipeline will be cataloged as a Data Asset in the Explore tab.
You can also open "View pipeline" in the pipeline list to review the summary, collected entities and columns, execution history, and micro-transformation history.