Features
| Feature | Supported | Notes |
|---|---|---|
| Collect source views | Yes | |
| Set pipeline frequency | Yes | For more details, see the guide. |
| Entity selection | Yes | |
| Full load synchronization | Yes | |
| Incremental - Append synchronization | Yes | |
| Column selection | Yes | |
| Micro-transformation: Hash | Yes | |
| VPN connection | Yes | Optional. Register your VPN to access your private database. |
| Native Oracle DB encryption | Yes | Encryption must be enabled to establish the connection. |
Quick Start
To create a pipeline, open the Collect module, go to "Pipelines", and click "New Pipeline".
Choose the Data Source
Use an existing source or register a new one
Connection Parameters (Source Registration)
| Field name | Description | Example |
|---|---|---|
| Endpoint | Public address for connecting to your database, in a valid IPv4 or DNS format | 32.142.63.4 |
| Port number | Database port | 1521 |
| Connection type | If your data is stored in a private network, choose a registered VPN authorized to access it. Otherwise, use a direct connection. | |
| User | Database user | admin |
| Password | Database user's password | |
| Database | Name of the database to connect to | my-db |
Pipeline Information
Assign a name and a brief description to your pipeline.
Pipeline Configuration Parameters
Choose the Connection Method
Requirements
Access a database version compatible with our connector.
If you are unsure whether your database is compatible, contact the support team.
Allow Dadosfera to connect to your database.
This depends on your network configuration. The best way to confirm whether Dadosfera can reach your database is to use the connection test in the interface.
- Public databases do not require any access configuration.
- For private databases, register your VPN in Dadosfera so it can be used to access the database. Contact the support team to register it. Contact support.
📘 Native Oracle DB Encryption
Encryption must be enabled to connect to Dadosfera. For instructions, see the Security section of the Oracle DB reference manual for your database version.
If the database is still unreachable, check whether the user has read access to the selected database.
Create a read-only Dadosfera user with access to all tables and views to be extracted.
This step is optional but strongly recommended for better permission control and auditing. If you prefer, connect using an existing database user.
To create a dedicated database user, run the following commands in your database:
CREATE USER <username> IDENTIFIED BY 'your_password';
GRANT CREATE SESSION TO <username>;
GRANT SELECT_CATALOG_ROLE TO <username>;
Done! You can now connect Dadosfera to your database through the interface using these parameters:
| Field name | Description | Example |
|---|---|---|
| Schema | Collection of database objects | nome_do_schema |
Click "Test Connection" and, if successful, click "Save and Continue".
If the connection test still fails after following the recommendations above, contact support.
Specify Entities
After testing the connection, you can view and select the entities — tables and views — and columns from the connected source to import during data collection.
You can also define the synchronization mode per entity. For more information, see the guide.
📘
When selecting more than one entity, a separate dataset is created in the catalog for each imported table or view.
📘 Data types accepted as incremental collection reference fields
- TIMESTAMP
- TIMESTAMP(0) WITH TIME ZONE
- TIMESTAMP(0) WITH LOCAL TIME ZONE
- DATE
- LONG
- NUMBER
- INTEGER
- SHORT
📘 General notes on data types:
A column is recognized as type number only when every element is an integer.
A column is recognized as type float only when every element follows the floating-point data format, i.e., a rational number (e.g., xxxx.xx).
Commas in numeric formatting (x,xxx.xx) cause the column to be converted to type text.
Micro-transformation
Dadosfera uses the ELT paradigm: transformations are performed after data is loaded. However, when creating a pipeline, you can apply the micro-transformations available in Dadosfera.
The micro-transformation available in the current version is "Encrypt" (technically called Hash). It transforms each value in the selected column consistently: repeated values in different rows produce the same result. It is applied event by event or in micro-batches.
📘
Example: "Alexandre" becomes "AdhSOidjdBWIOU&562)&y2396" in every row containing the word "Alexandre".
Frequency
- Finally, configure how often your pipeline should run. Choose one of the available options or enter a custom frequency using a cron expression. For more information, see the guide.
📘
The default time zone for scheduling is UTC.
All frequency methods define when extractions begin. They do not control how long replication runs or when the data actually reaches its destination.
Done! Wait for collection to run on the scheduled day and time.
To run the pipeline immediately, you can start it manually within 30 seconds after creating it. Go to "Pipelines", "List", and "Synchronize Pipeline".
After a few minutes, your pipeline will be cataloged as a Data Asset in the Explore tab.
You can also open "View pipeline" in the pipeline list to review the summary, collected entities and columns, execution history, and micro-transformation history.