Dadosfera Library Functions | Process | Documentation
bulk_delete_data_assets
This step defines the structure of a JSON payload for bulk deletion of data assets in Maestro. It keeps datasets organized by removing redundant or unwanted data in bulk for efficient data management.
Prerequisites
- A Dadosfera username and password.
- Permission to delete the data assets.
- An adequate understanding of the data assets you want to delete.
Configuration
Environment Variables
The following environment variables are required to run the step:
- DADOSFERA_USERNAME: The username used to log in to the Dadosfera environment.
- DADOSFERA_PASSWORD: The password used to log in to the Dadosfera environment.
Parameters:
-
maestro_base_url: The Maestro API base URL. Allowed values include
https://maestro.dadosfera.aiandhttps://maestro.stg.dadosfera.ai. -
additional_params: An array of filter parameters determining which data assets to delete. Each parameter consists of a key (filter name) and a value (filter criterion).
Note: Select the correct base URL and appropriate access information to prevent accidental data loss or incorrect redirects.
get_data_assets
About the project
This step retrieves data assets from Dadosfera's Maestro platform so that results can be stored and used in intelligence pipelines.
Prerequisites
- Access to the Maestro API.
- Knowledge of the specific data assets you intend to retrieve.
Configuration
Environment Variables
The following environment variables are required to run the step:
- DADOSFERA_USERNAME: The username used to log in to the Dadosfera environment.
- DADOSFERA_PASSWORD: The password used to log in to the Dadosfera environment.
Parameters:
-
maestro_base_url: The Maestro API base URL. Accepted values are
https://maestro.dadosfera.aiandhttps://maestro.stg.dadosfera.ai. -
output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath).
-
output_variable_name: Defines the output variable name.
-
additional_params: An array of filter parameters determining which data assets to retrieve. Each parameter is defined by a key (filter name) and a value (filter criterion).
Note: Set the correct base URL and specify the output variable carefully to ensure accurate extraction.
Configuration examples
Retrieving all data for a specific pipeline
{
"maestro_base_url": "https://maestro.dadosfera.ai",
"output_variable_name": "arbitrary_output_name",
"additional_params": [
{
"key": "pipeline_id",
"value": "28f8de4d-f46d-47a4-83fb-42069c1861ae"
}
]
}
Retrieving all dashboards
{
"maestro_base_url": "https://maestro.dadosfera.ai",
"output_variable_name": "arbitrary_output_name",
"additional_params": [
{
"key": "data_asset_type",
"value": "dashboard"
}
]
}
get_files_from_google_drive
This step accesses and extracts all files from a specific Google Drive folder, authenticating through a service account.
Prerequisites
- Access to a Google Drive service account.
- Knowledge of the Google Drive folder ID whose files you want to list.
Configuration
Parameters
The following parameters must be provided when running the step:
- service_account_file: Path to the Google Drive service account credentials JSON file.
- selected_folder_id: ID of the Google Drive folder whose files you want to list.
- outgoing_variable_name: Output variable name used to store the file list.
Note: This function integrates easily with other functions, so outgoing_variable_name must match the next step's input_variable_name.
get_objects_from_s3
This step lists and retrieves objects from an AWS S3 bucket based on a supplied prefix. It returns a list of dictionaries containing the files' contents, keys and names.
Prerequisites
- AWS credentials: AWS credentials must be configured in the environment so boto3 can authenticate and access the S3 bucket.
- Knowledge of which objects to import from AWS.
Configuration
Parameters
The following parameters must be provided when running the step:
- bucket_name: Name of the S3 bucket whose objects will be listed. (Required)
- prefix: Prefix used to filter objects. (Optional)
Response
The function returns a list of dictionaries, each representing a found object. Each dictionary contains:
- file_content: File contents in bytes.
- key: Object key in the S3 bucket.
- file_name: File name without the extension.
get_pipelines
This step retrieves a list of pipelines from Maestro.
Prerequisites
- Dadosfera credentials.
Configuration
Environment Variables
The following environment variables are required to run the step:
- DADOSFERA_USERNAME: The username used to log in to the Dadosfera environment.
- DADOSFERA_PASSWORD: The password used to log in to the Dadosfera environment.
Parameters
The following parameters must be provided when running the step:
- maestro_base_url: The Maestro API base URL. Accepted values are
https://maestro.dadosfera.aiandhttps://maestro.stg.dadosfera.ai. - additional_params_array: An array of dictionaries containing additional filter parameters.
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath).
get_status_of_all_pipelines
This step retrieves the status of Maestro pipelines.
Prerequisites
- Dadosfera credentials.
Configuration
Environment Variables
The following environment variables are required to run the step:
- DADOSFERA_USERNAME: The username used to log in to the Dadosfera environment.
- DADOSFERA_PASSWORD: The password used to log in to the Dadosfera environment.
Parameters
The following parameters must be provided when running the step:
- maestro_base_url: The Maestro API base URL. Accepted values are
https://maestro.dadosfera.aiandhttps://maestro.stg.dadosfera.ai. - additional_params_array: An array of dictionaries containing additional filter parameters.
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath).
get_table_from_snowflake
This step extracts data from a Snowflake table and returns a Pandas DataFrame.
Prerequisites
- Credentials with Snowflake access.
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- table_identifier: Identifier of the table to extract from Snowflake;
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath).
put_objects_in_s3
This step sends one or more objects to an AWS S3 bucket. It accepts the following parameters:
Prerequisites
- Credentials with AWS access;
- Files to upload to S3;
Configuration
Parameters
The following parameters must be provided when running the step:
- bucket_name: Name of the S3 bucket to which objects will be sent;
- objects: Objects to send to the bucket. All objects must have the keys
key: Object name in S3
body: Content body
content_type: Object content type; - input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- file_extension: Extension of the file to upload;
- prefix: Path inside the bucket to which files will be directed;
rfm_analysis
This step performs RFM (Recency, Frequency, Monetary Value) analysis on a Snowflake database table. It calculates each customer's RFM scores based on purchasing behavior and categorizes customers into different types.
Prerequisites
- Credentials with Snowflake access;
- A table with identifying keys for customers, purchases and amounts;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- customer_id_column: Name of the column containing the customer ID;
- date_col: Name of the column containing the purchase date;
- monetary_col: Name of the column containing the purchase amount;
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath);
save_data_in_snowflake
This step saves data in a Snowflake table by executing the supplied SQL queries.
Prerequisites
- Credentials with Snowflake access;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- table_identifier: Identifier of the table to extract from Snowflake;
- input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- snowflake_queries_object: SQL instruction for creating the objects needed to create tables in Snowflake;
snowflake_flatten_column
This step processes data to flatten nested Snowflake object-type columns in a dataset.
Prerequisites
- Credentials with Snowflake access;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- auto_discovery_flatten_mode: Enables automatic detection of columns to flatten;
- columns_to_flatten: Specifies columns to flatten when auto_discovery_flatten_mode is not enabled;
- incoming_variable_name: Name of the step variable containing the base table query;
- before_apply_operations (optional): Table modification settings:
columns_to_remove: List of columns to exclude.
column_prefixes_to_remove: Identify columns to remove based on their prefix.
remove_prefixes_from_columns: Remove prefixes from columns.
snowflake_remove_columns
This step removes columns in the Snowflake environment.
Prerequisites
- Credentials with Snowflake access;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- columns_to_remove: Defines the columns to remove;
- input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath);
snowflake_rename_columns
This step renames columns in a specific Snowflake table.
Prerequisites
- Credentials with Snowflake access;
- Names of the columns to rename;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- columns_to_rename: Defines columns to rename in a key-value structure ("column name to rename":"new name");
- input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath);
snowflake_text_transformation
This step transforms text-type columns in the Snowflake environment.
Prerequisites
- Credentials with Snowflake access;
- Names of the columns to transform;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- text_transformations: Defines the transformation to apply (lower, upper or trim);
- input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath);
snowflake_unpivot_data
This step "pivots" a table (turns rows into columns and vice versa).
Prerequisites
- Credentials with Snowflake access;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- categorical_column_name: Name of the categorical column used in the unpivot operation;
- numerical_column_name: Name of the associated numerical column;
- columns_to_unpivot: Names of the columns to recombine ("unpack"), combining their values into the new table structure;
- input_type: Defines the input type: from a file (from_filepath) or an environment variable (from_incoming_variable);
- output_type: Defines how output is generated: saved in a variable (to_outgoing_variable) or an output file (to_filepath);
standardize_tables_in_snowflake
This step standardizes tables in a Snowflake database. It accepts a list of tables and a dictionary of standardization settings. The function iterates over each table and applies the settings specified in the dictionary.
Prerequisites
- Credentials with Snowflake access;
- Column standardization file;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
- mapping_file_path: Path to the file containing the column standardization mapping;
standardized_tables_union_all
Complementing the previous step, this step selects all standardized tables (with the "STANDARDIZED_" prefix) and creates a view joining all of them;
Prerequisites
- Credentials with Snowflake access;
- Table standardization completed;
Configuration
Parameters
The following parameters must be provided when running the step:
- secret_id: Secret ID for connecting to Snowflake;
trigger_job
This step triggers a specific job (called the Target Job) based on another pipeline's execution. It normally appears at the end of a pipeline. It can trigger jobs in the same Intelligence Module project or in different projects.
Prerequisites
- Knowledge of the instance_url, target_pipeline_uuid and target_job_name of the job to trigger;
Configuration
Parameters
Select these parameters in the pipeline step:
- instance_url: Intelligence Module base URL. Usually in this format:
https://app-intelligence-CLIENTE.dadosfera.ai - target_pipeline_uuid: ID of the pipeline containing the job to trigger.
- target_job_name: Name of the job to trigger. Keep the same name whenever you recreate the job.
You can obtain the same results when running this step outside the Intelligence Module, but you must use a .json configuration file as shown below:
{
"instance_url": "https://app-intelligence-CLIENTE.dadosfera.ai",
"target_pipeline_uuid": "pipeline_uuid_a",
"target_job_name": "nome_do_job_01"
}
This example triggers the job named "nome_do_job_01" in the pipeline with ID "pipeline_uuid_a".