Skip to main content

Dadosfera Library | Process

This library was designed to simplify data extraction, processing and analysis from various sources inside and outside Dadosfera, such as:

  • Fetching data from Dadosfera S3 buckets and Snowflake;
  • Creating and retrieving data from our Catalog;
  • Integrating with APIs such as OpenAI;
  • It provides a modular interface for simple, efficient interaction with these sources, focusing on common tasks such as data collection, HTML parsing and communication with external APIs.

The general library workflow involves connecting to different data providers, retrieving information and converting it into formats ready for analysis or integration with other systems, following Dadosfera Platform standards. The library also structures data into datasets and simplifies handling large information volumes stored in the cloud.

Alongside Dadosfera Library, we provide Steps, a collection of ready-to-use scripts for Dadosfera.

All interactions are organized to reuse functions and components, making library maintenance and expansion simpler.

Available Version​

Keeping the library up to date is essential for access to the latest features, performance improvements and security fixes. Below are several ways to check the latest Dadosfera Library version.

In the Dadosfera platform's Intelligence Module, you can check which library version is installed in your project environment.

  1. Check the latest version released on Docker Hub: Dadosfera Library Docker Image
  • Click Tags to see something similar to the image below.

  1. In Jupyter Notebook, enter the code below to check the version installed in your environment. Running it shows something similar to the image below.
!pip show dadosfera

  • If your environment version is older than the latest Docker Hub version, follow these instructions:
  1. Click Enviroments and check the version in the Setup script section.

  1. Visit the official GitHub repository, Dadosfera Library Github, for detailed information about released versions.

  1. Change the Setup script code to the latest version released on GitHub. Then click Build to update.

You can now use the new Dadosfera Library features.

Release notes​

The dadosfera/base-kernel-py:1.0.3 image provides a Python 3.8 environment with various data science and web development packages. It was designed for Dadosfera's Intelligence and Transformation Module, a platform for building, running and deploying data science pipelines.

Included Packages​

Linux additions

  • Chrome Driver was added to the image as a dependency for web scraping tools such as Selenium.

Python Packages​

  • awscli
  • boto3
  • snowflake-snowpark-python
  • streamlit
  • orchest
  • dadosfera-library