Skip to main content

Transformation Lineage

Transformation lineage describes the relationships between jobs executed in the transformation module.

Dataset identity​

Following OpenLineage, a dataset is identified by its namespace and name. Two jobs share a dataset only when both values match. A shared namespace connects producers and consumers of the same asset.

When a writer and reader share the dataset namespace and name, the lineage graph connects them. Different namespaces identify separate datasets and produce isolated graphs.

Column lineage​

The runner validates column lineage for SQL transformations and derived tables, including:

  • CTAS with renamed columns;
  • derived expressions, such as multiplying columns to calculate a metric;
  • joins with alias resolution;
  • tracing output columns back to their actual input columns.

This identifies the fields that feed each output field, in addition to identifying the resulting table.

Files, Pandas, and external requests​

Transformation lineage also tracks local files and HTTP requests:

  • files read or written with open() and libraries such as Pandas become datasets;
  • configuration files and other artifacts that are not data can be filtered from the graph;
  • GET, HEAD, and OPTIONS requests are inputs;
  • write requests, such as POST, are outputs.

This supports tracing flows that combine files, networks, and databases.

Snowflake and Snowpark​

Snowflake scenarios validate column lineage facets from realistic SQL, including CTAS, join alias resolution, and identifier normalization in the Snowflake dialect.

The Snowpark scenario validates dataset emission and the schema facet. Column lineage remains a future expectation in that test, identifying a known instrumentation gap.

Current coverage​

The documented behavior covers shared datasets connecting jobs, namespace isolation, column lineage for supported SQL, tracking of files/Pandas/HTTP, and schema emission in Snowpark. Column instrumentation limitations remain explicit.

Return to the Lineage overview.