Skip to main content

Collection Lineage

Collection lineage records the jobs and datasets used by each collection DAG. The main reference is the configuration of the primary task and the lineage parameters passed to each job.

What happens in each collection DAG​

Every collection DAG is generated from a template that prepares it to emit lineage while collecting data. The collection job reports:

  • the pipeline being executed;
  • the specific job in that DAG;
  • the source being read;
  • the output being produced;
  • the execution that generated the lineage event.

This lets the platform display the data chain without requiring users to understand the collection implementation.

How the template prepares lineage​

The DAG template configures lineage in the call to the processing job. Each DAG has an identifier, and each execution creates a run identifier. Application and job names identify the flow, an event endpoint receives lineage metadata, and pipeline context is included as tags.

These parameters are injected when the DAG is generated.

How data reaches the lineage graph​

  1. The collection DAG is generated from the template.
  2. Its main task executes the source connector or job.
  3. During execution, instrumentation publishes input and output metadata.
  4. Lineage records the pipeline, job, dependencies, and datasets.
  5. The platform uses these records to show where data came from and where it went.

This applies to collection DAGs generated by the template, regardless of connector.

Reading a collection flow​

After a DAG runs, lineage normally shows which pipeline collected the data, which job produced or consumed a dataset, its inputs and outputs, and how the execution connects to subsequent processing.

For a job that reads a source table and writes a file or destination table, the source is an input, the collection job performs the operation, and the destination is an output. The current run remains associated with its pipeline.

Limitations​

  • The DAG must be correctly instrumented in the template.
  • Missing job metadata results in incomplete lineage.
  • When different DAGs use the same source, each execution remains linked to its own pipeline and job.

See Transformation Lineage.