Provenance System

When a pipeline produces thousands of results, you need to answer questions like “where did this output come from?”, “what parameters produced this result?”, and “which inputs failed?”. Provenance is how the framework records the full computational history so that every one of these questions is answerable -- without detective work, log parsing, or guesswork.

This page explains the two provenance systems, why both exist, how lineage edges are captured and validated, and how the provenance graph is stored and visualized.

Two complementary systems¶

The framework maintains two provenance systems because they answer fundamentally different questions.

Execution provenance records what ran: which operation, with what parameters, consuming which artifacts and producing which artifacts. It is an activity log -- ground truth directly observable from actual events.

Artifact provenance records where things came from: which specific input produced which specific output. It captures the individual derivation chains (A->D, B->E, C->F) that execution provenance cannot express.

Both align with the W3C PROV standard:

Why both systems are necessary¶

This is the core design constraint. Execution provenance cannot replace artifact provenance, and vice versa.

An operation processes a batch of artifacts. A single execution record shows “consumed {A, B, C}, produced {D, E, F}” -- but not which input produced which output. The correspondence A->D, B->E, C->F is invisible from the execution record alone.

Execution provenance              Artifact provenance

ExecutionRecord                   A ──→ D
  consumed: [A, B, C]            B ──→ E
  produced: [D, E, F]            C ──→ F

      ↑                               ↑
"What went in and out"          "Which produced which"

Why you cannot derive artifact provenance later. The information needed to match outputs to inputs -- filename stems and grouping indices -- is available only during the operation’s lineage phase. Once execution finishes, this context is gone. Attempting to reconstruct lineage after the fact by scanning filenames or guessing relationships is brittle and unreliable.

The framework’s solution: Lineage inference runs during a dedicated lineage phase (after postprocess), while context is still available. The resulting edges are staged alongside artifacts and committed atomically. There is no separate “lineage reconstruction” step.

Dual identity for executions¶

Each execution carries two identities because caching and provenance have conflicting requirements.

Same spec_id means “same request” -- a cache hit. Different run_id means “different attempt” -- distinct provenance even when the same computation runs twice. This separation lets the framework cache aggressively without losing the ability to distinguish separate executions in the provenance graph.

How lineage is captured¶

Lineage capture happens automatically for most operations. The framework uses filename stem matching -- the observation that operations naturally preserve filename stems through transformations. A file named sample_001.csv that gets transformed produces sample_001_transformed.csv. The shared stem connects them.

The algorithm¶

The stem matching algorithm strips extensions and probes prefixes:

• Strip all extensions from output and input filenames (sample_001_transformed.csv -> sample_001_transformed). Multi-part extensions like .tar.gz are fully stripped.

• Try exact match first -- if the output stem equals an input stem and there is exactly one candidate, that is the match

• If no exact match, try prefix matches longest-first: progressively shorten the output stem, checking whether the prefix matches any input stem

• Apply digit boundary protection at each prefix cut: the character being dropped must not be a digit (design_1 does not match design_10)

• At every level, require exactly one candidate -- if multiple candidates share the same stem or prefix, skip that level and keep shortening

• If no level yields a unique match, the output has no lineage mapping. The subsequent completeness validation then raises a LineageCompletenessError

Why the uniqueness requirement? Better to fail loudly than record incorrect lineage. If two candidates match a single output, the framework cannot determine which is the true parent, so it rejects the result rather than guessing. This ensures that every edge in the provenance graph is trustworthy.

Digit boundary protection¶

This rule prevents a common class of false matches when filenames contain numeric suffixes. When the algorithm shortens the output stem to test a prefix, it only considers a cut point if the character being dropped is not a digit:

Without this protection, design_1 would falsely match design_10, design_100, and design_1000. The digit boundary ensures that numeric identifiers are treated as indivisible tokens.

Explicit lineage¶

While stem matching handles most cases automatically, operations can also declare explicit parent-child relationships. A LineageMapping specifies the exact source artifact ID for each output draft, bypassing stem inference entirely.

This is useful when:

• Output filenames do not share stems with their inputs (the operation renames files entirely)

• The operation has custom grouping logic that stem matching cannot express

• Output-to-output edges need precise control

Explicit mappings use the same group_id mechanism as inferred edges, so co-input semantics (joint derivation) work identically in both paths.

Lineage declaration¶

Operations control how lineage edges are created through the infer_lineage_from field on OutputSpec. This field tells the framework which artifacts to consider when matching output filenames -- input roles, other output roles, or no parents (generative).

Three declaration patterns are supported:

None (no declaration) is valid for curator operations, which handle provenance through passthrough semantics.

The framework validates these declarations at class definition time. Creator operations must declare lineage explicitly on every output. Combined {"inputs": [...], "outputs": [...]} is not supported -- use separate output roles instead.

Output-to-output edges¶

When one output derives from another output of the same operation (for example, a metric artifact that summarizes a generated structure), the infer_lineage_from field references the output role. The framework matches the metric’s filename stem against the structure output’s filenames rather than the input filenames.

The lineage pipeline¶

Lineage capture is not a single step -- it flows through a multi-stage pipeline that progressively refines raw metadata into fully typed provenance edges.

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│     Capture      │     │      Build      │     │     Enrich      │
│                  │     │                 │     │                 │
│  Stem-match or   │──→  │  Resolve draft  │──→  │  Look up types  │
│  explicit mapping│     │  IDs to final   │     │  and add exec   │
│                  │     │  artifact IDs   │     │  context        │
│  → LineageMapping│     │  → SourceTarget │     │  → ArtifactProv │
│    (per role)    │     │    Pair         │     │    enanceEdge   │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Capture matches each output artifact to its source using stem matching or explicit declarations, producing LineageMapping entries keyed by output role. For multi-input operations with group_by pairing, co-input edges are created for all non-primary input roles at the matched index, all sharing the same group_id.

Build resolves draft references. During capture, output artifacts do not yet have their final content-addressed IDs. The build stage maps draft original_name values to finalized artifact_id values, producing lightweight SourceTargetPair records.

Enrich adds execution context and artifact types. Source and target artifact types are looked up (either from an in-memory dict or a bulk Delta Lake scan) and combined with the execution run ID to produce final ArtifactProvenanceEdge records ready for staging.

Lineage validation¶

The framework validates lineage at three levels, failing fast when something is wrong rather than staging incorrect provenance.

Artifact validation checks that output artifacts satisfy their declared specs: required roles are present, artifact types match, and no undeclared output roles exist.

Completeness validation verifies that every non-orphan output artifact has a lineage mapping. An orphan is an output whose infer_lineage_from is {"inputs": []} (generative). All other outputs must have at least one source edge, or the framework raises a LineageCompletenessError.

Integrity validation checks that all lineage references point to real artifacts: source IDs must exist in the input or output artifact sets, draft names must correspond to actual outputs, and no duplicate mappings for the same draft are allowed. Violations raise a LineageIntegrityError.

These validations run before any data is staged, so invalid lineage never reaches Delta Lake.

Edge types¶

All artifact provenance relationships are stored as directed ArtifactProvenanceEdge records. The edge direction follows W3C PROV wasDerivedFrom semantics: source is the parent, target is the derived artifact.

Four edge patterns appear in practice:

The first two are independent edges -- each parent artifact is sufficient on its own to explain the derivation. Co-input edges are different (see below). Config reference edges connect artifacts referenced in execution configurations to the config artifact that contains the reference, enabling “what configs used this artifact?” queries.

Co-input edges and joint derivation¶

Some outputs cannot be produced from any single input alone. When you compare two datasets, both are jointly necessary -- neither by itself could produce the comparison result. Co-input edges represent this joint derivation.

The test: can the output be produced from any proper subset of the inputs? If yes, use independent edges. If no -- if all inputs were jointly necessary -- use co-input edges.

Co-input edges share a group_id -- a deterministic hash computed from the set of source artifact IDs. Multiple ArtifactProvenanceEdge records with the same group_id and target_artifact_id represent a single joint derivation. This allows queries like “what were ALL the inputs to this derivation?” without requiring intermediate aggregate artifacts.

How multi-input pairing works¶

Operations declare how inputs across roles should be paired via a group_by class variable. The orchestrator pairs inputs before dispatch, and each pairing gets a deterministic group_id that flows through to the ArtifactProvenanceEdge.group_id field.

Composite provenance and step boundaries¶

When operations are composed into composites (multiple creators wired together via a CompositeDefinition), the provenance graph contains both internal edges between intermediate artifacts and shortcut edges from composite inputs to final outputs.

Each ArtifactProvenanceEdge carries a step_boundary flag that distinguishes these two kinds:

Whether internal edges are persisted depends on the composite’s intermediates configuration. When intermediates are discarded, only shortcut edges appear in the provenance graph. When intermediates are persisted or exposed, both internal and shortcut edges are stored, with step_boundary distinguishing them. This lets pipeline-level queries filter to step-boundary edges for a clean high-level view while preserving full detail for debugging.

Storage layout¶

Provenance data lives in three Delta Lake tables:

Artifact types are denormalized onto edge records (both source_artifact_type and target_artifact_type appear on each ArtifactProvenanceEdge). This avoids joins when filtering provenance queries by type -- a common pattern when you want “all metric descendants of artifact X” without scanning the full artifact index.

All provenance data is written through the staging-commit pattern: workers stage Parquet files, and the orchestrator commits them atomically. Provenance tables are committed after content and index tables, so partial failures leave artifacts reachable even if their provenance edges are incomplete.

Visualization¶

The framework provides two graph views built from provenance data, each answering a different question about pipeline structure.

Macro graphs show the pipeline at the step level. Each step appears as an execution node, each output role as a data node, and edges trace the data flow between steps. This view answers “what is the pipeline shape?” and comes from the steps table alone -- no artifact-level provenance needed.

Micro graphs show individual artifacts and executions. Every artifact and execution record appears as its own node, with both execution edges (artifact-to-execution links) and lineage edges (artifact-to-artifact derivations) overlaid. This view answers “what happened to this specific artifact?” and uses all three provenance tables.

Both graphs use left-to-right layout with strict column ordering (execution column -> data column -> next execution column) to maintain readability. Backward edges (such as passthrough artifacts consumed by a later step) are rendered as dashed lines to avoid breaking the layout.

For interactive exploration in Jupyter, a stepper widget lets you navigate the micro graph one step at a time, so you can watch the provenance graph build up as the pipeline progresses.

Querying provenance¶

The provenance graph supports several query patterns, from simple one-hop lookups to full transitive walks.

Backward queries (“where did this come from?”) start from a target artifact and follow edges to its sources. A single hop returns direct parents; a transitive walk returns all ancestors.

Forward queries (“what was derived from this?”) start from a source artifact and follow edges to its targets. These can be filtered by artifact type -- for example, finding all metrics derived from a specific data artifact.

Type-filtered queries combine forward or backward traversal with artifact type filtering, taking advantage of the denormalized type fields on edge records to avoid extra index scans.

Full graph maps load the entire backward or forward provenance map in a single Delta scan, enabling efficient batch analysis when you need to explore the full graph rather than starting from a single artifact.

For the practical API, see the inspecting provenance how-to guide.

Terminology¶

The provenance system uses two distinct vocabularies to avoid confusion between execution-level and artifact-level perspectives:

This distinction matters when reading code and querying provenance tables. An “input” is always relative to an execution. A “source” is always relative to a derivation edge.

Key design decisions¶

Cross-references¶

• Composites and Composition -- How composites handle provenance for internal operations

• Operations Model -- How operations declare inputs/outputs and lineage configuration

• Execution Flow -- When and how provenance is captured during the three execution phases

• Storage and Delta Lake -- Table layout and the staging-commit pattern

• Design Principles -- The “provenance is always captured, never reconstructed” principle

• Inspect Pipeline Results and Provenance -- Practical guide to querying and visualizing provenance

• Provenance Graphs Tutorial -- Interactive macro and micro provenance visualization