Provenance Graphs

What you’ll learn¶

Read macro graphs to understand pipeline topology at a glance
Read micro graphs and use step-by-step snapshots to trace individual artifact derivation

Artisan records two types of provenance as your pipeline runs: execution provenance tracks which step ran which operation, consuming and producing artifacts. Artifact provenance tracks which specific output artifact was derived from which specific input artifact. This tutorial shows you how to visualize both.

Prerequisites: Your First Pipeline, Exploring Results.
Estimated time: 10 minutes
GPU required: No.

from __future__ import annotations

from artisan.operations.curator import Filter
from artisan.operations.examples import (
    DataGenerator,
    DataTransformer,
    MetricCalculator,
)
from artisan.orchestration import Backend, PipelineManager
from artisan.utils import tutorial_setup
from artisan.visualization import (
    build_macro_graph,
    build_micro_graph,
    inspect_pipeline,
)

env = tutorial_setup("provenance_graphs")
DELTA_ROOT = env.delta_root

Build a sample pipeline¶

The pipeline below generates data, transforms it, computes metrics, filters by a metric threshold, and transforms the survivors. The branching topology — step 1’s output feeds both the metric computation and the filter — creates a non-trivial provenance graph to explore.

pipeline = PipelineManager.create(
    name="provenance_tutorial",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

# Step 0: Generate 5 source datasets
pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 5, "seed": 42},
    backend=Backend.LOCAL,
)

# Step 1: Transform each dataset
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,
)

# Step 2: Compute metrics for each transformed dataset
pipeline.run(
    operation=MetricCalculator,
    name="score",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.LOCAL,
)

# Step 3: Filter — keep datasets whose median > 0.3
pipeline.run(
    operation=Filter,
    name="filter",
    inputs={"passthrough": output("transform", "dataset")},
    params={
        "criteria": [
            {"metric": "distribution.median", "operator": "gt", "value": 0.3},
        ]
    },
    backend=Backend.LOCAL,
)

# Step 4: Transform the filtered results
pipeline.run(
    operation=DataTransformer,
    name="refine",
    inputs={"dataset": output("filter", "passthrough")},
    backend=Backend.LOCAL,
)

pipeline.finalize()
inspect_pipeline(DELTA_ROOT)

Pipeline topology: the macro graph¶

The macro graph shows the pipeline at the step level: which operations ran, what data flowed between them, and how many artifacts each step produced. It’s the fastest way to confirm your pipeline has the shape you intended.

build_macro_graph(DELTA_ROOT)

Reading the macro graph¶

The graph reads left to right:

Element	Meaning
Grey box	An execution step, labelled `(N) step_name`
Blue box	An artifact output, labelled with the output role and count
Arrow (step → data)	“This step produced these artifacts”
Arrow (data → step)	“This step consumed these artifacts”

Notice two things in this pipeline:

Branching at step 1. Step 1’s dataset output feeds both step 2 (MetricCalculator) and step 3 (Filter). You can see this as two arrows leaving the same data node.
Filter is passthrough. Step 3 doesn’t produce new artifact nodes with a distinct type — it routes a subset of existing artifacts through. The output count at step 3 is smaller than step 1’s count because some artifacts were filtered out.

The macro graph answers questions like: What’s the overall shape of my pipeline? Where does it branch? How many artifacts survived the filter? For artifact-level detail, you need the micro graph.

Artifact lineage: the micro graph¶

The macro graph groups artifacts into counts per step. The micro graph shows every individual artifact and the derivation edges between them. This is where you can trace exactly which input produced which output.

build_micro_graph renders the full artifact-level provenance graph. Use the max_step parameter to show only artifacts through a given step.

build_micro_graph(DELTA_ROOT)

Reading the micro graph¶

The micro graph above shows the full pipeline. Here’s what to look for at each step:

Step 0: Five data artifact nodes appear with no incoming edges. These are source artifacts — the starting points of all lineage chains.
Step 1: Five new data nodes appear. Each one is connected to a step-0 artifact by an orange arrow — that’s an artifact provenance edge meaning “derived from.” The grey execution box shows the DataTransformer operation.
Step 2: Five metric nodes appear, each derived from a step-1 dataset. The graph now shows the branch: step-1 artifacts have edges going both right (to metrics at step 2) and down (to the filter at step 3).
Step 3: No new artifact nodes appear. Filter is a passthrough — it selects a subset of step-1 artifacts without creating new ones. Only the surviving artifacts remain connected to downstream edges.
Step 4: New data nodes appear, derived from the filtered subset. Artifacts that were filtered out at step 3 have no step-4 descendants.

Visual legend:

Element	Meaning
Grey box	Execution record (an operation that ran)
Colored box	Artifact (a data, metric, or file artifact)
Black solid arrow	Execution provenance — “produced by” or “consumed by”
Orange arrow	Artifact provenance — “derived from”
Dashed arrow	Backward or passthrough reference

Step-by-step snapshots¶

Use build_micro_graph with max_step to render a cumulative snapshot showing all artifacts and edges through a given step. This lets you inspect how the provenance graph grows at each stage.

# After step 1: source generation + first transformation
build_micro_graph(DELTA_ROOT, max_step=1)

# After step 2: metrics branch off from the transformed datasets
build_micro_graph(DELTA_ROOT, max_step=2)

Which visualization for which question¶

Question	Tool
What’s the shape of my pipeline?	`build_macro_graph`
How many artifacts survived the filter?	`build_macro_graph` or `inspect_pipeline`
Which specific input produced this output?	`build_micro_graph`
How does the graph grow step by step?	`build_micro_graph` with `max_step`
Is my pipeline wired correctly?	`build_macro_graph` first, then `build_micro_graph` for details

For programmatic lineage queries (tracing ancestors, descendants, and impact analysis), see the Lineage Tracing tutorial.

Summary¶

You learned two ways to visualize provenance, each at a different level of detail:

Macro graph — step-level pipeline topology, for understanding the overall DAG shape and artifact flow.
Micro graph snapshots — individual artifact nodes and derivation edges, for tracing exactly which input produced which output.

Next steps¶

Lineage Tracing — Trace ancestors and descendants programmatically
Provenance System — How the dual provenance model works under the hood
Inspecting Provenance — Task-oriented recipes for common provenance queries
Exploring Results — Inspect helpers for pipeline data and metrics