Exploring Results

What you’ll learn¶

Get a one-line summary of an entire pipeline run
Drill into individual steps and their artifacts
Read dataset contents and metric values as Polars DataFrames
Trace an artifact’s lineage back through the pipeline
Visualize pipeline topology with provenance graphs
Analyze where time was spent with timing breakdowns

Prerequisites: Your First Pipeline
Estimated time: 15 minutes
GPU required: No

You’ve run a pipeline. Now what?

Artisan stores every artifact, metric, and provenance edge in Delta Lake tables. This tutorial shows how to explore those results — starting from a bird’s-eye view and progressively zooming in.

We’ll work through five levels of detail:

Pipeline overview   →  What steps ran? Did they succeed?
Step detail          →  What artifacts did step N produce?
Data & metrics       →  What are the actual values?
Lineage              →  Where did this artifact come from?
Visualization        →  What does the DAG look like?

Setup¶

First, we’ll build a small pipeline to generate results worth exploring. This is a simplified version of the pipeline from Your First Pipeline — generate data, transform it, compute metrics, filter, and transform again.

from __future__ import annotations

from artisan.operations.curator import Filter
from artisan.operations.examples import (
    DataGenerator,
    DataTransformer,
    MetricCalculator,
)
from artisan.orchestration import PipelineManager
from artisan.storage import ArtifactStore
from artisan.utils import tutorial_setup
from artisan.visualization import (
    PipelineTimings,
    build_macro_graph,
    build_micro_graph,
    inspect_data,
    inspect_metrics,
    inspect_pipeline,
    inspect_step,
)

env = tutorial_setup("exploring_results")
DELTA_ROOT = env.delta_root

pipeline = PipelineManager.create(
    name="exploring_results",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

pipeline.run(operation=DataGenerator, name="generate", params={"count": 5, "seed": 42})

pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    params={"scale_factor": 0.5, "variants": 1, "seed": 100},
)

pipeline.run(
    operation=MetricCalculator,
    name="metrics",
    inputs={"dataset": output("transform", "dataset")},
)

pipeline.run(
    operation=Filter,
    name="filter",
    inputs={"passthrough": output("transform", "dataset")},
    params={
        "criteria": [
            {"metric": "distribution.median", "operator": "gt", "value": 0.3},
        ]
    },
)

pipeline.run(
    operation=DataTransformer,
    name="rescale",
    inputs={"dataset": output("filter", "passthrough")},
    params={"scale_factor": 0.1, "variants": 1, "seed": 101},
)

result = pipeline.finalize()
print(
    f"Pipeline complete: {result['total_steps']} steps, success={result['overall_success']}"
)

Level 1: Pipeline overview¶

The first question after a run is: what happened?

inspect_pipeline returns a one-row-per-step summary showing the operation name, status, what was produced, and how long each step took.

inspect_pipeline(DELTA_ROOT)

Read this table column by column:

step — execution order (0-indexed)
operation — the step name
status — ok or skipped (a step is skipped when its inputs are empty, e.g. nothing passed the filter)
produced — artifact counts by type. Filter steps show “passed” instead — they route existing artifacts rather than creating new ones
duration — wall-clock time for that step

Level 2: Step detail¶

Next question: what artifacts did a specific step produce?

inspect_step lists every artifact at a given step. The details column shows type-specific information — row/column counts for datasets, key names for metrics, file sizes for file references.

# DataGenerator produced 5 datasets
inspect_step(DELTA_ROOT, step_number=0)

# MetricCalculator produced one metric per input dataset
inspect_step(DELTA_ROOT, step_number=2)

Level 3: Data and metrics¶

Now we can look at the actual values. Artisan stores dataset contents in Delta Lake tables and metric values as JSON — the inspect_data and inspect_metrics helpers load them as Polars DataFrames for interactive analysis.

Reading dataset contents¶

inspect_data loads the tabular content of data artifacts. You can query by step number (all datasets at that step) or by name (a specific dataset).

# All datasets at step 0 — the _source column shows which artifact each row came from
inspect_data(DELTA_ROOT, step_number=0)

# A single dataset by name
inspect_data(DELTA_ROOT, name="dataset_00000")

Reading metric values¶

inspect_metrics parses metric JSON into a flat DataFrame — nested keys like distribution.median become column names. This makes it easy to sort, filter, and compare across artifacts.

inspect_metrics(DELTA_ROOT, step_number=2)

Since the result is a standard Polars DataFrame, you can use any Polars operation on it — sorting, filtering, grouping, joining.

# Which artifacts scored highest?
inspect_metrics(DELTA_ROOT, step_number=2).sort("distribution.median", descending=True)

# Omit step_number to see metrics across all steps
inspect_metrics(DELTA_ROOT)

Level 4: Lineage¶

Every artifact knows where it came from. Artisan records provenance edges between artifacts so you can trace any result backward to its inputs.

The ArtifactStore provides programmatic lineage queries. This is useful for debugging unexpected results — you can follow the data backward step by step.

store = ArtifactStore(DELTA_ROOT)

# Pick a metric artifact from step 2
metric_ids = store.load_artifact_ids_by_type("metric", step_numbers=[2])
metric_id = sorted(metric_ids)[0]

# What was this metric computed from?
ancestors = store.get_ancestor_artifact_ids(metric_id)
print(f"Metric {metric_id[:12]}... was computed from:")
for aid in ancestors:
    art_type = store.get_artifact_type(aid)
    step_num = store.get_artifact_step_number(aid)
    print(f"  ← {aid[:12]}... (type={art_type}, step={step_num})")

# Going forward: what artifacts were derived from step 0's datasets?
data_ids = store.load_artifact_ids_by_type("data", step_numbers=[0])
descendants = store.get_descendant_artifact_ids(data_ids)

for source_id, child_ids in sorted(descendants.items()):
    print(f"Dataset {source_id[:12]}... produced {len(child_ids)} descendant(s)")

Level 5: Visualization¶

Graphs often communicate pipeline topology faster than tables. Artisan provides two levels of provenance graphs and a timing breakdown.

Macro graph (step-level)¶

The macro graph shows the pipeline at the step level — one node per step, one node per output role, connected by data-flow edges. This is the fastest way to confirm your pipeline has the shape you intended.

Element	Meaning
Grey box	An execution step, labeled `(N) step_name`
Blue box	An artifact group, labeled with the output role and count
Arrow from step to data	“This step produced these artifacts”
Arrow from data to step	“This step consumed these artifacts”

build_macro_graph(DELTA_ROOT)

Micro graph (artifact-level)¶

The micro graph shows every individual artifact and execution record as separate nodes. Where the macro graph groups artifacts into counts, the micro graph lets you trace the exact derivation chain for any artifact.

Element	Meaning
Grey box	An execution record (one operation invocation)
Blue box	An artifact (data, metric, or file reference)
Black arrow	Execution provenance — “produced by” or “consumed by”
Orange arrow	Artifact provenance — “derived from”
Dashed arrow	Backward or passthrough reference

build_micro_graph(DELTA_ROOT)

Timing analysis¶

PipelineTimings breaks down where time was spent in each step — input resolution, execution, and commit. This helps identify bottlenecks.

timings = PipelineTimings.from_delta(DELTA_ROOT)
timings.step_timings()

timings.plot_steps()

Bonus: Listing pipeline runs¶

A single Delta root can hold multiple pipeline runs. list_runs shows all of them, most recent first.

PipelineManager.list_runs(DELTA_ROOT)

Summary¶

Question	Tool	Returns
What steps ran?	`inspect_pipeline(delta_root)`	One row per step
What did step N produce?	`inspect_step(delta_root, N)`	One row per artifact
What’s in the data?	`inspect_data(delta_root, step_number=N)`	Polars DataFrame of contents
What are the metrics?	`inspect_metrics(delta_root, step_number=N)`	Flat metric table
Where did this come from?	`store.get_ancestor_artifact_ids(id)`	Parent artifact IDs
What was derived from this?	`store.get_descendant_artifact_ids(ids)`	Child artifact IDs
What’s the pipeline shape?	`build_macro_graph(delta_root)`	Step-level DAG
What’s the full artifact graph?	`build_micro_graph(delta_root)`	Artifact-level DAG
Where was time spent?	`PipelineTimings.from_delta(delta_root)`	Phase-level timing breakdown
What runs exist?	`PipelineManager.list_runs(delta_root)`	Run history table

All of these return Polars DataFrames or Graphviz objects — you can sort, filter, export, or render them however you like.

Next steps¶

Run vs Submit — Blocking vs non-blocking execution
Provenance System — How lineage tracking works under the hood
Storage and Delta Lake — How artifacts are persisted