Skipping the Cache

What you’ll learn¶

Force re-execution of cached steps with skip_cache
Apply skip_cache per step or pipeline-wide
Understand when and why to bypass the cache

Prerequisites: Your First Pipeline, Resume and Caching.
Estimated time: 10 minutes
GPU required: No.

Artisan’s two-level cache automatically skips steps whose inputs and parameters match a previous successful run. This is great for incremental development — but sometimes you need to force re-execution, even when the cache key hasn’t changed.

Scenario	Why the cache doesn’t help
Bug fix in operation code	The cache key is based on parameters, not code
Non-deterministic operation	You want a fresh sample, not the old one
External resource changed	A file or API the operation reads has been updated
Fresh pipeline run	Re-execute everything from scratch in the same delta root

from __future__ import annotations

from artisan.operations.examples import (
    DataGenerator,
    DataTransformer,
    MetricCalculator,
)
from artisan.orchestration import Backend, PipelineManager
from artisan.utils import tutorial_setup
from artisan.visualization import inspect_pipeline

env = tutorial_setup("skip_cache")

Establish a baseline¶

Run a 3-step pipeline so that future runs have cached results to match against.

pipeline = PipelineManager.create(
    name="skip_cache_demo",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 5, "seed": 42},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=MetricCalculator,
    name="score",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.LOCAL,
)

pipeline.finalize()
inspect_pipeline(env.delta_root)

Confirm caching works¶

Run the same pipeline again with the same parameters. Every step should show “CACHED” in the log because nothing changed.

env = tutorial_setup("skip_cache", clean=False)

pipeline = PipelineManager.create(
    name="cached_run",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 5, "seed": 42},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=MetricCalculator,
    name="score",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.LOCAL,
)

pipeline.finalize()

All three steps hit the cache. The pipeline completed in a fraction of the time with zero computation.

Skip cache for a single step¶

Pass skip_cache=True to pipeline.run() to force re-execution of that step. Other steps still use the cache normally.

This is the most common pattern — you fixed a bug in one operation and want to re-run that step without changing its parameters.

env = tutorial_setup("skip_cache", clean=False)

pipeline = PipelineManager.create(
    name="skip_one_step",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

# Step 0: cached (no change)
pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 5, "seed": 42},
    backend=Backend.LOCAL,
)

# Step 1: force re-execution
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,
    skip_cache=True,
)

# Step 2: cached (no change)
pipeline.run(
    operation=MetricCalculator,
    name="score",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.LOCAL,
)

pipeline.finalize()

In the log output, steps 0 and 2 show “CACHED” while step 1 executes fresh. The re-executed step’s results are written to delta storage — future runs without skip_cache will cache-hit against the new results.

Skip cache for the entire pipeline¶

Pass skip_cache=True to PipelineManager.create() to bypass the cache for every step in the pipeline. This is useful after upgrading a dependency or changing an external resource that affects all steps.

env = tutorial_setup("skip_cache", clean=False)

pipeline = PipelineManager.create(
    name="fresh_run",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
    skip_cache=True,
)
output = pipeline.output

pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 5, "seed": 42},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,
)
pipeline.run(
    operation=MetricCalculator,
    name="score",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.LOCAL,
)

pipeline.finalize()

No “CACHED” messages — every step re-executed. The pipeline-level flag overrides any per-step behavior.

How `skip_cache` interacts with the cache¶

A few details worth knowing:

Results are still written. Even with skip_cache=True, the step’s outputs are committed to delta storage. A future run without skip_cache will cache-hit against the new results.
OR logic. If either the pipeline-level skip_cache or the per-step skip_cache is True, the cache is bypassed. You don’t need both.
Both cache levels are bypassed. Artisan has a step-level cache (based on the step spec ID) and an execution-level cache (based on per-batch content hashing). skip_cache bypasses both.
Output isolation. When a step re-executes with skip_cache, the new outputs are tagged with a unique run ID. Downstream steps see only the outputs from this run, not a mix of old and new results.

Summary¶

Scope	API	Effect
Per step	`pipeline.run(..., skip_cache=True)`	Re-execute one step, others still cache
Pipeline-wide	`PipelineManager.create(..., skip_cache=True)`	Re-execute every step

Key takeaway: skip_cache forces re-execution when you know the cache is stale — after fixing a bug, updating a dependency, or working with non-deterministic operations. Results are still written to delta, so future runs benefit from the refreshed cache.

Next steps¶

Resume and Caching — How caching and resume work together
Step Overrides — Other per-step configuration options
Execution Flow — How the two-level cache works under the hood