Step Overrides

What you’ll learn¶

• Override operation parameters, names, and execution config per step

• Control failure handling with FailurePolicy

• Understand override precedence (pipeline defaults → operation defaults → step overrides)

Prerequisites: First Pipeline, Batching and Performance. Estimated time: 15 minutes GPU required: No.

Every pipeline.run() call accepts optional overrides that customize how the step executes. This tutorial demonstrates each override parameter with working examples.

Override parameters¶

Each pipeline.run() call accepts override parameters for execution, resources, and backend. For the complete list, see Configuring Execution.

The most commonly used overrides are params, name, execution, backend, resources, and failure_policy. This tutorial demonstrates each one with working examples.

params — operation parameters¶

The params dict is passed directly to the operation. Each operation defines its own parameter schema. Passing different params to the same operation creates distinct steps with different behavior.

Both steps use DataGenerator but produce different outputs: step 0 generates 3 datasets with seed 42, step 1 generates 10 with seed 99. The pipeline overview shows the different artifact counts.

name — custom step name¶

By default, each step is named after the operation (e.g., data_generator). Use name to give steps descriptive labels, especially when the same operation appears multiple times.

The pipeline overview now shows initial_candidates, normalize, and augment instead of generic operation names. Custom names make pipelines easier to read, especially in provenance graphs.

execution — batching configuration¶

The execution dict accepts any ExecutionConfig field. See Batching and Performance for a deep dive.

Step 1 groups 12 artifacts into 3 execution units (4 per unit) and runs at most 2 concurrently. Without the override, MetricCalculator’s default artifacts_per_unit of 10,000 would put all 12 artifacts into a single unit.

backend — execution backend¶

The backend parameter overrides the pipeline-level default backend for a single step. This allows mixing backends within one pipeline — for example, running lightweight steps locally while submitting heavy computation to SLURM.

# Pipeline default is SLURM, but run this step locally
pipeline.run(
    MetricCalculator,
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.LOCAL,  # Override for this step only
)

Available backends: Backend.LOCAL (process pool on current machine), Backend.SLURM (submits jobs to a SLURM cluster), and Backend.SLURM_INTRA (distributes work via srun within an existing SLURM allocation).

resources — SLURM resource allocation¶

The resources dict controls SLURM job resources (CPUs, memory, GPUs, time limit). These have no effect when running locally. For all ResourceConfig fields, see Configuring Execution.

pipeline.run(
    MyGPUOperation,
    inputs={"data": datasets},
    resources={"gpus": 1, "memory_gb": 32, "extra": {"partition": "gpu"}},
    backend=Backend.SLURM,
)

failure_policy — handling failures¶

By default, the pipeline uses the policy set at creation time (default: FailurePolicy.CONTINUE). Override per-step for fine-grained control.

Step 1 uses FAIL_FAST — if any execution unit fails, the entire step aborts with an exception. Step 2 uses CONTINUE — failures are logged and surviving results are committed. Use FAIL_FAST for critical steps where partial results are meaningless, and CONTINUE for steps where you’d rather keep what succeeded.

environment — container and environment configuration¶

The environment parameter selects which execution environment to use for a step. Pass a string to select a pre-configured environment, or a dict to override environment settings.

Available environments: "local", "docker", "apptainer", "pixi".

# Run this step in a Docker container
pipeline.run(
    MyOperation,
    inputs={"data": datasets},
    environment={
        "active": "docker",
        "docker": {"image": "my_image:latest"},
    },
)

Since environment overrides require specific container or environment setups, this tutorial does not include a runnable example.

tool — external tool overrides¶

The tool dict overrides the executable, interpreter, or subcommand for operations that wrap external programs.

pipeline.run(
    ToolAOp,
    inputs={"data": datasets},
    tool={"executable": "/opt/tool_a/bin/tool_a"},
)

The operation must already define a tool in its class — this override only changes specific fields.

Override precedence¶

When the same setting is specified at multiple levels, step-level overrides win:

Pipeline defaults  →  Operation class defaults  →  Step overrides (wins)

For example, if the pipeline default failure policy is CONTINUE, the operation class has no override, and a step specifies FAIL_FAST, that step uses FAIL_FAST.

This applies to all override parameters: backend, resources, execution, environment, tool, and failure_policy. The params parameter has no defaults — it is always explicit per step.

Summary¶

Key takeaway: Step overrides give you fine-grained control without changing operation code. Start with pipeline-level defaults, and override individual steps only where needed.

Next steps¶

• Batching and Performance — Deep dive into execution config tuning

• Configuring Execution — Complete configuration reference

• Error Handling in Practice — Runtime failures, failure logs, and FailurePolicy