Running Inside a SLURM Allocation

What you’ll learn¶

Use Backend.SLURM_INTRA to dispatch work via srun within an existing allocation
Understand when to use SLURM_INTRA vs SLURM
Override resources carved from the allocation (GPUs, CPUs, memory)
Control batching and concurrency for srun steps
Debug srun-based runs

Prerequisites: Running on SLURM, active salloc or sbatch session. Estimated time: 10 minutes GPU required: No.

Note

This tutorial requires an active SLURM allocation (salloc or sbatch). Code cells are shown for reference but are not executed in the docs build.

To run the demo yourself, copy the companion script:

salloc --cpus-per-node=8 --time=00:30:00
pixi run python docs/tutorials/execution/demo_slurm_intra.py

from __future__ import annotations

from artisan.operations.examples import (
    DataGenerator,
    DataTransformer,
    MetricCalculator,
)
from artisan.orchestration import Backend, PipelineManager
from artisan.utils import tutorial_setup
from artisan.visualization import inspect_metrics, inspect_pipeline

env = tutorial_setup("slurm_intra_execution")
DELTA_ROOT = env.delta_root

When to use `SLURM_INTRA`¶

The standard Backend.SLURM submits each step as independent sbatch jobs to the SLURM queue. This works well for production pipelines, but creates unnecessary overhead when you already have resources reserved:

salloc --cpus-per-node=8 --time=00:30:00
# 8 CPUs are now reserved and idle, waiting for work

With Backend.SLURM, each task still goes through the scheduler queue, even though the resources are already in hand. Backend.SLURM_INTRA eliminates this by distributing work directly to allocated resources using srun.

	`Backend.SLURM`	`Backend.SLURM_INTRA`
Dispatch	`sbatch` (job arrays)	`srun` (within allocation)
Queue wait	Yes (seconds to minutes)	None
Requires	Cluster access	Active `salloc`/`sbatch`
Multi-node	Yes	Yes
GPU isolation	SLURM scheduler	`--gres=gpu:N` per step

Use SLURM_INTRA when: You’re in an interactive salloc session and want zero-latency dispatch across your reserved nodes.

Use SLURM when: You’re submitting a pipeline from a login node and want SLURM to manage scheduling and resource allocation.

The one-parameter switch¶

If you’ve already built a pipeline with Backend.SLURM, switching to intra-allocation dispatch is a single parameter change:

# Queue-based (submits sbatch jobs)
step = pipeline.run(MyOperation, inputs=..., backend=Backend.SLURM)

# Intra-allocation (dispatches via srun, zero queue wait)
step = pipeline.run(MyOperation, inputs=..., backend=Backend.SLURM_INTRA)

Operations, inputs, params, output wiring, and result inspection all stay identical.

Building a pipeline inside an allocation¶

A typical workflow: allocate resources interactively, then run a pipeline that dispatches across the reserved CPUs.

# Step 1: Reserve resources
salloc --cpus-per-node=8 --time=00:30:00

# Step 2: Run your pipeline script
pixi run python docs/tutorials/execution/demo_slurm_intra.py

The pipeline mixes local and intra-allocation steps:

Step	Operation	Backend	Why
0	`DataGenerator`	LOCAL	Fast -- no reason to use srun overhead
1	`DataTransformer`	SLURM_INTRA	Compute-intensive, dispatched via srun
2	`MetricCalculator`	SLURM_INTRA	Compute-intensive, dispatched via srun

pipeline = PipelineManager.create(
    name="slurm_intra_tutorial",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

Step 0: Generate data (local)¶

DataGenerator is fast -- run it locally to avoid srun launch overhead.

step0 = pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 8, "seed": 42},
    backend=Backend.LOCAL,
)
print(f"Generated {step0.succeeded_count} datasets locally")

Step 1: Transform data (srun)¶

Switch to Backend.SLURM_INTRA. Each execution unit becomes an srun step dispatched to one of the allocated nodes. SLURM picks a node with available resources and binds GPUs automatically.

step1 = pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
    params={"scale_factor": 0.5, "variants": 2, "seed": 100},
    backend=Backend.SLURM_INTRA,
    resources={"cpus": 2, "memory_gb": 2},
)
print(f"Transformed {step1.succeeded_count} datasets via srun")

Step 2: Compute metrics (srun)¶

Output wiring works the same across backends -- output("transform", "dataset") resolves to the artifacts produced by the srun step.

step2 = pipeline.run(
    operation=MetricCalculator,
    name="metrics",
    inputs={"dataset": output("transform", "dataset")},
    backend=Backend.SLURM_INTRA,
    resources={"cpus": 1, "memory_gb": 1},
)
print(f"Computed metrics for {step2.succeeded_count} datasets via srun")

summary = pipeline.finalize()
print(
    f"Pipeline complete: {summary['total_steps']} steps, success={summary['overall_success']}"
)

Inspect results¶

The same inspection tools work regardless of backend. Results land in the same Delta Lake tables whether steps ran locally, via sbatch, or via srun.

inspect_pipeline(DELTA_ROOT)

inspect_metrics(DELTA_ROOT, step_number=2)

Overriding resources¶

With SLURM_INTRA, resource overrides carve resources from the existing allocation rather than requesting them from the scheduler. Each srun step gets the specified CPUs and memory from the pool of reserved resources.

The partition field is ignored -- the allocation’s partition is already determined by the salloc/sbatch that created it.

pipeline = PipelineManager.create(
    name="slurm_intra_overrides",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

step0 = pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 8, "seed": 42},
    backend=Backend.LOCAL,
)
print(f"Generated {step0.succeeded_count} datasets for override examples")

# Each srun step gets 2 CPUs and 2 GB from the 8-CPU allocation
step = pipeline.run(
    operation=DataTransformer,
    inputs={"dataset": output("generate", "datasets")},
    params={"scale_factor": 0.5, "variants": 1, "seed": 200},
    backend=Backend.SLURM_INTRA,
    resources={
        "cpus": 2,
        "memory_gb": 2,
        "time_limit": "00:10:00",
    },
)
print(f"Transformed {step.succeeded_count} datasets with custom resources")

These translate to srun flags:

`resources` field	srun flag	Effect
`gpus`	`--gres=gpu:N`	Binds N GPUs, sets `CUDA_VISIBLE_DEVICES`
`cpus`	`--cpus-per-task`	CPU cores per step
`memory_gb`	`--mem`	Memory per step
`time_limit`	`--time`	Wall-clock limit per step

With an 8-CPU allocation and cpus=2 per step, up to 4 srun steps can run concurrently. Excess steps wait in Python until a slot opens.

Controlling batching¶

Batching works the same as with Backend.SLURM:

artifacts_per_unit controls how many artifacts each execution unit processes.
units_per_worker controls how many units each srun step handles sequentially.

100 artifacts
  / artifacts_per_unit=10  ->  10 execution units
  / units_per_worker=2    ->   5 srun steps

With SLURM_INTRA, there is no array size limit -- each srun step is launched independently. However, the srun_launch_concurrency setting (default 128) caps how many srun processes run simultaneously to avoid overwhelming the SLURM controller. Excess steps queue in Python until a slot opens.

# 4 artifacts per unit, 2 units per srun step
step = pipeline.run(
    operation=MetricCalculator,
    inputs={"dataset": output("generate", "datasets")},
    backend=Backend.SLURM_INTRA,
    execution={"artifacts_per_unit": 4, "units_per_worker": 2},
)
print(f"Computed metrics for {step.succeeded_count} datasets with custom batching")

Non-blocking dispatch with `submit()`¶

pipeline.submit() works with SLURM_INTRA the same way it works with other backends -- it returns a StepFuture immediately while the srun steps run in the background. Use this to overlap independent steps.

pipeline = PipelineManager.create(
    name="slurm_intra_submit_demo",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 8, "seed": 42},
    backend=Backend.LOCAL,
)

# Submit two independent srun steps
future_a = pipeline.submit(
    operation=DataTransformer,
    name="transform_a",
    inputs={"dataset": output("generate", "datasets")},
    params={"scale_factor": 0.5, "variants": 1, "seed": 100},
    backend=Backend.SLURM_INTRA,
)

future_b = pipeline.submit(
    operation=DataTransformer,
    name="transform_b",
    inputs={"dataset": output("generate", "datasets")},
    params={"scale_factor": 2.0, "variants": 1, "seed": 200},
    backend=Backend.SLURM_INTRA,
)

# Both are running on the allocated nodes concurrently
print(f"Step 1 status: {future_a.status}")
print(f"Step 2 status: {future_b.status}")

# Collect results
result_a = future_a.result()
result_b = future_b.result()
print(f"Step 1: {result_a.succeeded_count} artifacts")
print(f"Step 2: {result_b.succeeded_count} artifacts")

pipeline.finalize()

Debugging srun runs¶

Develop locally first¶

Run with Backend.LOCAL until the pipeline logic is correct, then switch to SLURM_INTRA for the full allocation.

Allocation validation¶

SLURM_INTRA checks for SLURM_JOB_ID in the environment. If you run outside an allocation, you’ll see a warning:

UserWarning: Backend 'slurm_intra' selected for 'data_transformer'
but SLURM_JOB_ID is not set. Are you inside an salloc/sbatch?

The SlurmTaskRunner raises a RuntimeError if SLURM_JOB_ID is missing, so the pipeline fails fast rather than submitting broken jobs.

Monitor steps with `squeue`¶

srun steps appear as job steps in SLURM. Use squeue -s to see them:

$ squeue -s -j $SLURM_JOB_ID
  STEPID     NAME    STATE   TIME
  12345.0    srun    RUNNING 0:02
  12345.1    srun    RUNNING 0:02
  12345.2    srun    RUNNING 0:01

Preserve working directories¶

Pass preserve_working=True when creating the pipeline to keep worker sandbox directories after execution.

debug_pipeline = PipelineManager.create(
    name="slurm_intra_debug",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
    preserve_working=True,
    preserve_staging=True,
)

print(f"preserve_working: {debug_pipeline.config.preserve_working}")
print(f"preserve_staging: {debug_pipeline.config.preserve_staging}")

Summary¶

Backend.SLURM_INTRA dispatches work via srun within an existing allocation
Zero queue wait -- srun steps launch on reserved resources immediately
Operations, inputs, params, and output wiring are identical to other backends
Override resources per step to carve GPUs, CPUs, and memory from the allocation
Batching works the same as SLURM, with a concurrency cap of 128 simultaneous steps
Use submit() for non-blocking steps that run concurrently
Develop locally first, switch to SLURM_INTRA inside an salloc session

Next steps¶

Running on SLURM -- Queue-based SLURM execution for production pipelines
Configuring Execution -- Complete configuration reference
Execution Flow -- How the framework dispatches and tracks work