Multi-Input Operations

What you’ll learn¶

Why operations sometimes need multiple input roles
How group_by strategies control which artifacts get paired together
How the LINEAGE strategy pairs artifacts that share a common ancestor
How LINEAGE handles 1:N expansion for parameter sweeps

Prerequisites: Sources and Sequencing, Branching and Merging Estimated time: 15 minutes

Every tutorial so far has used single-input operations — each step consumes one stream of artifacts. But some operations need artifacts from multiple streams simultaneously. A transformation might need both a dataset and a config file. A comparison might need outputs from two different branches.

When an operation declares two input roles, the framework faces a pairing problem: if role A has 3 artifacts and role B has 6 artifacts, which A goes with which B? The group_by strategy answers this question.

from __future__ import annotations

from artisan.operations.examples import (
    DataGenerator,
    DataTransformerConfig,
    DataTransformerScript,
)
from artisan.orchestration import PipelineManager
from artisan.utils import tutorial_setup
from artisan.visualization import build_macro_graph, build_micro_graph

Graph legend: See Sources and Sequencing for box/arrow key.

The pairing problem¶

Consider an operation that needs two inputs: a dataset and a config. If the pipeline has produced 3 datasets and 3 configs, the framework needs to decide which dataset goes with which config before dispatching execution units.

The operation declares this via group_by, which accepts one of three strategies:

Strategy	Pairing rule	Result count	Use when
LINEAGE	Match artifacts that share a common ancestor	Varies (depends on lineage graph)	Config was derived from the dataset
CROSS_PRODUCT	Every A × every B	N × M	You want all combinations (parameter sweeps)
ZIP	Positional: 1st with 1st, 2nd with 2nd	N (lengths must match)	Artifacts are already aligned by position

The operation author chooses the strategy when writing the operation. As a pipeline builder, you wire inputs — the framework handles the rest.

LINEAGE pairing¶

LINEAGE is the most common multi-input strategy. It pairs artifacts that share a common ancestor in the provenance graph.

The canonical example is the config + execute pattern:

DataGenerator produces 3 datasets
DataTransformerConfig reads each dataset and produces a config for it
DataTransformerScript needs both the dataset and its config

Because each config was derived from a specific dataset, they share lineage. The framework uses this lineage to pair them automatically — dataset A gets config A, dataset B gets config B, and so on.

env_lineage = tutorial_setup("lineage_basic")


def run_lineage_basic():
    pipeline = PipelineManager.create(
        name="lineage_basic",
        delta_root=env_lineage.delta_root,
        staging_root=env_lineage.staging_root,
        working_root=env_lineage.working_root,
    )
    output = pipeline.output

    # Step 0: Generate 3 datasets
    pipeline.run(
        operation=DataGenerator, name="generate", params={"count": 3, "seed": 42}
    )

    # Step 1: Produce a config for each dataset (1:1 lineage)
    pipeline.run(
        operation=DataTransformerConfig,
        name="configure",
        inputs={"dataset": output("generate", "datasets")},
        params={"scale_factors": [0.5]},
    )

    # Step 2: Execute with paired dataset + config (multi-input, group_by=LINEAGE)
    pipeline.run(
        operation=DataTransformerScript,
        name="execute",
        inputs={
            "dataset": output("generate", "datasets"),
            "config": output("configure", "config"),
        },
    )

    return pipeline.finalize()


result_lineage = run_lineage_basic()

Let’s look at the macro graph. Notice that step 2 (data_transformer_script) has two incoming edges — one from the datasets (step 0) and one from the configs (step 1). This is the visual signature of a multi-input operation.

build_macro_graph(env_lineage.delta_root)

Use the stepper to walk through each step. At step 2, you can see the orange lineage arrows that connect each config back to its source dataset. These arrows are what LINEAGE pairing follows.

build_micro_graph(env_lineage.delta_root)

LINEAGE with 1:N expansion (parameter sweep)¶

LINEAGE pairing also handles the case where one dataset produces multiple configs. DataTransformerConfig creates a config for every combination of scale_factors × noise_amplitudes. Each config traces back to its source dataset, so LINEAGE pairs correctly even with unequal counts.

With 1 dataset and 3 scale factors × 2 noise amplitudes = 6 configs, the framework produces 6 execution units — each pairing the single dataset with one of its 6 configs.

env_sweep = tutorial_setup("lineage_sweep")


def run_lineage_sweep():
    pipeline = PipelineManager.create(
        name="lineage_sweep",
        delta_root=env_sweep.delta_root,
        staging_root=env_sweep.staging_root,
        working_root=env_sweep.working_root,
    )
    output = pipeline.output

    # 1 dataset
    pipeline.run(
        operation=DataGenerator, name="generate", params={"count": 1, "seed": 42}
    )

    # 6 configs (3 scale_factors × 2 noise_amplitudes)
    pipeline.run(
        operation=DataTransformerConfig,
        name="configure",
        inputs={"dataset": output("generate", "datasets")},
        params={
            "scale_factors": [0.1, 0.5, 1.0],
            "noise_amplitudes": [1, 2],
        },
    )

    # 6 execution units (each dataset paired with one config via lineage)
    pipeline.run(
        operation=DataTransformerScript,
        name="execute",
        inputs={
            "dataset": output("generate", "datasets"),
            "config": output("configure", "config"),
        },
    )

    return pipeline.finalize()


result_sweep = run_lineage_sweep()

build_macro_graph(env_sweep.delta_root)

The graph shows 6 configs flowing into step 2 alongside the 1 dataset. Use the stepper to trace the lineage arrows — each config points back to the same source dataset.

build_micro_graph(env_sweep.delta_root)

Scaling to multiple datasets¶

The same pattern works with multiple datasets. With 3 datasets and 2 scale factors, DataTransformerConfig produces 2 configs per dataset (6 total). LINEAGE correctly pairs each dataset with only its configs, yielding 6 execution units — not the 18 you’d get from a cross product.

env_multi = tutorial_setup("lineage_multi_dataset")


def run_lineage_multi_dataset():
    pipeline = PipelineManager.create(
        name="lineage_multi_dataset",
        delta_root=env_multi.delta_root,
        staging_root=env_multi.staging_root,
        working_root=env_multi.working_root,
    )
    output = pipeline.output

    # 3 datasets
    pipeline.run(
        operation=DataGenerator, name="generate", params={"count": 3, "seed": 42}
    )

    # 2 configs per dataset = 6 configs total
    pipeline.run(
        operation=DataTransformerConfig,
        name="configure",
        inputs={"dataset": output("generate", "datasets")},
        params={"scale_factors": [0.5, 2.0]},
    )

    # 6 execution units (each dataset paired with its 2 configs)
    pipeline.run(
        operation=DataTransformerScript,
        name="execute",
        inputs={
            "dataset": output("generate", "datasets"),
            "config": output("configure", "config"),
        },
    )

    return pipeline.finalize()


result_multi = run_lineage_multi_dataset()

build_macro_graph(env_multi.delta_root)

build_micro_graph(env_multi.delta_root)

The stepper shows the key difference from a cross product: each dataset is paired only with the configs that were derived from it. Dataset 0 gets configs 0 and 1, dataset 1 gets configs 2 and 3, and so on. LINEAGE uses the provenance graph to enforce this — no manual wiring needed.

Summary¶

In this tutorial you learned:

Multi-input operations consume artifacts from multiple input roles, and the framework pairs them using a group_by strategy
LINEAGE pairs artifacts that share a common ancestor — the most common strategy, used for config+execute and derived-data patterns
LINEAGE handles 1:N expansion — parameter sweeps produce multiple configs per dataset, and lineage pairs them correctly
The framework also supports CROSS_PRODUCT (all combinations) and ZIP (positional matching) strategies — see the strategy table above

Next: Diamonds and Iteration — diamond DAGs and iterative refinement loops

Deeper understanding: Operations Model — how multi-input pairing is resolved