Tutorials are hands-on, learning-oriented guides. Work through them from start to finish in a Jupyter notebook. Each tutorial builds on concepts from the previous ones.
Recommended order: Start with Getting Started (in order), then Pipeline Design, then Execution, then Analysis, then Writing Operations.
Getting Started¶
Build your first pipeline and explore the results.
First Pipeline (~15 min) -- Build and run a seven-step pipeline with example operations
Exploring Results (~15 min) -- Query Delta Lake tables and inspect artifacts
Pipeline Design¶
Reusable patterns for common pipeline topologies, each with provenance graph illustrations.
Sources and Sequencing (~10 min) -- Linear pipelines and data ingestion
Branching and Merging (~10 min) -- Fan-out and fan-in patterns
Metrics and Filtering (~10 min) -- Score-based filtering with auto-discovered metrics
Multi-Input Operations (~15 min) -- Operations that consume multiple input roles
Diamonds and Iteration (~15 min) -- Diamond DAGs, output lineage, and iterative refinement
Composable Operations (~15 min) -- Compose operations into reusable units with collapsed or expanded execution
Execution¶
Running pipelines, caching, batching, error handling, and step-level configuration.
Run vs Submit (~10 min) -- Blocking vs non-blocking step execution
Resume and Caching (~15 min) -- List runs, resume pipelines, and step-level caching
Batching and Performance (~20 min) -- Two-level batching model, ExecutionConfig fields, and tuning guidelines
Error Handling in Practice (~15 min) -- Runtime failures, failure logs, FailurePolicy, and empty input cascades
Storage Layout and Logging (~15 min) -- Three-path model, Delta Lake tables, directory layout, and logging configuration
Step Overrides (~15 min) -- All
pipeline.run()override parameters: params, name, execution, backend, and moreSLURM Execution (~10 min) -- Run operations on a SLURM cluster
Pipeline Cancellation (~10 min) -- Cancel pipelines with
cancel()or Ctrl+C, inspect results, and re-runSLURM Intra-Allocation (~10 min) -- Dispatch work via srun within an existing SLURM allocation
Analysis¶
Provenance visualization, diagnostics, and interactive analysis tools.
Provenance Graphs (~10 min) -- Macro and micro provenance graph rendering
Lineage Tracing (~15 min) -- Trace ancestors and descendants programmatically
Interactive Filter (~15 min) -- Explore metric distributions and commit filter thresholds interactively
Timing Analysis (~10 min) -- Profile step and execution performance with PipelineTimings
Writing Operations¶
Build custom operations and composites from scratch.
Writing an Operation (~15 min) -- Build a custom operation from scratch
Writing a Composite (~20 min) -- Build a reusable CompositeDefinition that composes multiple operations