Storage Layout and Logging

What you’ll learn¶

The three-path model: delta, staging, and working directories
What’s inside the runs directory after a pipeline completes
Delta Lake table structure and what each table contains
Working directory defaults and when to preserve them
Configuring logging: level, file handler, noise suppression

Prerequisites: First Pipeline, Exploring Results
Estimated time: 15 minutes
GPU required: No.

Every pipeline run produces files in three directory trees. Understanding this layout helps you debug failures, manage disk space, and configure paths for your environment.

from __future__ import annotations

import tempfile
from pathlib import Path

import polars as pl

from artisan.operations.examples import DataGenerator, DataTransformer, MetricCalculator
from artisan.orchestration import PipelineManager
from artisan.schemas.enums import TablePath
from artisan.utils import configure_logging, tutorial_setup
from artisan.visualization import inspect_pipeline, inspect_step

The three-path model¶

Every pipeline is configured with three root directories:

Path	Purpose	Lifetime
`delta_root`	Delta Lake tables — artifacts, metadata, provenance	Permanent
`staging_root`	Worker output staging area before commit	Temporary (cleared after commit)
`working_root`	Sandbox directories where operations execute	Temporary (cleaned after each execution)

The delta root is the pipeline’s durable store. All artifacts, execution records, and provenance edges live here as Delta Lake tables.

The staging root is scratch space where workers write outputs before they’re committed to the delta store. After a successful commit, staging files are removed.

The working root is where each operation’s sandbox directory is created. Operations write intermediate files here during execution. By default, sandboxes are cleaned up after each execution completes.

env = tutorial_setup("storage_layout")

print(f"delta_root:   {env.delta_root}")
print(f"staging_root: {env.staging_root}")
print(f"working_root: {env.working_root}")
print(f"\nAll under: {env.runs_dir}")

Running a pipeline and exploring the output¶

Let’s run a 3-step pipeline and inspect the directory tree afterward.

pipeline = PipelineManager.create(
    name="storage_demo",
    delta_root=env.delta_root,
    staging_root=env.staging_root,
    working_root=env.working_root,
)
output = pipeline.output

pipeline.run(
    operation=DataGenerator,
    name="generate",
    params={"count": 3, "seed": 42},
)
pipeline.run(
    operation=DataTransformer,
    name="transform",
    inputs={"dataset": output("generate", "datasets")},
)
pipeline.run(
    operation=MetricCalculator,
    name="metrics",
    inputs={"dataset": output("transform", "dataset")},
)

result = pipeline.finalize()
print(f"Pipeline complete: {result['total_steps']} steps")

def show_tree(path: Path, prefix: str = "", max_depth: int = 2, depth: int = 0) -> None:
    """Display a directory tree up to max_depth."""
    if depth >= max_depth or not path.is_dir():
        return
    entries = sorted(path.iterdir())
    for i, entry in enumerate(entries):
        connector = (
            "\u2514\u2500\u2500 " if i == len(entries) - 1 else "\u251c\u2500\u2500 "
        )
        suffix = "/" if entry.is_dir() else ""
        print(f"{prefix}{connector}{entry.name}{suffix}")
        if entry.is_dir():
            extension = "    " if i == len(entries) - 1 else "\u2502   "
            show_tree(entry, prefix + extension, max_depth, depth + 1)


print(f"{env.runs_dir.name}/")
show_tree(env.runs_dir, max_depth=3)

Key observations:

delta/ contains the Delta Lake tables — this is the permanent record
staging/ is empty — staging files were cleaned up after commit
working/ is empty — sandbox directories were cleaned up after execution
logs/ contains pipeline.log from the pipeline’s logging

What’s in the Delta Lake tables¶

The framework stores data in several Delta Lake tables under delta_root:

Table	Path	Purpose
Artifact index	`artifacts/index`	Global registry — artifact ID → type, step, metadata
Artifact edges	`provenance/artifact_edges`	Derivation edges between artifacts
Execution edges	`provenance/execution_edges`	Execution-level provenance
Executions	`orchestration/executions`	One row per execution attempt
Steps	`orchestration/steps`	Step state transitions

Content tables (e.g., artifacts/data/, artifacts/metrics/) are managed by the artifact type registry — one table per artifact type.

The TablePath enum lists the framework tables:

for tp in TablePath:
    table_dir = env.delta_root / tp
    exists = "\u2713" if table_dir.exists() else "\u2717"
    print(f"  {exists} {tp.value}")

The inspect_pipeline and inspect_step functions are convenience wrappers that query these tables:

inspect_pipeline(env.delta_root)

inspect_step(env.delta_root, step_number=0)

You can also read the tables directly with Polars:

artifact_index = pl.read_delta(str(env.delta_root / TablePath.ARTIFACT_INDEX))
print(artifact_index.head())

Working directory defaults¶

When you create a pipeline without specifying working_root, it defaults to the system temp directory. This is intentional — sandbox directories are ephemeral scratch space.

print(f"Default working_root: {Path(tempfile.gettempdir())}")
print(f"Tutorial working_root: {env.working_root}")

For debugging, two flags control cleanup:

preserve_working=True — keep sandbox directories after execution so you can inspect intermediate files
preserve_staging=True — keep staged Parquet files so you can inspect them before they’re committed to Delta Lake

Pass these to PipelineManager.create() when debugging.

Configuring logging¶

The configure_logging function sets up artisan’s logging. It accepts:

Parameter	Default	Purpose
`level`	`"INFO"`	Log level for artisan loggers
`suppress_noise`	`True`	Quiet Prefect, httpx, asyncio loggers
`loggers`	`("artisan",)`	Which logger hierarchies to configure
`logs_root`	`None`	When set with `level="DEBUG"`, adds a rotating file handler

PipelineManager.create() calls configure_logging automatically, setting logs_root to <delta_root>/../logs. The console handler is always active with Rich-colored output.

# configure_logging is idempotent — safe to call multiple times
configure_logging(level="INFO", suppress_noise=True)

The suppress_noise=True default sets noisy third-party loggers (Prefect, httpx, httpcore, asyncio) to CRITICAL level. Without this, debug output is dominated by HTTP client chatter.

# Check if the pipeline wrote a log file
log_path = env.runs_dir / "logs" / "pipeline.log"
if log_path.exists():
    lines = log_path.read_text().splitlines()
    print(f"Pipeline log: {len(lines)} lines")
    print(f"First: {lines[0]}")
    print(f"Last:  {lines[-1]}")
else:
    print("No pipeline.log (file handler requires level='DEBUG')")

Crash recovery¶

If a pipeline crashes mid-run, staging files from completed steps may be left on disk. The recover_staging config flag (default: True) tells PipelineManager.create() to auto-commit any leftover staging files when the next pipeline is initialized.

from artisan.schemas.orchestration.pipeline_config import PipelineConfig

print(
    f"recover_staging default: {PipelineConfig.model_fields['recover_staging'].default}"
)

This means you generally don’t need to worry about partial writes from crashes — the next pipeline run cleans them up automatically.

Summary¶

Concept	Detail
`delta_root`	Permanent Delta Lake tables — artifacts, metadata, provenance
`staging_root`	Temporary worker output area, cleared after commit
`working_root`	Temporary sandbox directories, defaults to system temp
`preserve_working`	Keep sandbox contents for debugging
`preserve_staging`	Keep staged Parquet for inspection
`TablePath` enum	Framework tables: artifact index, provenance edges, executions, steps
Console logging	Always active, Rich-colored
Pipeline file log	Written at `<runs_dir>/logs/pipeline.log`
`configure_logging`	Set level, suppress noisy third-party loggers
`recover_staging`	Auto-commit leftover staging from crashed runs (default: True)

Next steps¶

Error Handling in Practice — Runtime failures, failure logs, and FailurePolicy
Step Overrides — Override execution config, backend, and parameters per step
Resume and Caching — How caching interacts with the delta store