Object Hierarchy

🧱 Metaflow Object Hierarchy

Here’s the hierarchy in top-down order:

Flow
 └── Run
      └── Step
           └── Task
                └── Artifact

Think of it like a tree-shaped object model that helps you programmatically navigate past executions, debug, inspect data, or build automation.

1. Flow

Represents the entire flow class (your pipeline).

from metaflow import Flow
flow = Flow('MyFlow')

• .latest_run
• .runs → list of all runs
• .name → name of the flow

2. Run

Represents a single execution of your flow.

run = Flow('MyFlow').latest_run

• .id → unique run ID (e.g., 1699374571123927)
• .steps → dictionary of steps in this run
• .successful → boolean
• .created_at → timestamp
• .user_tags, .system_tags

You can loop through runs:

for run in Flow('MyFlow').runs():
    print(run.id, run.successful)

3. Step

Represents a single step (e.g., start, train, join) inside a run.

step = run['train']

• .name → step name
• .tasks → list of tasks for this step (can be more than one for foreach)
• .successful → if step finished without error

4. Task

Represents a single task (actual execution unit of a step).

task = step.task  # or `step.tasks[0]`

• .attempt → retry attempt number
• .finished → if task finished
• .stdout, .stderr
• .tags
• .metadata

Most importantly:

task.data.<artifact_name>

This is how you retrieve artifacts from past runs.

5. Artifact

This is the actual data (variable) persisted by a step.

acc = task.data.accuracy

These are the self.var = ... assignments in your step code.

Artifacts:

• Are stored in Metaflow's data store
• Are tied to the task, step, and run that created them
• Are read-only from the perspective of another run

🧪 Example: Traversing the Object Tree

from metaflow import Flow

flow = Flow('GridSearchFlow')
run = flow.latest_run
step = run['train_model']  # Step where training occurred
for task in step.tasks:
    result = task.data.result
    print(result)

🏷️ ✅ Common Properties (Available on All Objects)

These are useful when writing generic traversal utilities.

🔄 Flow-Level (Flow object)

Accessed via:

from metaflow import Flow
flow = Flow('MyFlow')

➡️ Great for programmatically pulling the last N runs, analyzing outputs, or rerunning failed jobs.

🔁 Run-Level (Run object)

run = Flow('MyFlow').latest_run

➡️ The data property is especially handy to get final results fast without needing to drill down manually.

🔁 Step-Level (Step object)

step = run['train']

➡️ If using foreach, this is where .tasks() becomes important for aggregating parallel task results.

⚙️ Task-Level (Task object)

task = step.task

➡️ The data and stdout properties are the most common to use in inspection scripts or dashboards.

🧪 Minimal Working Example (From Docs)

from metaflow import Step

step = Step('DebugFlow/2/start')  # format: FlowName/RunID/StepName

if step.task.successful:
    print("Finished at:", step.task.finished_at)
    print("Stdout:")
    print(step.task.stdout)
    print("Artifacts:", list(step.task.data.__dict__.keys()))

🧠 Bonus Tips

• pathspec can be used to fetch any object directly using Step(...), Task(...), etc.
• Use .tags in combination with filters to group runs or detect anomalies
• code and environment_info are useful for reproducibility and audit logging
• Combine .data with pandas.DataFrame or plotly to visualize experiment results

✅ Summary Table