Data Collection & Analytics

Data collection is defined via a collect.py script placed inside each stage directory.

1. Custom collect.py Implementation

Your collect() function executes inside each individual SPoint run directory and returns a dictionary of values or numpy arrays to compile:

def collect():
    import numpy as np
    # Loads output files from the local SPoint directory
    loss = np.load("loss_history.npy")
    accuracy = np.load("accuracy.npy")

    return {
        "loss": loss,
        "accuracy": accuracy
    }

2. Analyzing Data with the Data_Manager API

After running workflow data collect, data is compiled into a single data container located at data/{stage_name}.dm. Load and query it:

from snap_workflow import Data_Manager

# Load stage output
dm = Data_Manager("simulation")

# Filter by variable criteria
high_res_data = dm[dm["mesh_size"] >= 128]

# Split by unique parameter values
mesh_sizes, sub_dms = dm.split("mesh_size")
for size, sub_dm in zip(mesh_sizes, sub_dms):
    print(f"Mesh size: {size}")
    # Iterate through individual runs
    for run in sub_dm:
        print(f"Run variables: {run.keys()}")
        loss_history = run["loss"]