Walkthrough

This walkthrough is based on the experimaestro-demo repository, which provides a complete working example of using experimaestro for hyperparameter search in deep learning.

Tip

Running the walkthrough To follow along, clone the demo repository:

git clone https://github.com/experimaestro/experimaestro-demo.git
cd experimaestro-demo

Setup

Follow these steps to ensure a working environment:

I - Installation

Clone this repository.

git clone https://github.com/experimaestro/experimaestro-demo.git && cd experimaestro-demo

II - Setting up workspaces

Experimaestro automatically creates folders for all jobs. We still have to specify where those folders will be created !

Two options:

1. Define a default workspace (recommended)

Workspace settings are stored in $HOME/.config/experimaestro/settings.yaml (see documentation).

This repository contains a default settings.yaml for quick testing. It uses triggers: so that running the MNIST_train experiment automatically picks the right workspace — no --workspace / --workdir flag needed:

workspaces:
  - id: mnist
    path: ~/experiments/mnist_xp
    triggers:
      - "MNIST_*"   # any experiment whose id matches this glob picks this workspace

💡 Tip: This command will write the default config above in your settings.yaml if it doesn’t already exist.

FILE="$HOME/.config/experimaestro/settings.yaml"; if [ ! -f $FILE ] ; then cat ./xpm_settings.yaml > $FILE ; else echo "$FILE already exists !"; fi

2. Specify workspace inline

If there is no settings.yaml, you can still specify where you want your experiments to run with:

mkdir $HOME/experiments
uv run experimaestro run-experiment --workdir $HOME/experiments ...

III - Launchers

Launchers, specify how a task should be launched. There exist two types of launchers at the moment, direct launcher (starting a new process) or through slurm. The launchers.py file is there to tell experimaestro how to launch your tasks. Without it, it will just launch everything in parallel (subprocesses): This is actually fine for this lightweight experiment, so this section is optional.

1. Generating a launchers file (single host)

If you run on your laptop / a single machine, experimaestro can auto-detect your hardware (CPU, CUDA GPUs, Apple Silicon MPS) and generate a launchers.py for you:

uv run experimaestro launchers direct generate

It writes ~/.config/experimaestro/launchers.py with a memory-based token system so concurrent tasks don’t oversubscribe RAM / GPU memory. After this you can use find_launcher(...) in experiment.py without any cluster configuration — see the launcher section below.

For SLURM clusters, the equivalent is experimaestro launchers slurm generate (interactive TUI). See the launchers documentation for advanced setups.

The experiment structure

Lets now have a look at how an experimaestro project can be structured, you can also choose to begin by running the experiment by jumping directly to running the experiment, and come back later for details.

We will now have a closer look at the key files of this demo repository. In short, in the mnist_xp folder we have

learn.py contains the core code, here a CNN configuration and the learning and evaluation tasks
data.py contains the code that provides dataset
experiment.py orchestrates the experiment
params.yamlcontains configuration parameters for our experiment

We will also point out the most important objects that allow us to run the experiment.

`learn.py`: defining the model and tasks

This file contains the code that defines a CNN model, and specifies how to learn and evaluate an image classification model.

The most important concept at this stage is that of a configuration object, that serves as a structured template to define parameters and settings for tasks and experiments.

Let’s see a first configuration that defines a CNN model

class CNN(Config, nn.Module):
    """Defines a CNN model"""

    n_layers: Param[int] = 2
    """Number of Hidden layers"""

    hidden_dim: Param[int] = 64
    """Number of hidden units"""

    kernel_size: Param[int] = 3
    """Kernel size of the CNN"""
    ...

A configuration is characterized by:

Deriving from the experimaestro.Config class
Defining experimental parameters (number of layer, hidden dimension, kernel size) that can change the outcome of an experiment.
Use docstring to document the parameters – which can be used to generate a documentation when the number of experimental configuration becomes high (e.g. the documentation of a cross-scorer in the IR library)

This configuration can latter be used as a dataclass when configuring the experiments, e.g. CNN.C(n_layers=3).For each instance of a configuration, we can compute a unique identifier that changes only if one or more experimental parameter changes. For instance, CNN.C() and CNN.C(n_layers=2) have the same identifier, which is different from CNN.C(n_layers=1).

Another important type of object are Task objects. They correspond to the code that can be run, e.g. on a SLURM cluster or locally, to perform a part of the experiment, e.g. learning the CNN model from data. Let us take a closer look at some bits of the code:

class Learn(Task):
    """Learn to classify an image into a pre-defined set of classes"""

    parameters_path: Meta[Path] = field(default_factory=PathGenerator("parameters.pth"))
    """Path to store the model parameters"""

    data: Param[LabelledImages]
    """Train data are labelled images"""

    model: Param[CNN]
    """The model we are training"""

    ...

    def execute(self):
        ...

You can notice that tasks are specific types of configuration. You can also notice that parameters can be other configurations, allowing to compose experimental components easily.

The main difference between Config and Task is the execute method. The latter contain the code to be run when the task is run by the task scheduler. Another important thing in the above example

Please have a look at the Task documentation for more details.

Dataset handling: `data.py`

The first code in this file defines a labelled image dataset, which is a light abstraction of the basic datasets we manipulate in our experiment. We use the datamaestro library, which is tightly associated with experimaestro, for this purpose.

class LabelledImages(Base, ABC):
    @abstractmethod
    def torchvision_dataset(self, **kwargs) -> VisionDataset:
        ...

The datamaestro.Base class is the central class that any dataset class should derive from.

Leveraging torchvision, We then define the MNIST data with the following code:

class MNISTLabelledImages(LabelledImages):
    root: Meta[Path]
    train: Param[bool]

    def torchvision_dataset(self, **kwargs) -> VisionDataset:
        return MNIST(self.root, train=self.train, **kwargs)


def download_mnist(context: Context, root: Path, force=False):
    logging.info("Downloading in %s", root)
    for train in [False, True]:
        MNIST(root, train=train, download=True)


@custom_download("root", download_mnist)
@dataset(id="com.lecun.mnist")
def mnist(root: Path) -> Supervised[LabelledImages, None, LabelledImages]:
    """This corresponds to a dataset with an ID `com.lecun.mnist`"""
    return Supervised(
        train=MNISTLabelledImages.C(root=root, train=True),
        test=MNISTLabelledImages.C(root=root, train=False),
    )

🚀 Going further datamaestro can help managing many datasets by providing a unified interface to many datasets, as well as providing many utilities to download files from various sources. One such example is datamaestro-image that contains e.g. the MNIST dataset and datamaestro-text that contains NLP and IR datasets. Feel free to contribute!

👓 See how MNIST is defined in datamaestro-image

from datamaestro_image.data import ImageClassification, LabelledImages, Base, IDXImage
from datamaestro.download.single import filedownloader
from datamaestro.definitions import dataset
from datamaestro.data.tensor import IDX


@filedownloader(
    "train_images.idx",
    "https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz",
)
@filedownloader(
    "train_labels.idx",
    "https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz",
)
@filedownloader(
    "test_images.idx",
    "https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz",
)
@filedownloader(
    "test_labels.idx",
    "https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz",
)
@dataset()
def mnist(train_images, train_labels, test_images, test_labels) -> ImageClassification:
    """The MNIST database

    The MNIST database of handwritten digits, available from this page, has a
    training set of 60,000 examples, and a test set of 10,000 examples. It is a
    subset of a larger set available from NIST. The digits have been
    size-normalized and centered in a fixed-size image.
    """
    return ImageClassification(
        train=LabelledImages(
            images=IDXImage(path=train_images), labels=IDX(path=train_labels)
        ),
        test=LabelledImages(
            images=IDXImage(path=test_images), labels=IDX(path=test_labels)
        ),
    )

`experiment.py`

This file describe the experiment, an experiment may load and launch several Tasks .. There are two key elements in experiment.py:

Configuration

The most important concept in Experimaestro is that of a configuration. In Experimaestro, a configuration object is a fundamental concept used to specify parameters and settings for tasks and experiments. It acts as a structured way to input the necessary details to execute a task or a series of tasks.

💡 Have you noticed ? Under the hood, the Task that we defined above contains a Configuration ! This enables experimaestro to create a unique Task ID for each set of Params.

Here, we first define a ConfigurationBase object that describes our experiment settings.

from experimaestro import ConfigurationBase
...
@configuration
class Configuration(ConfigurationBase):
	# experimaestro Task parameters
	## Model
	n_layers: int = 3 # number of Hidden layers
	...
	# Misc
	launcher: str = """duration=3h & cuda(mem=4G)*1 & cpu(cores=2)"""
...

This class describes the configuration needed to run our experiment, The values for this configuration Please have a look at the Config documentation for more details.

Launchers

A launcher turns a hardware requirement string ("duration=1h & gpu(mem=4G)*1 & cpu(cores=4)") into a concrete way of running a task — locally, on SLURM, on OAR, etc. The mapping lives in a launchers.py file you control (see Installation for the auto-generated version).

gpulauncher = find_launcher(cfg.launcher)

The same line works on a laptop (direct launcher generated by experimaestro launchers direct generate) and on a SLURM cluster (configured by experimaestro launchers slurm generate): the launcher file is responsible for picking a partition / configuring tokens that satisfy cfg.launcher. See the launchers documentation for the requirement DSL and advanced setups (including per-cluster tags=).

Submitting tasks

Now we are ready to launch our tasks. We use a grid search over the hyper-parameters defined in params.yaml:

for n_layer in cfg.n_layers:
    for hidden_dim in cfg.hidden_dim:
        for kernel_size in cfg.kernel_size:
            task = Learn.C(...)
            task.submit(launcher=gpulauncher)  # send to the scheduler

`params.yaml`

This file contains the values of the parameters that will be used to run our experiment.

id: MNIST_train
title: "MNIST training"
description: "Training a simple CNN model on MNIST"
# what experiment file to run
file: experiment # will run experiment.py

# Launcher configuration: what resources we need to run a Task
launcher: "duration=1h & cuda(mem=4G)*1 & cpu(cores=4)"

# Experimental parameters
hidden_dim:        # number of hidden units in the model
    - 32
n_layers:          # number of layers in the model
    - 1
    - 2
kernel_size:       # kernel size of the convolutions
    - 3

# Training parameters
epochs: 1       # number of epochs to train
lr: 1e-3        # learning rate
batch_size: 64  # batch size
n_val: 100      # number of steps between validation and logging

We will launch one job for each possible combination of hidden_dim,n_layers and kernel_size.

Running the Experiment

Once your workspace and launcher are configured (see Installation). You can run the experiment !

Dry-Run

Before actually running the tasks, you can first launch a dry-run to verify what will be launched: experimaestro will list all tasks in the experiment.

uv run experimaestro run-experiment mnist_xp/params.yaml --run-mode dry_run 

the dry-run mode notably features:

Colors for not done / done / failed tasks.
Preview of Launcher that will be used (e.g DirectLauncher or Slurm(cuda=32G, cpu...))

Launch Tasks

If you are happy with the output of the dry-run, you can now actually launch them !

uv run experimaestro run-experiment mnist_xp/params.yaml --run-mode dry_run 

Now experimaestro will:

Lock the experiment MNIST_train so that you cannot relaunch it while this one is running.
Run experiment.py with the configuration values from params.yaml
For each Task submitted in the experiment:
- A unique hash ID is created depending on the parameters given to the task. (This ensures that you don’t run the Task with the same params twice.)
- A folder is created in the workspace/jobs/task-class/task-id, it will be the working directory for the Task in which you can find the logs and the outputs.

Monitor with the TUI:

Experimaestro also features a tool for monitoring all your tasks and experiments:

You can run a TUI-based monitor in the terminal with

uv run experimaestro experiments monitor --console

🚀 Ready for more? Check out the Advanced Guide (ADVANCED.md) to learn about:

Declarative Grid Search: Using the GridSearch component.
Post-Experiment Actions: Automatically exporting the best model.
Complex DAGs: Visualizing task dependencies.

Going further

These features extend the demo and are worth knowing once the basic flow above is comfortable:

Long-running / preemptible jobs — wrap Learn as a ResumableTask, check remaining_time() and raise GracefulTimeout to checkpoint cleanly when a SLURM walltime is about to expire.
Shared workspace settings — imports: in settings.yaml lets multiple project-level YAML files inherit a base workspace block. See Settings.
Archiving completed jobs (beta) — declare auxiliary folders with mode: backup (or move) on a workspace to automatically copy/migrate finished job directories to long-term storage. See Settings.
Moving experiments across workspaces — experimaestro experiments copy transfers a finished experiment (and its job directories) between workspaces, useful when promoting a laptop prototype to a cluster run. See CLI.

Reference

Configurations — deep dive into configuration objects
Tasks — task definition, ResumableTask, dynamic outputs
Launchers — direct, SLURM, OAR launchers and the requirement DSL
Settings — workspaces, triggers, imports, folders
CLI — command-line interface reference