Experimaestro

Experimaestro is a Python framework designed for researchers and engineers who need to manage complex, large-scale experimental workflows without losing track of reproducibility.

Unlike traditional schedulers, Experimaestro focuses on the experimental logic: how configurations relate to each other and how results are organized.

Experimaestro TUI

Why Experimaestro?

Experimaestro solves common pain points in research workflows by providing:

🚀 Configuration-as-Code: Define your experiments using strongly-typed Python objects. Forget about fragile JSON/YAML files; benefit from IDE autocompletion, type checking, and recursive parameter management.
🛡️ Deduplication & Reproducibility: Every task is assigned a unique identifier based on its parameters. If you try to run the same experiment twice, Experimaestro knows—ensuring you never waste compute time on results you already have.
📁 Organized by Design Results are automatically cached in a predictable directory structure derived from task identifiers. No more “results_v2_final_fixed.pt”—your file system stays as clean as your code.
🏗️ Built-in Scalability Seamlessly transition from local testing to high-performance clusters. Use Connectors (Local, SSH) and Launchers (Direct, Slurm) to run the same experimental code across different environments.

Difference with Other Projects

Experimaestro differentiates itself from traditional job scheduling software like OAR and Slurm, which focus more on resource allocation than on managing experimental workflows. It also stands apart from other experiment management tools like Comet, Sacred, FGLab, and Sumatra. For instance, Comet emphasizes collaboration and note-taking for machine learning experiments but is not open-source and focuses on single-shot experiments. Sumatra and FGLab, based on parameter files, offer less flexibility. Sacred, though open-source and allowing for pre-processing steps, doesn’t support the construction of complex experimental plans like Experimaestro.

Experimaestro’s distinct features include:

Comprehensive Task Composition: It allows for the composition of types and tasks within an experimental plan.
Parameter Monitoring: Offers a clear method to monitor experimental parameters using tags.
Automated Output Organization: Efficiently manages task outputs in the file system, simplifying result storage.
Imperative Experiment Definition: Unlike other tools that define experiments declaratively, Experimaestro adopts an imperative approach, enhancing flexibility in complex experimental planning.

Guide to the Documentation

🏁 Getting Started If you are new to the project, start with the Tutorial. It walks you through setting up your first workspace and running a basic experiment: training a CNN on MNIST.

🧪 Building Experiments Learn how to define your workflow:

Configurations: The heart of Experimaestro. Define parameters, nested structures, and value classes.
Tasks: Define the execution logic and manage dependencies.
Experimental Plans: Compose tasks into complex matrices and track them using tags.

⚙️ Execution & Infrastructure Control where and how your code runs:

Launchers: Manage execution environments (Direct, Slurm).
Connectors: Abstract file access and command execution (Local, SSH).

🛠️ Advanced Tools

Jupyter Integration: Interact with your experiments from notebooks.
API Reference: Deep dive into the classes and methods.

Detailed Outline

Experiments

Execution

Misc

Integration

Reference

Development

Note

Experimaestro and datamaestro are described in the following paper

Benjamin Piwowarski. 2020. Experimaestro and Datamaestro: Experiment and Dataset Managers (for IR). In Proceedings of the 43rd International ACM SIGIR