Changelog

Version 2.3.0 (2026)

New Features

Experiment Actions (alpha)

Actions are user-defined operations that can be registered during task submission and executed after experiment completion. They support interactive user input via CLI (rich prompts) and TUI (modal dialogs).

from experimaestro import Task, Action, Interaction, Param

class ExportToHub(Action):
    model: Param[Model]

    def describe(self) -> str:
        return "Export model to HF Hub"

    def execute(self, interaction: Interaction) -> None:
        name = interaction.text("name", "Hub model name:")
        self.model.push_to_hub(name)

class TrainModel(Task):
    model: Param[Model]

    def __submit__(self, dep, add_action, **kwargs):
        add_action(ExportToHub.C(model=self.model))
        return self

Run actions via CLI or TUI:

experimaestro experiments actions list
experimaestro experiments actions run

Streaming config serialization (objects.jsonl)

Job configs and actions are now streamed to objects.jsonl as they are submitted, replacing the old configs.json batch serialization at experiment finalization. Use load_xp_info() to load experiment data:

from experimaestro import load_xp_info

info = load_xp_info("/path/to/run_dir")
info.jobs     # dict[str, Config]
info.actions  # dict[str, Action]

__submit__ method

The new __submit__(self, dep, add_action, **kwargs) method on tasks replaces task_outputs. It receives an add_action callable for registering actions. Legacy task_outputs methods continue to work.

Bug Fixes

• Fixed race condition where on_completed callbacks could run after experiment exit. Exit notifications are now routed through the notification executor to ensure all callbacks complete before wait() returns.

Deprecations

• load_configs() is deprecated in favor of load_xp_info().

• configs.json is replaced by objects.jsonl (old format still loadable for backward compatibility).

---

Version 2.0 (2025)

Version 2.0 is a major release with significant new features, improved developer experience, and important breaking changes.

New Features

Terminal UI (TUI)

A new interactive terminal interface built with Textual for monitoring experiments:

• Real-time experiment and job monitoring

• Push notifications via file watching

• SQLite persistence for tracking multiple experiments

• Orphan job detection and cleanup

• Clipboard support via OSC 52 (works over SSH/tmux)

• Thread-safe updates

Launch with:

experimaestro experiments --workdir /path/to/workdir monitor --console

with workdir one of the directories defined in the Settings

Read more about the TUI…

Resumable Tasks

Tasks can now handle timeouts gracefully and resume from where they left off:

from experimaestro import ResumableTask, GracefulTimeout, Meta, PathGenerator, field

class LongTraining(ResumableTask):
    epochs: Param[int] = 1000
    checkpoint: Meta[Path] = field(default_factory=PathGenerator("checkpoint.pth"))

    def execute(self):
        start_epoch = load_checkpoint(self.checkpoint) if self.checkpoint.exists() else 0

        for epoch in range(start_epoch, self.epochs):
            remaining = self.remaining_time()  # Query remaining walltime (SLURM)
            if remaining is not None and remaining < 300:
                save_checkpoint(self.checkpoint, epoch)
                raise GracefulTimeout("Not enough time for another epoch")

            train_one_epoch()
            save_checkpoint(self.checkpoint, epoch)

• remaining_time() method to query remaining walltime

• GracefulTimeout exception for clean timeout handling

• Configurable max_retries (default: 3) for automatic retry on timeout

• Log file rotation when resuming tasks

• Graceful termination: Handle SIGTERM/SIGINT with TaskCancelled for custom cleanup

Read more about resumable tasks…

Dynamic Task Outputs

Tasks can now produce outputs during execution with callback support, useful for triggering evaluation on intermediate checkpoints while training is still running:

from experimaestro import ResumableTask, Config, Param, DependentMarker

class Validation(Config):
    model: Param[Model]

    def checkpoint(self, dep: DependentMarker, *, step: int) -> Checkpoint:
        # dep() marks the output as depending on the producing task,
        # so different learning rates produce different checkpoint identifiers
        return dep(Checkpoint.C(model=self.model, step=step))

    def compute(self, step: int):
        self.register_task_output(self.checkpoint, step=step)

class Learn(ResumableTask):
    validation: Param[Validation]

    def execute(self):
        for step in range(100):
            train_step()
            if step % 10 == 0:
                self.validation.compute(step)  # Signals output, triggers callbacks

# Register a callback and submit
def on_checkpoint(checkpoint: Checkpoint):
    Evaluate.C(checkpoint=checkpoint).submit()

learn = Learn.C(model=model, validation=validation)
learn.watch_output(validation.checkpoint, on_checkpoint)
learn.submit()

• Only available on ResumableTask

• Callbacks are replayed when tasks restart, ensuring no outputs are missed

• Multiple callbacks can watch the same output method

• Callbacks run in a dedicated worker thread

Read more about dynamic task outputs…

Mypy Plugin

Full mypy support for experimaestro’s Config.C pattern:

# pyproject.toml
[tool.mypy]
plugins = ["experimaestro.mypy"]

Provides:

• Type inference for Config.C, XPMConfig, XPMValue

• Proper __init__ signatures with Param and Meta fields

• submit() return type inference (including task_outputs)

• ConfigMixin methods available on all Config subclasses

Partial Identifiers

Share directories across tasks that differ only in excluded parameters. This is particularly useful for:

• Checkpoint sharing: Training runs with different iteration counts can share the same checkpoint directory, enabling seamless resume from any previous run

• Resource efficiency: Avoid duplicating large files (model weights, preprocessed data) across similar experiments

• Hyperparameter sweeps: When varying certain parameters (e.g., number of epochs), other outputs remain shared

class Learn(Task):
    checkpoints = partial(exclude_groups=["iter"])

    max_iter: Param[int] = field(group="iter")
    learning_rate: Param[float]

    checkpoints_path: Meta[Path] = field(
        default_factory=PathGenerator(),
        partial=checkpoints
    )

Read more about partial identifiers…

Configuration Composition

Compose configurations using the @ operator. The operator automatically finds the parameter in the outer configuration that accepts the inner configuration’s type:

class Encoder(Config):
    hidden_size: Param[int]

class Model(Config):
    encoder: Param[Encoder]

# These two are equivalent:
model1 = Model.C(encoder=Encoder.C(hidden_size=256))
model2 = Model.C() @ Encoder.C(hidden_size=256)

Read more about configuration composition…

Flexible Deprecation

Deprecate and redirect configurations with automatic migration:

@deprecate(NewConfig, replace=True)
class OldConfig(Config):
    ...

Read more about deprecation…

Instance-Based Identity

Distinguish between shared and separate instances with identical parameters using InstanceConfig. Essential for workflows where components can be tied or independent:

class Encoder(InstanceConfig):
    hidden_size: Param[int]

class DualEncoderModel(Config):
    query_encoder: Param[Encoder]
    doc_encoder: Param[Encoder]

enc = Encoder.C(hidden_size=128)
shared = DualEncoderModel.C(query_encoder=enc, doc_encoder=enc)      # tied weights
separate = DualEncoderModel.C(query_encoder=enc, doc_encoder=Encoder.C(hidden_size=128))  # independent
# shared and separate have different identifiers

Backwards-compatible: first instance keeps its original identifier.

Read more about instance-based configurations…

Workspace Auto-Selection

Workspaces can be automatically selected based on experiment ID patterns:

# ~/.config/experimaestro/settings.yaml
workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"
      - "ir-experiment"

Read more about workspace settings…

Carbon Tracking

Track the environmental impact of your experiments with optional carbon emissions monitoring:

# Enable in ~/.config/experimaestro/settings.yaml
carbon:
  enabled: true
  country_iso_code: FRA  # Optional: override detected country

When enabled, experimaestro tracks CO2 emissions, energy consumption, and power usage for each job using CodeCarbon. Metrics are stored per-job and can be aggregated across experiments.

SSH Remote Monitoring

Monitor experiments running on remote servers directly from your local machine:

experimaestro experiments ssh-monitor user@cluster /path/to/workspace

Features include JSON-RPC communication over SSH, on-demand file synchronization for logs, and support for both Web UI and TUI. See Remote Monitoring via SSH for details.

Transient Tasks

Tasks that don’t need persistent storage can be marked as transient at submission time:

from experimaestro import TransientMode

# Skip if no dependent tasks need it
task.submit(transient=TransientMode.TRANSIENT)

# Run but remove directory after dependents complete
task.submit(transient=TransientMode.REMOVE)

Transient tasks are only executed if they have non-transient dependents, saving resources for intermediate computations.

Experiment Run History

Track multiple runs of the same experiment with automatic run numbering:

• New layout: experiments/{experiment-id}/{run-id}/

• View run history with d key in TUI

• Each run maintains its own jobs and state

Improved Stability

The scheduler has been refactored to use asyncio instead of threads for job management, resulting in more predictable behavior and reduced race conditions.

Other Features

• Environment tracking: Git state and environment info saved for reproducibility

• Conflict warnings: Warns on conflicting tag values with source location tracking

• Override warnings: Warns when overriding arguments without overrides=True flag

• value_class decorator: Register external types as experimaestro values

• Explicit default behavior: field(ignore_default=...) and field(default=...)

• DynamicLauncher: Select launchers dynamically based on priority

• AcceleratorSpecification: Cross-platform GPU support (CUDA, MPS, generic)

• Protocol version checking: Client-server compatibility for remote monitoring

• Colored logging: ISO timestamps and colored output for CLI (--logging option)

• Events viewer: Stream experiment events to console (--events-viewer option)

• Workspace version checking: Prevents running v2 code against v1 workspaces

• Job dependencies display: View job dependencies in TUI/Web UI

• SLURM TUI configuration: Interactive terminal-based SLURM launcher configuration

Breaking Changes

Configuration Constructor Pattern (Config.C())

Configurations must now be created using the .C() constructor instead of direct instantiation:

# v1.x (no longer works)
model = MyModel(hidden_size=256)
task = TrainTask(model=model, epochs=10)

# v2.0 (required)
model = MyModel.C(hidden_size=256)
task = TrainTask.C(model=model, epochs=10)

The .C() constructor returns an XPMConfig wrapper that tracks configuration state, handles serialization, and manages dependencies. The actual instance is created when the task executes.

Removed Deprecated Decorators

The following decorators, deprecated since v1.x, have been removed in v2:

• @config, @task, @param, @option, @pathoption

Use modern class-based syntax instead:

class MyTask(Task):
    x: Param[int]

Init Tasks Affect Identifiers

Init tasks are now properly included in identifier computation. This may cause identifier changes for tasks that use init_tasks.

Default Values Require field()

Bare default values are deprecated:

# Deprecated (warning)
class MyConfig(Config):
    x: Param[int] = 23

# Correct
class MyConfig(Config):
    x: Param[int] = field(ignore_default=23)  # Ignored in identifier if value==23
    # or
    x: Param[int] = field(default=23)  # Always included in identifier

Run experimaestro refactor default-values to automatically fix bare defaults.

Tags Are Now Experiment-Specific

Tags are now scoped to individual experiments rather than being global. This means tags set in one experiment are no longer visible in other experiments. Additionally, tags are no longer stored in the params.json file.

Workspace Database Migration

v2 uses a new workspace-level SQLite database format (stored in .experimaestro). This SQLite is automatically synced based on the actual disk state.

For detailed commit-level changes, see the GitHub releases.