# Changelog

## Version 2.3.0 (2026)

### New Features

#### [Experiment Actions](experiments/actions.md) (alpha)

Actions are user-defined operations that can be registered during task submission
and executed after experiment completion. They support interactive user input
via CLI (rich prompts) and TUI (modal dialogs).

```python
from experimaestro import Task, Action, Interaction, Param

class ExportToHub(Action):
    model: Param[Model]

    def describe(self) -> str:
        return "Export model to HF Hub"

    def execute(self, interaction: Interaction) -> None:
        name = interaction.text("name", "Hub model name:")
        self.model.push_to_hub(name)

class TrainModel(Task):
    model: Param[Model]

    def __submit__(self, dep, add_action, **kwargs):
        add_action(ExportToHub.C(model=self.model))
        return self
```

Run actions via CLI or TUI:
```bash
experimaestro experiments actions list
experimaestro experiments actions run
```

#### Streaming config serialization (`objects.jsonl`)

Job configs and actions are now streamed to `objects.jsonl` as they are submitted,
replacing the old `configs.json` batch serialization at experiment finalization.
Use `load_xp_info()` to load experiment data:

```python
from experimaestro import load_xp_info

info = load_xp_info("/path/to/run_dir")
info.jobs     # dict[str, Config]
info.actions  # dict[str, Action]
```

#### `__submit__` method

The new `__submit__(self, dep, add_action, **kwargs)` method on tasks replaces
`task_outputs`. It receives an `add_action` callable for registering actions.
Legacy `task_outputs` methods continue to work.

### Bug Fixes

- Fixed race condition where `on_completed` callbacks could run after experiment
  exit. Exit notifications are now routed through the notification executor to
  ensure all callbacks complete before `wait()` returns.

### Deprecations

- `load_configs()` is deprecated in favor of `load_xp_info()`.
- `configs.json` is replaced by `objects.jsonl` (old format still loadable for
  backward compatibility).

---

## Version 2.0 (2025)

Version 2.0 is a major release with significant new features, improved developer experience, and important breaking changes.

### New Features

#### Terminal UI (TUI)

A new interactive terminal interface built with [Textual](https://textual.textualize.io/) for monitoring experiments:

- Real-time experiment and job monitoring
- Push notifications via file watching
- SQLite persistence for tracking multiple experiments
- Orphan job detection and cleanup
- Clipboard support via OSC 52 (works over SSH/tmux)
- Thread-safe updates

![Experiment screen](img/tui-experiments.png)

![Jobs screen](img/tui-jobs.png)

![Log screen](img/tui-logs.png)

Launch with:
```bash
experimaestro experiments --workdir /path/to/workdir monitor --console
```

with workdir one of the directories defined in the [Settings](settings.md)

[Read more about the TUI...](interfaces.md#terminal-ui-tui)

#### [Resumable Tasks](experiments/task.md#resumable-tasks)

Tasks can now handle timeouts gracefully and resume from where they left off:

```python
from experimaestro import ResumableTask, GracefulTimeout, Meta, PathGenerator, field

class LongTraining(ResumableTask):
    epochs: Param[int] = 1000
    checkpoint: Meta[Path] = field(default_factory=PathGenerator("checkpoint.pth"))

    def execute(self):
        start_epoch = load_checkpoint(self.checkpoint) if self.checkpoint.exists() else 0

        for epoch in range(start_epoch, self.epochs):
            remaining = self.remaining_time()  # Query remaining walltime (SLURM)
            if remaining is not None and remaining < 300:
                save_checkpoint(self.checkpoint, epoch)
                raise GracefulTimeout("Not enough time for another epoch")

            train_one_epoch()
            save_checkpoint(self.checkpoint, epoch)
```

- `remaining_time()` method to query remaining walltime
- `GracefulTimeout` exception for clean timeout handling
- Configurable `max_retries` (default: 3) for automatic retry on timeout
- Log file rotation when resuming tasks
- [Graceful termination](experiments/task.md#graceful-termination): Handle SIGTERM/SIGINT with `TaskCancelled` for custom cleanup

[Read more about resumable tasks...](experiments/task.md#resumable-tasks)

#### [Dynamic Task Outputs](experiments/task.md#dynamic-task-outputs)

Tasks can now produce outputs *during* execution with callback support, useful for triggering evaluation on intermediate checkpoints while training is still running:

```python
from experimaestro import ResumableTask, Config, Param, DependentMarker

class Validation(Config):
    model: Param[Model]

    def checkpoint(self, dep: DependentMarker, *, step: int) -> Checkpoint:
        # dep() marks the output as depending on the producing task,
        # so different learning rates produce different checkpoint identifiers
        return dep(Checkpoint.C(model=self.model, step=step))

    def compute(self, step: int):
        self.register_task_output(self.checkpoint, step=step)

class Learn(ResumableTask):
    validation: Param[Validation]

    def execute(self):
        for step in range(100):
            train_step()
            if step % 10 == 0:
                self.validation.compute(step)  # Signals output, triggers callbacks

# Register a callback and submit
def on_checkpoint(checkpoint: Checkpoint):
    Evaluate.C(checkpoint=checkpoint).submit()

learn = Learn.C(model=model, validation=validation)
learn.watch_output(validation.checkpoint, on_checkpoint)
learn.submit()
```

- Only available on `ResumableTask`
- Callbacks are replayed when tasks restart, ensuring no outputs are missed
- Multiple callbacks can watch the same output method
- Callbacks run in a dedicated worker thread

[Read more about dynamic task outputs...](experiments/task.md#dynamic-task-outputs)

#### Mypy Plugin

Full mypy support for experimaestro's `Config.C` pattern:

```toml
# pyproject.toml
[tool.mypy]
plugins = ["experimaestro.mypy"]
```

Provides:

- Type inference for `Config.C`, `XPMConfig`, `XPMValue`
- Proper `__init__` signatures with `Param` and `Meta` fields
- `submit()` return type inference (including `task_outputs`)
- `ConfigMixin` methods available on all Config subclasses

#### [Partial Identifiers](experiments/config.md#partial-identifiers)

Share directories across tasks that differ only in excluded parameters. This is particularly useful for:

- **Checkpoint sharing**: Training runs with different iteration counts can share the same checkpoint directory, enabling seamless resume from any previous run
- **Resource efficiency**: Avoid duplicating large files (model weights, preprocessed data) across similar experiments
- **Hyperparameter sweeps**: When varying certain parameters (e.g., number of epochs), other outputs remain shared

```python
class Learn(Task):
    checkpoints = partial(exclude_groups=["iter"])

    max_iter: Param[int] = field(group="iter")
    learning_rate: Param[float]

    checkpoints_path: Meta[Path] = field(
        default_factory=PathGenerator(),
        partial=checkpoints
    )
```

[Read more about partial identifiers...](experiments/config.md#partial-identifiers)

#### [Configuration Composition](experiments/config.md#composition-operator)

Compose configurations using the `@` operator. The operator automatically finds the parameter in the outer configuration that accepts the inner configuration's type:

```python
class Encoder(Config):
    hidden_size: Param[int]

class Model(Config):
    encoder: Param[Encoder]

# These two are equivalent:
model1 = Model.C(encoder=Encoder.C(hidden_size=256))
model2 = Model.C() @ Encoder.C(hidden_size=256)
```

[Read more about configuration composition...](experiments/config.md#composition-operator)

#### [Flexible Deprecation](experiments/config.md#deprecating-a-configuration-or-attributes)

Deprecate and redirect configurations with automatic migration:

```python
@deprecate(NewConfig, replace=True)
class OldConfig(Config):
    ...
```

[Read more about deprecation...](experiments/config.md#deprecating-a-configuration-or-attributes)

#### [Instance-Based Identity](experiments/config.md#instance-based-configurations)

Distinguish between *shared* and *separate* instances with identical parameters using `InstanceConfig`. Essential for workflows where components can be tied or independent:

```python
class Encoder(InstanceConfig):
    hidden_size: Param[int]

class DualEncoderModel(Config):
    query_encoder: Param[Encoder]
    doc_encoder: Param[Encoder]

enc = Encoder.C(hidden_size=128)
shared = DualEncoderModel.C(query_encoder=enc, doc_encoder=enc)      # tied weights
separate = DualEncoderModel.C(query_encoder=enc, doc_encoder=Encoder.C(hidden_size=128))  # independent
# shared and separate have different identifiers
```

Backwards-compatible: first instance keeps its original identifier.

[Read more about instance-based configurations...](experiments/config.md#instance-based-configurations)

#### [Workspace Auto-Selection](settings.md)

Workspaces can be automatically selected based on experiment ID patterns:

```yaml
# ~/.config/experimaestro/settings.yaml
workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"
      - "ir-experiment"
```

[Read more about workspace settings...](settings.md)

#### Carbon Tracking

Track the environmental impact of your experiments with optional carbon emissions monitoring:

```yaml
# Enable in ~/.config/experimaestro/settings.yaml
carbon:
  enabled: true
  country_iso_code: FRA  # Optional: override detected country
```

When enabled, experimaestro tracks CO2 emissions, energy consumption, and power usage for each job using [CodeCarbon](https://codecarbon.io/). Metrics are stored per-job and can be aggregated across experiments.

#### SSH Remote Monitoring

Monitor experiments running on remote servers directly from your local machine:

```bash
experimaestro experiments ssh-monitor user@cluster /path/to/workspace
```

Features include JSON-RPC communication over SSH, on-demand file synchronization for logs, and support for both Web UI and TUI. See [Remote Monitoring via SSH](interfaces.md#remote-monitoring-via-ssh) for details.

#### Transient Tasks

Tasks that don't need persistent storage can be marked as transient at submission time:

```python
from experimaestro import TransientMode

# Skip if no dependent tasks need it
task.submit(transient=TransientMode.TRANSIENT)

# Run but remove directory after dependents complete
task.submit(transient=TransientMode.REMOVE)
```

Transient tasks are only executed if they have non-transient dependents, saving resources for intermediate computations.

#### Experiment Run History

Track multiple runs of the same experiment with automatic run numbering:

- New layout: `experiments/{experiment-id}/{run-id}/`
- View run history with `d` key in TUI
- Each run maintains its own jobs and state

#### Improved Stability

The scheduler has been refactored to use asyncio instead of threads for job management,
resulting in more predictable behavior and reduced race conditions.

#### Other Features

- **Environment tracking**: Git state and environment info saved for reproducibility
- **Conflict warnings**: Warns on conflicting tag values with source location tracking
- **Override warnings**: Warns when overriding arguments without `overrides=True` flag
- **[`value_class` decorator](experiments/config.md#value-classes)**: Register external types as experimaestro values
- **[Explicit default behavior](experiments/config.md#parameters)**: `field(ignore_default=...)` and `field(default=...)`
- **[DynamicLauncher](launchers/index.md#dynamic-launcher)**: Select launchers dynamically based on priority
- **[AcceleratorSpecification](launchers/index.md#accelerator-types)**: Cross-platform GPU support (CUDA, MPS, generic)
- **[Protocol version checking](interfaces.md#version-compatibility)**: Client-server compatibility for remote monitoring
- **Colored logging**: ISO timestamps and colored output for CLI (`--logging` option)
- **Events viewer**: Stream experiment events to console (`--events-viewer` option)
- **Workspace version checking**: Prevents running v2 code against v1 workspaces
- **Job dependencies display**: View job dependencies in TUI/Web UI
- **SLURM TUI configuration**: Interactive terminal-based SLURM launcher configuration

### Breaking Changes

#### Configuration Constructor Pattern (`Config.C()`)

Configurations must now be created using the `.C()` constructor instead of direct instantiation:

```python
# v1.x (no longer works)
model = MyModel(hidden_size=256)
task = TrainTask(model=model, epochs=10)

# v2.0 (required)
model = MyModel.C(hidden_size=256)
task = TrainTask.C(model=model, epochs=10)
```

The `.C()` constructor returns an `XPMConfig` wrapper that tracks configuration state, handles serialization, and manages dependencies. The actual instance is created when the task executes.

#### Removed Deprecated Decorators

The following decorators, deprecated since v1.x, have been removed in v2:

- `@config`, `@task`, `@param`, `@option`, `@pathoption`

Use modern class-based syntax instead:

```python
class MyTask(Task):
    x: Param[int]
```

#### Init Tasks Affect Identifiers

Init tasks are now properly included in identifier computation. This may cause identifier changes for tasks that use `init_tasks`.

#### Default Values Require `field()`

Bare default values are deprecated:

```python
# Deprecated (warning)
class MyConfig(Config):
    x: Param[int] = 23

# Correct
class MyConfig(Config):
    x: Param[int] = field(ignore_default=23)  # Ignored in identifier if value==23
    # or
    x: Param[int] = field(default=23)  # Always included in identifier
```

Run `experimaestro refactor default-values` to automatically fix bare defaults.

#### Tags Are Now Experiment-Specific

Tags are now scoped to individual experiments rather than being global. This means tags set in one experiment are no longer visible in other experiments. Additionally, tags are no longer stored in the `params.json` file.

#### Workspace Database Migration

v2 uses a new workspace-level SQLite database format (stored in
`.experimaestro`). This SQLite is automatically synced based on the actual disk
state.

For detailed commit-level changes, see the [GitHub releases](https://github.com/experimaestro/experimaestro-python/releases).