Experimaestro configuration

This pages describes how experimaestro itself can be configured for your need.

Settings file

Default settings can be stored in the $HOME/.config/experimaestro/settings.yaml file. All the settings are optional

# Experiment server settings
server:
    port: 12345
    token: 2134inmd8132323
    host: 192.168.1.1

# Environment variables for experimental tasks
env:
  JAVA_HOME: /path/to/java

workspaces:
    # First workspace is the default
  - id: neuralir
    path: ~/experiments/xpmir
    # Specific environment for this workspace
    env:
      VARNAME: VALUE
    # Auto-select this workspace when experiment ID matches these patterns
    triggers:
      - "neuralir-*"
      - "my-awesome-experiment"

  - id: other_project
    path: ~/experiments/other
    triggers:
      - "other-*"

Workspace Selection

When running experiments with experimaestro run-experiment, the workspace is selected using the following priority:

Explicit --workspace flag: If provided, uses the workspace with that ID
Explicit --workdir flag: If provided, uses that directory directly
Auto-selection via triggers: If the experiment ID matches a workspace trigger pattern, that workspace is selected
Default workspace: The first workspace in the settings list is used as fallback

Workspace Triggers

Workspace triggers allow automatic workspace selection based on experiment ID patterns. This is useful when you have multiple workspaces for different projects and want experiments to automatically run in the correct workspace.

Triggers use glob-style pattern matching:

* matches any characters (e.g., "base_id-*" matches "base_id-123", "base_id-test")
? matches a single character
[abc] matches any character in the brackets

Example:

workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"          # Matches neuralir-123, neuralir-test, etc.
      - "ir-experiment"       # Exact match
      - "test-ir-*"          # Matches test-ir-1, test-ir-baseline, etc.

  - id: nlp
    path: ~/experiments/nlp
    triggers:
      - "nlp-*"
      - "transformer-*"

  - id: default
    path: ~/experiments/default
    # No triggers - this is the fallback

If an experiment’s ID matches multiple workspace triggers, the first matching workspace in the list wins.

Auxiliary Folders (Beta)

Warning

This feature is beta. The YAML schema and Python API may change based on user feedback. Track issue #55 for updates.

Each workspace can declare additional folders where successful jobs are archived, looked up read-only, or migrated. This is useful for time-limited fast partitions on a cluster, sharing results across workspaces, or rotating storage tiers.

workspaces:
  - id: nlp
    path: ~/experiments/nlp
    folders:
      - path: /scratch/old-workspace/jobs   # alternative read-only location
        mode: use
      - path: /slow/backup/nlp
        mode: backup
      - path: /fast-read/nlp
        mode: move

Modes

Mode	Behaviour
`use`	Read-only attachment. Jobs found here are picked up by the scheduler if the primary workspace has no copy. Nothing is ever written or deleted.
`backup`	After a job completes successfully, its directory is copied into the folder. The folder acts as a recovery source if the primary copy is deleted.
`move`	After a job completes successfully, its directory is moved into the folder and replaced by a symlink in the primary workspace. Use case: fast-read SSD tier with archive on slow storage.

Copies and moves are atomic (staged in a sibling .tmp.* directory, then renamed) and run on a background worker thread, so the scheduler is never blocked on filesystem I/O.

The on-disk layout mirrors the primary workspace (<folder>/jobs/<task_id>/<hash>/). Experimaestro-internal files (*.lock, *.pid, .scheduler/) are excluded from copies; the .experimaestro/ directory (which holds task-outputs.jsonl) is preserved so dynamic-output callbacks can replay correctly.

Automatic recovery

When the scheduler submits a job whose primary directory is missing but a copy exists in any attached folder, it is restored transparently before execution. This lets you delete the primary workspace contents and still benefit from previously computed results.

Orphan handling

experimaestro orphans and the TUI orphans tab treat a job as recoverable, not orphan, when it appears in a backup or move folder. Use experimaestro orphans --no-folders to ignore folders and list everything missing from the primary workspace. Jobs in use folders are never considered orphan candidates.

Per-job opt-out

Pass backup=False to .submit() to skip archival for a single job:

task = MyTask.C(...).submit(backup=False)

Manual recovery

experimaestro jobs recover mymodule.MyTask/abc123

Copies the named job from the first folder that has it back into the primary workspace.

Migration from `alt_workspaces`

WorkspaceSettings.alt_workspaces is deprecated. Replace it with folders entries of mode: use:

# Old (deprecated)
workspaces:
  - id: nlp
    alt_workspaces: ["other-workspace-id"]

# New
workspaces:
  - id: nlp
    folders:
      - path: /path/to/other/workspace
        mode: use

Caveats

Only params.json files written by experimaestro v3+ (paths encoded relative to the job or workspace) are guaranteed to survive a copy or move. Older absolute-path encodings may need manual rewriting.
A move folder must remain mounted for the primary workspace to resolve job paths (they are reached through a symlink).
use folders should never be cleaned by experimaestro orphans --clean against a different workspace.

Remote Workspaces (SSH)

Workspaces can be configured with SSH settings for remote monitoring. A workspace with ssh settings is a remote workspace — it is automatically excluded from local experiment workspace discovery (trigger matching and default workspace selection).

workspaces:
  - id: my-cluster
    path: /remote/path/to/experiments
    ssh:
      host: user@cluster
      shell_init: "source /etc/profile; module load python/3.10"
      uv_offline: true
      python: python3

  - id: local-workspace
    path: ~/experiments/local
    # No ssh — this is a local workspace

SSH Settings

Setting	Description
`host`	SSH host (e.g., `user@cluster` or SSH config alias). Required.
`shell_init`	Shell commands to run before the remote experimaestro command (e.g., `source /etc/profile; module load python/3.10`)
`uv_offline`	Pass `--offline` to `uv tool run` (use cached packages, no network). Useful on HPC clusters without direct PyPI access.
`python`	Python interpreter for `uv tool run --python`
`xpm_path`	Path to experimaestro on remote host (bypasses `uv tool run` entirely)
`options`	Additional SSH options (e.g., `["-p", "2222"]`)

Using Workspace SSH Settings

Once configured, use --workspace with ssh-monitor to avoid repeating host/options:

# All settings come from the workspace configuration
experimaestro experiments ssh-monitor --workspace my-cluster

# CLI flags override workspace settings
experimaestro experiments ssh-monitor --workspace my-cluster --uv-offline

All SSH settings can be overridden by CLI flags (--remote-shell-init, --uv-offline, --remote-python, --remote-xpm, -o).