# Experimaestro configuration

This pages describes how experimaestro itself can be configured for your need.

## Settings file

Default settings can be stored in the `$HOME/.config/experimaestro/settings.yaml file`.
All the settings are optional

```yaml

# Experiment server settings
server:
    port: 12345
    token: 2134inmd8132323
    host: 192.168.1.1

# Environment variables for experimental tasks
env:
  JAVA_HOME: /path/to/java

workspaces:
    # First workspace is the default
  - id: neuralir
    path: ~/experiments/xpmir
    # Specific environment for this workspace
    env:
      VARNAME: VALUE
    # Auto-select this workspace when experiment ID matches these patterns
    triggers:
      - "neuralir-*"
      - "my-awesome-experiment"

  - id: other_project
    path: ~/experiments/other
    triggers:
      - "other-*"
```

## Workspace Selection

When running experiments with `experimaestro run-experiment`, the workspace is selected using the following priority:

1. **Explicit `--workspace` flag**: If provided, uses the workspace with that ID
2. **Explicit `--workdir` flag**: If provided, uses that directory directly
3. **Auto-selection via triggers**: If the experiment ID matches a workspace trigger pattern, that workspace is selected
4. **Default workspace**: The first workspace in the settings list is used as fallback

### Workspace Triggers

Workspace triggers allow automatic workspace selection based on experiment ID patterns. This is useful when you have multiple workspaces for different projects and want experiments to automatically run in the correct workspace.

Triggers use glob-style pattern matching:
- `*` matches any characters (e.g., `"base_id-*"` matches `"base_id-123"`, `"base_id-test"`)
- `?` matches a single character
- `[abc]` matches any character in the brackets

**Example:**

```yaml
workspaces:
  - id: neuralir
    path: ~/experiments/xpmir
    triggers:
      - "neuralir-*"          # Matches neuralir-123, neuralir-test, etc.
      - "ir-experiment"       # Exact match
      - "test-ir-*"          # Matches test-ir-1, test-ir-baseline, etc.

  - id: nlp
    path: ~/experiments/nlp
    triggers:
      - "nlp-*"
      - "transformer-*"

  - id: default
    path: ~/experiments/default
    # No triggers - this is the fallback
```

If an experiment's ID matches multiple workspace triggers, the first matching workspace in the list wins.

## Auxiliary Folders (Beta)

```{warning}
This feature is **beta**. The YAML schema and Python API may change
based on user feedback. Track [issue #55](https://github.com/experimaestro/experimaestro-python/issues/55)
for updates.
```

Each workspace can declare additional **folders** where successful
jobs are archived, looked up read-only, or migrated. This is useful
for time-limited fast partitions on a cluster, sharing results across
workspaces, or rotating storage tiers.

```yaml
workspaces:
  - id: nlp
    path: ~/experiments/nlp
    folders:
      - path: /scratch/old-workspace/jobs   # alternative read-only location
        mode: use
      - path: /slow/backup/nlp
        mode: backup
      - path: /fast-read/nlp
        mode: move
```

### Modes

| Mode | Behaviour |
|------|-----------|
| `use` | Read-only attachment. Jobs found here are picked up by the scheduler if the primary workspace has no copy. Nothing is ever written or deleted. |
| `backup` | After a job completes successfully, its directory is copied into the folder. The folder acts as a recovery source if the primary copy is deleted. |
| `move` | After a job completes successfully, its directory is moved into the folder and replaced by a symlink in the primary workspace. Use case: fast-read SSD tier with archive on slow storage. |

Copies and moves are **atomic** (staged in a sibling `.tmp.*`
directory, then renamed) and run on a **background worker thread**, so
the scheduler is never blocked on filesystem I/O.

The on-disk layout mirrors the primary workspace
(`<folder>/jobs/<task_id>/<hash>/`). Experimaestro-internal files
(`*.lock`, `*.pid`, `.scheduler/`) are excluded from copies; the
`.experimaestro/` directory (which holds `task-outputs.jsonl`) is
preserved so dynamic-output callbacks can replay correctly.

### Automatic recovery

When the scheduler submits a job whose primary directory is missing
but a copy exists in any attached folder, it is restored transparently
before execution. This lets you delete the primary workspace contents
and still benefit from previously computed results.

### Orphan handling

`experimaestro orphans` and the TUI orphans tab treat a job as
**recoverable, not orphan**, when it appears in a `backup` or `move`
folder. Use `experimaestro orphans --no-folders` to ignore folders and
list everything missing from the primary workspace. Jobs in `use`
folders are never considered orphan candidates.

### Per-job opt-out

Pass `backup=False` to `.submit()` to skip archival for a single job:

```python
task = MyTask.C(...).submit(backup=False)
```

### Manual recovery

```bash
experimaestro jobs recover mymodule.MyTask/abc123
```

Copies the named job from the first folder that has it back into the
primary workspace.

### Migration from `alt_workspaces`

`WorkspaceSettings.alt_workspaces` is deprecated. Replace it with
`folders` entries of `mode: use`:

```yaml
# Old (deprecated)
workspaces:
  - id: nlp
    alt_workspaces: ["other-workspace-id"]

# New
workspaces:
  - id: nlp
    folders:
      - path: /path/to/other/workspace
        mode: use
```

### Caveats

- Only `params.json` files written by experimaestro **v3+** (paths
  encoded relative to the job or workspace) are guaranteed to survive
  a copy or move. Older absolute-path encodings may need manual
  rewriting.
- A `move` folder must remain mounted for the primary workspace to
  resolve job paths (they are reached through a symlink).
- `use` folders should never be cleaned by `experimaestro orphans
  --clean` against a different workspace.

## Remote Workspaces (SSH)

Workspaces can be configured with SSH settings for remote monitoring. A workspace
with `ssh` settings is a **remote workspace** — it is automatically excluded from
local experiment workspace discovery (trigger matching and default workspace selection).

```yaml
workspaces:
  - id: my-cluster
    path: /remote/path/to/experiments
    ssh:
      host: user@cluster
      shell_init: "source /etc/profile; module load python/3.10"
      uv_offline: true
      python: python3

  - id: local-workspace
    path: ~/experiments/local
    # No ssh — this is a local workspace
```

### SSH Settings

| Setting | Description |
|---------|-------------|
| `host` | SSH host (e.g., `user@cluster` or SSH config alias). Required. |
| `shell_init` | Shell commands to run before the remote experimaestro command (e.g., `source /etc/profile; module load python/3.10`) |
| `uv_offline` | Pass `--offline` to `uv tool run` (use cached packages, no network). Useful on HPC clusters without direct PyPI access. |
| `python` | Python interpreter for `uv tool run --python` |
| `xpm_path` | Path to experimaestro on remote host (bypasses `uv tool run` entirely) |
| `options` | Additional SSH options (e.g., `["-p", "2222"]`) |

### Using Workspace SSH Settings

Once configured, use `--workspace` with `ssh-monitor` to avoid repeating host/options:

```bash
# All settings come from the workspace configuration
experimaestro experiments ssh-monitor --workspace my-cluster

# CLI flags override workspace settings
experimaestro experiments ssh-monitor --workspace my-cluster --uv-offline
```

All SSH settings can be overridden by CLI flags (`--remote-shell-init`, `--uv-offline`,
`--remote-python`, `--remote-xpm`, `-o`).