Command Line Interface

Running Experiments

The general command to launch experiments is:

experimaestro run-experiment [OPTIONS] YAML_FILE [ARGS]

run-experiment Options

Option	Description
`--run-mode MODE`	Set run mode: `NORMAL`, `DRY_RUN`, `GENERATE_ONLY`, or `PREPARE`
`--workspace ID`	Use a workspace from settings by ID
`--workdir PATH`	Specify working directory directly
`--port PORT`	Port for web monitoring interface
`--console`	Use terminal TUI instead of web interface
`--host HOST`	Server hostname (for remote job access)
`--pre-yaml FILE`	Load YAML file before the main one (can be repeated)
`--post-yaml FILE`	Load YAML file after the main one (can be repeated)
`-c KEY=VALUE`	Override configuration values
`--file PATH`	Python file containing the `run` function
`--module-name NAME`	Python module containing the `run` function
`--env KEY VALUE`	Set environment variable (can be repeated)
`--xpm-config-dir PATH`	Custom experimaestro config directory
`--show`	Print merged configuration and exit
`--debug`	Enable debug logging

Run Modes

experimaestro run-experiment --run-mode [DRY_RUN|NORMAL|GENERATE_ONLY|PREPARE]

DRY_RUN - Display task hashes and dependencies without launching
GENERATE_ONLY - Generate job folders without launching tasks
NORMAL - Generate and launch jobs (default)
PREPARE - Run only data-preparation steps (every Prepare instance referenced by submitted tasks), skipping the tasks themselves. Useful for pre-warming caches before running jobs on an offline cluster. See Prepare configurations.

Job Control

Besides the web interface, it is possible to use the command line to check the job status and control jobs.

Database Synchronization

Important: The job commands read from the workspace database. If jobs are not visible, you may need to synchronize the database with the filesystem:

experimaestro experiments sync --workdir /path/to/workspace

This is typically needed when:

Jobs were run before the database was introduced
Jobs were created by an external process
The database is out of sync with the filesystem

Available Commands

Command	Description
`experimaestro jobs list`	List all jobs in the workspace (sorted by date, most recent first)
`experimaestro jobs kill`	Kill running jobs
`experimaestro jobs clean`	Clean (delete) finished jobs
`experimaestro jobs log JOBID`	View job log (stderr by default)
`experimaestro jobs path JOBID`	Print the path to a job directory

Common Options

All job commands support these options:

--workdir PATH: Specify the workspace directory
--workspace ID: Use a workspace from settings by ID
--experiment ID: Filter by experiment ID
--filter EXPR: Filter using a filter expression (see below)
--tags: Show job tags in output
--fullpath: Show full paths instead of task.id/hash format

List Command Options

The list command has additional options:

--count N or -c N: Limit output to the N most recent jobs

# Show only the 10 most recent jobs
experimaestro jobs list -c 10

# Show the 5 most recent running jobs
experimaestro jobs list --filter '@state = "RUNNING"' -c 5

Kill and Clean Commands

The kill and clean commands require --perform to actually execute:

# Dry run - shows what would be killed
experimaestro jobs kill --filter '@state="RUNNING"'

# Actually kill the jobs
experimaestro jobs kill --filter '@state="RUNNING"' --perform

# Dry run - shows what would be cleaned
experimaestro jobs clean --filter '@state="DONE"'

# Actually clean the jobs (deletes directories and database entries)
experimaestro jobs clean --filter '@state="DONE"' --perform

Log and Path Commands

The log and path commands take a JOBID in the format task.name/hash:

# View stderr log (default)
experimaestro jobs log mymodule.MyTask/abc123def

# View stdout log
experimaestro jobs log --std mymodule.MyTask/abc123def

# Follow log (like tail -f)
experimaestro jobs log -f mymodule.MyTask/abc123def

# Get job directory path
experimaestro jobs path mymodule.MyTask/abc123def

Filter Expressions

The job filter is a boolean expression that allows filtering jobs by state, name, or tags.

Special Variables

@state: Job state (values: UNSCHEDULED, WAITING, READY, RUNNING, DONE, ERROR)
@name: Task identifier (e.g., mymodule.MyTask)

Operators

=: Equality comparison
~: Regex matching
in: Membership in a list
not in: Non-membership in a list
and: Logical AND
or: Logical OR

Examples

Filter by state:

# List only running jobs
experimaestro jobs list --filter '@state = "RUNNING"'

# List completed jobs
experimaestro jobs list --filter '@state = "DONE"'

# List failed jobs
experimaestro jobs list --filter '@state = "ERROR"'

Filter by tag:

# Filter by a single tag
experimaestro jobs list --filter 'model = "bm25"'

# Filter by multiple tags
experimaestro jobs list --filter 'model = "bm25" and dataset = "msmarco"'

Complex filters:

# Running or waiting jobs
experimaestro jobs list --filter '@state = "RUNNING" or @state = "WAITING"'

# Jobs with model in a list
experimaestro jobs list --filter 'model in ["bm25", "splade"]'

# Jobs NOT matching a value
experimaestro jobs list --filter 'mode not in ["debug", "test"]'

# Combine state and tag filters
experimaestro jobs list --filter '@state = "ERROR" and model = "bm25"'

Tags

Using the --tags flag, lists all jobs and shows their tagged arguments:

experimaestro jobs list --tags

Output example:

DONE    mymodule.MyTask/abc123 [my_experiment] model=bm25 dataset=msmarco
RUNNING mymodule.MyTask/def456 [my_experiment] model=splade dataset=msmarco

Cleanup Partial Directories

Partial directories are shared checkpoint locations created by partial(). When all jobs using a partial are deleted, the partial becomes orphaned:

# Dry run - shows what would be deleted
experimaestro jobs cleanup-partials

# Actually delete orphan partials
experimaestro jobs cleanup-partials --perform

Experiment Management

The experiments command group provides tools for managing experiments:

experimaestro experiments [OPTIONS] COMMAND

Available Commands

Command	Description
`list`	List experiments in the workspace
`monitor`	Monitor experiments with web UI or console TUI
`ssh-monitor`	Monitor experiments on a remote server via SSH
`monitor-server`	Start SSH monitoring server (internal, for remote monitoring)
`sync`	Synchronize workspace database from disk state
`copy`	Copy an experiment between workspaces

List Command

List experiments with their status and metadata:

experimaestro experiments list --workdir /path/to/workspace

Output format:

experiment-id [hostname] (finished_jobs/total_jobs jobs)

experiment-id: The experiment identifier
hostname: The host where the experiment was launched (if recorded)
Job counts: Number of completed jobs out of total

Monitor Command

Launch an interactive monitoring interface for experiments:

# Web interface (default) on port 12345
experimaestro experiments monitor --workdir /path/to/workspace

# Console TUI interface
experimaestro experiments monitor --workdir /path/to/workspace --console

# Custom port for web interface
experimaestro experiments monitor --workdir /path/to/workspace --port 8080

# Force sync before starting
experimaestro experiments monitor --workdir /path/to/workspace --sync

SSH Monitor Command

Monitor experiments running on a remote server through SSH:

experimaestro experiments ssh-monitor HOST REMOTE_WORKDIR [OPTIONS]

Option	Description
`--web`	Use web UI instead of console TUI (console is default)
`--port PORT`	Port for local web server (default: 12345)
`--remote-xpm PATH`	Path to experimaestro on remote host
`-o, --ssh-option OPT`	Additional SSH options (can be repeated)
`--workspace ID`	Load SSH settings from a workspace in settings.yaml
`--remote-shell-init CMD`	Shell commands to run before remote experimaestro
`--uv-offline`	Pass `--offline` to `uv tool run` (no network)
`--remote-python NAME`	Python interpreter for `uv tool run --python`
`--request-timeout SECS`	Timeout for remote RPC requests (default: 60)

Examples:

# Basic monitoring (console TUI by default)
experimaestro experiments ssh-monitor myserver /home/user/experiments

# With web UI
experimaestro experiments ssh-monitor user@host /workspace --web

# Using workspace SSH settings from settings.yaml
experimaestro experiments ssh-monitor --workspace my-cluster

# With shell init for HPC environments
experimaestro experiments ssh-monitor host /workspace --remote-shell-init "source /etc/profile; module load python/3.10"

# Offline mode (no PyPI access)
experimaestro experiments ssh-monitor host /workspace --uv-offline --remote-python python3

# With SSH port and identity file
experimaestro experiments ssh-monitor myserver /workspace -o "-p 2222" -o "-i ~/.ssh/mykey"

# With custom experimaestro path on remote
experimaestro experiments ssh-monitor host /workspace --remote-xpm /path/to/venv/bin/experimaestro

See Remote Monitoring via SSH for detailed documentation.

Copy Command

Copy an experiment (run directory and all referenced job directories) between workspaces. Supports local and remote (SSH) workspaces.

experimaestro experiments copy [OPTIONS] [EXPERIMENT_ID] [RUN_ID]

Option	Description
`--from-workspace ID`	Source workspace ID (from settings.yaml)
`--to-workspace ID`	Destination workspace ID
`-y, --yes`	Actually perform the copy (default is dry-run)

When arguments or options are omitted, the command prompts for interactive selection.

Examples:

# Dry-run (default) — shows what would be copied
experimaestro experiments copy --from-workspace cluster --to-workspace local my_experiment

# Actually copy
experimaestro experiments copy --from-workspace cluster --to-workspace local my_experiment -y

# Interactive mode — prompts for everything
experimaestro experiments copy

What gets copied:

The experiment run directory (experiments/<id>/<run_id>/) — metadata, events, configs
All job directories referenced in jobs.jsonl — outputs, logs, params
Jobs that already exist at the destination are skipped

Path encoding: Since experimaestro 2.4, params.json stores paths relative to the job root or workspace (see serialization), so copying a job between workspaces does not require rewriting paths inside the file. The top-level "workspace" field — and any absolute paths produced by older versions — are still rewritten for backward compatibility.

Atomic staging: Each job is first rsynced into a temporary staging directory (.xpm-staging/) inside the destination workspace, then moved to its final location. This prevents partial copies from leaving broken state.

See also the Python API for programmatic use.

Cleaning Up Orphans

Check for tasks that are not part of any experimental plan in the given experimental folder:

Usage: experimaestro orphans [OPTIONS] PATH

Options:
  --size      Show size of each folder
  --clean     Prune the orphan folders
  --show-all  Show even not orphans

Difference between two configurations

It is possible to look at the differences (that explain that two tasks have a different identifier) by using the parameters-difference command

Refactoring Code

Fix Bare Default Values

The refactor default-values command helps migrate from deprecated bare default values to the explicit field(ignore_default=...) syntax:

# Dry run - shows what would be changed (default)
experimaestro refactor default-values /path/to/project

# Actually apply the changes
experimaestro refactor default-values /path/to/project --perform

This converts patterns like:

x: Param[int] = 23

To the explicit form:

x: Param[int] = field(ignore_default=23)

The tool:

Scans Python files for Param, Meta, or Option annotations with bare default values
Suggests converting to field(ignore_default=...) to maintain backwards compatibility
Runs in dry-run mode by default (use --perform to apply changes)
Handles simple single-line defaults; multi-line defaults require manual intervention

Installing the Agent Skill

Experimaestro ships an Agent Skill that provides LLM coding assistants (Claude Code, Cursor, Gemini CLI, VS Code Copilot, etc.) with experimaestro best practices and patterns. The skill teaches the assistant how to correctly write Config/Task classes, set up experiments, configure launchers, and more.

Install

# Install to ~/.agents/skills/ (open standard, works with all compatible tools)
experimaestro install-skill

# Install for a specific tool
experimaestro install-skill claude

# Install to multiple targets at once
experimaestro install-skill agents claude cursor

Available targets

Target	Directory	Description
`agents`	`~/.agents/skills/`	Cross-client open standard (default)
`claude`	`~/.claude/skills/`	Claude Code
`cursor`	`~/.cursor/skills/`	Cursor

List targets and status

experimaestro install-skill --list

Shows all targets, which tools are detected on your system, and whether the skill is already installed.

Updating

Run the same install-skill command again after upgrading experimaestro to update the skill to the latest version.

(Sphinx) Checking undocumented configurations

Use experimaestro check-documentation objects.inv my_package to check for undocumented members. The objects.inv file should be generated by sphinx