View on GitHub

Monitoring Interfaces

Experimaestro provides two interfaces for monitoring experiments and jobs: a Web UI and a Terminal UI (TUI). Both can be used during experiment execution or to monitor a workspace after the fact.

Web Interface

The web interface provides a browser-based dashboard for monitoring experiments.

Web UI tasks screen

Tasks view: filter, sort and monitor jobs by status, tags, duration and CO₂, with live progress and per-job logs/actions.

Starting the Web UI

During experiment execution:

from experimaestro import experiment

# Port enables the web interface
with experiment(workdir, "my_experiment", port=12345) as xp:
    # Tasks submitted here can be monitored at http://localhost:12345
    task = MyTask.C(...).submit()

Standalone monitoring:

# Monitor an existing workspace
experimaestro --workdir /path/to/workspace monitor --web --port 12345

Then open http://localhost:12345 in your browser.

Features

  • Experiments List: Status (failed, running, done, stopped — no longer running but with unfinished jobs — or empty, as in the TUI), job counts, host, duration and CO₂ per experiment, with buttons to kill all the running jobs of an experiment or delete it (both ask for confirmation; deleting keeps the job directories, and is refused while the experiment is running); searchable by ID or tag and sortable by status, name or start date (newest first by default), by clicking the column headers; Group by prefix or tag, with collapsible groups showing their number of experiments per status and their total CO₂

  • Job List: View all jobs with their status (pending, running, done, error), sortable by status, task, submission date or job ID

  • Keyboard Shortcuts: The TUI keys, outside text fields: / to search the experiments or filter the jobs (Esc clears it), and S (status), D (date), O (name) and, for jobs, T (task) to sort; the key of the current sort reverses it; g cycles the grouping of the experiments

  • Job Filter: The same filter expressions as the TUI search bar, evaluated by the server so that both interfaces select the same jobs; invalid expressions are reported below the input. Suggestions appear while typing (or with Ctrl+Space / ↓): the fields (@state, @name, @host, @partition and the tag keys), the operators, then the values of the field — job states, task identifiers, hosts or tag values of the displayed jobs — and and/or; ↑/↓ select one, Tab or Enter accepts it, Esc closes the list (a second Esc clears the filter)

  • Grouped View: Group by status, task or tag value, with collapsible groups showing job counts (and a status breakdown); the flat list is the default

  • Tags shown: the Tags menu hides tag keys (e.g. a tag common to all the jobs) in the job list of every experiment; hidden tags can still be used to filter and group

  • Real-time Updates: WebSocket-based updates as job states change

  • Failure reasons: the status of a failed job tells why it failed with the icon of the TUI (❌ the job failed, 🔗 a dependency failed, ⏱ timeout, 💾 memory, 🚫 rejected by the launcher); hover it for the description

  • Job actions: hovering a job in the job list shows its actions at the end of the row, each with a key acting on the hovered job (shown when hovering the icon): details (i), logs (l), copy <task id>/<job id> (c), copy the job folder (f; for a workspace monitored over SSH, the remote folder), kill (k, running jobs) and delete (d); kill and delete ask for confirmation, and a notification at the top right tells what was copied. The duration and CO₂ of a job are below its submission date

  • Job Details: The details of a job show its parameters, logs, output paths, where it runs (see Hosts and resources), its failure reason with what happened (see below) and its dependencies with their status (click a dependency to open its details, e.g. to find the job at the source of a failure)

  • Experiment Filtering: Filter jobs by experiment ID and run

  • Log Viewing: View stdout/stderr logs directly in the browser (stderr first), once the job has started (the logs of a waiting or scheduled job are those of an earlier run, and are not shown). The end of a log is shown first and followed as it grows; earlier lines are loaded when scrolling to the top. Progress bars (tqdm, nested ones included) are shown as in a terminal, on one line

  • Orphan & Stray Jobs: Jobs no experiment references, with their disk usage; kill or delete them one by one, or all at once (with confirmation)

  • Services: The services of the selected experiment (of all experiments by default) and run (the current one by default, or the one chosen with the run selector), also when the experiment is not running; start, stop and open them (through /services/<experiment>/<service>/, with ?runId=<run> for a service of a given run), or view their logs

  • Active Services: The services starting or running, of all the experiments, are listed under the navigation bar on every page (with their experiment), with links to open them, see their logs or stop them

  • Links: Each page has its own URL, which can be bookmarked or shared, and the browser’s back and forward buttons move between the pages (see below)

Page URLs

URL

Page

/

Experiments

/xps/<experiment>/jobs

Jobs of an experiment (?run=<run> for a past run)

/xps/<experiment>/graph

Dependency graph (?run=<run> for a past run)

/xps/<experiment>/services

Services of an experiment

/xps/<experiment>/actions

Actions of an experiment

/xps/<experiment>/jobs/<task>/<job>

Details of a job (.../logs for its logs)

/all/jobs, /all/services, …

The same pages, for all the experiments

/orphans

Orphan and stray jobs

When the server asks for the token, the login page goes back to the page that was opened.

Killing and deleting jobs or experiments, and starting and stopping services, behave the same in the web UI, the TUI and (for jobs) the experimaestro jobs commands: the same operations are refused (e.g. deleting a running job) and reported with the same messages. Deleting a job also removes the partial directories it leaves orphaned.

Both interfaces also show the same job status — the execution state resolved against the state the experiment tracks — and sort it in the same order: running, scheduled, ready, waiting, error (timeouts and memory errors first), done, unscheduled and transient.

Remote Access

If your jobs run on remote machines (e.g., SLURM cluster), use the --host option to make the server accessible:

experimaestro run --host 0.0.0.0 --port 12345 experiment.yaml

Terminal UI (TUI)

The TUI provides an interactive terminal-based interface using Textual. It’s useful when working over SSH or when you prefer staying in the terminal.

Starting the TUI

During experiment execution:

# Use --console to launch the terminal UI
experimaestro run --console experiment.yaml

Standalone monitoring:

experimaestro --workdir /path/to/workspace monitor --console

Features

  • Experiments Table: Lists experiments with ID (and tags), hostname, jobs count, status, and duration; grouped by prefix or tag with g (Enter on a group collapses or expands it)

  • Job Table: Scrollable list of jobs with status indicators, filtered with filter expressions (/)

  • Tree View: Jobs grouped by status, task and/or tag (t)

  • Job Details: Where the job runs (see Hosts and resources), process, carbon, tags, progress, the failure reason with what happened (see below) and the job dependencies with their status and failure reason (click a dependency to open its details, Esc to come back to the job it was opened from)

  • Live Logs: Real-time log output panel

  • Keyboard Navigation: Navigate and interact using keyboard shortcuts

  • Status Bar: Shows experiment status and connection info

Keyboard Shortcuts

Press ? in the TUI to show the help screen with all shortcuts.

Navigation:

Key

Action

q

Quit application

Esc

Go back / Close dialog

r

Refresh data

?

Show help

j

Switch to Jobs tab

s

Switch to Services tab

Experiments:

Key

Action

Enter

Select experiment

d

Show experiment runs

Ctrl+d

Delete experiment

Ctrl+k

Kill all running jobs

/

Search (in the experiment IDs and tags; Esc clears it)

S

Sort by status

D

Sort by start date (default, newest first)

O

Sort alphabetically

g

Group by nothing, prefix or tag (in turn)

Pressing the key of the current sort again reverses the order.

Jobs:

Key

Action

l

View job logs (once the job has started)

Ctrl+d

Delete job

Ctrl+k

Kill running job

/

Open search filter

c

Clear search filter

S

Sort by status

T

Sort by task

D

Sort by date

O

Sort alphabetically

#

Choose the tags shown (hidden tag keys, for all the experiments)

f

Copy folder path

Log Viewer:

Key

Action

f

Toggle follow mode

g

Go to top (earlier lines are loaded at the top)

G

Go to bottom

r

Check the logs for changes now

Esc/q

Close viewer

Services:

Key

Action

Ctrl+S

Start service

Ctrl+K

Stop service

l

View logs

u

Copy URL

Orphan Jobs:

Key

Action

r

Refresh

T

Sort by task

Z

Sort by size

Ctrl+d

Delete selected

Ctrl+D

Delete all

f

Copy folder path

Clipboard Support

The TUI supports clipboard operations (like f to copy path) that work both locally and in remote/tmux environments.

How it works:

  1. First tries pyperclip for local clipboard access

  2. Falls back to OSC 52 escape sequences when pyperclip fails (e.g., over SSH)

OSC 52 supported terminals:

  • iTerm2 (macOS)

  • kitty

  • alacritty

  • Windows Terminal

  • Most modern terminal emulators

tmux configuration:

For clipboard to work inside tmux over SSH, add these lines to ~/.tmux.conf:

set -g set-clipboard on
set -g allow-passthrough on   # Required for tmux 3.2+

Then reload tmux configuration: tmux source-file ~/.tmux.conf

Python API

The TUI can also be launched programmatically:

from experimaestro.tui import ExperimentTUI

tui = ExperimentTUI(
    workdir=Path("/path/to/workspace"),
    show_logs=True,
)
tui.run()

Experiment Tags

Experiments can be tagged, with experiment(..., tags=["eval", "mice"]) or the tags field of the run YAML file. The tags are stored with each run and shown in the experiment list of both interfaces, where the search also matches them. The experiments can be grouped:

  • by prefix: what comes before the first -, _ or . of the experiment ID (msmarco-splade and msmarco_bm25 are both in msmarco);

  • by tag: an experiment is in the group of each of its tags, and those without tags are in the last group, (untagged).

The header of a group shows its total CO₂: all the runs of its experiments, a job they share being counted once. On the command line, --group selects the experiments with a tag (experiments list --group, experiments carbon summary --group).

Job Filter Expressions

The TUI search bar (/) and the web Filter input accept the same expressions (also used by experimaestro jobs --filter, see CLI):

Expression

Meaning

@state = 'done'

Job state (unscheduled, waiting, ready, scheduled, running, done, error)

@name ~ 'pkg\.Learn'

Task identifier matches a regular expression (anchored at the start)

@host = 'node042'

Host running the job (one of them, for a job on several nodes; see Hosts and resources)

@partition = 'gpu'

SLURM partition of the job

model = 'bm25'

Tag value of the job in the displayed experiment run

@state in ['running', 'waiting']

Value in a list (not in for the converse)

... and ..., ... or ...

Combine conditions

Tag values are compared as text, whatever their type: a tag phase=1 is matched by phase = '1' or phase ~ '1'.

For example, @state = 'error' and model = 'bm25' selects the failed jobs whose model tag is bm25.

Choosing Between Web and TUI

Use Case

Recommended

Local development

Web UI

Remote SSH session

TUI

Sharing with team

Web UI (with --host)

Minimal dependencies

TUI

Detailed log analysis

Web UI

Quick status check

TUI

Remote Monitoring via SSH

When experiments run on remote servers (e.g., HPC clusters, cloud instances), you can monitor them from your local machine using SSH tunneling. Monitoring a remote workspace establishes an SSH connection, starts a monitoring server on the remote host, and communicates via JSON-RPC over the SSH channel.

Basic Usage

experimaestro [SSH OPTIONS] --ssh HOST:REMOTE_WORKDIR monitor (--console | --web)
  • HOST: SSH host in standard format (e.g., user@server, server, or SSH config alias)

  • REMOTE_WORKDIR: Path to the workspace directory on the remote server

Examples

# Basic SSH monitoring (web UI)
experimaestro --ssh myserver:/home/user/experiments monitor --web

# With the console TUI instead of the web UI
experimaestro --ssh user@cluster.example.com:/scratch/experiments monitor --console

# Custom port for web interface
experimaestro --ssh myserver:/workspace monitor --web --port 8080

# With SSH options (e.g., custom port, identity file)
experimaestro --ssh myserver:/workspace -o "-p 2222" -o "-i ~/.ssh/cluster_key" monitor --web

# Specify path to experimaestro on remote host
experimaestro --ssh myserver:/workspace --remote-xpm /opt/conda/bin/experimaestro monitor --web

Options

Option

Description

--console / --web

Terminal (Textual) UI or web UI (one is required)

--port PORT

Port for local web server (default: 12345)

The SSH options (--remote-xpm, -o, …) are given before the command, see Remote Workspaces.

How It Works

  1. Connection: The client establishes an SSH connection to the remote host

  2. Server Start: On the remote host, experimaestro starts in server mode, reading commands from stdin and writing responses to stdout

  3. JSON-RPC Protocol: Commands and responses are exchanged using JSON-RPC 2.0

  4. Job logs: The remote experimaestro renders the logs as a terminal would and sends them through the same connection, by pages (the end of a log first, earlier pages when scrolling up, the new content while following): only the lines shown are sent, not the bytes of the progress bars that rewrite them. When the end of a log is a progress bar running for hours, the lines before it are shown with the bar as last drawn. An open log is checked every second, less and less often (up to every 30 s) while it does not change. A log rewritten by a restarted job is detected and read again. With an older remote experimaestro, logs are read by blocks into a local copy that lasts as long as the monitor, or, older still, copied with rsync.

  5. File Sync: When needed (e.g., for TensorBoard logs), files are synchronized on-demand using rsync over SSH

  6. Real-time Updates: The server sends notifications when job states change — only for what the UI shows: the experiment list is always kept up to date, but job, action and service events are only sent for the experiment shown (and for the started services, shown on every page). The same applies to the web UI, for each browser page.

Lost Connections

When the SSH connection drops (network loss, laptop sleep, the remote server exiting), the monitor reconnects on its own: it starts a new remote server, declares again the experiments shown, and reads the state again (the events sent while disconnected were missed). Both interfaces show the connection state on every page — the TUI below its header, the web UI below its navigation bar:

  • reconnecting: the connection was lost, with the number of the attempt (e.g. attempt 2/5) and why the connection was lost or the last attempt failed;

  • disconnected: all the attempts failed; restart the monitor to try again.

Nothing is shown while connected. The reconnection policy — at most max_retries attempts within window seconds, with a delay doubling from delay up to max_delay — is set in settings.yaml, see Reconnection. SSH is run with ServerAliveInterval=15 and ServerAliveCountMax=3, so that a dead connection is detected within a minute (your own -o options take precedence).

Shared Workspaces

Like local monitoring, remote monitoring runs the workspace cleanup, so that crashed jobs are recovered without waiting for the next experiment. This is safe in a workspace shared with other users: a job whose process cannot be checked from the monitoring machine (its PID belongs to another host, or its launcher is not reachable) is considered active, and job files owned by another user are never modified.

See Sharing a workspace for the filesystem permissions a shared workspace needs, and Lock file permissions for the lock_mode setting.

Configuring SSH Settings in settings.yaml

Instead of passing all options on the command line, you can configure SSH settings per-workspace in your ~/.config/experimaestro/settings.yaml:

workspaces:
  - id: my-cluster
    path: /scratch/experiments
    ssh:
      host: user@cluster
      shell_init: "source /etc/profile; module load python/3.10"
      uv_offline: true
      python: python3

Then use --workspace to load settings automatically:

experimaestro -w my-cluster monitor --web

See Remote Workspaces (SSH) for all available settings.

Remote experimaestro Installation

By default, the SSH client runs uv tool run experimaestro==<version> on the remote host. This requires:

  • uv installed on the remote host

  • Network access to PyPI (or a local mirror)

For environments without direct PyPI access (common on HPC clusters), use --uv-offline or configure uv_offline: true in workspace SSH settings. If the remote host requires environment setup before running commands (e.g., module load), use --remote-shell-init or configure shell_init in workspace SSH settings.

Alternatively, specify a pre-installed experimaestro path:

# If experimaestro is installed in a virtualenv
experimaestro --ssh host:/workspace --remote-xpm /path/to/venv/bin/experimaestro monitor --web

# If installed system-wide
experimaestro --ssh host:/workspace --remote-xpm experimaestro monitor --web

Version Compatibility

The local and remote experimaestro versions should be compatible. The protocol version is checked on connection, and the connection is refused if it differs. For best results, use the same version on both sides.

A remote experimaestro older than the local one may lack some features (e.g. the dependency graph): the remote server lists what it handles when the connection starts, and the interfaces then report the missing features (the remote experimaestro does not support …) instead of failing. Update experimaestro on the remote host, or point --remote-xpm to a more recent one.

SSH Output

Remote server output (logs, warnings) is displayed locally with a colored [SSH] prefix to distinguish it from local output. This helps debug connection issues or see remote server activity.

Troubleshooting

Slow or stuck interface: trace the requests on both sides. Each one is logged with its time and answer size: on the remote side, the time spent answering it; on the local side, the round trip (which includes the wait in the queue — requests are answered one at a time).

experimaestro --logging xpm.remote.trace=DEBUG -w my-cluster monitor \
    --remote-logging xpm.remote.trace=DEBUG

The xpm.remote.payload logger shows the messages themselves (very verbose).

uv tool run hangs (no output):

  • The remote host likely has no outbound HTTPS access to PyPI

  • Use uv_offline: true in workspace SSH settings or --uv-offline CLI flag

  • Ensure the package is already cached: ssh host "uv tool install experimaestro==X.Y.Z" (run once with network access)

Connection times out on a cluster:

  • Starting Python can take tens of seconds on a shared file system (Lustre, GPFS, NFS) when its cache is cold

  • Increase connect_timeout in the workspace SSH settings (or use --connect-timeout), 60 seconds by default

  • Keep the bytecode cache on a local disk with export PYTHONPYCACHEPREFIX=/tmp/$USER/pycache (in shell_init)

  • With Python 3.15+, the command line loads far fewer modules (lazy imports)

Wrong Python version or missing modules:

  • Non-interactive SSH skips .profile/.bash_profile

  • Use shell_init in workspace SSH settings or --remote-shell-init to run setup commands (e.g., source /etc/profile; module load python/3.10)

Connection refused or timeout:

  • Verify SSH access: ssh user@host echo "connected"

  • Check if the remote workspace exists

  • Ensure experimaestro is accessible on the remote host

Local copies: each monitor keeps its copies of the remote files in a temporary directory (xpm_remote_*, marked by a .experimaestro-remote-cache file), removed when it stops. The directories left by monitors that did not stop normally are removed by the next monitor started on the same host.

Slow logs: to see how the interfaces behave on a slow network, set XPM_LOG_READ_DELAY to a delay (in seconds) added to each read of a log file, e.g. XPM_LOG_READ_DELAY=1 experimaestro --ssh host:/path monitor --web.

Services not working (e.g., TensorBoard):

  • Services are recreated locally from saved state

  • Some services may require file synchronization; this happens automatically

  • Check that rsync is available on both machines

Experiment DAG Visualization

The Graph tab of the web UI shows the dependency graph of the selected experiment run: jobs coloured by status (with the progress of the running ones), init tasks (thin solid border, tinted background), the configurations linking them (optional, dashed border), stacks of similar nodes, and a grid search configuration selector. The graph follows the run: jobs submitted while it runs are added. Clicking a node highlights its lineage and opens an interactive parameter inspector in the right panel.

Experimaestro DAG Web Viewer

Dependency graph: the selected stack of evaluations, with its lineage highlighted.

Integrated Parameters Inspector

Clicking any node or stack opens the inspector panel on the right:

  • Hierarchical drill-down: recursively expands nested sub-configurations, showing their parameters, data paths, and initialization tasks while preventing circular references.

  • Categorization: automatically separates parameters into logical groups (Hyperparameters, Paths & Outputs, Sub-Configurations, Collections, Init Tasks).

  • Search & Highlighting: filter parameters by name or value in real time. Matching text is highlighted in yellow, collapsed parent configs auto-unfold, and a match counter (1 of N) provides previous/next buttons (▲ / ▼) and keyboard shortcuts (Enter / Shift+Enter) to cycle matches.

  • Code Generation: switch between an interactive Tree, reproducible Python (.C) synthesized code, or raw JSON.

  • Stacked Runs: for grouped nodes representing multiple runs (e.g. multi-seed evaluations), member tabs (#1, #2, …) allow inspecting individual run configurations.

Standalone HTML Export

The same interactive view, including graph navigation and the full parameters inspector, can be exported as a completely self-contained HTML page that requires no server:

experimaestro dag my_experiment --html dag.html

It can also visualize standalone serialized models or Hugging Face Hub models without a workspace:

experimaestro dag VictorMorand/Mice-P-61m --html mice_dag.html

See Experiment DAG Visualization in the CLI reference for details and options.

Hosts and resources

The job details of both interfaces show where a job runs (Runs on), e.g. node042 (partition gpu, 8 CPUs, GPUs 0,1):

  • the host of the job process or, in a SLURM allocation (pilot jobs included), its nodes — the one running the job process first;

  • for SLURM jobs, the partition and the CPUs and GPUs allocated on that node.

The job process records this when it starts (in its status.json, from its host name and the SLURM environment), so it is shown by any monitor — also through SSH, or on a machine without the SLURM commands — and kept once the job has ended. Only the last run of a job is described: a retry replaces it. Jobs can be selected by host or partition with the @host and @partition filter fields.

Failure details

The job details of both interfaces show what happened to a failed job, read from the .failed marker in the job folder:

  • for an exception raised by the task, its type and message (e.g. ValueError: the learning rate must be positive) and where it was raised (file.py:42 in execute) — the full traceback is in the job logs;

  • for a timeout, a cancellation or a crashed process, the recorded message;

  • for a job the launcher refused to start (e.g. a SLURM pending reason such as PartitionTimeLimit) or stopped (e.g. the SLURM time limit), the reason given by the launcher, followed by what the job reported (a job SLURM stops at its time limit receives SIGTERM and reports it was cancelled: it is shown as timed out, and a resumable task is retried);

  • the exit code of the job process.

Jobs that failed with an older experimaestro only show their exit code. Over SSH, the details are read by the remote experimaestro, if recent enough.