Monitoring Interfaces
Experimaestro provides two interfaces for monitoring experiments and jobs: a Web UI and a Terminal UI (TUI). Both can be used during experiment execution or to monitor a workspace after the fact.
Web Interface
The web interface provides a browser-based dashboard for monitoring experiments.
Tasks view: filter, sort and monitor jobs by status, tags, duration and CO₂, with live progress and per-job logs/actions.
Starting the Web UI
During experiment execution:
from experimaestro import experiment
# Port enables the web interface
with experiment(workdir, "my_experiment", port=12345) as xp:
# Tasks submitted here can be monitored at http://localhost:12345
task = MyTask.C(...).submit()
Standalone monitoring:
# Monitor an existing workspace
experimaestro --workdir /path/to/workspace monitor --web --port 12345
Then open http://localhost:12345 in your browser.
Features
Experiments List: Status (failed, running, done, stopped — no longer running but with unfinished jobs — or empty, as in the TUI), job counts, host, duration and CO₂ per experiment, with buttons to kill all the running jobs of an experiment or delete it (both ask for confirmation; deleting keeps the job directories, and is refused while the experiment is running); searchable by ID or tag and sortable by status, name or start date (newest first by default), by clicking the column headers; Group by prefix or tag, with collapsible groups showing their number of experiments per status and their total CO₂
Job List: View all jobs with their status (pending, running, done, error), sortable by status, task, submission date or job ID
Keyboard Shortcuts: The TUI keys, outside text fields:
/to search the experiments or filter the jobs (Escclears it), andS(status),D(date),O(name) and, for jobs,T(task) to sort; the key of the current sort reverses it;gcycles the grouping of the experimentsJob Filter: The same filter expressions as the TUI search bar, evaluated by the server so that both interfaces select the same jobs; invalid expressions are reported below the input. Suggestions appear while typing (or with
Ctrl+Space/↓): the fields (@state,@name,@host,@partitionand the tag keys), the operators, then the values of the field — job states, task identifiers, hosts or tag values of the displayed jobs — andand/or;↑/↓select one,TaborEnteraccepts it,Esccloses the list (a secondEscclears the filter)Grouped View: Group by status, task or tag value, with collapsible groups showing job counts (and a status breakdown); the flat list is the default
Tags shown: the Tags menu hides tag keys (e.g. a tag common to all the jobs) in the job list of every experiment; hidden tags can still be used to filter and group
Real-time Updates: WebSocket-based updates as job states change
Failure reasons: the status of a failed job tells why it failed with the icon of the TUI (❌ the job failed, 🔗 a dependency failed, ⏱ timeout, 💾 memory, 🚫 rejected by the launcher); hover it for the description
Job actions: hovering a job in the job list shows its actions at the end of the row, each with a key acting on the hovered job (shown when hovering the icon): details (
i), logs (l), copy<task id>/<job id>(c), copy the job folder (f; for a workspace monitored over SSH, the remote folder), kill (k, running jobs) and delete (d); kill and delete ask for confirmation, and a notification at the top right tells what was copied. The duration and CO₂ of a job are below its submission dateJob Details: The details of a job show its parameters, logs, output paths, where it runs (see Hosts and resources), its failure reason with what happened (see below) and its dependencies with their status (click a dependency to open its details, e.g. to find the job at the source of a failure)
Experiment Filtering: Filter jobs by experiment ID and run
Log Viewing: View stdout/stderr logs directly in the browser (stderr first), once the job has started (the logs of a waiting or scheduled job are those of an earlier run, and are not shown). The end of a log is shown first and followed as it grows; earlier lines are loaded when scrolling to the top. Progress bars (tqdm, nested ones included) are shown as in a terminal, on one line
Orphan & Stray Jobs: Jobs no experiment references, with their disk usage; kill or delete them one by one, or all at once (with confirmation)
Services: The services of the selected experiment (of all experiments by default) and run (the current one by default, or the one chosen with the run selector), also when the experiment is not running; start, stop and open them (through
/services/<experiment>/<service>/, with?runId=<run>for a service of a given run), or view their logsActive Services: The services starting or running, of all the experiments, are listed under the navigation bar on every page (with their experiment), with links to open them, see their logs or stop them
Links: Each page has its own URL, which can be bookmarked or shared, and the browser’s back and forward buttons move between the pages (see below)
Page URLs
URL |
Page |
|---|---|
|
Experiments |
|
Jobs of an experiment ( |
|
Dependency graph ( |
|
Services of an experiment |
|
Actions of an experiment |
|
Details of a job ( |
|
The same pages, for all the experiments |
|
Orphan and stray jobs |
When the server asks for the token, the login page goes back to the page that was opened.
Killing and deleting jobs or experiments, and starting and stopping services,
behave the same in the web UI, the TUI and (for jobs) the experimaestro jobs
commands: the same operations are refused
(e.g. deleting a running job) and reported with the same messages. Deleting a
job also removes the partial directories
it leaves orphaned.
Both interfaces also show the same job status — the execution state resolved against the state the experiment tracks — and sort it in the same order: running, scheduled, ready, waiting, error (timeouts and memory errors first), done, unscheduled and transient.
Remote Access
If your jobs run on remote machines (e.g., SLURM cluster), use the --host option
to make the server accessible:
experimaestro run --host 0.0.0.0 --port 12345 experiment.yaml
Terminal UI (TUI)
The TUI provides an interactive terminal-based interface using Textual. It’s useful when working over SSH or when you prefer staying in the terminal.
Starting the TUI
During experiment execution:
# Use --console to launch the terminal UI
experimaestro run --console experiment.yaml
Standalone monitoring:
experimaestro --workdir /path/to/workspace monitor --console
Features
Experiments Table: Lists experiments with ID (and tags), hostname, jobs count, status, and duration; grouped by prefix or tag with
g(Enteron a group collapses or expands it)Job Table: Scrollable list of jobs with status indicators, filtered with filter expressions (
/)Tree View: Jobs grouped by status, task and/or tag (
t)Job Details: Where the job runs (see Hosts and resources), process, carbon, tags, progress, the failure reason with what happened (see below) and the job dependencies with their status and failure reason (click a dependency to open its details,
Escto come back to the job it was opened from)Live Logs: Real-time log output panel
Keyboard Navigation: Navigate and interact using keyboard shortcuts
Status Bar: Shows experiment status and connection info
Keyboard Shortcuts
Press ? in the TUI to show the help screen with all shortcuts.
Navigation:
Key |
Action |
|---|---|
|
Quit application |
|
Go back / Close dialog |
|
Refresh data |
|
Show help |
|
Switch to Jobs tab |
|
Switch to Services tab |
Experiments:
Key |
Action |
|---|---|
|
Select experiment |
|
Show experiment runs |
|
Delete experiment |
|
Kill all running jobs |
|
Search (in the experiment IDs and tags; |
|
Sort by status |
|
Sort by start date (default, newest first) |
|
Sort alphabetically |
|
Group by nothing, prefix or tag (in turn) |
Pressing the key of the current sort again reverses the order.
Jobs:
Key |
Action |
|---|---|
|
View job logs (once the job has started) |
|
Delete job |
|
Kill running job |
|
Open search filter |
|
Clear search filter |
|
Sort by status |
|
Sort by task |
|
Sort by date |
|
Sort alphabetically |
|
Choose the tags shown (hidden tag keys, for all the experiments) |
|
Copy folder path |
Log Viewer:
Key |
Action |
|---|---|
|
Toggle follow mode |
|
Go to top (earlier lines are loaded at the top) |
|
Go to bottom |
|
Check the logs for changes now |
|
Close viewer |
Services:
Key |
Action |
|---|---|
|
Start service |
|
Stop service |
|
View logs |
|
Copy URL |
Orphan Jobs:
Key |
Action |
|---|---|
|
Refresh |
|
Sort by task |
|
Sort by size |
|
Delete selected |
|
Delete all |
|
Copy folder path |
Clipboard Support
The TUI supports clipboard operations (like f to copy path) that work both
locally and in remote/tmux environments.
How it works:
First tries
pyperclipfor local clipboard accessFalls back to OSC 52 escape sequences when pyperclip fails (e.g., over SSH)
OSC 52 supported terminals:
iTerm2 (macOS)
kitty
alacritty
Windows Terminal
Most modern terminal emulators
tmux configuration:
For clipboard to work inside tmux over SSH, add these lines to ~/.tmux.conf:
set -g set-clipboard on
set -g allow-passthrough on # Required for tmux 3.2+
Then reload tmux configuration: tmux source-file ~/.tmux.conf
Python API
The TUI can also be launched programmatically:
from experimaestro.tui import ExperimentTUI
tui = ExperimentTUI(
workdir=Path("/path/to/workspace"),
show_logs=True,
)
tui.run()
Job Filter Expressions
The TUI search bar (/) and the web Filter input accept the same expressions
(also used by experimaestro jobs --filter, see CLI):
Expression |
Meaning |
|---|---|
|
Job state ( |
|
Task identifier matches a regular expression (anchored at the start) |
|
Host running the job (one of them, for a job on several nodes; see Hosts and resources) |
|
SLURM partition of the job |
|
Tag value of the job in the displayed experiment run |
|
Value in a list ( |
|
Combine conditions |
Tag values are compared as text, whatever their type: a tag phase=1 is
matched by phase = '1' or phase ~ '1'.
For example, @state = 'error' and model = 'bm25' selects the failed jobs whose
model tag is bm25.
Choosing Between Web and TUI
Use Case |
Recommended |
|---|---|
Local development |
Web UI |
Remote SSH session |
TUI |
Sharing with team |
Web UI (with |
Minimal dependencies |
TUI |
Detailed log analysis |
Web UI |
Quick status check |
TUI |
Remote Monitoring via SSH
When experiments run on remote servers (e.g., HPC clusters, cloud instances), you can monitor them from your local machine using SSH tunneling. Monitoring a remote workspace establishes an SSH connection, starts a monitoring server on the remote host, and communicates via JSON-RPC over the SSH channel.
Basic Usage
experimaestro [SSH OPTIONS] --ssh HOST:REMOTE_WORKDIR monitor (--console | --web)
HOST: SSH host in standard format (e.g.,
user@server,server, or SSH config alias)REMOTE_WORKDIR: Path to the workspace directory on the remote server
Examples
# Basic SSH monitoring (web UI)
experimaestro --ssh myserver:/home/user/experiments monitor --web
# With the console TUI instead of the web UI
experimaestro --ssh user@cluster.example.com:/scratch/experiments monitor --console
# Custom port for web interface
experimaestro --ssh myserver:/workspace monitor --web --port 8080
# With SSH options (e.g., custom port, identity file)
experimaestro --ssh myserver:/workspace -o "-p 2222" -o "-i ~/.ssh/cluster_key" monitor --web
# Specify path to experimaestro on remote host
experimaestro --ssh myserver:/workspace --remote-xpm /opt/conda/bin/experimaestro monitor --web
Options
Option |
Description |
|---|---|
|
Terminal (Textual) UI or web UI (one is required) |
|
Port for local web server (default: 12345) |
The SSH options (--remote-xpm, -o, …) are given before the command, see
Remote Workspaces.
How It Works
Connection: The client establishes an SSH connection to the remote host
Server Start: On the remote host, experimaestro starts in server mode, reading commands from stdin and writing responses to stdout
JSON-RPC Protocol: Commands and responses are exchanged using JSON-RPC 2.0
Job logs: The remote experimaestro renders the logs as a terminal would and sends them through the same connection, by pages (the end of a log first, earlier pages when scrolling up, the new content while following): only the lines shown are sent, not the bytes of the progress bars that rewrite them. When the end of a log is a progress bar running for hours, the lines before it are shown with the bar as last drawn. An open log is checked every second, less and less often (up to every 30 s) while it does not change. A log rewritten by a restarted job is detected and read again. With an older remote experimaestro, logs are read by blocks into a local copy that lasts as long as the monitor, or, older still, copied with rsync.
File Sync: When needed (e.g., for TensorBoard logs), files are synchronized on-demand using rsync over SSH
Real-time Updates: The server sends notifications when job states change — only for what the UI shows: the experiment list is always kept up to date, but job, action and service events are only sent for the experiment shown (and for the started services, shown on every page). The same applies to the web UI, for each browser page.
Lost Connections
When the SSH connection drops (network loss, laptop sleep, the remote server exiting), the monitor reconnects on its own: it starts a new remote server, declares again the experiments shown, and reads the state again (the events sent while disconnected were missed). Both interfaces show the connection state on every page — the TUI below its header, the web UI below its navigation bar:
reconnecting: the connection was lost, with the number of the attempt (e.g.
attempt 2/5) and why the connection was lost or the last attempt failed;disconnected: all the attempts failed; restart the monitor to try again.
Nothing is shown while connected. The reconnection policy — at most
max_retries attempts within window seconds, with a delay doubling from
delay up to max_delay — is set in settings.yaml, see
Reconnection. SSH is run with
ServerAliveInterval=15 and ServerAliveCountMax=3, so that a dead
connection is detected within a minute (your own -o options take
precedence).
Configuring SSH Settings in settings.yaml
Instead of passing all options on the command line, you can configure SSH settings
per-workspace in your ~/.config/experimaestro/settings.yaml:
workspaces:
- id: my-cluster
path: /scratch/experiments
ssh:
host: user@cluster
shell_init: "source /etc/profile; module load python/3.10"
uv_offline: true
python: python3
Then use --workspace to load settings automatically:
experimaestro -w my-cluster monitor --web
See Remote Workspaces (SSH) for all available settings.
Remote experimaestro Installation
By default, the SSH client runs uv tool run experimaestro==<version> on the
remote host. This requires:
uvinstalled on the remote hostNetwork access to PyPI (or a local mirror)
For environments without direct PyPI access (common on HPC clusters), use
--uv-offline or configure uv_offline: true in workspace SSH settings.
If the remote host requires environment setup before running commands
(e.g., module load), use --remote-shell-init or configure shell_init
in workspace SSH settings.
Alternatively, specify a pre-installed experimaestro path:
# If experimaestro is installed in a virtualenv
experimaestro --ssh host:/workspace --remote-xpm /path/to/venv/bin/experimaestro monitor --web
# If installed system-wide
experimaestro --ssh host:/workspace --remote-xpm experimaestro monitor --web
Version Compatibility
The local and remote experimaestro versions should be compatible. The protocol version is checked on connection, and the connection is refused if it differs. For best results, use the same version on both sides.
A remote experimaestro older than the local one may lack some features (e.g.
the dependency graph): the remote server lists what it handles when the
connection starts, and the interfaces then report the missing features (the
remote experimaestro does not support …) instead of failing. Update
experimaestro on the remote host, or point --remote-xpm to a more recent one.
SSH Output
Remote server output (logs, warnings) is displayed locally with a colored
[SSH] prefix to distinguish it from local output. This helps debug
connection issues or see remote server activity.
Troubleshooting
Slow or stuck interface: trace the requests on both sides. Each one is logged with its time and answer size: on the remote side, the time spent answering it; on the local side, the round trip (which includes the wait in the queue — requests are answered one at a time).
experimaestro --logging xpm.remote.trace=DEBUG -w my-cluster monitor \
--remote-logging xpm.remote.trace=DEBUG
The xpm.remote.payload logger shows the messages themselves (very verbose).
uv tool run hangs (no output):
The remote host likely has no outbound HTTPS access to PyPI
Use
uv_offline: truein workspace SSH settings or--uv-offlineCLI flagEnsure the package is already cached:
ssh host "uv tool install experimaestro==X.Y.Z"(run once with network access)
Connection times out on a cluster:
Starting Python can take tens of seconds on a shared file system (Lustre, GPFS, NFS) when its cache is cold
Increase
connect_timeoutin the workspace SSH settings (or use--connect-timeout), 60 seconds by defaultKeep the bytecode cache on a local disk with
export PYTHONPYCACHEPREFIX=/tmp/$USER/pycache(inshell_init)With Python 3.15+, the command line loads far fewer modules (lazy imports)
Wrong Python version or missing modules:
Non-interactive SSH skips
.profile/.bash_profileUse
shell_initin workspace SSH settings or--remote-shell-initto run setup commands (e.g.,source /etc/profile; module load python/3.10)
Connection refused or timeout:
Verify SSH access:
ssh user@host echo "connected"Check if the remote workspace exists
Ensure experimaestro is accessible on the remote host
Local copies: each monitor keeps its copies of the remote files in a
temporary directory (xpm_remote_*, marked by a .experimaestro-remote-cache
file), removed when it stops. The directories left by monitors that did not
stop normally are removed by the next monitor started on the same host.
Slow logs: to see how the interfaces behave on a slow network, set
XPM_LOG_READ_DELAY to a delay (in seconds) added to each read of a log file,
e.g. XPM_LOG_READ_DELAY=1 experimaestro --ssh host:/path monitor --web.
Services not working (e.g., TensorBoard):
Services are recreated locally from saved state
Some services may require file synchronization; this happens automatically
Check that rsync is available on both machines
Experiment DAG Visualization
The Graph tab of the web UI shows the dependency graph of the selected experiment run: jobs coloured by status (with the progress of the running ones), init tasks (thin solid border, tinted background), the configurations linking them (optional, dashed border), stacks of similar nodes, and a grid search configuration selector. The graph follows the run: jobs submitted while it runs are added. Clicking a node highlights its lineage and opens an interactive parameter inspector in the right panel.
Dependency graph: the selected stack of evaluations, with its lineage highlighted.
Integrated Parameters Inspector
Clicking any node or stack opens the inspector panel on the right:
Hierarchical drill-down: recursively expands nested sub-configurations, showing their parameters, data paths, and initialization tasks while preventing circular references.
Categorization: automatically separates parameters into logical groups (Hyperparameters, Paths & Outputs, Sub-Configurations, Collections, Init Tasks).
Search & Highlighting: filter parameters by name or value in real time. Matching text is highlighted in yellow, collapsed parent configs auto-unfold, and a match counter (
1 of N) provides previous/next buttons (▲/▼) and keyboard shortcuts (Enter/Shift+Enter) to cycle matches.Code Generation: switch between an interactive Tree, reproducible Python (.C) synthesized code, or raw JSON.
Stacked Runs: for grouped nodes representing multiple runs (e.g. multi-seed evaluations), member tabs (
#1,#2, …) allow inspecting individual run configurations.
Standalone HTML Export
The same interactive view, including graph navigation and the full parameters inspector, can be exported as a completely self-contained HTML page that requires no server:
experimaestro dag my_experiment --html dag.html
It can also visualize standalone serialized models or Hugging Face Hub models without a workspace:
experimaestro dag VictorMorand/Mice-P-61m --html mice_dag.html
See Experiment DAG Visualization in the CLI reference for details and options.
Hosts and resources
The job details of both interfaces show where a job runs (Runs on), e.g.
node042 (partition gpu, 8 CPUs, GPUs 0,1):
the host of the job process or, in a SLURM allocation (pilot jobs included), its nodes — the one running the job process first;
for SLURM jobs, the partition and the CPUs and GPUs allocated on that node.
The job process records this when it starts (in its status.json, from its
host name and the SLURM environment), so it is shown by any monitor — also
through SSH, or on a machine without the SLURM commands — and kept once the
job has ended. Only the last run of a job is described: a retry replaces it.
Jobs can be selected by host or partition with the @host and @partition
filter fields.
Failure details
The job details of both interfaces show what happened to a failed job, read
from the .failed marker in the job folder:
for an exception raised by the task, its type and message (e.g.
ValueError: the learning rate must be positive) and where it was raised (file.py:42 in execute) — the full traceback is in the job logs;for a timeout, a cancellation or a crashed process, the recorded message;
for a job the launcher refused to start (e.g. a SLURM pending reason such as
PartitionTimeLimit) or stopped (e.g. the SLURM time limit), the reason given by the launcher, followed by what the job reported (a job SLURM stops at its time limit receives SIGTERM and reports it was cancelled: it is shown as timed out, and a resumable task is retried);the exit code of the job process.
Jobs that failed with an older experimaestro only show their exit code. Over SSH, the details are read by the remote experimaestro, if recent enough.