Running experiments

The main class is experimaestro.experiment

experiment

Main experiment object

It is a context object, i.e. experiments is run with

    with experiment(...) as xp:
        ...
alt_jobspaths property

Return potential other directories

jobsbakpath property

Return the directory in which results can be stored for this experiment

jobspath property

Return the directory in which results can be stored for this experiment

resultspath property

Return the directory in which results can be stored for this experiment

__init__(env, name, *, host=None, port=None, token=None, run_mode=None, launcher=None)
Parameters:
  • env (Union[Path, str, WorkspaceSettings]) –

    an environment -- or a working directory for a local environment

  • name (str) –

    the identifier of the experiment

  • launcher –

    The launcher (if not provided, inferred from path)

  • host (Optional[str], default: None ) –

    The host for the web server (overrides the environment if set)

  • port (Optional[int], default: None ) –

    the port for the web server (overrides the environment if set). Use negative number to avoid running a web server (default when dry run).

  • run_mode (Optional[RunMode], default: None ) –

    The run mode for the experiment (normal, generate run files, dry run)
add_service(service)

Adds a service (e.g. tensorboard viewer) to the experiment

Parameters:
  • service (ServiceClass) –

    A service instance
Returns:
  • ServiceClass –

    The same service instance
current() staticmethod

Returns the current experiment, but checking first if set

If there is no current experiment, raises an AssertError

load(reference, name='default')

Serializes configurations.

Loads configuration objects from an experimental directory

Parameters:
  • reference (str) –

    The name of the experiment

  • name (str, default: 'default' ) –

    The name of the saving directory (default to default)
prepare(job)

Generate the file

save(obj, name='default')

Serializes configurations.

Saves configuration objects within the experimental directory

Parameters:
  • obj (Any) –

    The object to save

  • name (str, default: 'default' ) –

    The name of the saving directory (default to default)
setenv(name, value, override=True)

Shortcut to set the environment value

stop()

Stop the experiment as soon as possible

token(name, count)

Returns a token for this experiment

The token is the default token of the workspace connector

wait()

Wait until the running processes have finished

When using the command line interface to run experiment, the main object of interaction is the ExperimentHelper:

ExperimentHelper

Helper for experiments

xp: experiment instance-attribute

The experiment object

decorator(*args, **kwargs) classmethod

Decorator for the run(helper, configuration) method

Experiment services

Service

An experiment service

Services can be associated with an experiment. They send notifications to service listeners.

add_listener(listener)

Adds a listener

Parameters:
remove_listener(listener)

Removes a listener

Parameters:

WebService

Bases: Service

Web service

ServiceListener

A service listener

Experiment configuration

The module experimaestro.experiments contain code factorizing boilerplate for launching experiments. It allows to setup the experimental environment and read YAML configuration files to setup some experimental parameters.

This can be extended to support more specific experiment helpers (see e.g. experimaestro-ir for an example).

ConfigurationBase should be the parent class of any configuration.

ConfigurationBase

Base configuration for any experiment

description: str = '' class-attribute instance-attribute

Description of the experiment

file: str = 'experiment' class-attribute instance-attribute

Relative path of the file containing a run function

id: str = MISSING class-attribute instance-attribute

ID of the experiment

This ID is used by experimaestro when running as the experiment.

paper: str = '' class-attribute instance-attribute

Source paper for this experiment

parent: Optional[str] = None class-attribute instance-attribute

Relative path of a YAML file that should be merged

subtitle: str = '' class-attribute instance-attribute

Allows to give some more details about the experiment

title: str = '' class-attribute instance-attribute

Short description of the experiment

Example

An experiment.py file:

    from experimaestro.experiments import ExperimentHelper, configuration, ConfigurationBase

    @configuration
    class Configuration(ConfigurationBase):
        #: Default learning rate
        learning_rate: float = 1e-3

    def run(
        helper: ExperimentHelper, cfg: Configuration
    ):
        # Experimental code
        ...

With full.yaml located in the same folder as experiment.py

    file: experiment
    learning_rate: 1e-4

The experiment can be started with

    experimaestro run-experiment --run-mode normal full.yaml

Common handling

cli

ExperimentCallable

Bases: Protocol

Protocol for the run function

ExperimentHelper

Helper for experiments

xp: experiment instance-attribute

The experiment object

decorator(*args, **kwargs) classmethod

Decorator for the run(helper, configuration) method

experiments_cli(yaml_file, xp_file, host, port, xpm_config_dir, workdir, workspace, env, run_mode, extra_conf, pre_yaml, post_yaml, args, show, debug)

Run an experiment

load(yaml_file)

Loads a YAML file, and parents one if they exist