Skip to main content

ClearML Agent Reference

This reference page provides detailed information about ClearML Agent commands, including:

  • build - Create a worker environment, without executing an experiment.
  • config - List your ClearML Agent configuration.
  • daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
  • execute - Execute an experiment, locally without a queue.
  • list - List the current workers.

build#

Use the build subcommand to create a worker environment without executing the experiment.

Syntax#

clearml-agent build [-h] --id TASK_ID [--target TARGET]
[--install-globally]
[--docker [DOCKER [DOCKER ...]]] [--force-docker]
[--python-version PYTHON_VERSION]
[--entry-point {reuse_task,clone_task}] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]

Arguments#

id (mandatory)

  • Build a worker environment for this Task ID.

cpu-only

  • Disable GPU access for the Docker container.

docker

  • Docker mode. A Docker container that a worker will execute at launch.

    To specify the image name and optional arguments, either:

    • Use --docker <image_name> <args> on the command line, or

    • Use --docker on the command line, and specify the image name and arguments in the configuration file.

      Environment variable settings for Dockers:

    • CLEARML_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.

    • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.

    • CLEARML_AGENT_GIT_USER and CLEARML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

      To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.

entry-point

  • Used in conjunction with --docker, indicates how to run the Task specified by task-id on Docker startup. The entry-point options are:
    • reuse - Overwrite the existing Task data.
    • clone_task - Clone the Task, and execute the cloned Task.

force-docker

  • Force using the agent-specified docker image (either explicitly in the --docker argument or using the agent's default docker image). If provided, the agent will not use any docker container information stored in the task itself (default False)

git-pass

  • Git password for repository access.

git-user

  • Git username for repository access.

gpus

  • Specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example:
    • --gpus 0
    • --gpu 0,1,2
    • --gpus all

h, help

  • Get help for this command.

install-globally

  • Install the required Python packages before creating the virtual environment. Use agent.package_manager.system_site_packages to control the installation of the system packages. When --docker is used, install-globally is always true.

log-level

  • SDK log level. The values are:
    • DEBUG
    • INFO
    • WARN
    • WARNING
    • ERROR
    • CRITICAL

python-version

  • Virtual environment Python version to use.

O

  • Compile optimized pyc code (see python documentation). Repeat for more optimization.

target

  • The target folder for the virtual environment and source code that will be used at launch.

config#

Use the config subcommand to list your ClearML Agent configuration.

Syntax#

clearml-agent config [-h]

daemon#

Use the daemon subcommand to run a worker, optionally in a Docker container, listening to a queue.

Syntax#

clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-fairness]
[--standalone-mode] [--services-mode [SERVICES_MODE]]
[--child-report-tags CHILD_REPORT_TAGS [CHILD_REPORT_TAGS ...]]
[--create-queue] [--detached] [--stop] [--dynamic-gpus]
[--uptime [UPTIME [UPTIME ...]]] [--downtime [DOWNTIME [DOWNTIME ...]]]
[--status] [--use-owner-token] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]
[--docker [DOCKER [DOCKER ...]]] [--force-current-version]

Arguments#

child-report-tags

List of tags to send with the status reports from the worker that executes a task.

cpu-only

  • If running in Docker mode (see the docker option), disable GPU access for the Docker container or virtual environment.

create-queue

  • If the queue name provided does not exist in the server, create and use it.

detached

  • Run agent in the background. The clearml-agent command returns immediately.

docker

  • Run in Docker mode. Execute the Task inside a Docker container.

    To specify the image name and optional arguments, either:

    • use --docker <image_name> <args> on the command line, or

    • use --docker on the command line, and specify the image name and arguments in the configuration file.

      Environment variable settings for Dockers:

    • CLEARML_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.

    • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.

    • ClearML_AGENT_GIT_USER and ClearML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

downtime

  • Specify downtime for clearml-agent in <hours> <days> format.

    For example, use 09-13 TUE to set Tuesday's downtime to 09-13.

info
  • This feature is available under the ClearML Enterprise plan
  • Make sure to have only one of uptime / downtime configuration and not both.

dynamic-gpus

  • Allow to dynamically allocate gpus based on queue properties, configure with --queue <queue_name>=<num_gpus>.

    For example: --dynamic-gpus --queue dual_gpus=2 single_gpu=1

Enterprise Feature

This feature is available under the ClearML Enterprise plan

force-current-version

  • To use your current version of ClearML Agent when running in Docker mode (the docker argument is specified), instead of the latest ClearML Agent version which is automatically installed, specify force-current-version.

    For example, if your current ClearML Agent version is 0.13.1, but the latest version of ClearML Agent is 0.13.3, use --force-current-version and your Task will execute in the Docker container with ClearML Agent version 0.13.1.

foreground

  • Pipe full log to stdout/stderr. Do not use this option if running in background.

git-pass

  • Git password for repository access.

git-user

  • Git username for repository access.

gpus

  • If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example:
    • --gpus 0
    • --gpu 0,1,2
    • --gpus all

h, help

  • Get help for this command.

log-level

  • SDK log level. The values are:
    • DEBUG
    • INFO
    • WARN
    • WARNING
    • ERROR
    • CRITICAL

O

  • Compile optimized pyc code (see python documentation). Repeat for more optimization.

order-fairness

  • Pull from each queue in a round-robin order, instead of priority order.

queue

  • Specify the queues which the worker is listening to. The values can be any combination of:

    • One or more queue IDs.
    • One or more queue names.
    • default indicating the default queue.

services-mode

  • Launch multiple long-term docker services. Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is verified. Use in CPU mode (--cpu-only), only.

  • To limit the number of simultaneous tasks run in services mode, pass the maximum number immediately after the --services-mode option (e.g. --services-mode 5)

standalone-mode

  • Do not use any network connects. This assumes all requirements are pre-installed.

status

  • Print the worker's schedule (uptime properties, server's runtime properties and listening queues)

stop

  • Terminate a running ClearML Agent, if other arguments are the same. If no additional arguments are provided, agents are terminated in lexicographical order.

uptime

  • Specify uptime for clearml-agent in <hours> <days> format. for example, use 17-20 TUE to set Tuesday's uptime to 17-20
info
  • This feature is available under the ClearML Enterprise plan
  • Make sure to have only one of uptime / downtime configuration and not both.

use-owner-token

Generate and use the task owner's token for the execution of the task.

execute#

Use the execute subcommand to build and execute an experiment without a queue.

Syntax#

clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitoring]
[--full-monitoring] [--require-queue]
[--standalone-mode] [--docker [DOCKER [DOCKER ...]]] [--clone]
[-O] [--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]

Arguments#

id (mandatory)

  • The ID of the Task to build.

clone

  • Clone the Task specified by id, and then execute that cloned Task.

cpu-only

  • Disable GPU access for the daemon, only use CPU in either docker or virtual environment.

docker

  • Run in Docker mode. Execute the Task inside a Docker container.

    To specify the image name and optional arguments, either:

    • use --docker <image_name> <args> on the command line, or

    • use --docker on the command line, and specify the default image name and arguments in the configuration file.

      Environment variable settings for Dockers containers:

    • ClearML_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.

    • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.

    • ClearML_AGENT_GIT_USER and ClearML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

disable-monitoring

  • Disable logging and monitoring, except for stdout.

full-monitoring

  • Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout.

git-pass

  • Git password for repository access.

git-user

  • Git username for repository access.

gpus

  • Specify active GPUs for the daemon to use (docker / virtual environment), Equivalent to setting NVIDIA_VISIBLE_DEVICES. Examples: --gpus 0 or --gpu 0,1,2 or --gpus all

h, help

  • Get help for this command.

log-file

  • The log file for Task execution output (stdout / stderr) to a text file.

log-level

  • SDK log level. The values are:
    • DEBUG
    • INFO
    • WARN
    • WARNING
    • ERROR
    • CRITICAL

O

  • Compile optimized pyc code (see python documentation). Repeat for more optimization.

require-queue

  • If the specified task is not queued (in any Queue), the execution will fail. (Used for 3rd party scheduler integration, e.g. K8s, SLURM, etc.)

standalone-mode

  • Do not use any network connects, assume everything is pre-installed

list#

Use the list subcommand to list information about all workers

Syntax#

clearml-agent list [-h]
Edit this page