Skip to main content

ClearML Agent CLI

The following page provides a reference to clearml-agent's CLI commands:

  • build - Create a worker environment without executing an experiment.
  • config - List your ClearML Agent configuration data.
  • daemon - Run a worker daemon listening to a queue for Tasks (experiments) to execute.
  • execute - Execute an experiment, locally without a queue.
  • list - List the current workers.

build

Use the build command to create worker environments without executing tasks.

You can build Docker containers according to the execution environments of specific tasks, which an agent can later use to execute other tasks. See tutorial.

You can also create a Docker container that executes a specific task when launched. See tutorial.

clearml-agent build [-h] --id TASK_ID [--target TARGET]
[--install-globally]
[--docker [DOCKER [DOCKER ...]]] [--force-docker]
[--python-version PYTHON_VERSION]
[--entry-point {reuse_task,clone_task}] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]

Parameters

NameDescriptionOptional
--idBuild a worker environment for this Task ID.No
--cpu-onlyDisable GPU access for the Docker container.Yes
--dockerRun agent in Docker mode. Specify a Docker container that a worker will execute at launch. To specify the image name and optional arguments, use one of the following:
  • --docker <image_name> <args> on the command line
  • --docker on the command line, and specify the default image name and arguments in the configuration file.
Environment variable settings for Docker containers:
  • CLEARML_DOCKER_SKIP_GPUS_FLAG - Ignore the --gpus flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • CLEARML_AGENT_GIT_USER and CLEARML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.
To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.
Yes
--entry-pointUsed in conjunction with --docker, indicates how to run the Task specified by --task-id on Docker startup. The --entry-point options are:
  • reuse - Overwrite the existing Task data.
  • clone_task - Clone the Task, and execute the cloned Task.
Yes
--force-dockerForce using the agent-specified docker image (either explicitly in the --docker argument or using the agent's default docker image). If provided, the agent will not use any docker container information stored in the task itself (default False)Yes
--git-passGit password for repository access.Yes
--git-userGit username for repository access.Yes
--gpusSpecify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example:
  • --gpus 0
  • --gpu 0,1,2
  • --gpus all
Yes
-h, --helpGet help for this command.Yes
--install-globallyInstall the required Python packages before creating the virtual environment. Use agent.package_manager.system_site_packages to control the installation of the system packages. When --docker is used, --install-globally is always true.Yes
--log-levelSDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
Yes
--python-versionVirtual environment Python version to use.Yes
-OCompile optimized pyc code (see python documentation). Repeat for more optimization.Yes
--targetThe target folder for the virtual environment and source code that will be used at launch.Yes

config

List your ClearML Agent configuration.

clearml-agent config [-h]

daemon

Use the daemon command to spin up an agent on any machine: on-prem and/or cloud instance. When spinning up an agent, assign it a queue(s) to service, and when experiments are added to its queues, the agent will pull and execute them.

With the daemon command you can configure your agent's behavior: allocate resources, prioritize queues, set it to run in a Docker, and more.

clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-fairness] 
[--standalone-mode] [--services-mode [SERVICES_MODE]]
[--child-report-tags CHILD_REPORT_TAGS [CHILD_REPORT_TAGS ...]]
[--create-queue] [--detached] [--stop] [--dynamic-gpus]
[--uptime [UPTIME [UPTIME ...]]] [--downtime [DOWNTIME [DOWNTIME ...]]]
[--status] [--use-owner-token] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]
[--docker [DOCKER [DOCKER ...]]] [--force-current-version]

Parameters

NameDescriptionOptional
--child-report-tagsList of tags to send with the status reports from the worker that executes a task.Yes
--cpu-onlyIf running in Docker mode (see the --docker option), disable GPU access for the Docker container or virtual environment.Yes
--create-queueIf the queue name provided with --queue does not exist in the server, create it on-the-fly and use it.Yes
--detachedRun agent in the background. The clearml-agent command returns immediately.Yes
--dockerRun in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either:
  • Use --docker <image_name> <args> on the command line, or
  • Use --docker on the command line, and specify the default image name and arguments in the configuration file.
Environment variable settings for Docker containers:
  • CLEARML_DOCKER_SKIP_GPUS_FLAG - Ignore the --gpus flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • CLEARML_AGENT_GIT_USER and CLEARML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.
Yes
--downtimeSpecify downtime for clearml-agent in <hours> <days> format. For example, use 09-13 TUE to set Tuesday's downtime to 09-13.

NOTES:
  • This feature is available under the ClearML Enterprise plan
  • Make sure to configure only --uptime or --downtime, but not both.
Yes
--dynamic-gpusAllow to dynamically allocate GPUs based on queue properties, configure with --queue <queue_name>=<num_gpus>. For example: --dynamic-gpus --queue dual_gpus=2 single_gpu=1

NOTE: This feature is available under the ClearML Enterprise plan
Yes
--force-current-versionTo use your current version of ClearML Agent when running in Docker mode (the --docker argument is specified), instead of the latest ClearML Agent version which is automatically installed, specify force-current-version.

For example, if your current ClearML Agent version is 0.13.1, but the latest version of ClearML Agent is 0.13.3, use --force-current-version and your Task will execute in the Docker container with ClearML Agent version 0.13.1.
Yes
--foregroundPipe full log to stdout/stderr. Do not use this option if running in background.Yes
--git-passGit password for repository access.Yes
--git-userGit username for repository accessYes
--gpusIf running in Docker mode (see the --docker option), specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example:
  • --gpus 0
  • --gpu 0,1,2
  • --gpus all
Yes
-h, --helpGet help for this command.Yes
--log-levelSDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
Yes
-OCompile optimized pyc code (see python documentation). Repeat for more optimization.Yes
--order-fairnessPull from each queue in a round-robin order, instead of priority order.Yes
--queueSpecify the queues which the worker is listening to. The values can be any combination of:
  • One or more queue IDs
  • One or more queue names
  • default indicating the default queue
Yes
--services-modeLaunch multiple long-term docker services. Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is verified. Use in CPU mode (--cpu-only) only.
To limit the number of simultaneous tasks run in services mode, pass the maximum number immediately after the --services-mode option (e.g. --services-mode 5)
Yes
--standalone-modeDo not use any network connects. This assumes all requirements are pre-installed.Yes
--statusPrint the worker's schedule (uptime properties, server's runtime properties and listening queues)Yes
--stopTerminate a running ClearML Agent, if other arguments are the same. If no additional arguments are provided, agents are terminated in lexicographical order.Yes
--uptimeSpecify uptime for clearml-agent in <hours> <days> format. For example, use 17-20 TUE to set Tuesday's uptime to 17-20.

NOTES:
  • This feature is available under the ClearML Enterprise plan
  • Make sure to configure only --uptime or --downtime, but not both.
Yes
--use-owner-tokenGenerate and use the task owner's token for the execution of the task.Yes

execute

Use the execute command to set an agent to build and execute specific tasks directly without listening to a queue.

clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitoring] 
[--full-monitoring] [--require-queue]
[--standalone-mode] [--docker [DOCKER [DOCKER ...]]] [--clone]
[-O] [--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]

Parameters

NameDescriptionOptional
--idThe ID of the Task to buildNo
--cloneClone the Task specified by --id, and then execute that cloned Task.Yes
--cpu-onlyDisable GPU access for the daemon, only use CPU in either docker or virtual environment.Yes
--dockerRun in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, use one of the following:
  • --docker <image_name> <args> on the command line
  • --docker on the command line, and specify the default image name and arguments in the configuration file.
Environment variable settings for Dockers containers:
  • CLEARML_DOCKER_SKIP_GPUS_FLAG - Ignore the --gpus flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • CLEARML_AGENT_GIT_USER and CLEARML_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.
Yes
--disable-monitoringDisable logging and monitoring, except for stdout.Yes
--full-monitoringCreate a full log, including the environment setup log, Task log, and monitoring, as well as stdout.Yes
--git-passGit password for repository access.Yes
--git-userGit username for repository access.Yes
--gpusSpecify active GPUs for the daemon to use (docker / virtual environment), Equivalent to setting NVIDIA_VISIBLE_DEVICES. For example:
  • --gpus 0
  • --gpu 0,1,2
  • --gpus all
Yes
-h, --helpGet help for this command.Yes
--log-fileThe log file for Task execution output (stdout / stderr) to a text file.Yes
--log-levelSDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
Yes
-OCompile optimized pyc code (see python documentation). Repeat for more optimization.Yes
--require-queueIf the specified task is not queued, the execution will fail (used for 3rd party scheduler integration, e.g. K8s, SLURM, etc.)Yes
--standalone-modeDo not use any network connects, assume everything is pre-installedYes

list

List information about all active workers.

clearml-agent list [-h]