Skip to main content

ClearML Task

Using only the command line and zero additional lines of code, easily track your work and integrate ClearML with your existing code.

clearml-task automatically imports any script or Python repository into ClearML. clearml-task lets you enqueue your code for execution by an available ClearML Agent. You can even provide command line arguments, Python module dependencies, and a requirements.txt file!

What Is ClearML Task For?#

  • Launching off-the-shelf code on a remote machine with dedicated resources (e.g. GPU)
  • Running hyperparameter optimization on a codebase that is still not in ClearML
  • Creating a pipeline from an assortment of scripts, that you need to turn into ClearML tasks
  • Running some code on a remote machine, either using an on-prem cluster or on the cloud

How Does ClearML Task Work?#

  1. Execute clearml-task, specifying the ClearML target project and task name, along with your script (and repository / commit / branch). Optionally, specify an execution queue and Docker image to use.
  2. clearml-task does its magic! It creates a new task on the ClearML Server, and, if so directed, enqueues it for execution by a ClearML Agent.
  3. While the Task is running on the remote machine, all its console outputs are logged in real-time, alongside your TensorBoard and matplotlib. You can track your script’s progress and results in the ClearML Web UI (a link to your task details page in the ClearML Web UI is printed as ClearML Task creates the task).

Execution Configuration#

Docker#

Specify a Docker container to run the code in with the --docker <docker_image> option. The ClearML Agent pulls it from Docker Hub or a Docker artifactory automatically.

Package Dependencies#

clearml-task automatically finds the requirements.txt file in remote repositories.

If a local script requires certain packages, or the remote repository doesn't have a requirements.txt file, manually specify the required Python packages using --packages "<package_name>", for example --packages "keras" "tensorflow>2.2".

Queue#

Tasks are passed to ClearML Agents via Queues. Specify a queue in which to enqueue the task. If you don't input a queue, the task is created in draft status, and you can enqueue it at a later point.

Branch and Working Directory#

To specify your code's branch and commit ID, pass the --branch <branch_name> --commit <commit_id> options. If unspecified, clearml-task will use the latest commit from the 'master' branch.

Github Default Branch

For Github repositories, it is recommended to explicitly specify your default branch (e.g. --branch main) to avoid errors in identifying the correct default branch.

Command Line Options#

NameDescriptionOptional
--versionDisplay the clearml-task utility versionYes
--projectSet the project name for the task (required, unless using --base-task-id). If the named project does not exist, it is created on-the-flyNo
--nameSet a target name for the new taskNo
--repoURL of remote repository. Example: --repo https://github.com/allegroai/clearml.gitYes
--branchSelect repository branch / tag. By default, latest commit from the master branchYes
--commitSelect commit ID to use. By default, latest commit, or local commit ID when using local repositoryYes
--folderExecute the code from a local folder. Notice, it assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and replicated on the remote machineYes
--scriptEntry point script for the remote execution. When used with --repo, input the script's relative path inside the repository. For example: --script source/train.py. When used with --folder, it supports a direct path to a file inside the local repository itself, for example: --script ~/project/source/train.pyNo
--cwdWorking directory to launch the script from. Relative to repo root or local --folderYes
--argsArguments to pass to the remote task, list of <argument>=<value> strings. Currently only argparse arguments are supportedYes
--queueSelect a task's execution queue. If not provided, a task is created but not launchedYes
--requirementsSpecify requirements.txt file to install when setting the session. By default, the requirements.txt from the repository will be usedYes
--packagesManually specify a list of required packages. Example: --packages "tqdm>=2.1" "scikit-learn"Yes
--dockerSelect the Docker image to use in the remote taskYes
--docker_argsAdd Docker arguments, pass a single stringYes
--docker_bash_setup_scriptAdd a bash script to be executed inside the Docker before setting up the task's environmentYes
--output-uriSet the task output_uri, upload destination for task models and artifactsYes
--task-typeSet the task type. Optional values: training, testing, inference, data_processing, application, monitor, controller, optimizer, service, qc, customYes
--skip-task-initIf set, Task.init() call is not added to the entry point, and is assumed to be called within the scriptYes
--base-task-idUse a pre-existing task in the system, instead of a local repo / script. Essentially clones an existing task and overrides arguments / requirementsYes
--import-offline-sessionSpecify the path to the offline session you want to import.Yes

Usage#

These commands demonstrate a few useful use cases for clearml-task

Executing Code from a Remote Repository#

clearml-task --project examples --name remote_test --repo https://github.com/allegroai/events.git --branch master --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default

The keras_mnist.py script from the events GitHub repository is imported as a ClearML task named remote_test in the examples project. Its command line arguments batch_size and epochs values are set, and the task is enqueued for execution on the default queue.

Executing a Local Script#

Using clearml-task to execute a local script is very similar to using it with a remote repo.

clearml-task --project examples --name local_test --script keras_mnist.py --branch master --requirements requirements.txt --args epochs=1 --queue default

The keras_mnist.py script on the user's local machine is imported as a ClearML task named local_test in the examples project.

Its Python requirements are taken from the local requiremnts.txt file, and its epochs command line argument value is set.

The task is enqueued for execution on the default queue.

Pushing a Script to the Server#

clearml-task --project examples --name no_execute --script keras_mnist.py --branch master --requirements requirements.txt --args epochs=1

The keras_mnist.py script on the user's local machine is imported as a ClearML task named no_execute in the examples project.

Its Python requirements are taken from the local requiremnts.txt file, and its epochs command line argument value is set. The task is created in a draft status (i.e. can be modified in the WebApp UI and later be enqueued).