ClearML Task CLI

Using only the command line and zero additional lines of code, easily track your work and integrate ClearML with your existing code.

clearml-task automatically imports any script or Python repository into ClearML. clearml-task lets you enqueue your code for execution by an available ClearML Agent. You can even provide command line arguments, Python module dependencies, and a requirements.txt file!

What Is ClearML Task For?

• Launching off-the-shelf code on a remote machine with dedicated resources (e.g. GPU)
• Running hyperparameter optimization on a codebase that is still not in ClearML
• Creating a pipeline from an assortment of scripts, that you need to turn into ClearML tasks
• Running some code on a remote machine, either using an on-prem cluster or on the cloud

How Does ClearML Task Work?

1. Execute clearml-task, specifying the ClearML target project and task name, along with your script (and repository / commit / branch). Optionally, specify an execution queue and Docker image to use.
2. clearml-task does its magic! It creates a new ClearML Task, and, if so directed, enqueues it for execution by a ClearML Agent.
3. While the Task is running on the remote machine, all its console outputs are logged in real-time, alongside your TensorBoard and matplotlib. You can track your script's progress and results in the ClearML Web UI (a link to your task details page in the ClearML Web UI is printed as ClearML Task creates the task).

Execution Configuration

Docker

Specify a Docker container to run the code in with the --docker <docker_image> option. The ClearML Agent pulls it from Docker Hub or a Docker artifactory automatically.

Package Dependencies

clearml-task automatically finds the requirements.txt file in remote repositories.

If a local script requires certain packages, or the remote repository doesn't have a requirements.txt file, manually specify the required Python packages using --packages "<package_name>", for example --packages "keras" "tensorflow>2.2".

Queue

Tasks are passed to ClearML Agents via Queues. Specify a queue in which to enqueue the task. If you don't input a queue, the task is created in draft status, and you can enqueue it at a later point.

Branch and Working Directory

To specify your code's branch and commit ID, pass the --branch <branch_name> --commit <commit_id> options. If unspecified, clearml-task will use the latest commit from the 'master' branch.

Github Default Branch

For GitHub repositories, it is recommended to explicitly specify your default branch (e.g. --branch main) to avoid errors in identifying the correct default branch.

Command Line Options

Name	Description	Optional
--version	Display the clearml-task utility version
--project	Set the project name for the task (required, unless using --base-task-id). If the named project does not exist, it is created on-the-fly
--name	Set a target name for the new task
--repo	URL of remote repository. Example: --repo https://github.com/allegroai/clearml.git
--branch	Select repository branch / tag. By default, latest commit from the master branch
--commit	Select commit ID to use. By default, latest commit, or local commit ID when using local repository
--folder	Execute the code from a local folder. Notice, it assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and replicated on the remote machine
--script	Entry point script for the remote execution. When used with --repo, input the script's relative path inside the repository. For example: --script source/train.py. When used with --folder, it supports a direct path to a file inside the local repository itself, for example: --script ~/project/source/train.py
--cwd	Working directory to launch the script from. Relative to repo root or local --folder
--args	Arguments to pass to the remote task, list of <argument>=<value> strings. Currently only argparse arguments are supported
--queue	Select a task's execution queue. If not provided, a task is created but not launched
--requirements	Specify requirements.txt file to install when setting the session. By default, the requirements.txt from the repository will be used
--packages	Manually specify a list of required packages. Example: --packages "tqdm>=2.1" "scikit-learn"
--docker	Select the Docker image to use in the remote task
--docker_args	Add Docker arguments. Pass a single string in the following format: --docker-args "<argument_string>". For example: --docker-args "-v some_dir_1:other_dir_1 -v some_dir_2:other_dir_2"
--docker_bash_setup_script	Add a bash script to be executed inside the Docker before setting up the task's environment
--output-uri	Set the task output_uri, upload destination for task models and artifacts
--task-type	Set the task type. Optional values: training, testing, inference, data_processing, application, monitor, controller, optimizer, service, qc, custom
--skip-task-init	If set, Task.init() call is not added to the entry point, and is assumed to be called within the script
--base-task-id	Use a pre-existing task in the system, instead of a local repo / script. Essentially clones an existing task and overrides arguments / requirements
--import-offline-session	Specify the path to the offline session you want to import.

Usage

These commands demonstrate a few useful use cases for clearml-task.

Executing Code from a Remote Repository

clearml-task --project examples --name remote_test --repo https://github.com/allegroai/events.git --branch master --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default

The keras_mnist.py script from the events GitHub repository is imported as a ClearML task named remote_test in the examples project. Its command line arguments batch_size and epochs values are set, and the task is enqueued for execution on the default queue.

Executing a Local Script

Using clearml-task to execute a local script is very similar to using it with a remote repo.

clearml-task --project examples --name local_test --script keras_mnist.py --branch master --requirements requirements.txt --args epochs=1 --queue default

The keras_mnist.py script on the user's local machine is imported as a ClearML task named local_test in the examples project.

Its Python requirements are taken from the local requiremnts.txt file, and its epochs command line argument value is set.

The task is enqueued for execution on the default queue.

Pushing a Script to the Server

clearml-task --project examples --name no_execute --script keras_mnist.py --branch master --requirements requirements.txt --args epochs=1 

The keras_mnist.py script on the user's local machine is imported as a ClearML task named no_execute in the examples project.

Its Python requirements are taken from the local requiremnts.txt file, and its epochs command line argument value is set. The task is created in a draft status (i.e. can be modified in the WebApp UI and later be enqueued).