This tutorial demonstrates using
command to build a Docker container replicating the execution environment of an existing task. ClearML Agents can make
use of such containers to execute tasks without having to set up their environment every time.
A use case for this would be manual hyperparameter optimization, where a base task can be used to create a container to be used when running optimization tasks.
clearml-agentinstalled and configured
clearmlinstalled and configured
- clearml repo cloned (
git clone https://github.com/allegroai/clearml.git)
Creating the ClearML Experiment
Set up the experiment’s execution environment:
pip install -r requirements.txt
Run the experiment:
This creates a ClearML task called "Keras with TensorBoard example" in the "examples" project.
Note the task ID in the console output when running the script above:
ClearML Task: created new task id=<TASK_ID>
This ID will be used in the following section.
Building the Docker Container
Execute the following command to build the container. Input the ID of the task created above.
clearml-agent build --id <TASK_ID> --docker --target new_docker
If the container will not make use of a GPU, add the
This will create a container with the specified task’s execution environment in the
When the Docker build completes, the console output shows:
Docker build done
Committing docker container to: new_docker
Using the New Docker Container
Make use of the container you've just built by having a ClearML agent make use of it for executing a new experiment:
Clone the experiment.
In the cloned experiment, go to the EXECUTION tab > CONTAINER section. Under IMAGE, insert the name of the new Docker image,
new_docker. See Tuning Experiments for more task modification options.
Enqueue the cloned experiment to the
clearml-agentin Docker Mode and assign it to the
clearml-agent daemon --docker --queue defaulttip
If the agent will not make use of a GPU, add the
This agent will pull the enqueued task and run it using the
new_dockerimage to create the execution environment. In the task’s CONSOLE tab, one of the first logs should be:
Executing: ['docker', 'run', ..., 'CLEARML_DOCKER_IMAGE=new_docker', ...].