Skip to main content

Experiment Environment Containers

This tutorial demonstrates using clearml-agent's build command to build a Docker container replicating the execution environment of an existing task. ClearML Agents can make use of such containers to execute tasks without having to set up their environment every time.

A use case for this would be manual hyperparameter optimization, where a base task can be used to create a container to be used when running optimization tasks.

Prerequisites​

  • clearml-agent installed and configured
  • clearml installed and configured
  • clearml repo cloned (git clone https://github.com/allegroai/clearml.git)

Creating the ClearML Experiment​

  1. Set up the experiment's execution environment:

    cd clearml/examples/frameworks/keras
    pip install -r requirements.txt

  2. Run the experiment:

    python keras_tensorboard.py

    This creates a ClearML task called "Keras with TensorBoard example" in the "examples" project.

    Note the task ID in the console output when running the script above:

    ClearML Task: created new task id=<TASK_ID>

    This ID will be used in the following section.

Building the Docker Container​

Execute the following command to build the container. Input the ID of the task created above. 

clearml-agent build --id <TASK_ID> --docker --target new_docker
tip

If the container will not make use of a GPU, add the --cpu-only flag

This will create a container with the specified task's execution environment in the --target folder. When the Docker build completes, the console output shows:

Docker build done
Committing docker container to: new_docker
sha256:460453b93ct1989fd1c6637c236e544031c4d378581433fc0b961103ce206af1

Using the New Docker Container​

Make use of the container you've just built by having a ClearML agent make use of it for executing a new experiment:

  1. In the ClearML Web UI, go to the "examples" project, "Keras with TensorBoard example" task (the one executed above).

  2. Clone the experiment.

  3. In the cloned experiment, go to the EXECUTION tab > CONTAINER section. Under IMAGE, insert the name of the new Docker image, new_docker. See Tuning Experiments for more task modification options.

  4. Enqueue the cloned experiment to the default queue.

  5. Launch a clearml-agent in Docker Mode and assign it to the default queue:

    clearml-agent daemon --docker --queue default
    tip

    If the agent will not make use of a GPU, add the --cpu-only flag

    This agent will pull the enqueued task and run it using the new_docker image to create the execution environment. In the task's CONSOLE tab, one of the first logs displays the following:

    Executing: ['docker', 'run', ..., 'CLEARML_DOCKER_IMAGE=new_docker', ...].