Skip to main content

Extra Docker Shell Script

When using clearml-agent, an agent recreates an entire execution environment, be it by pulling the docker container or installing specified packages, and then executes the code on a remote machine. The Agent takes into account required Python packages, but sometimes, when using a Docker container, a user may need to use additional, non-Python tools.

Tutorial#

In this tutorial, we will learn how to use extra_docker_shell_script, with which we will reconfigure an Agent to execute a shell script when a docker is started, but before an experiment is run.

Prerequisites#

  • clearml-agent downloaded and configured - work on a machine which has access to the configuration file of the Agent you want to configure
  • Any code with a ClearML Task.

Steps#

  1. Open your ClearML configuration file for editing. Depending upon your operating system, it is:
  • Linux - ~/clearml.conf
  • Mac - $HOME/clearml.conf
  • Windows - \User\<username>\clearml.conf

When you open up the file, the first line should say: # CLEARML-AGENT configuration file

  1. In the file, search for and go to, "extra_docker_shell_script:", which is where we will be putting our extra script. If it is commented out, make sure to uncomment the line. We will use the example script that is already there ["apt-get install -y bindfs", ].

  2. Search for and go to "docker_force_pull" in the document, and make sure that it is set to "true", so that your docker image will be updated.

  3. Run the clearml-agent in docker mode: clearml-agent daemon --docker --queue default. The agent will use the default Cuda/Nvidia Docker Image.

  4. Enqueue any Clearml Task to the default queue, which the Agent is now listening to. The Agent pulls the Task, and then reproduces it, and now it will execute the extra_docker_shell_script that was put in the configuration file. Then the code will be executed in the updated docker container. If we look at the console output in the web UI, the third entry should start with Executing: ['docker', 'run', '-t', '--gpus...', and towards the end of the entry, where the downloaded packages are mentioned, we can see the additional shell-script apt-get install -y bindfs.