clearml-agent, an agent recreates an entire execution environment, be it by pulling the docker container or
installing specified packages, and then executes the code on a remote machine. The Agent takes into account required Python packages,
but sometimes, when using a Docker container, a user may need to use additional, non-Python tools.
In this tutorial, we will learn how to use
extra_docker_shell_script, with which we will reconfigure an Agent to execute
a shell script when a docker is started, but before an experiment is run.
clearml-agentdownloaded and configured - work on a machine which has access to the configuration file of the Agent you want to configure
- Any code with a ClearML Task.
- Open your ClearML configuration file for editing. Depending upon your operating system, it is:
- Linux - ~/clearml.conf
- Mac - $HOME/clearml.conf
- Windows - \User\<username>\clearml.conf
When you open up the file, the first line should say:
# CLEARML-AGENT configuration file
In the file, search for and go to, "extra_docker_shell_script:", which is where we will be putting our extra script. If it is commented out, make sure to uncomment the line. We will use the example script that is already there ["apt-get install -y bindfs", ].
Search for and go to "docker_force_pull" in the document, and make sure that it is set to "true", so that your docker image will be updated.
clearml-agentin docker mode:
clearml-agent daemon --docker --queue default. The agent will use the default Cuda/Nvidia Docker Image.
Enqueue any Clearml Task to the default queue, which the Agent is now listening to. The Agent pulls the Task, and then reproduces it, and now it will execute the
extra_docker_shell_scriptthat was put in the configuration file. Then the code will be executed in the updated docker container. If we look at the console output in the web UI, the third entry should start with
Executing: ['docker', 'run', '-t', '--gpus...', and towards the end of the entry, where the downloaded packages are mentioned, we can see the additional shell-script
apt-get install -y bindfs.