Machine Learning and Deep Learning development is sometimes more challenging than traditional software development. If you are working on an average laptop or computer, and you have a sizeable dataset that requires significant computation, your local machine may not be able to provide you with the resources for an effective workflow.
If you can run and debug your code on your own machine, congrats you are lucky! Continue doing that, then clone your code in the UI and send it for long-term training on a remote machine.
If you are not that lucky, this section is for you :)
clearml-session is a feature that allows to launch a session of Jupyterlab and VS Code, and to execute code on a remote
machine that better meets resource needs. With this feature, local links are provided, which can be used to access
JupyterLab and VS Code on a remote machine over a secure and encrypted SSH connection.
VS Code Window
ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the ClearML Agent.
A ClearML Agent will run on a target machine, and ClearML Session will instruct it to execute the Jupyter \ VS Code
server to develop remotely.
After entering a
clearml-session command with all specifications:
clearml-sessioncreates a new Task that is responsible for setting up the SSH and JupyterLab / VS Code environment according to your specifications on the host machine.
The Task is enqueued, and a ClearML Agent pulls and executes it. The agent downloads the appropriate server and launches it.
Once the agent finishes the initial setup of the interactive Task, the local
cleaml-sessionconnects to the host machine via SSH, and tunnels both SSH and JupyterLab over the SSH connection. If a specific Docker was specified, the JupyterLab environment will run inside the Docker.
The CLI outputs access links to the remote JupyterLab and VS Code sessions:Interactive session is running:SSH: ssh root@localhost -p 8022 [password: c5d19b3c0fa9784ba4f6aeb568c1e036a4fc2a4bc7f9bfc54a2c198d64ceb9c8]Jupyter Lab URL: http://localhost:8878/?token=ff7e5e8b9e5493a01b1a72530d18181320630b95f442b419VSCode server available at http://localhost:8898/
Notice the links are to 'localhost' since all communication to the remote server itself is done over secure SSH connection.
Now start working on the code as if you're running on the target machine itself!
To run a session inside a Docker container, use the
--docker flag and enter the docker image to use in the interactive
clearml-session can install required Python packages when setting up the remote environment. A
can be attached to the command using
Alternatively, packages can be manually specified, using
--packages "keras" "clearml"), and they'll be automatically installed.
To access a git repository remotely, add a
--git-credentials flag and set it to
true, so the local .git-credentials
file will be sent to the interactive session. This is helpful if working on private git repositories, and it allows for seamless
cloning and tracking of git references, including untracked changes.
clearml-session was launched locally and is still running on a remote machine, users can easily reconnect to it.
To reconnect to a previous session, execute
clearml-session with no additional flags, and the option of reconnecting
to an existing session will show up:
If multiple sessions were launched from a local machine and are still active, choose the desired session:
To shut down a remote session, which will free the
clearml-agent and close the CLI, enter "Shutdown". If a session
is shutdown, there is no option to reconnect to it.
clearml-session is running remotely, it's possible to continue working on the session from any machine.
clearml-session is launched, it initializes a task with a unique ID in the ClearML Server.
To connect to an existing session:
- Go to the web UI, find the interactive session task (by default, it's in project "DevOps").
- Click on the ID button to the right of the task name, and copy the unique ID.
- Enter the following command:
clearml-session --attach <session_id>.
- Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session
Previously executed experiments in the ClearML system can be debugged on a remote interactive session.
clearml-session the ID of a Task to debug, then
clearml-session clones the experiment's git repository and
replicates the environment on a remote machine. Then the code can be interactively executed and debugged on JupyterLab / VS Code.
The Task must be connected to a git repository, since currently single script debugging is not supported.
- In the ClearML web UI, find the experiment (Task) that needs debugging.
- Click on the ID button next to the Task name, and copy the unique ID.
- Enter the following command:
clearml-session --debugging-session <experiment_id_here>
- Click on the JupyterLab / VS Code link, or connect directly to the SSH session.
- In JupyterLab / VS Code, access the experiment's repository in the
|Command line options||Description||Default value|
|Download a Jupyter-Lab environment|
|Download a VSCode environment|
|Register the public IP of the remote machine (if you are running the session on a public cloud)||Session runs on the machine whose agent is executing the session|
|Specify a BASH init script file to be executed when the interactive session is being set up|
|Specify the path for the session's remote base folder for the session||Home folder(|
|Specify a path to another configuration file for |
|Specify a gateway IP to pass to the interactive session, if an external address needs to be accessed|
|Pass the ID of a task that will become the base task, so the session will use its configurations|
|Disable transparent proxy that keep sockets alive to maintain the connection to the remote resource|
|The queue option list will exclude queues with specified tags. See the |
|The queue option list will include only queues with specified tags. See the |
|Pass the |
|Set your own SSH username for the interactive session|
|Set your own SSH password for the interactive session||A randomly generated password or a previously used one|