Google Colab is a common development environment for data scientists. It offers a convenient IDE as well as compute provided by google.
Users can transform a Google Colab instance into an available resource in ClearML using Clearml Agent.
In this tutorial, we will go over how to create a ClearML worker node in a Google Colab notebook. Once the worker is up and running, users can send Tasks to be executed on the Google Colab's HW.
- Be signed up for ClearML (Or have a server deployed).
- Have a Google account to access Google Colab
Open up this Google Colab notebook.
Run the first cell, which installs all the necessary packages:!pip install git+https://github.com/allegroai/clearml!pip install clearml-agent
Run the second cell, which exports this environment variable:! export MPLBACKEND=TkAg
This environment variable makes Matplotlib work in headless mode, so it won't output graphs to the screen.
Create new credentials.
Go to your profile in the ClearML WebApp. Under the WORKSPACES section, go to App Credentials, click + Create new credentials, and copy the information that pops up.
Set the credentials.
In the third cell, enter your own credentials:from clearml import TaskTask.set_credentials(api_host="https://api.community.clear.ml",web_host="https://app.community.clear.ml",files_host="https://files.community.clear.ml",key='6ZHX9UQMYL874A1NE8',secret='=2h6#%@Y&m*tC!VLEXq&JI7QhZPKuJfbaYD4!uUk(t7=9ENv')
In the fourth cell, launch a
clearml-agentthat will listen to the
defaultqueue:!clearml-agent daemon --queue default
For additional options for running
clearml-agent, see the clearml-agent reference.
After cell 4 is executed, the worker should now appear in the Workers & Queues page of your server. Clone experiments and enqueue them to your hearts content! The
clearml-agentwill fetch experiments and execute them using the Google Colab hardware.