Q&A: How to Integrate Google Colab with ClearML

One of the best advances in recent years was Google Colab. I know that many see this is a community style jupyter notebook, however I don’t see that as a negative. I love jupyter notebooks as well. The trick it did was allowing people to easily share notebooks and collaborate. The side affect was that it made bootstrapping a new worker node for ClearML trivial.

I know that’s a bold claim. Nothing these days seems trivial, least of all infrastructure. However I have made a bold claim, so let me defend this, and quickly go through the steps.

The first step is signing into the ClearML Community dashboard. This obviously assumes you have signed up, but if you haven’t, then that would obviously be step 1. Once you are at the dashboard,
then goto your profile, and create new credentials for your workspace. You will need the api keys for the second cell to connect.

Now you have your credentials, let’s fire up a Google CoLab. The code that you need is detailed below, with your api keys in cell 2.

First Cell

!pip install clearml numpy clearml-agent 'pyjwt<1.8.0'

Second Cell

%%bash

export api=`cat <<EOF
api {
    # Stef Telford's workspace
    web_server: https://app.community.clear.ml
    api_server: https://api.community.clear.ml
    files_server: https://files.community.clear.ml
    credentials {
        "access_key" = "3ZJX3UQ9YS87PD4A1G"
        "secret_key" = "=OfK#N@Y&mFtC)VHWVqTJIb52QUZPfuJ&bSYD4!uUk(t7ENv"
    }
}
EOF
`
echo "$api" > /root/clearml.conf

and lastly, the Third Cell

!clearml-agent daemon --queue default

After you start the execution of cell 3, you should notice that it does not ever finish. This is to be expected, as the worker should now appear in your dashboard. The worker id is rather cryptic sadly, but we can’t have everything.

Now that this is in-place, you can happily clone experiments and enqueue them to your hearts content. No need to run clearml-agent on your own hardware or spin up AWS nodes.

Nice, neat, simple.