Skip to main content

Monitoring Service Posting Slack Alerts

The Slack alerts example demonstrates how to use the clearml.automation.monitor class to implement a service that monitors the completion and failure of tasks, and posts alert messages on a Slack channel.

Slack alert example

Creating a Slack Bot​

Before configuring and running the Slack alert service, create a Slack Bot (ClearML Bot).

info

The Slack API token and channel you create are required to configure the Slack alert service.

  1. Login to your Slack account.
  2. Go to https://api.slack.com/apps/new.
  3. In App Name, enter an app name; for example, "ClearML Bot".
  4. In Development Slack Workspace, select a workspace.
  5. Click Create App.
  6. In Basic Information, under Display Information, complete the following:
    • In Short description, enter "Allegro Train Bot".
    • In Background color, enter "#202432".
  7. Click Save Changes.
  8. In OAuth & Permissions, under Scopes, click Add an OAuth Scope, and then select the following permissions on the list:
    • channels:join
    • channels:read
    • chat:write
  9. In OAuth Tokens & Redirect URLs:
    1. Click Install App to Workspace
    2. In the confirmation dialog, click Allow.
    3. Click Copy to copy the Bot User OAuth Access Token.

Running the Script​

Self deployed ClearML server

A template Slack Alerts task is available in the DevOps Services project. You can clone it, adapt its configuration to your needs and enqueue for execution directly from the ClearML UI.

Run the monitoring service in one of these ways:

To run the monitoring service:

python slack_alerts.py --channel <Slack-channel-name> --slack-api <Slack-API-token> --local True [...]
  • channel - The Slack channel where alerts will be posted.
  • slack_api - Slack API key.
  • local - If True, run monitoring services locally. If False, enqueue the task in the queue passed as the service_queue (by default services queue) for remote execution.

The script supports the following additional command line options:

  • service_queue - The queue to use when running remotely as a service. The default value is services (make sure your workspace has such a queue and to assign a ClearML Agent to this queue).
  • message_prefix - A message prefix for Slack alerts. For example, to alert all channel members use: "Hey <!here>".
  • min_num_iterations - Minimal iteration threshold below which experiments are ignored. Use this option to eliminate debug sessions that fail quickly. The default value is 0.
  • project - The name of the project to monitor. By default, all projects are monitored.
  • include_manual_experiments - Whether to include experiments that are running locally:
    • True - Monitor all experiments (both local and remote, executed by ClearML Agent).
    • False (default) - Monitor only remote experiments.
  • include_completed_experiments - If False (default), send alerts only for failed tasks. If True, send an alert for completed and failed tasks.
  • include_archived - If False (default), only tasks that are not archived will be reported. This option can be useful if a task is archived between polling.
  • refresh_rate - How often to monitor the experiments in seconds. The default value is 10.0.
  • include_users - Only report tasks that were initiated by these users (usernames and user IDs are accepted). Mutually exclusive to exclude_users.
  • exclude_users - Only report tasks that were NOT initiated by these users (usernames and user IDs are accepted). Mutually exclusive to include_users.
  • verbose - If True, will increase verbosity of messages (such as when tasks are polled but filtered away).

Configuration​

ClearML automatically logs command line options defined with argparse. They appear in the experiment's CONFIGURATION page under HYPERPARAMETERS > Args.

Monitoring configuration

The task can be reused to launch another monitor instance: clone the task, edit its parameters, and enqueue the task for execution (you'll typically want to use a ClearML Agent running in services mode for such service tasks).

Console​

All console output appears in the experiment's CONSOLE page.

Additional Information about slack_alerts.py​

In slack_alerts.py, the class SlackMonitor inherits from the Monitor class in clearml.automation.monitor. SlackMonitor overrides the following Monitor class methods:

  • get_query_parameters - Get the query parameters for Task monitoring.
  • process_task - Get the information for a Task, post a Slack message, and output to console.

You can run the example remotely by calling Task.execute_remotely().

To interface to Slack, the example uses slack_sdk.WebClient and slack_sdk.errors.SlackApiError.