Skip to main content

TriggerScheduler

class automation.TriggerScheduler(pooling_frequency_minutes=3.0, sync_frequency_minutes=15, force_create_task_name=None, force_create_task_project=None)​

A TriggerScheduler launches Tasks (or calls functions) when an event event in your ClearML system.

Examples:

  • A new model is published or tagged

  • A new Dataset is created

  • A Task fails

  • A Task metric goes above/below a threshold

Create a Task trigger service.

  • Parameters

    • pooling_frequency_minutes (float) – Check for new events every X minutes (default 3).

    • sync_frequency_minutes (float) – Sync the scheduler configuration every X minutes. Allows changing the schedule at runtime by editing the Task’s configuration object.

    • force_create_task_name (Optional[str]) – Name to force-create the Task Scheduler service under, even if a main Task.init() already exists.

    • force_create_task_project (Optional[str]) – Project to force-create the Task Scheduler service under, even if a main Task.init() already exists.

add_dataset_trigger​

add_dataset_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_publish=None, trigger_on_tags=None, trigger_on_archive=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job-like scheduling for a pre-existing Task or function. Trigger the Task/function execution on changes in the dataset repository (this does not include hyper-datasets). It is recommended to give the trigger a descriptive, unique name; if not provided, a task ID is used.

task_overrides can accept a reference to the trigger’s dataset ID, for example: task_overrides={'Args/dataset_id': '${dataset.id}'}. If schedule_function is passed, it must follow this interface:

def schedule_function(dataset_id):
    pass

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution.

    • schedule_queue (Optional[str]) – Name or ID of the queue to put the Task into (i.e. schedule it on).

    • schedule_function (Optional[Callable[[str], None]]) – A function to call instead of providing a Task ID to schedule. The function is called from the scheduler context (i.e. runs on the same machine as the scheduler).

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided, otherwise randomly generated).

    • trigger_project (Optional[str]) – Only monitor datasets from this specific project (not recursive).

    • trigger_name (Optional[str]) – Trigger only on datasets with name matching (regexp).

    • trigger_on_publish (Optional[bool]) – Trigger when dataset is published.

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present.

    • trigger_on_archive (Optional[bool]) – Trigger when dataset is archived.

    • trigger_required_tags (Optional[List[str]]) – Trigger only on datasets with the following additional tags (must include all tags).

    • target_project (Optional[str]) – The project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Tag to add to the executed Task. Provide a specific tag (str), or pass True (default) to use the trigger name as the tag.

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running (skip until the next scheduled time period) (default False).

    • reuse_task (bool) – If True, re-enqueue the same Task (i.e. do not clone it) every time (default False).

    • task_parameters (Optional[dict]) – Configuration parameters for the executed Task, for example: {'Args/batch': '12'}. Not available when reuse_task is True.

    • task_overrides (Optional[dict]) – Change the Task’s definition, for example: {'script.version_num': None, 'script.branch': 'main'}. Not available when reuse_task is True.

  • Return type

    None

add_model_trigger​

add_model_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_publish=None, trigger_on_tags=None, trigger_on_archive=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job-like scheduling for a pre-existing Task or function. Trigger the Task/function execution on changes in the model repository. It is recommended to give the trigger a descriptive, unique name; if not provided, a task ID is used.

task_overrides can accept a reference to the trigger’s model ID, for example: task_overrides={'Args/model_id': '${model.id}'}. If schedule_function is passed, it must follow this interface:

def schedule_function(model_id):
    pass

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution.

    • schedule_queue (Optional[str]) – Name or ID of the queue to put the Task into (i.e. schedule it on).

    • schedule_function (Optional[Callable[[str], None]]) – A function to call instead of providing a Task ID to schedule. The function is called from the scheduler context (i.e. runs on the same machine as the scheduler).

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided, otherwise randomly generated).

    • trigger_project (Optional[str]) – Only monitor models from this specific project (not recursive).

    • trigger_name (Optional[str]) – Trigger only on models with name matching (regexp).

    • trigger_on_publish (Optional[bool]) – Trigger when model is published.

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present.

    • trigger_on_archive (Optional[bool]) – Trigger when model is archived.

    • trigger_required_tags (Optional[List[str]]) – Trigger only on models with the following additional tags (must include all tags).

    • target_project (Optional[str]) – The project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Tag to add to the executed Task. Provide a specific tag (str), or pass True (default) to use the trigger name as the tag.

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running (skip until the next scheduled time period) (default False).

    • reuse_task (bool) – If True, re-enqueue the same Task (i.e. do not clone it) every time (default False).

    • task_parameters (Optional[dict]) – Configuration parameters for the executed Task, for example: {'Args/batch': '12'}. Not available when reuse_task is True.

    • task_overrides (Optional[dict]) – Change the Task’s definition, for example: {'script.version_num': None, 'script.branch': 'main'}. Not available when reuse_task is True.

  • Return type

    None

add_task_trigger​

add_task_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_tags=None, trigger_on_status=None, trigger_exclude_dev_tasks=None, trigger_on_metric=None, trigger_on_variant=None, trigger_on_threshold=None, trigger_on_sign=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job-like scheduling for a pre-existing Task or function. Trigger the Task/function execution on changes to a Task. It is recommended to give the trigger a descriptive, unique name; if not provided, a task ID is used.

task_overrides can accept a reference to the trigger’s task ID, for example: task_overrides={'Args/task_id': '${task.id}'}. If schedule_function is passed, it must follow this interface:

def schedule_function(task_id):
    pass

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution.

    • schedule_queue (Optional[str]) – Name or ID of the queue to put the Task into (i.e. schedule it on).

    • schedule_function (Optional[Callable[[str], None]]) – A function to call instead of providing a Task ID to schedule. The function is called from the scheduler context (i.e. runs on the same machine as the scheduler).

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided, otherwise randomly generated).

    • trigger_project (Optional[str]) – Only monitor tasks from this specific project (not recursive).

    • trigger_name (Optional[str]) – Trigger only on tasks with name matching (regexp).

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present.

    • trigger_required_tags (Optional[List[str]]) – Trigger only on tasks with the following additional tags (must include all tags).

    • trigger_on_status (Optional[List[str]]) – Trigger on Task status change. Expects a list of status strings, e.g. ['failed', 'published']. Valid values (Task.TaskStatusEnum): "created", "in_progress", "stopped", "closed", "failed", "completed", "queued", "published", "publishing", "unknown".

    • trigger_exclude_dev_tasks (Optional[bool]) – If True, only trigger on Tasks executed by a clearml-agent (not on manually-run Tasks).

    • trigger_on_metric (Optional[str]) – Metric title to monitor, used together with trigger_on_variant and trigger_on_threshold.

    • trigger_on_variant (Optional[str]) – Metric variant (series) to monitor, used together with trigger_on_metric and trigger_on_threshold.

    • trigger_on_threshold (Optional[float]) – Threshold value (float) to trigger on, used together with trigger_on_metric, trigger_on_variant, and trigger_on_sign.

    • trigger_on_sign (Optional[str]) – Whether to trigger when the metric goes above or below the threshold. Pass 'max'/'maximum' to trigger when the metric rises above the threshold, or 'min'/'minimum' to trigger when it falls below it (default 'minimum').

    • target_project (Optional[str]) – The project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Tag to add to the executed Task. Provide a specific tag (str), or pass True (default) to use the trigger name as the tag.

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running (skip until the next scheduled time period) (default False).

    • reuse_task (bool) – If True, re-enqueue the same Task (i.e. do not clone it) every time (default False).

    • task_parameters (Optional[dict]) – Configuration parameters for the executed Task, for example: {'Args/batch': '12'}. Not available when reuse_task is True.

    • task_overrides (Optional[dict]) – Change the Task’s definition, for example: {'script.version_num': None, 'script.branch': 'main'}. Not available when reuse_task is True.

  • Return type

    None

get_triggers​

get_triggers()

Return all triggers (models, datasets, and tasks).

  • Return type

    List[BaseTrigger]

  • Returns

    A list of BaseTrigger objects.

start​

start()

Start the TriggerScheduler loop.

Notice: this method does not return.

  • Return type

    None

start_remotely​

start_remotely(queue='services')

Start the Task TaskScheduler loop (notice this function does not return)

  • Parameters

    queue (str) – Remote queue to run the scheduler on, default ‘services’ queue.

  • Return type

    None