Skip to main content

TriggerScheduler

class automation.TriggerScheduler()#

Trigger Task execution if an event happens in the system Examples:

New model is published/tagged, New Dataset is created, General Task failed, Task metric below/above threshold, alert every X minutes

Create a Task trigger service

  • Parameters

    • pooling_frequency_minutes (float) – Check for new events every X minutes (default 2)

    • sync_frequency_minutes (float) – Sync task scheduler configuration every X minutes.

Allow to change scheduler in runtime by editing the Task configuration object :type force_create_task_name: Optional[str] :param force_create_task_name: Optional, force creation of Task Scheduler service, even if main Task.init already exists. :type force_create_task_project: Optional[str] :param force_create_task_project: Optional, force creation of Task Scheduler service, even if main Task.init already exists.


add_dataset_trigger#

add_dataset_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_publish=None, trigger_on_tags=None, trigger_on_archive=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job alike scheduling for a pre existing Task or function. Trigger the Task/function execution on changes in the dataset repository (notice this is not the hyper-datasets) Notice it is recommended to give the trigger a descriptive unique name, if not provided a task ID is used.

Notice task_overrides can except reference to the trigger model ID: example: task_overrides={‘Args/dataset_id’: ‘${dataset.id}’} Notice if schedule_function is passed, use the following function interface: ```py def schedule_function(dataset_id):

pass

```

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution

    • schedule_queue (Optional[str]) – Queue name or ID to put the Task into (i.e. schedule)

    • schedule_function (Optional[Callable[[str], None]]) – Optional, instead of providing Task ID to be scheduled, provide a function to be called. Notice the function is called from the scheduler context (i.e. running on the same machine as the scheduler)

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided otherwise randomly generated)

    • trigger_project (Optional[str]) – Only monitor datasets from this specific project (not recursive)

    • trigger_name (Optional[str]) – Trigger only on datasets with name matching (regexp)

    • trigger_on_publish (Optional[bool]) – Trigger when dataset is published.

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present

    • trigger_on_archive (Optional[bool]) – Trigger when dataset is archived

    • trigger_required_tags (Optional[List[str]]) – Trigger only on datasets with the following additional tags (must include all tags)

    • target_project (Optional[str]) – Specify target project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Add tag to the executed Task. Provide specific tag (str) or pass True (default) to use the trigger name as tag

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running

    • reuse_task (bool ) –

    • task_parameters (Optional [ dict ] ) –

    • task_overrides (Optional [ dict ] ) –

  • Return type

    None

(skip until the next scheduled time period). Default False. :type reuse_task: bool :param reuse_task: If True, re-enqueue the same Task (i.e. do not clone it) every time, default False. :type task_parameters: Optional[dict] :param task_parameters: Configuration parameters to the executed Task. for example: {‘Args/batch’: ‘12’} Notice: not available when reuse_task=True/ :type task_overrides: Optional[dict] :param task_overrides: Change task definition. for example {‘script.version_num’: None, ‘script.branch’: ‘main’} Notice: not available when reuse_task=True :rtype: None :return: True if job is successfully added to the scheduling list


add_model_trigger#

add_model_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_publish=None, trigger_on_tags=None, trigger_on_archive=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job alike scheduling for a pre existing Task or function. Trigger the Task/function execution on changes in the model repository Notice it is recommended to give the trigger a descriptive unique name, if not provided a task ID is used.

Notice task_overrides can except reference to the trigger model ID: example: task_overrides={‘Args/model_id’: ‘${model.id}’} Notice if schedule_function is passed, use the following function interface: ```py def schedule_function(model_id):

pass

```

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution

    • schedule_queue (Optional[str]) – Queue name or ID to put the Task into (i.e. schedule)

    • schedule_function (Optional[Callable[[str], None]]) – Optional, instead of providing Task ID to be scheduled, provide a function to be called. Notice the function is called from the scheduler context (i.e. running on the same machine as the scheduler)

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided otherwise randomly generated)

    • trigger_project (Optional[str]) – Only monitor models from this specific project (not recursive)

    • trigger_name (Optional[str]) – Trigger only on models with name matching (regexp)

    • trigger_on_publish (Optional[bool]) – Trigger when model is published.

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present

    • trigger_on_archive (Optional[bool]) – Trigger when model is archived

    • trigger_required_tags (Optional[List[str]]) – Trigger only on models with the following additional tags (must include all tags)

    • target_project (Optional[str]) – Specify target project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Add tag to the executed Task. Provide specific tag (str) or pass True (default) to use the trigger name as tag

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running

    • reuse_task (bool ) –

    • task_parameters (Optional [ dict ] ) –

    • task_overrides (Optional [ dict ] ) –

  • Return type

    None

(skip until the next scheduled time period). Default False. :type reuse_task: bool :param reuse_task: If True, re-enqueue the same Task (i.e. do not clone it) every time, default False. :type task_parameters: Optional[dict] :param task_parameters: Configuration parameters to the executed Task. for example: {‘Args/batch’: ‘12’} Notice: not available when reuse_task=True :type task_overrides: Optional[dict] :param task_overrides: Change task definition. for example {‘script.version_num’: None, ‘script.branch’: ‘main’} Notice: not available when reuse_task=True :rtype: None :return: True if job is successfully added to the scheduling list


add_task_trigger#

add_task_trigger(schedule_task_id=None, schedule_queue=None, schedule_function=None, trigger_project=None, trigger_name=None, trigger_on_tags=None, trigger_on_status=None, trigger_exclude_dev_tasks=None, trigger_on_metric=None, trigger_on_variant=None, trigger_on_threshold=None, trigger_on_sign=None, trigger_required_tags=None, name=None, target_project=None, add_tag=True, single_instance=False, reuse_task=False, task_parameters=None, task_overrides=None)

Create a cron job alike scheduling for a pre existing Task or function. Trigger the Task/function execution on changes in the Task Notice it is recommended to give the trigger a descriptive unique name, if not provided a task ID is used.

Notice task_overrides can except reference to the trigger model ID: example: task_overrides={‘Args/task_id’: ‘${task.id}’} Notice if schedule_function is passed, use the following function interface: ```py def schedule_function(task_id):

pass

```

  • Parameters

    • schedule_task_id (Union[str, Task, None]) – Task/task ID to be cloned and scheduled for execution

    • schedule_queue (Optional[str]) – Queue name or ID to put the Task into (i.e. schedule)

    • schedule_function (Optional[Callable[[str], None]]) – Optional, instead of providing Task ID to be scheduled, provide a function to be called. Notice the function is called from the scheduler context (i.e. running on the same machine as the scheduler)

    • name (Optional[str]) – Name or description for the cron Task (should be unique if provided otherwise randomly generated)

    • trigger_project (Optional[str]) – Only monitor tasks from this specific project (not recursive)

    • trigger_name (Optional[str]) – Trigger only on tasks with name matching (regexp)

    • trigger_on_tags (Optional[List[str]]) – Trigger when all tags in the list are present

    • trigger_required_tags (Optional[List[str]]) – Trigger only on tasks with the following additional tags (must include all tags)

    • trigger_on_status (Optional[List[str]]) – Trigger on Task status change. expect list of status strings, e.g. [‘failed’, ‘published’]

    • trigger_exclude_dev_tasks (Optional[bool]) – If True only trigger on Tasks executed by clearml-agent (and not manually)

    • trigger_on_metric (Optional[str]) – Trigger on metric/variant above/under threshold (metric=title, variant=series)

    • trigger_on_variant (Optional[str]) – Trigger on metric/variant above/under threshold (metric=title, variant=series)

    • trigger_on_threshold (Optional[float]) – Trigger on metric/variant above/under threshold (float number)

    • trigger_on_sign (Optional[str]) – possible values “max”/”maximum” or “min”/”minimum”, trigger Task if metric below “min” or “above” maximum. Default: “minimum”

    • target_project (Optional[str]) – Specify target project to put the cloned scheduled Task in.

    • add_tag (Union[bool, str]) – Add tag to the executed Task. Provide specific tag (str) or pass True (default) to use the trigger name as tag

    • single_instance (bool) – If True, do not launch the Task job if the previous instance is still running

    • reuse_task (bool ) –

    • task_parameters (Optional [ dict ] ) –

    • task_overrides (Optional [ dict ] ) –

  • Return type

    None

(skip until the next scheduled time period). Default False. :type reuse_task: bool :param reuse_task: If True, re-enqueue the same Task (i.e. do not clone it) every time, default False. :type task_parameters: Optional[dict] :param task_parameters: Configuration parameters to the executed Task. for example: {‘Args/batch’: ‘12’} Notice: not available when reuse_task=True/ :type task_overrides: Optional[dict] :param task_overrides: Change task definition. for example {‘script.version_num’: None, ‘script.branch’: ‘main’} Notice: not available when reuse_task=True :rtype: None :return: True if job is successfully added to the scheduling list


get_triggers#

get_triggers()

Return all triggers (models, datasets, tasks) :rtype: List[BaseTrigger] :return: List of trigger objects

  • Return type

    List[clearml.automation.trigger.BaseTrigger]


start#

start()

Start the Task trigger loop (notice this function does not return)


start_remotely#

start_remotely(queue='services')

Start the Task TaskScheduler loop (notice this function does not return)

  • Parameters

    queue (str) – Remote queue to run the scheduler on, default ‘services’ queue.

  • Return type

    None