Skip to main content

Task

class Task()#

The Task class is a code template for a Task object which, together with its connected experiment components, represents the current running experiment. These connected components include hyperparameters, loggers, configuration, label enumeration, models, and other artifacts.

The term “main execution Task” refers to the Task context for current running experiment. Python experiment scripts can create one, and only one, main execution Task. It is a traceable, and after a script runs and ClearML stores the Task in the ClearML Server (backend), it is modifiable, reproducible, executable by a worker, and you can duplicate it for further experimentation.

The Task class and its methods allow you to create and manage experiments, as well as perform advanced experimentation functions, such as autoML.

warning

Do not construct Task objects directly. Use one of the methods listed below to create experiments or reference existing experiments.

For detailed information about creating Task objects, see the following methods:

  • Create a new reproducible Task - Task.init
important

In some cases, Task.init may return a Task object which is already stored in ClearML Server (already initialized), instead of creating a new Task. For a detailed explanation of those cases, see the Task.init method.

  • Manually create a new Task (no auto-logging will apply) - Task.create

  • Get the current running Task - Task.current_task

  • Get another (different) Task - Task.get_task

info

The ClearML documentation often refers to a Task as, “Task (experiment)”.

“Task” refers to the class in the ClearML Python Client Package, the object in your Python experiment script, and the entity with which ClearML Server and ClearML Agent work.

“Experiment” refers to your deep learning solution, including its connected components, inputs, and outputs, and is the experiment you can view, analyze, compare, modify, duplicate, and manage using the ClearML Web-App (UI).

Therefore, a “Task” is effectively an “experiment”, and “Task (experiment)” encompasses its usage throughout the ClearML.

The exception to this Task behavior is sub-tasks (non-reproducible Tasks), which do not use the main execution Task. Creating a sub-task always creates a new Task with a new Task ID.

warning

Do not construct Task manually! Please use Task.init or Task.get_task


Task.add_requirements#

classmethod add_requirements(package_name, package_version=None)

Force the adding of a package to the requirements list. If package_version is None, use the installed package version, if found. Example: Task.add_requirements(‘tensorflow’, ‘2.4.0’) Example: Task.add_requirements(‘tensorflow’, ‘>=2.4’) Example: Task.add_requirements(‘tensorflow’) -> use the installed tensorflow version Example: Task.add_requirements(‘tensorflow’, ‘’) -> no version limit

  • Parameters

    • package_name (str ) – The package name to add to the “Installed Packages” section of the task.

    • package_version (Optional[str]) – The package version requirements. If None, then use the installed version.

  • Return type

    None


add_tags#

add_tags(tags)

Add Tags to this task. Old tags are not deleted. When executing a Task (experiment) remotely, this method has no effect).

  • Parameters

    tags (Union[Sequence[str], str]) – A list of tags which describe the Task to add.

  • Return type

    None


artifacts#

property artifacts

A read-only dictionary of Task artifacts (name, artifact).

  • Return type

    Dict[str, Artifact]

  • Returns

    The artifacts.


cache_dir#

property cache_dir

The cache directory which is used to store the Task related files.

  • Return type

    Path


Task.clone#

classmethod clone(source_task=None, name=None, comment=None, parent=None, project=None)

Create a duplicate (a clone) of a Task (experiment). The status of the cloned Task is Draft and modifiable.

Use this method to manage experiments and for autoML.

  • Parameters

    • source_task (str ) – The Task to clone. Specify a Task object or a Task ID. (Optional)

    • name (str ) – The name of the new cloned Task. (Optional)

    • comment (str ) – A comment / description for the new cloned Task. (Optional)

    • parent (str ) – The Id of the parent Task of the new Task.

      * If `parent` is not specified, then `parent` is set to `source_task.parent`.
      * If `parent` is not specified and `source_task.parent` is not available, then

      parent set to to source_task.

    • project (str ) – The Id of the project in which to create the new Task. If None, the new task inherits the original Task’s project. (Optional)

  • Return type

    Task

  • Returns

    The new cloned Task (experiment).


close#

close()

Close the current Task. Enables you to manually shutdown the task.

warning

Only call Task.close if you are certain the Task is not needed.


completed#

completed(ignore_errors=True)

The signal indicating that this Task completed.

  • Return type

    ()

  • Parameters

    ignore_errors (bool ) –


connect#

connect(mutable, name=None)

Connect an object to a Task object. This connects an experiment component (part of an experiment) to the experiment. For example, connect hyperparameters or models.

  • Parameters

    • mutable (object ) – The experiment component to connect. The object can be any object Task supports integrating, including:

      * argparse - An argparse object for parameters.
      * dict - A dictionary for parameters.
      * TaskParameters - A TaskParameters object.
      * Model - A model object for initial model warmup, or for model update/snapshot uploading.
      * Class type - A Class type, storing all class properties (excluding ‘_’ prefix properties)
      * Object - A class instance, storing all instance properties (excluding ‘_’ prefix properties)
    • name (str ) – A section name associated with the connected object. Default: ‘General’ Currently only supported for dict / TaskParameter objects Examples: name=’General’ will put the connected dictionary under the General section in the hyper-parameters name=’Train’ will put the connected dictionary under the Train section in the hyper-parameters

  • Return type

    Any

  • Returns

    The result returned when connecting the object, if supported.

  • Raise

    Raise an exception on unsupported objects.


connect_configuration#

connect_configuration(configuration, name=None, description=None)

Connect a configuration dictionary or configuration file (pathlib.Path / str) to a Task object. This method should be called before reading the configuration file.

Later, when creating an output model, the model will include the contents of the configuration dictionary or file.

For example, a local file:

config_file = task.connect_configuration(config_file)
my_params = json.load(open(config_file,'rt'))

A parameter dictionary:

my_params = task.connect_configuration(my_params)
  • Parameters

    • configuration (Union[Mapping, Path, str]) – The configuration. This is usually the configuration used in the model training process. Specify one of the following:

      * A dictionary - A dictionary containing the configuration. ClearML stores the configuration in

      the ClearML Server (backend), in a HOCON format (JSON-like format) which is editable.

      * A `pathlib2.Path` string - A path to the configuration file. ClearML stores the content of the file.

      A local path must be relative path. When executing a Task remotely in a worker, the contents brought from the ClearML Server (backend) overwrites the contents of the file.

    • name (str ) – Configuration section name. default: ‘General’ Allowing users to store multiple configuration dicts/files

    • description (str ) – Configuration section description (text). default: None

  • Return type

    Union[dict, Path, str]

  • Returns

    If a dictionary is specified, then a dictionary is returned. If pathlib2.Path / string is specified, then a path to a local configuration file is returned. Configuration object.


connect_label_enumeration#

connect_label_enumeration(enumeration)

Connect a label enumeration dictionary to a Task (experiment) object.

Later, when creating an output model, the model will include the label enumeration dictionary.

  • Parameters

    enumeration (dict ) – A label enumeration dictionary of string (label) to integer (value) pairs.

    For example:

    {
    'background': 0,
    'person': 1
    }
  • Return type

    Dict[str, int]

  • Returns

    The label enumeration dictionary (JSON).


Task.create#

classmethod create(project_name=None, task_name=None, task_type=None, repo=None, branch=None, commit=None, script=None, working_directory=None, packages=None, requirements_file=None, docker=None, docker_args=None, docker_bash_setup_script=None, argparse_args=None, base_task_id=None, add_task_init_call=True)

Manually create and populate a new Task (experiment) in the system. If the code does not already contain a call to Task.init, pass add_task_init_call=True, and the code will be patched in remote execution (i.e. when executed by clearml-agent

info

This method always creates a new Task. Use Task.init method to automatically create and populate task for the running process. To reference an existing Task, call the Task.get_task method .

  • Parameters

    • project_name – Set the project name for the task. Required if base_task_id is None.

    • task_name – Set the name of the remote task. Required if base_task_id is None.

    • task_type – Optional, The task type to be created. Supported values: ‘training’, ‘testing’, ‘inference’, ‘data_processing’, ‘application’, ‘monitor’, ‘controller’, ‘optimizer’, ‘service’, ‘qc’, ‘custom’

    • repo – Remote URL for the repository to use, or path to local copy of the git repository Example: ‘https://github.com/allegroai/clearml.git’ or ‘~/project/repo’

    • branch – Select specific repository branch/tag (implies the latest commit from the branch)

    • commit – Select specific commit id to use (default: latest commit, or when used with local repository matching the local commit id)

    • script – Specify the entry point script for the remote execution. When used in tandem with remote git repository the script should be a relative path inside the repository, for example: ‘./source/train.py’ . When used with local repository path it supports a direct path to a file inside the local repository itself, for example: ‘~/project/source/train.py’

    • working_directory – Working directory to launch the script from. Default: repository root folder. Relative to repo root or local folder.

    • packages – Manually specify a list of required packages. Example: [“tqdm>=2.1”, “scikit-learn”] or True to automatically create requirements based on locally installed packages (repository must be local).

    • requirements_file – Specify requirements.txt file to install when setting the session. If not provided, the requirements.txt from the repository will be used.

    • docker – Select the docker image to be executed in by the remote session

    • docker_args – Add docker arguments, pass a single string

    • docker_bash_setup_script – Add bash script to be executed inside the docker before setting up the Task’s environement

    • argparse_args – Arguments to pass to the remote execution, list of string pairs (argument, value) Notice, only supported if the codebase itself uses argparse.ArgumentParser

    • base_task_id – Use a pre-existing task in the system, instead of a local repo/script. Essentially clones an existing task and overrides arguments/requirements.

    • add_task_init_call – If True, a ‘Task.init()’ call is added to the script entry point in remote execution.

  • Return type

    Task

  • Returns

    The newly created Task (experiment)


create_function_task#

create_function_task(func, func_name=None, task_name=None, kwargs)**

Create a new task, and call func with the specified kwargs. One can think of this call as remote forking, where the newly created instance is the new Task calling the specified func with the appropriate kwargs and leave once the func terminates. Notice that a remote executed function cannot create another child remote executed function.

info

Must be called from the main Task, i.e. the one created by Task.init(…)

The remote Tasks inherits the environment from the creating Task

In the remote Task, the entrypoint is the same as the creating Task

In the remote Task, the execution is the same until reaching this function call

  • Parameters

    • func (Callable) – A function to execute remotely as a single Task. On the remote executed Task the entry-point and the environment are copied from this calling process, only this function call redirect the the execution flow to the called func, alongside the passed arguments

    • func_name (Optional[str]) – A unique identifier of the function. Default the function name without the namespace. For example Class.foo() becomes ‘foo’

    • task_name (Optional[str]) – The newly create Task name. Default: the calling Task name + function name

    • kwargs (Optional[Any]) – name specific arguments for the target function. These arguments will appear under the configuration, “Function” section

  • Return Task

    Return the newly created Task or None if running remotely and execution is skipped

  • Return type

    Optional[Task]


Task.current_task#

classmethod current_task()

Get the current running Task (experiment). This is the main execution Task (task context) returned as a Task object.

  • Return type

    Task

  • Returns

    The current running Task (experiment).


Task.debug_simulate_remote_task#

classmethod debug_simulate_remote_task(task_id, reset_task=False)

Simulate remote execution of a specified Task. This call will simulate the behaviour of your Task as if executed by the ClearML-Agent This means configurations will be coming from the backend server into the code (the opposite from manual execution, where the backend logs the code arguments) Use with care.

  • Parameters

    • task_id (str ) – Task ID to simulate, notice that all configuration will be taken from the specified Task, regardless of the code initial values, just like it as if executed by ClearML agent

    • reset_task (bool ) – If True target Task, is automatically cleared / reset.

  • Return type

    ()


delete#

delete(delete_artifacts_and_models=True, skip_models_used_by_other_tasks=True, raise_on_error=False)

Delete the task as well as it’s output models and artifacts. Models and artifacts are deleted from their storage locations, each using its URI.

Note: in order to delete models and artifacts using their URI, make sure the proper storage credentials are configured in your configuration file (e.g. if an artifact is stored in S3, make sure sdk.aws.s3.credentials are properly configured and that you have delete permission in the related buckets).

  • Parameters

    • delete_artifacts_and_models (bool) – If True, artifacts and models would also be deleted (default True)

    • skip_models_used_by_other_tasks (bool) – If True, models used by other tasks would not be deleted (default True)

    • raise_on_error (bool) – If True an exception will be raised when encountering an error. If False an error would be printed and no exception will be raised.

  • Return type

    bool

  • Returns

    True if the task was deleted successfully.


delete_parameter#

delete_parameter(name)

Delete a parameter byt it’s full name Section/name.

  • Parameters

    name (str) – Parameter name in full, i.e. Section/name. For example, ‘Args/batch_size’

  • Return type

    bool

  • Returns

    True if the parameter was deleted successfully


delete_user_properties#

delete_user_properties(*iterables)

Delete hyper-parameters for this task. :type iterables: Iterable[Union[dict, Iterable[str, str]]] :param iterables: Hyper parameter key iterables. Each an iterable whose possible values each represent a hyper-parameter entry to delete, value formats are:

  • A dictionary containing a ‘section’ and ‘name’ fields
  • An iterable (e.g. tuple, list etc.) whose first two items denote ‘section’ and ‘name’
  • Return type

    bool

  • Parameters

    iterables (Iterable [ Union [ dict , Iterable [ str , str ] ] ] ) –


Task.dequeue#

classmethod dequeue(task)

Dequeue (remove) a Task from an execution queue.

  • Parameters

    task (Task/str ) – The Task to dequeue. Specify a Task object or Task ID.

  • Return type

    Any

  • Returns

    A dequeue JSON response.

{
"dequeued": 1,
"updated": 1,
"fields": {
"status": "created",
"status_reason": "",
"status_message": "",
"status_changed": "2020-02-24T16:43:43.057320+00:00",
"last_update": "2020-02-24T16:43:43.057320+00:00",
"execution.queue": null
}
}
  • dequeued - The number of Tasks enqueued (an integer or null).

  • fields

    • status - The status of the experiment.

    • status_reason - The reason for the last status change.

    • status_message - Information about the status.

    • status_changed - The last status change date and time in ISO 8601 format.

    • last_update - The last time the Task was created, updated,

      changed or events for this task were reported.

    • execution.queue - The Id of the queue where the Task is enqueued. null indicates not enqueued.

  • updated - The number of Tasks updated (an integer or null).


Task.enqueue#

classmethod enqueue(task, queue_name=None, queue_id=None)

Enqueue a Task for execution, by adding it to an execution queue.

info

A worker daemon must be listening at the queue for the worker to fetch the Task and execute it, see Use Case Examples on the “ClearML Agent Reference page.

  • Parameters

    • task (Task/str ) – The Task to enqueue. Specify a Task object or Task ID.

    • queue_name (str ) – The name of the queue. If not specified, then queue_id must be specified.

    • queue_id (str ) – The Id of the queue. If not specified, then queue_name must be specified.

  • Return type

    Any

  • Returns

    An enqueue JSON response.

    {
    "queued": 1,
    "updated": 1,
    "fields": {
    "status": "queued",
    "status_reason": "",
    "status_message": "",
    "status_changed": "2020-02-24T15:05:35.426770+00:00",
    "last_update": "2020-02-24T15:05:35.426770+00:00",
    "execution.queue": "2bd96ab2d9e54b578cc2fb195e52c7cf"
    }
    }
    • queued - The number of Tasks enqueued (an integer or null).

    • updated - The number of Tasks updated (an integer or null).

    • fields

      * `status` - The status of the experiment.
      * `status_reason` - The reason for the last status change.
      * `status_message` - Information about the status.
      * `status_changed` - The last status change date and time (ISO 8601 format).
      * `last_update` - The last Task update time, including Task creation, update, change, or events for

      this task (ISO 8601 format).

      * `execution.queue` - The Id of the queue where the Task is enqueued. `null` indicates not enqueued.

execute_remotely#

execute_remotely(queue_name=None, clone=False, exit_process=True)

If task is running locally (i.e., not by clearml-agent), then clone the Task and enqueue it for remote execution; or, stop the execution of the current Task, reset its state, and enqueue it. If exit==True, exit this process.

info

If the task is running remotely (i.e., clearml-agent is executing it), this call is a no-op (i.e., does nothing).

  • Parameters

    • queue_name (Optional[str]) – The queue name used for enqueueing the task. If None, this call exits the process without enqueuing the task.

    • clone (bool) – Clone the Task and execute the newly cloned Task

      The values are:

      * `True` - A cloned copy of the Task will be created, and enqueued, instead of this Task.
      * `False` - The Task will be enqueued.
    • exit_process (bool) – The function call will leave the calling process at the end

      * `True` - Exit the process (exit(0)).
      * `False` - Do not exit the process.
      warning

      If clone==False, then exit_process must be True.

  • Return Task

    return the task object of the newly generated remotely executing task

  • Return type

    Optional[Task]


export_task#

export_task()

Export Task’s configuration into a dictionary (for serialization purposes). A Task can be copied/modified by calling Task.import_task() Notice: Export task does not include the tasks outputs, such as results (scalar/plots etc.) or Task artifacts/models

  • Return type

    dict

  • Returns

    dictionary of the Task’s configuration.


flush#

flush(wait_for_uploads=False)

Flush any outstanding reports or console logs.

  • Parameters

    wait_for_uploads (bool ) – Wait for all outstanding uploads to complete

    • True - Wait

    • False - Do not wait (default)

  • Return type

    bool


Task.force_requirements_env_freeze#

classmethod force_requirements_env_freeze(force=True)

Force using pip freeze / conda list to store the full requirements of the active environment (instead of statically analyzing the running code and listing directly imported packages) Notice: Must be called before Task.init !

  • Parameters

    force (bool) – Set force using pip freeze flag on/off

  • Return type

    None


Task.get_all#

classmethod get_all(session=None, log=None, kwargs)**

List all the Tasks based on specific projection.

  • Parameters

    • session (Session ) – The session object used for sending requests to the API.

    • log (logging.Logger ) – The Log object.

    • kwargs (dict ) – Keyword args passed to the GetAllRequest (see backend_api.services.v2_5.tasks.GetAllRequest)

      For example:

      status='completed', 'search_text'='specific_word', 'user'='user_id', 'project'='project_id'
  • Return type

    Any

  • Returns

    The API response.


get_archived#

get_archived()

Return the Archive state of the Task

  • Return type

    bool

  • Returns

    If True the Task is archived, otherwise it is not.


get_base_docker#

get_base_docker()

Get the base Docker command (image) that is set for this experiment.

  • Return type

    str


get_configuration_object#

get_configuration_object(name)

Get the Task’s configuration object section as a blob of text Use only for automation (externally), otherwise use Task.connect_configuration.

  • Parameters

    name (str ) – Configuration section name

  • Return type

    Optional[str]

  • Returns

    The Task’s configuration as a text blob (unconstrained text string) return None if configuration name is not valid


get_configuration_objects#

get_configuration_objects()

Get the Task’s configuration object section as a blob of text Use only for automation (externally), otherwise use Task.connect_configuration.

  • Return type

    Optional[Mapping[str, str]]

  • Returns

    The Task’s configurations as a

dict (config name as key) and text blob as value (unconstrained text string)


get_initial_iteration#

get_initial_iteration()

Return the initial iteration offset, default is 0 Useful when continuing training from previous checkpoints

  • Return type

    int

  • Returns

    Initial iteration offset.


get_label_num_description#

get_label_num_description()

Get a dictionary of label number to a string pairs representing all labels associated with this number on the model labels.


get_labels_enumeration#

get_labels_enumeration()

Get the label enumeration dictionary label enumeration dictionary of string (label) to integer (value) pairs.

  • Return type

    Mapping[str, int]

  • Returns

    A dictionary containing the label enumeration.


get_last_iteration#

get_last_iteration()

Get the last reported iteration, which is the last iteration for which the Task reported a metric.

info

The maximum reported iteration is not in the local cache. This method sends a request to the ClearML Server (backend).

  • Return type

    int

  • Returns

    The last reported iteration number.


get_last_scalar_metrics#

get_last_scalar_metrics()

Get the last scalar metrics which the Task reported. This is a nested dictionary, ordered by title and series.

For example:

{
'title': {
'series': {
'last': 0.5,
'min': 0.1,
'max': 0.9
}
}
}
  • Return type

    Dict[str, Dict[str, Dict[str, float]]]

  • Returns

    The last scalar metrics.


get_logger#

get_logger()

Get a Logger object for reporting, for this task context. You can view all Logger report output associated with the Task for which this method is called, including metrics, plots, text, tables, and images, in the ClearML Web-App (UI).

  • Return type

    Logger

  • Returns

    The Logger for the Task (experiment).


get_model_config_dict#

get_model_config_dict()

Deprecated: Deprecated since version 0.14.1: Use Task.connect_configuration instead.

  • Return type

    Dict


get_model_config_text#

get_model_config_text()

Deprecated: Deprecated since version 0.14.1: Use Task.connect_configuration instead.

  • Return type

    str


get_model_design#

get_model_design()

Get the model configuration as blob of text.

  • Return type

    str

  • Returns

    The model configuration as blob of text.


get_models#

get_models()

Return a dictionary with {‘input’: [], ‘output’: []} loaded/stored models of the current Task Input models are files loaded in the task, either manually or automatically logged Output models are files stored in the task, either manually or automatically logged Automatically logged frameworks are for example: TensorFlow, Keras, PyTorch, ScikitLearn(joblib) etc.

  • Return type

    Mapping[str, Sequence[Model]]

  • Returns

    A dictionary-like object with “input”/”output” keys and input/output properties, pointing to a list-like object containing of Model objects. Each list-like object also acts as a dictionary, mapping model name to a appropriate model instance.

    Example:

    {'input': [clearml.Model()], 'output': [clearml.Model()]}

get_num_of_classes#

get_num_of_classes()

number of classes based on the task’s labels


get_offline_mode_folder#

get_offline_mode_folder()

Return the folder where all the task outputs and logs are stored in the offline session. :rtype: Optional[Path] :return: Path object, local folder, later to be used with report_offline_session()

  • Return type

    Optional[pathlib2.Path]


get_output_destination#

get_output_destination(extra_path=None, kwargs)**

Get the task’s output destination, with an optional suffix


get_output_log_web_page#

get_output_log_web_page()

Return the Task results & outputs web page address. For example: https://demoapp.demo.clear.ml/projects/216431/experiments/60763e04/output/log

  • Return type

    str

  • Returns

    http/s URL link.


get_parameter#

get_parameter(name, default=None)

Get a value for a parameter.

  • Parameters

    • name (str) – Parameter name

    • default (Optional[Any]) – Default value

  • Return type

    Any

  • Returns

    The Parameter value (or default value if parameter is not defined).


get_parameters#

get_parameters(backwards_compatibility=True)

Get the parameters for a Task. This method returns a complete group of key-value parameter pairs, but does not support parameter descriptions (the result is a dictionary of key-value pairs). Notice the returned parameter dict is flat: i.e. {‘Args/param’: ‘value’} is the argument “param” from section “Args”

  • Parameters

    backwards_compatibility (bool) – If True (default) parameters without section name (API version < 2.9, clearml-server < 0.16) will be at dict root level. If False, parameters without section name, will be nested under “Args/” key.

  • Return type

    Optional[dict]

  • Returns

    dict of the task parameters, all flattened to key/value. Different sections with key prefix “section/”


get_parameters_as_dict#

get_parameters_as_dict()

Get the Task parameters as a raw nested dictionary.

info

The values are not parsed. They are returned as is.

  • Return type

    Dict


Task.get_project_id#

classmethod get_project_id(project_name)

Return a project’s unique ID (str). If more than one project matched the project_name, return the last updated project If no project matched the requested name, returns None

  • Return type

    Optional[str]

  • Returns

    Project unique ID (str), or None if no project was found.

  • Parameters

    project_name (str ) –


Task.get_projects#

classmethod get_projects()

Return a list of projects in the system, sorted by last updated time

  • Return type

    List[ForwardRef]

  • Returns

    A list of all the projects in the system. Each entry is a services.projects.Project object.


get_registered_artifacts#

get_registered_artifacts()

Get a dictionary containing the Task’s registered (dynamically synchronized) artifacts (name, artifact object).

info

After calling get_registered_artifacts, you can still modify the registered artifacts.

  • Return type

    Dict[str, Artifact]

  • Returns

    The registered (dynamically synchronized) artifacts.


get_reported_console_output#

get_reported_console_output(number_of_reports=1)

Return a list of console outputs reported by the Task. Retrieved outputs are the most updated console outputs.

  • Parameters

    number_of_reports (int ) – The number of reports to return. The default value is 1, indicating the last (most updated) console output

  • Return type

    Sequence[str]

  • Returns

    A list of strings, each entry corresponds to one report.


get_reported_scalars#

get_reported_scalars(max_samples=0, x_axis='iter')

Return a nested dictionary for the scalar graphs, where the first key is the graph title and the second is the series name. Value is a dict with ‘x’: values and ‘y’: values

info

This call is not cached, any call will retrieve all the scalar reports from the back-end. If the Task has many scalars reported, it might take long for the call to return.

Example:

{'title': {'series': {
'x': [0, 1 ,2],
'y': [10, 11 ,12],
}}}
  • Parameters

    • max_samples (int ) – Maximum samples per series to return. Default is 0 returning all scalars. With sample limit, average scalar values inside sampling window.

    • x_axis (str ) – scalar x_axis, possible values: ‘iter’: iteration (default), ‘timestamp’: seconds from start, ‘iso_time’: absolute time

  • Return type

    Mapping[str, Mapping[str, Mapping[str, Sequence[float]]]]

  • Returns

    dict: Nested scalar graphs: dict[title(str), dict[series(str), dict[axis(str), list(float)]]]


get_status#

get_status()

Return The task status without refreshing the entire Task object object (only the status property)

TaskStatusEnum: [“created”, “in_progress”, “stopped”, “closed”, “failed”, “completed”,

“queued”, “published”, “publishing”, “unknown”]
  • Return type

    str

  • Returns

    str: Task status as string (TaskStatusEnum)


Task.get_task#

classmethod get_task(task_id=None, project_name=None, task_name=None, allow_archived=True, task_filter=None)

Get a Task by Id, or project name / task name combination.

For example:

The following code demonstrates calling Task.get_task to report a scalar to another Task. The output of Logger.report_scalar from testing is associated with the Task named training. It allows training and testing to run concurrently, because they initialized different Tasks (see Task.init for information about initializing Tasks).

The training script:

# initialize the training Task
task = Task.init('myProject', 'training')
# do some training

The testing script:

# initialize the testing Task
task = Task.init('myProject', 'testing')
# get the training Task
train_task = Task.get_task(project_name='myProject', task_name='training')
# report metrics in the training Task
for x in range(10):
train_task.get_logger().report_scalar('title', 'series', value=x * 2, iteration=x)
  • Parameters

    • task_id (str ) – The Id (system UUID) of the experiment to get. If specified, project_name and task_name are ignored.

    • project_name (str ) – The project name of the Task to get.

    • task_name (str ) – The name of the Task within project_name to get.

    • allow_archived (bool ) – Only applicable if not using specific task_id, If True (default) allow to return archived Tasks, if False filter out archived Tasks

    • task_filter (bool ) – Only applicable if not using specific task_id, Pass additional query filters, on top of project/name. See details in Task.get_tasks.

  • Return type

    Task

  • Returns

    The Task specified by ID, or project name / experiment name combination.


Task.get_tasks#

classmethod get_tasks(task_ids=None, project_name=None, task_name=None, task_filter=None)

Get a list of Tasks by one of the following:

  • A list of specific Task IDs.

  • All Tasks in a project matching a full or partial Task name.

  • All Tasks in any project matching a full or partial Task name.

  • Parameters

    • task_ids (list ( str ) ) – The Ids (system UUID) of experiments to get. If task_ids specified, then project_name and task_name are ignored.

    • project_name (str ) – The project name of the Tasks to get. To get the experiment in all projects, use the default value of None. (Optional) Use a list of string for multiple optional project names.

    • task_name (str ) – The full name or partial name of the Tasks to match within the specified project_name (or all projects if project_name is None). This method supports regular expressions for name matching. (Optional)

    • task_ids – list of unique task id string (if exists other parameters are ignored)

    • project_name – project name (str) the task belongs to (use None for all projects)

    • task_name – task name (str) in within the selected project Return any partial match of task_name, regular expressions matching is also supported If None is passed, returns all tasks within the project

    • task_filter (dict ) – filter and order Tasks. See service.tasks.GetAllRequest for details

  • Return type

    Sequence[Task]

  • Returns

    The Tasks specified by the parameter combinations (see the parameters).


get_user_properties#

get_user_properties(value_only=False)

Get user properties for this task. Returns a dictionary mapping user property name to user property details dict. :type value_only: bool :param value_only: If True, returned user property details will be a string representing the property value.

  • Return type

    Dict[str, Union[str, dict]]

  • Parameters

    value_only (bool ) –


Task.ignore_requirements#

classmethod ignore_requirements(package_name)

Ignore a specific package when auto generating the requirements list. Example: Task.ignore_requirements(‘pywin32’)

  • Parameters

    package_name (str ) – The package name to remove/ignore from the “Installed Packages” section of the task.

  • Return type

    None


Task.import_offline_session#

classmethod import_offline_session(session_folder_zip)

Upload an off line session (execution) of a Task. Full Task execution includes repository details, installed packages, artifacts, logs, metric and debug samples.

  • Parameters

    session_folder_zip (str) – Path to a folder containing the session, or zip-file of the session folder.

  • Return type

    Optional[str]

  • Returns

    Newly created task ID (str)


Task.import_task#

classmethod import_task(task_data, target_task=None, update=False)

Import (create) Task from previously exported Task configuration (see Task.export_task) Can also be used to edit/update an existing Task (by passing target_task and update=True).

  • Parameters

    • task_data (dict) – dictionary of a Task’s configuration

    • target_task (Union[Task, str, None]) – Import task_data into an existing Task. Can be either task_id (str) or Task object.

    • update (bool) – If True, merge task_data with current Task configuration.

  • Return type

    Optional[Task]

  • Returns

    return True if Task was imported/updated


Task.init#

classmethod init(project_name=None, task_name=None, task_type=<TaskTypes.training: 'training'>, tags=None, reuse_last_task_id=True, continue_last_task=False, output_uri=None, auto_connect_arg_parser=True, auto_connect_frameworks=True, auto_resource_monitoring=True, auto_connect_streams=True)

Creates a new Task (experiment) if:

  • The Task never ran before. No Task with the same task_name and project_name is stored in ClearML Server.

  • The Task has run before (the same task_name and project_name), and (a) it stored models and / or artifacts, or (b) its status is Published , or (c) it is Archived.

  • A new Task is forced by calling Task.init with reuse_last_task_id=False.

Otherwise, the already initialized Task object for the same task_name and project_name is returned.

info

To reference another Task, instead of initializing the same Task more than once, call Task.get_task. For example, to “share” the same experiment in more than one script, call Task.get_task. See the Task.get_task method for an example.

For example: The first time the following code runs, it will create a new Task. The status will be Completed.

from clearml import Task
task = Task.init('myProject', 'myTask')

If this code runs again, it will not create a new Task. It does not store a model or artifact, it is not Published (its status Completed) , it was not Archived, and a new Task is not forced.

If the Task is Published or Archived, and run again, it will create a new Task with a new Task ID.

The following code will create a new Task every time it runs, because it stores an artifact.

task = Task.init('myProject', 'myOtherTask')
d = {'a': '1'}
task.upload_artifact('myArtifact', d)
  • Parameters

    • project_name (str ) – The name of the project in which the experiment will be created. If the project does not exist, it is created. If project_name is None, the repository name is used. (Optional)

    • task_name (str ) – The name of Task (experiment). If task_name is None, the Python experiment script’s file name is used. (Optional)

    • task_type (TaskTypes ) – The task type.

      Valid task types:

      * `TaskTypes.training` (default)
      * `TaskTypes.testing`
      * `TaskTypes.inference`
      * `TaskTypes.data_processing`
      * `TaskTypes.application`
      * `TaskTypes.monitor`
      * `TaskTypes.controller`
      * `TaskTypes.optimizer`
      * `TaskTypes.service`
      * `TaskTypes.qc`
      * `TaskTypes.custom`
    • tags (Optional[Sequence[str]]) – Add a list of tags (str) to the created Task. For example: tags=[‘512x512’, ‘yolov3’]

    • reuse_last_task_id (bool ) – Force a new Task (experiment) with a previously used Task ID, and the same project and Task name.

      info

      If the previously executed Task has artifacts or models, it will not be reused (overwritten) and a new Task will be created. When a Task is reused, the previous execution outputs are deleted, including console outputs and logs.

      The values are:

      * `True` - Reuse the last Task ID. (default)
      * `False` - Force a new Task (experiment).
      * A string - You can also specify a Task ID (string) to be reused,
      instead of the cached ID based on the project/name combination.
    • continue_last_task (bool ) – Continue the execution of a previously executed Task (experiment)

      info

      When continuing the executing of a previously executed Task, all previous artifacts / models/ logs are intact. New logs will continue iteration/step based on the previous-execution maximum iteration value. For example: The last train/loss scalar reported was iteration 100, the next report will be iteration 101.

      The values are:

      * `True` - Continue the the last Task ID.
      specified explicitly by reuse_last_task_id or implicitly with the same logic as reuse_last_task_id
      * `False` - Overwrite the execution of previous Task (default).
      * A string - You can also specify a Task ID (string) to be continued.
      This is equivalent to continue_last_task=True and reuse_last_task_id=a_task_id_string.
    • output_uri (str ) – The default location for output models and other artifacts. If True is passed, the default files_server will be used for model storage. In the default location, ClearML creates a subfolder for the output. The subfolder structure is the following: <output destination name> / <project name> / <task name>.<Task ID>

      The following are examples of output_uri values for the supported locations:

      * A shared folder: `/mnt/share/folder`
      * S3: `s3://bucket/folder`
      * Google Cloud Storage: `gs://bucket-name/folder`
      * Azure Storage: `azure://company.blob.core.windows.net/folder/`
      * Default file server: True
      important

      For cloud storage, you must install the ClearML package for your cloud storage type, and then configure your storage credentials. For detailed information, see ClearML Python Client Extras in the “ClearML Python Client Reference” section.

    • auto_connect_arg_parser (Union[bool, Mapping[str, bool]]) – Automatically connect an argparse object to the Task

      The values are:

      * `True` - Automatically connect. (default)
      * `False` - Do not automatically connect.
      * A dictionary - In addition to a boolean, you can use a dictionary for fined grained control of connected
      arguments. The dictionary keys are argparse variable names and the values are booleans.
      The `False` value excludes the specified argument from the Task’s parameter section.
      Keys missing from the dictionary default to `True`, and an empty dictionary defaults to `False`.

      For example:

      auto_connect_arg_parser={'do_not_include_me': False, }
      info

      To manually connect an argparse, use Task.connect.

    • auto_connect_frameworks (Union[bool, Mapping[str, bool]]) – Automatically connect frameworks This includes patching MatplotLib, XGBoost, scikit-learn, Keras callbacks, and TensorBoard/X to serialize plots, graphs, and the model location to the ClearML Server (backend), in addition to original output destination.

      The values are:

      * `True` - Automatically connect (default)
      * `False` - Do not automatically connect
      * A dictionary - In addition to a boolean, you can use a dictionary for fined grained control of connected
      frameworks. The dictionary keys are frameworks and the values are booleans.
      Keys missing from the dictionary default to `True`, and an empty dictionary defaults to `False`.

      For example:

      auto_connect_frameworks={
      'matplotlib': True, 'tensorflow': True, 'tensorboard': True, 'pytorch': True,
      'xgboost': True, 'scikit': True, 'fastai': True, 'lightgbm': True,
      'hydra': True, 'detect_repository': True,
      }
    • auto_resource_monitoring (bool ) – Automatically create machine resource monitoring plots These plots appear in in the ClearML Web-App (UI), RESULTS tab, SCALARS sub-tab, with a title of :resource monitor:.

      The values are:

      * `True` - Automatically create resource monitoring plots. (default)
      * `False` - Do not automatically create.
      * Class Type - Create ResourceMonitor object of the specified class type.
    • auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr

      The values are:

      * `True` - Automatically connect (default)
      * `False` - Do not automatically connect
      * A dictionary - In addition to a boolean, you can use a dictionary for fined grained control of stdout and
      stderr. The dictionary keys are ‘stdout’ , ‘stderr’ and ‘logging’, the values are booleans.
      Keys missing from the dictionary default to `False`, and an empty dictionary defaults to `False`.
      Notice, the default behaviour is logging stdout/stderr the
      logging module is logged as a by product of the stderr logging

      For example:

      auto_connect_streams={'stdout': True, 'stderr': True, 'logging': False}
  • Return type

    Task

  • Returns

    The main execution Task (Task context).


is_current_task#

is_current_task()

Deprecated: Deprecated since version 0.13.0: This method is deprecated. Use Task.is_main_task instead.

Is this Task object the main execution Task (initially returned by Task.init)

  • Return type

    bool

  • Returns

    Is this Task object the main execution Task

    • True - Is the main execution Task.

    • False - Is not the main execution Task.


is_main_task#

is_main_task()

Is this Task object the main execution Task (initially returned by Task.init)

info

If Task.init was never called, this method will not create it, making this test more efficient than:

Task.init() == task

  • Return type

    bool

  • Returns

    Is this Task object the main execution Task

    • True - Is the main execution Task.

    • False - Is not the main execution Task.


Task.is_offline#

classmethod is_offline()

Return offline-mode state, If in offline-mode, no communication to the backend is enabled.

  • Return type

    bool

  • Returns

    boolean offline-mode state


labels_stats#

property labels_stats

Get accumulated label stats for the current/last frames iteration

  • Return type

    dict


logger#

property logger

Get a Logger object for reporting, for this task context. You can view all Logger report output associated with the Task for which this method is called, including metrics, plots, text, tables, and images, in the ClearML Web-App (UI).

  • Return type

    Logger

  • Returns

    The Logger object for the current Task (experiment).


mark_failed#

mark_failed(ignore_errors=True, status_reason=None, status_message=None, force=False)

The signal that this Task stopped.

  • Return type

    ()

  • Parameters

    • ignore_errors (bool ) –

    • status_reason (Optional [ str ] ) –

    • status_message (Optional [ str ] ) –

    • force (bool ) –


mark_started#

mark_started(force=False)

Manually mark a Task as started (happens automatically)

  • Parameters

    force (bool ) – If True the task status will be changed to started regardless of the current Task state.

  • Return type

    ()


mark_stopped#

mark_stopped(force=False)

Manually mark a Task as stopped (also used in _at_exit)

  • Parameters

    force (bool ) – If True the task status will be changed to stopped regardless of the current Task state.

  • Return type

    ()


metrics_manager#

property metrics_manager

A metrics manager used to manage the metrics related to this task

  • Return type

    Metrics


models#

property models

Read-only dictionary of the Task’s loaded/stored models.

  • Return type

    Mapping[str, Sequence[Model]]

  • Returns

    A dictionary-like object with “input”/”output” keys and input/output properties, pointing to a list-like object containing of Model objects. Each list-like object also acts as a dictionary, mapping model name to a appropriate model instance.

    Get input/output models:

    task.models.input
    task.models["input"]
    task.models.output
    task.models["output"]

    Get the last output model:

    task.models.output[-1]

    Get a model by name:

    task.models.output["model name"]

publish#

publish(ignore_errors=True)

The signal that this Task will be published

  • Return type

    ()

  • Parameters

    ignore_errors (bool ) –


register_artifact#

register_artifact(name, artifact, metadata=None, uniqueness_columns=True)

Register (add) an artifact for the current Task. Registered artifacts are dynamically sychronized with the ClearML Server (backend). If a registered artifact is updated, the update is stored in the ClearML Server (backend). Registered artifacts are primarily used for Data Audition.

The currently supported registered artifact object type is a pandas.DataFrame.

See also Task.unregister_artifact and Task.get_registered_artifacts.

info

ClearML also supports uploaded artifacts which are one-time uploads of static artifacts that are not dynamically sychronized with the ClearML Server (backend). These static artifacts include additional object types. For more information, see Task.upload_artifact.

  • Parameters

    • name (str ) – The name of the artifact.

      warning

      If an artifact with the same name was previously registered, it is overwritten.

    • artifact (object ) – The artifact object.

    • metadata (dict ) – A dictionary of key-value pairs for any metadata. This dictionary appears with the experiment in the ClearML Web-App (UI), ARTIFACTS tab.

    • uniqueness_columns (Union [ bool , Sequence [ str ] ] ) – A Sequence of columns for artifact uniqueness comparison criteria, or the default value of True. If True, the artifact uniqueness comparison criteria is all the columns, which is the same as artifact.columns.

  • Return type

    None


reload#

reload()

Reload current Task’s state from clearml-server. Refresh all task’s fields, including artifacts / models / parameters etc.

  • Return type

    ()


reset#

reset(set_started_on_success=False, force=False)

Reset a Task. ClearML reloads a Task after a successful reset. When a worker executes a Task remotely, the Task does not reset unless the force parameter is set to True (this avoids accidentally clearing logs and metrics).

  • Parameters

    • set_started_on_success (bool ) – If successful, automatically set the Task to started

      • True - If successful, set to started.

      • False - If successful, do not set to started. (default)

    • force (bool ) – Force a Task reset, even when executing the Task (experiment) remotely in a worker

      • True - Force

      • False - Do not force (default)

  • Return type

    None


running_locally#

static running_locally()

Is the task running locally (i.e., clearml-agent is not executing it)

  • Return type

    bool

  • Returns

    True, if the task is running locally. False, if the task is not running locally.


save_exec_model_design_file#

save_exec_model_design_file(filename='model_design.txt', use_cache=False)

Save execution model design to file


set_archived#

set_archived(archive)

Archive the Task or remove it from the archived folder.

  • Parameters

    archive (bool ) – If True archive the Task, If False make sure it is removed from the archived folder

  • Return type

    ()


set_artifacts#

set_artifacts(artifacts_list=None)

List of artifacts (tasks.Artifact) to update the task

  • Parameters

    artifacts_list (list ) – list of artifacts (type tasks.Artifact)

  • Return type

    Optional[List[Artifact]]

  • Returns

    List of current Task’s Artifacts or None if error.


set_base_docker#

set_base_docker(docker_cmd, docker_arguments=None, docker_setup_bash_script=None)

Set the base docker image for this experiment If provided, this value will be used by clearml-agent to execute this experiment inside the provided docker image. When running remotely the call is ignored

  • Parameters

    • docker_cmd (str ) – docker container image (example: ‘nvidia/cuda:11.1’)

    • docker_arguments (Optional [ Union [ str , Sequence [ str ] ] ] ) – docker execution parameters (example: ‘-e ENV=1’)

    • docker_setup_bash_script (Optional [ Union [ str , Sequence [ str ] ] ] ) – bash script to run at the beginning of the docker before launching the Task itself. example: [‘apt update’, ‘apt-get install -y gcc’]

  • Return type

    ()


set_comment#

set_comment(comment)

Set a comment / description for the Task.

  • Parameters

    comment (str ) – The comment / description for the Task.

  • Return type

    ()


set_configuration_object#

set_configuration_object(name, config_text=None, description=None, config_type=None)

Set the Task’s configuration object as a blob of text. Use only for automation (externally), otherwise use Task.connect_configuration.

  • Parameters

    • name (str ) – Configuration section name

    • config_text (Optional[str]) – configuration as a blob of text (unconstrained text string) usually the content of a configuration file of a sort

    • description (str ) – Configuration section description

    • config_type (str ) – Optional configuration format type

  • Return type

    None


Task.set_credentials#

classmethod set_credentials(api_host=None, web_host=None, files_host=None, key=None, secret=None, store_conf_file=False)

Set new default ClearML Server (backend) host and credentials.

These credentials will be overridden by either OS environment variables, or the ClearML configuration file, clearml.conf.

warning

Credentials must be set before initializing a Task object.

For example, to set credentials for a remote computer:

Task.set_credentials(
api_host='http://localhost:8008', web_host='http://localhost:8080', files_host='http://localhost:8081',
key='optional_credentials', secret='optional_credentials'
)
task = Task.init('project name', 'experiment name')
  • Parameters

    • api_host (str ) – The API server url. For example, host='http://localhost:8008'

    • web_host (str ) – The Web server url. For example, host='http://localhost:8080'

    • files_host (str ) – The file server url. For example, host='http://localhost:8081'

    • key (str ) – The user key (in the key/secret pair). For example, key='thisisakey123'

    • secret (str ) – The user secret (in the key/secret pair). For example, secret='thisisseceret123'

    • store_conf_file (bool ) – If True store the current configuration into the ~/clearml.conf file. If the configuration file exists, no change will be made (outputs a warning). Not applicable when running remotely (i.e. clearml-agent).

  • Return type

    None


set_initial_iteration#

set_initial_iteration(offset=0)

Set initial iteration, instead of zero. Useful when continuing training from previous checkpoints

  • Parameters

    offset (int ) – Initial iteration (at starting point)

  • Return type

    int

  • Returns

    Newly set initial offset.


set_input_model#

set_input_model(model_id=None, model_name=None, update_task_design=True, update_task_labels=True, name=None)

Set a new input model for the Task. The model must be “ready” (status is Published) to be used as the Task’s input model.

  • Parameters

    • model_id (str ) – The Id of the model on the ClearML Server (backend). If model_name is not specified, then model_id must be specified.

    • model_name (Optional [ str ] ) – The model name in the artifactory. The model_name is used to locate an existing model in the ClearML Server (backend). If model_id is not specified, then model_name must be specified.

    • update_task_design (bool ) – Update the Task’s design

      • True - ClearML copies the Task’s model design from the input model.

      • False - ClearML does not copy the Task’s model design from the input model.

    • update_task_labels (bool ) – Update the Task’s label enumeration

      • True - ClearML copies the Task’s label enumeration from the input model.

      • False - ClearML does not copy the Task’s label enumeration from the input model.

    • name (Optional [ str ] ) – Model section name to be stored on the Task (unrelated to the model object name itself) Default: the the model weight filename is used (excluding file extension)

  • Return type

    ()


set_model_config#

set_model_config(config_text=None, config_dict=None)

Deprecated: Deprecated since version 0.14.1: Use Task.connect_configuration instead.

  • Return type

    None

  • Parameters

    • config_text (Optional [ str ] ) –

    • config_dict (Optional [ Mapping ] ) –


set_model_label_enumeration#

set_model_label_enumeration(enumeration=None)

Set the label enumeration for the Task object before creating an output model. Later, when creating an output model, the model will inherit these properties.

  • Parameters

    enumeration (dict ) – A label enumeration dictionary of string (label) to integer (value) pairs.

    For example:

    {
    'background': 0,
    'person': 1
    }
  • Return type

    ()


set_name#

set_name(name)

Set the Task name.

  • Parameters

    name (str ) – The name of the Task.

  • Return type

    ()


Task.set_offline#

classmethod set_offline(offline_mode=False)

Set offline mode, where all data and logs are stored into local folder, for later transmission

  • Parameters

    offline_mode (bool) – If True, offline-mode is turned on, and no communication to the backend is enabled.

  • Return type

    None

  • Returns


set_parameter#

set_parameter(name, value, description=None, value_type=None)

Set a single Task parameter. This overrides any previous value for this parameter.

  • Parameters

    • name (str ) – The parameter name.

    • value (str ) – The parameter value.

    • description (Optional [ str ] ) – The parameter description.

    • value_type (Optional [ Any ] ) – The type of the parameters (cast to string and store)

  • Return type

    ()


set_parameters#

set_parameters(*args, kwargs)**

Set the parameters for a Task. This method sets a complete group of key-value parameter pairs, but does not support parameter descriptions (the input is a dictionary of key-value pairs). Notice the parameter dict is flat: i.e. {‘Args/param’: ‘value’} will set the argument “param” in section “Args” to “value”

  • Parameters

    • args (dict ) – Positional arguments, which are one or more dictionary or (key, value) iterable. They are merged into a single key-value pair dictionary.

    • kwargs (Any ) – Key-value pairs, merged into the parameters dictionary created from args.

  • Return type

    ()


set_parameters_as_dict#

set_parameters_as_dict(dictionary)

Set the parameters for the Task object from a dictionary. The dictionary can be nested. This does not link the dictionary to the Task object. It does a one-time update. This is the same behavior as the Task.connect method.

  • Return type

    None

  • Parameters

    dictionary (Dict ) –


set_parent#

set_parent(parent)

Set the parent task for the Task. :type parent: Optional[Union[str, Task]] :param parent: The parent task id (or parent Task object) for the Task. Set None for no parent. :type parent: str or Task

  • Return type

    ()

  • Parameters

    parent (Optional [ Union [ str , Task ] ] ) –


set_resource_monitor_iteration_timeout#

set_resource_monitor_iteration_timeout(seconds_from_start=1800)

Set the ResourceMonitor maximum duration (in seconds) to wait until first scalar/plot is reported. If timeout is reached without any reporting, the ResourceMonitor will start reporting machine statistics based on seconds from Task start time (instead of based on iteration)

  • Parameters

    seconds_from_start (float) – Maximum number of seconds to wait for scalar/plot reporting before defaulting to machine statistics reporting based on seconds from experiment start time

  • Return type

    bool

  • Returns

    True if success


set_task_type#

set_task_type(task_type)

Set the task_type for the Task.

  • Parameters

    task_type (str or TaskTypes ) – The task_type of the Task (see optional values in TaskTypes).

  • Return type

    ()


set_user_properties#

set_user_properties(*iterables, properties)**

Set user properties for this task. A user property can contain the following fields (all of type string): name / value / description / type

Examples:

task.set_user_properties(backbone=’great’, stable=True)
task.set_user_properties(backbone={“type”: int, “description”: “network type”, “value”: “great”}, )
task.set_user_properties(
{“name”: “backbone”, “description”: “network type”, “value”: “great”},
{“name”: “stable”, “description”: “is stable”, “value”: True},
)
  • Parameters

    • iterables (Union[Mapping[str, Union[str, dict, None]], Iterable[dict]]) – Properties iterables, each can be: * A dictionary of string key (name) to either a string value (value) a dict (property details). If the value is a dict, it must contain a “value” field. For example:

      {
      "property_name": {"description": "This is a user property", "value": "property value"},
      "another_property_name": {"description": "This is user property", "value": "another value"},
      "yet_another_property_name": "some value"
      }
      * An iterable of dicts (each representing property details). Each dict must contain a “name” field and a
      ”value” field. For example:
      ```javascript
      [
      {
      "name": "property_name",
      "description": "This is a user property",
      "value": "property value"
      },
      {
      "name": "another_property_name",
      "description": "This is another user property",
      "value": "another value"
      }
      ]
      ```
    • properties (Union[str, dict, int, float, None]) – Additional properties keyword arguments. Key is the property name, and value can be a string (property value) or a dict (property details). If the value is a dict, it must contain a “value” field. For example:

      {

      “property_name”: “string as property value”,
      “another_property_name”: {
      ”type”: “string”,
      “description”: “This is user property”,
      “value”: “another value”
      }

      }

  • Return type

    bool


started#

started(ignore_errors=True, force=False)

The signal that this Task started.

  • Return type

    ()

  • Parameters

    • ignore_errors (bool ) –

    • force (bool ) –


status#

property status

The Task’s status. To keep the Task updated. ClearML reloads the Task status information only, when this value is accessed.

return str: TaskStatusEnum status

  • Return type

    str


stopped#

stopped(ignore_errors=True, force=False)

The signal that this Task stopped.

  • Return type

    ()

  • Parameters

    • ignore_errors (bool ) –

    • force (bool ) –


unregister_artifact#

unregister_artifact(name)

Unregister (remove) a registered artifact. This removes the artifact from the watch list that ClearML uses to synchronize artifacts with the ClearML Server (backend).

important

Calling this method does not remove the artifact from a Task. It only stops ClearML from monitoring the artifact.

When this method is called, ClearML immediately takes the last snapshot of the artifact.

  • Return type

    None

  • Parameters

    name (str ) –


update_model_desc#

update_model_desc(new_model_desc_file=None)

Change the Task’s model description.

  • Return type

    ()

  • Parameters

    new_model_desc_file (Optional [ str ] ) –


update_output_model#

update_output_model(model_path, name=None, comment=None, tags=None, model_name=None, iteration=None)

Update the Task’s output model weights file. First, ClearML uploads the file to the preconfigured output destination (see the Task’s output.destination property or call the setup_upload method), then ClearML updates the model object associated with the Task an API call. The API call uses with the URI of the uploaded file, and other values provided by additional arguments.

  • Parameters

    • model_path (str) – A local weights file or folder to be uploaded. If remote URI is provided (e.g. http:// or s3: // etc) then the URI is stored as is, without any upload

    • name (Optional[str]) – The updated model name. If not provided, the name is the model weights file filename without the extension.

    • comment (Optional[str]) – The updated model description. (Optional)

    • tags (Optional[Sequence[str]]) – The updated model tags. (Optional)

    • model_name (Optional[str]) – If provided the model name as it will appear in the model artifactory. (Optional) Default: Task.name - name

    • iteration (Optional[int]) – iteration number for the current stored model (Optional)

  • Return type

    str

  • Returns

    The URI of the uploaded weights file. Notice: upload is done is a background thread, while the function call returns immediately


update_parameters#

update_parameters(*args, kwargs)**

Update the parameters for a Task. This method updates a complete group of key-value parameter pairs, but does not support parameter descriptions (the input is a dictionary of key-value pairs). Notice the parameter dict is flat: i.e. {‘Args/param’: ‘value’} will set the argument “param” in section “Args” to “value”

  • Parameters

    • args (dict ) – Positional arguments, which are one or more dictionary or (key, value) iterable. They are merged into a single key-value pair dictionary.

    • kwargs (Any ) – Key-value pairs, merged into the parameters dictionary created from args.

  • Return type

    ()


update_task#

update_task(task_data)

Update current task with configuration found on the task_data dictionary. See also export_task() for retrieving Task configuration.

  • Parameters

    task_data (dict) – dictionary with full Task configuration

  • Return type

    bool

  • Returns

    return True if Task update was successful


upload_artifact#

upload_artifact(name, artifact_object, metadata=None, delete_after_upload=False, auto_pickle=True, preview=None, wait_on_upload=False)

Upload (add) a static artifact to a Task object. The artifact is uploaded in the background.

The currently supported upload (static) artifact types include:

  • string / pathlib2.Path - A path to artifact file. If a wildcard or a folder is specified, then ClearML creates and uploads a ZIP file.

  • dict - ClearML stores a dictionary as .json file and uploads it.

  • pandas.DataFrame - ClearML stores a pandas.DataFrame as .csv.gz (compressed CSV) file and uploads it.

  • numpy.ndarray - ClearML stores a numpy.ndarray as .npz file and uploads it.

  • PIL.Image - ClearML stores a PIL.Image as .png file and uploads it.

  • Any - If called with auto_pickle=True, the object will be pickled and uploaded.

  • Parameters

    • name (str ) – The artifact name.

      warning

      If an artifact with the same name was previously uploaded, then it is overwritten.

    • artifact_object (object ) – The artifact object.

    • metadata (dict ) – A dictionary of key-value pairs for any metadata. This dictionary appears with the experiment in the ClearML Web-App (UI), ARTIFACTS tab.

    • delete_after_upload (bool ) – After the upload, delete the local copy of the artifact

      • True - Delete the local copy of the artifact.

      • False - Do not delete. (default)

    • auto_pickle (bool ) – If True (default) and the artifact_object is not one of the following types: pathlib2.Path, dict, pandas.DataFrame, numpy.ndarray, PIL.Image, url (string), local_file (string) the artifact_object will be pickled and uploaded as pickle file artifact (with file extension .pkl)

    • preview (Any ) – The artifact preview

    • wait_on_upload (bool ) – Whether or not the upload should be synchronous, forcing the upload to complete before continuing.

  • Return type

    bool

  • Returns

    The status of the upload.

  • True - Upload succeeded.

  • False - Upload failed.

  • Raise

    If the artifact object type is not supported, raise a ValueError.

  • Parameters

    • name (str ) –

    • artifact_object (Union [ str , Mapping , pandas.DataFrame , numpy.ndarray , Image.Image , Any ] ) –

    • metadata (Optional [ Mapping ] ) –

    • delete_after_upload (bool ) –

    • auto_pickle (bool ) –

    • preview (Any ) –

    • wait_on_upload (bool ) –

  • Return type

    bool


wait_for_status#

wait_for_status(status=(<TaskStatusEnum.completed: 'completed'>, <TaskStatusEnum.stopped: 'stopped'>, <TaskStatusEnum.closed: 'closed'>), raise_on_status=(<TaskStatusEnum.failed: 'failed'>, ), check_interval_sec=60.0)

Wait for a task to reach a defined status.

  • Parameters

    • status (Iterable [ Task.TaskStatusEnum ] ) – Status to wait for. Defaults to (‘completed’, ‘stopped’, ‘closed’, )

    • raise_on_status (Optional [ Iterable [ Task.TaskStatusEnum ] ] ) – Raise RuntimeError if the status of the tasks matches one of these values. Defaults to (‘failed’).

    • check_interval_sec (float ) – Interval in seconds between two checks. Defaults to 60 seconds.

  • Raise

    RuntimeError if the status is one of {raise_on_status}.

  • Return type

    ()