HyperParameterOptimizer

class automation.HyperParameterOptimizer(base_task_id, hyper_parameters, objective_metric_title, objective_metric_series, objective_metric_sign='min', optimizer_class=<class 'clearml.automation.optimization.RandomSearch'>, max_number_of_concurrent_tasks=10, execution_queue='default', optimization_time_limit=None, compute_time_limit=None, auto_connect_task=True, always_create_task=False, spawn_project=None, save_top_k_tasks_only=None, **optimizer_kwargs)

Hyperparameter search controller. Clones the base experiment, changes its arguments, and tries to maximize / minimize the defined objective.

Create a new hyperparameter controller. The newly created object will launch and monitor the new experiments.

Parameters
- base_task_id (str) – The Task ID to be used as template experiment to optimize.
- hyper_parameters (Sequence[Parameter]) – The list of parameter objects to optimize over.
- objective_metric_title (Union[str, Sequence[str]]) – The Objective metric title(s) to maximize / minimize (for example, validation, ["validation", "loss"]). If objective_metric_title is a sequence (used to optimize multiple objectives at the same time), then objective_metric_series and objective_metric_sign have to be sequences of the same length. Each title will be matched with the respective series and sign.
- objective_metric_series (Union[str, Sequence[str]]) – The Objective metric series to maximize / minimize (for example, loss_series, ["validation_series", "loss_series"]).
- objective_metric_sign (Union[str, Sequence[str]]) – The objectives to maximize / minimize.
  
  The values are:
  - min - Minimize the last reported value for the specified title/series scalar.
  - max - Maximize the last reported value for the specified title/series scalar.
  - min_global - Minimize the min value of all reported values for the specific title/series scalar.
  - max_global - Maximize the max value of all reported values for the specific title/series scalar.
- optimizer_class (Union[SearchStrategy, type]) – The SearchStrategy optimizer class (or instance) to use for the hyperparameter search.
- max_number_of_concurrent_tasks (int) – The maximum number of concurrent Tasks (experiments) running at the same time.
- execution_queue (str) – The execution queue to use for launching Tasks (experiments).
- optimization_time_limit (Optional[float]) – The maximum time (minutes) for the entire optimization process. The default is None, indicating no time limit.
- compute_time_limit (Optional[float]) – The maximum compute time in minutes. When the time limit is exceeded, all jobs are aborted.
- auto_connect_task (Union[bool, Task]) – Store optimization arguments and configuration in the Task.
  
  The values are:
  - True - The optimization argument and configuration will be stored in the Task. All arguments will be under the hyperparameter section opt, and the optimization hyper_parameters space will be stored in the Task configuration object section.
  - False - Do not store with Task.
  - Task - A specific Task object to connect the optimization process with.
- always_create_task (bool) – Always create a new Task.
  
  The values are:
  - True - No current Task initialized. Create a new task named optimization in the base_task_id project.
  - False - Use the Task.current_task (if exists) to report statistics.
- spawn_project (Optional[str]) – If project name is specified, create all optimization Jobs (Tasks) in the specified project instead of the original base_task_id project.
- save_top_k_tasks_only (Optional[int]) – If specified and greater than 0, keep only that many top-performing Tasks, and archive the rest of the created Tasks. If not specified (default), keep everything; nothing will be archived.
- optimizer_kwargs (Any) – Arguments passed directly to the optimizer constructor.
Example:
```
:linenos:
:caption: Example

from clearml import Task
from clearml.automation import UniformParameterRange, DiscreteParameterRange
from clearml.automation import GridSearch, RandomSearch, HyperParameterOptimizer

task = Task.init('examples', 'HyperParameterOptimizer example')
an_optimizer = HyperParameterOptimizer(
    base_task_id='fa30fa45d95d4927b87c323b5b04dc44',
    hyper_parameters=[
        UniformParameterRange('lr', min_value=0.01, max_value=0.3, step_size=0.05),
        DiscreteParameterRange('network', values=['ResNet18', 'ResNet50', 'ResNet101']),
    ],
    objective_metric_title='title',
    objective_metric_series='series',
    objective_metric_sign='min',
    max_number_of_concurrent_tasks=5,
    optimizer_class=RandomSearch,
    execution_queue='workers', time_limit_per_job=120, pool_period_min=0.2)

# This will automatically create and print the optimizer new task id
# for later use. if a Task was already created, it will use it.
an_optimizer.set_time_limit(in_minutes=10.)
an_optimizer.start()
# we can create a pooling loop if we like
while not an_optimizer.reached_time_limit():
    top_exp = an_optimizer.get_top_experiments(top_k=3)
    print(top_exp)
# wait until optimization completed or timed-out
an_optimizer.wait()
# make sure we stop all jobs
an_optimizer.stop()
```

elapsed

elapsed()

Return minutes elapsed from controller starting time stamp.

Return type

float
Returns

The minutes from controller start time. A negative value means the process has not started yet.

get_active_experiments

get_active_experiments()

Return a list of Tasks of the current active experiments.

Return type

Sequence[Task]
Returns

A list of Task objects, representing the current active experiments.

get_num_active_experiments

get_num_active_experiments()

Return the number of current active experiments.

Return type

int
Returns

The number of active experiments.

get_optimizer

get_optimizer()

Return the currently used optimizer object.

Return type

SearchStrategy
Returns

The SearchStrategy object used.

HyperParameterOptimizer.get_optimizer_top_experiments

classmethod get_optimizer_top_experiments(objective_metric_title, objective_metric_series, objective_metric_sign, optimizer_task_id, top_k)

Return a list of Tasks of the top performing experiments for a specific HyperParameter Optimization session (i.e. Task ID), based on the title/series objective.

Parameters
- objective_metric_title (Union[str, List[str]]) – The Objective metric title to maximize / minimize (for example, validation).
- objective_metric_series (Union[str, List[str]]) – The Objective metric series to maximize / minimize (for example, loss).
- objective_metric_sign (Union[str, List[str]]) – The objective to maximize / minimize.
  
  The values are:
  - min - Minimize the last reported value for the specified title/series scalar.
  - max - Maximize the last reported value for the specified title/series scalar.
  - min_global - Minimize the min value of all reported values for the specific title/series scalar.
  - max_global - Maximize the max value of all reported values for the specific title/series scalar.
- optimizer_task_id (str) – Parent optimizer Task ID.
- top_k (int) – The number of Tasks (experiments) to return.
Return type

Sequence[Task]
Returns

A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_time_limit

get_time_limit()

Return the controller optimization time limit.

Return type

datetime
Returns

The absolute datetime limit of the controller optimization process.

get_top_experiments

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

Parameters

top_k (int) – The number of Tasks (experiments) to return.
Return type

Sequence[Task]
Returns

A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_top_experiments_details

get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)

Return a list of dictionaries of the top performing experiments. Example: [{'task_id': TASK_ID, 'metrics': SCALAR_METRIC_DICT, 'hyper_parameters': HYPER_PARAMETERS},]

Order is based on the controller Objective object.

Parameters
- top_k (int) – The number of Tasks (experiments) to return.
- all_metrics (bool) – If False (default), only the objective metric is included in the metrics dictionary. If True, all scalar metrics of the experiment are included.
- all_hyper_parameters (bool) – If True, return all the hyperparameters from all the sections. If False (default), return only the hyperparameters that are part of the optimization search space.
- only_completed (bool) – If True, return only completed Tasks. Default: False.
Return type

Sequence[Union[str, dict]]

Returns

A list of dictionaries ({task_id: '', hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example with all_metrics=False:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    },
]

Example with all_metrics=True:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    },
]

is_active

is_active()

Return whether the optimization procedure is active (still running).

The values are:

True - The optimization procedure is active (still running).
False - The optimization procedure is not active (not still running).

info

If the daemon thread has not yet started, is_active returns True.

Return type

bool
Returns

A boolean indicating whether the optimization procedure is active (still running) or stopped.

is_running

is_running()

Return whether the optimization controller is running.

The values are:

True - The optimization procedure is running.
False - The optimization procedure is not running.

Return type

bool
Returns

A boolean indicating whether the optimization procedure is active (still running) or stopped.

reached_time_limit

reached_time_limit()

Return whether the optimizer has reached the time limit.

The values are:

True - The time limit passed.
False - The time limit did not pass.

This method returns immediately, it does not wait for the optimizer.

Return type

bool
Returns

True, if the optimizer is running and the time limit has passed. False, otherwise.

set_default_job_class

set_default_job_class(job_class)

Set the Job class to use when the optimizer spawns new Jobs.

Parameters

job_class (ClearmlJob) – The Job class to use.
Return type

None

set_report_period

set_report_period(report_period_minutes)

Set reporting period for the accumulated objective report (minutes). This report is sent on the Optimizer Task, and collects the Objective metric from all running jobs.

Parameters

report_period_minutes (float) – The reporting period (minutes). The default is once every 10 minutes.
Return type

None

set_time_limit

set_time_limit(in_minutes=None, specific_time=None)

Set a time limit for the HyperParameterOptimizer controller. Once the time limit is reached, stop the optimization process. If specific_time is provided, use it; otherwise, use the in_minutes.

Parameters
- in_minutes (Optional[float]) – The maximum processing time from current time (minutes).
- specific_time (Optional[datetime]) – The specific date/time limit.
Return type

None

start

start(job_complete_callback=None)

Start the HyperParameterOptimizer controller. If the calling process is stopped, then the controller stops as well.

Parameters

job_complete_callback (Optional[Callable[[str, float, int, dict, str], None]]) – Callback function, called when a job is completed.

def job_complete_callback(
    job_id,                 # type: str
    objective_value,        # type: float
    objective_iteration,    # type: int
    job_parameters,         # type: dict
    top_performance_job_id  # type: str
):
    pass

Return type

bool
Returns

True, if the controller started. False, if the controller did not start.

start_locally

start_locally(job_complete_callback=None)

Start the HyperParameterOptimizer controller completely locally. Both the optimizer task and all spawned subtasks are run on the local machine using the current environment. If the calling process is stopped, then the controller stops as well.

Parameters

job_complete_callback (Optional[Callable[[str, float, int, dict, str], None]]) – Callback function, called when a job is completed.

def job_complete_callback(
    job_id,                 # type: str
    objective_value,        # type: float
    objective_iteration,    # type: int
    job_parameters,         # type: dict
    top_performance_job_id  # type: str
):
    pass

Return type

bool
Returns

True, if the controller started. False, if the controller did not start.

stop

stop(timeout=None, wait_for_reporter=True)

Stop the HyperParameterOptimizer controller and the optimization thread.

Parameters
- timeout (Optional[float]) – Wait timeout for the optimization thread to exit (minutes). The default is None, indicating do not wait to terminate immediately.
- wait_for_reporter (Optional[bool]) – If True (default), wait for the reporter thread to flush its data before returning.
Return type

None

wait

wait(timeout=None)

Wait for the optimizer to finish.

info

This method does not stop the optimizer. Call stop to terminate the optimizer.

Parameters

timeout (Optional[float]) – The timeout to wait for the optimization to complete (minutes). If None, wait indefinitely until the optimization completes.
Return type

bool
Returns

True, if the optimization finished. False, if the optimization timed out.

elapsed​

get_active_experiments​

get_num_active_experiments​

get_optimizer​

HyperParameterOptimizer.get_optimizer_top_experiments​

get_time_limit​

get_top_experiments​

get_top_experiments_details​

is_active​

is_running​

reached_time_limit​

set_default_job_class​

set_report_period​

set_time_limit​

start​

start_locally​

stop​

wait​

elapsed

get_active_experiments

get_num_active_experiments

get_optimizer

HyperParameterOptimizer.get_optimizer_top_experiments

get_time_limit

get_top_experiments

get_top_experiments_details

is_active

is_running

reached_time_limit

set_default_job_class

set_report_period

set_time_limit

start

start_locally

stop

wait