HyperParameterOptimizer

class automation.HyperParameterOptimizer()

Hyperparameter search controller. Clones the base experiment, changes arguments and tries to maximize/minimize the defined objective.

Create a new hyperparameter controller. The newly created object will launch and monitor the new experiments.

Parameters
- base_task_id (str ) – The Task ID to be used as template experiment to optimize.
- hyper_parameters (list ) – The list of Parameter objects to optimize over.
- Sequence [ str ] ] objective_metric_title (Union [ str , ) – The Objective metric title(s) to maximize / minimize (for example, validation, ["validation", "loss"]). If objective_metric_title is a sequence (used to optimize multiple objectives at the same time), then objective_metric_series and objective_metric_sign have to be sequences of the same length. Each title will be matched with the respective series and sign
- Sequence [ str ] ] objective_metric_series (Union [ str , ) – The Objective metric series to maximize / minimize (for example, loss_series, ["validation_series", "loss_series"]).
- Sequence [ str ] ] objective_metric_sign (Union [ str , ) – The objectives to maximize / minimize.
  The values are:
  - min - Minimize the last reported value for the specified title/series scalar.
  - max - Maximize the last reported value for the specified title/series scalar.
  - min_global - Minimize the min value of all reported values for the specific title/series scalar.
  - max_global - Maximize the max value of all reported values for the specific title/series scalar.
- optimizer_class (class.SearchStrategy ) – The SearchStrategy optimizer to use for the hyperparameter search
- max_number_of_concurrent_tasks (int ) – The maximum number of concurrent Tasks (experiments) running at the same time.
- execution_queue (str ) – The execution queue to use for launching Tasks (experiments).
- optimization_time_limit (float ) – The maximum time (minutes) for the entire optimization process. The default is None, indicating no time limit.
- compute_time_limit (float ) – The maximum compute time in minutes. When time limit is exceeded, all jobs aborted. (Optional)
- auto_connect_task (bool ) – Store optimization arguments and configuration in the Task.
  The values are:
  - True - The optimization argument and configuration will be stored in the Task. All arguments will
    be under the hyperparameter section opt, and the optimization hyper_parameters space will be stored in the Task configuration object section.
  - False - Do not store with Task.
  - Task - A specific Task object to connect the optimization process with.
- always_create_task (bool ) – Always create a new Task.
  The values are:
  - True - No current Task initialized. Create a new task named optimization in the base_task_id
    project.
  - False - Use the task.Task.current_task (if exists) to report statistics.
- spawn_project (str ) – If project name is specified, create all optimization Jobs (Tasks) in the specified project instead of the original base_task_id project.
- save_top_k_tasks_only (int ) – If specified and above 0, keep only the top_k performing Tasks, and archive the rest of the created Tasks. Default: -1 keep everything, nothing will be archived.
- optimizer_kwargs (**** ) – Arguments passed directly to the optimizer constructor.
  Example:
```
:linenos:
:caption: Example

from clearml import Task
from clearml.automation import UniformParameterRange, DiscreteParameterRange
from clearml.automation import GridSearch, RandomSearch, HyperParameterOptimizer

task = Task.init('examples', 'HyperParameterOptimizer example')
an_optimizer = HyperParameterOptimizer(
    base_task_id='fa30fa45d95d4927b87c323b5b04dc44',
    hyper_parameters=[
        UniformParameterRange('lr', min_value=0.01, max_value=0.3, step_size=0.05),
        DiscreteParameterRange('network', values=['ResNet18', 'ResNet50', 'ResNet101']),
    ],
    objective_metric_title='title',
    objective_metric_series='series',
    objective_metric_sign='min',
    max_number_of_concurrent_tasks=5,
    optimizer_class=RandomSearch,
    execution_queue='workers', time_limit_per_job=120, pool_period_min=0.2)

# This will automatically create and print the optimizer new task id
# for later use. if a Task was already created, it will use it.
an_optimizer.set_time_limit(in_minutes=10.)
an_optimizer.start()
# we can create a pooling loop if we like
while not an_optimizer.reached_time_limit():
    top_exp = an_optimizer.get_top_experiments(top_k=3)
    print(top_exp)
# wait until optimization completed or timed-out
an_optimizer.wait()
# make sure we stop all jobs
an_optimizer.stop()
```

elapsed

elapsed()

Return minutes elapsed from controller stating time stamp.

Return type
float
Returns
The minutes from controller start time. A negative value means the process has not started yet.

get_active_experiments

get_active_experiments()

Return a list of Tasks of the current active experiments.

Return type
Sequence[Task]
Returns
A list of Task objects, representing the current active experiments.

get_num_active_experiments

get_num_active_experiments()

Return the number of current active experiments.

Return type
int
Returns
The number of active experiments.

get_optimizer

get_optimizer()

Return the currently used optimizer object.

Return type
SearchStrategy
Returns
The SearchStrategy object used.

HyperParameterOptimizer.get_optimizer_top_experiments

classmethod get_optimizer_top_experiments(objective_metric_title, objective_metric_series, objective_metric_sign, optimizer_task_id, top_k)

Return a list of Tasks of the top performing experiments for a specific HyperParameter Optimization session (i.e. Task ID), based on the title/series objective.

Parameters
- objective_metric_title (str ) – The Objective metric title to maximize / minimize (for example, validation).
- objective_metric_series (str ) – The Objective metric series to maximize / minimize (for example, loss).
- objective_metric_sign (str ) – The objective to maximize / minimize.
  The values are:
  - min - Minimize the last reported value for the specified title/series scalar.
  - max - Maximize the last reported value for the specified title/series scalar.
  - min_global - Minimize the min value of all reported values for the specific title/series scalar.
  - max_global - Maximize the max value of all reported values for the specific title/series scalar.
- optimizer_task_id (str ) – Parent optimizer Task ID
- top_k (int) – The number of Tasks (experiments) to return.
Return type
Sequence[Task]
Returns
A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_time_limit

get_time_limit()

Return the controller optimization time limit.

Return type
datetime
Returns
The absolute datetime limit of the controller optimization process.

get_top_experiments

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

Parameters
top_k (int ) – The number of Tasks (experiments) to return.
Return type
Sequence[Task]
Returns
A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_top_experiments_details

get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)

Return a list of dictionaries of the top performing experiments. Example: [{'task_id': Task-ID, 'metrics': scalar-metric-dict, 'hyper_parameters': Hyper-Parameters},]

Order is based on the controller Objective object.

Parameters
- top_k (int ) – The number of Tasks (experiments) to return.
- all_metrics (bool ) – Default False, only return the objective metric on the metrics dictionary. If True, return all scalar metrics of the experiment
- all_hyper_parameters (bool ) – Default False. If True, return all the hyperparameters from all the sections.
- only_completed (bool ) – return only completed Tasks. Default False.
Return type
Sequence[(str, dict)]

Returns

A list of dictionaries ({task_id: ‘’, hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example w/ all_metrics=False:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    },
]

Example w/ all_metrics=True:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    },
]

is_active

is_active()

Is the optimization procedure active (still running)

The values are:

True - The optimization procedure is active (still running).
False - The optimization procedure is not active (not still running).

info

If the daemon thread has not yet started, is_active returns True.

Return type
bool
Returns
A boolean indicating whether the optimization procedure is active (still running) or stopped.

is_running

is_running()

Is the optimization controller is running

The values are:

True - The optimization procedure is running.
False - The optimization procedure is running.

Return type
bool
Returns
A boolean indicating whether the optimization procedure is active (still running) or stopped.

reached_time_limit

reached_time_limit()

Did the optimizer reach the time limit

The values are:

True - The time limit passed.
False - The time limit did not pass.

This method returns immediately, it does not wait for the optimizer.

Return type
bool
Returns
True, if optimizer is running and we passed the time limit, otherwise returns False.

set_default_job_class

set_default_job_class(job_class)

Set the Job class to use when the optimizer spawns new Jobs.

Parameters
job_class (ClearmlJob ) – The Job Class type.
Return type
()

set_report_period

set_report_period(report_period_minutes)

Set reporting period for the accumulated objective report (minutes). This report is sent on the Optimizer Task, and collects the Objective metric from all running jobs.

Parameters
report_period_minutes (float ) – The reporting period (minutes). The default is once every 10 minutes.
Return type
()

set_time_limit

set_time_limit(in_minutes=None, specific_time=None)

Set a time limit for the HyperParameterOptimizer controller. If we reached the time limit, stop the optimization process. If specific_time is provided, use it; otherwise, use the in_minutes.

Parameters
- in_minutes (float ) – The maximum processing time from current time (minutes).
- specific_time (datetime ) – The specific date/time limit.
Return type
()

start

start(job_complete_callback=None)

Start the HyperParameterOptimizer controller. If the calling process is stopped, then the controller stops as well.

Parameters

job_complete_callback (Callable ) – Callback function, called when a job is completed.

def job_complete_callback(
    job_id,                 # type: str
    objective_value,        # type: float
    objective_iteration,    # type: int
    job_parameters,         # type: dict
    top_performance_job_id  # type: str
):
    pass

Return type
bool
Returns
True, if the controller started. False, if the controller did not start.

start_locally

start_locally(job_complete_callback=None)

Start the HyperParameterOptimizer controller completely locally. Both the optimizer task and all spawned substasks are run on the local machine using the current environment. If the calling process is stopped, then the controller stops as well.

Parameters

job_complete_callback (Callable ) – Callback function, called when a job is completed.

def job_complete_callback(
    job_id,                 # type: str
    objective_value,        # type: float
    objective_iteration,    # type: int
    job_parameters,         # type: dict
    top_performance_job_id  # type: str
):
    pass

Return type
bool
Returns
True, if the controller started. False, if the controller did not start.

stop

stop(timeout=None, wait_for_reporter=True)

Stop the HyperParameterOptimizer controller and the optimization thread.

Parameters
- timeout (float ) – Wait timeout for the optimization thread to exit (minutes). The default is None, indicating do not wait to terminate immediately.
- wait_for_reporter (Optional [ bool ] ) – Wait for reporter to flush data.
Return type
()

wait

wait(timeout=None)

Wait for the optimizer to finish.

info

This method does not stop the optimizer. Call stop to terminate the optimizer.

Parameters
timeout (float ) – The timeout to wait for the optimization to complete (minutes). If None, then wait until we reached the timeout, or optimization completed.
Return type
bool
Returns
True, if the optimization finished. False, if the optimization timed out.

class automation.HyperParameterOptimizer()​

elapsed​

get_active_experiments​

get_num_active_experiments​

get_optimizer​

HyperParameterOptimizer.get_optimizer_top_experiments​

get_time_limit​

get_top_experiments​

get_top_experiments_details​

is_active​

is_running​

reached_time_limit​

set_default_job_class​

set_report_period​

set_time_limit​

start​

start_locally​

stop​

wait​

class automation.HyperParameterOptimizer()

elapsed

get_active_experiments

get_num_active_experiments

get_optimizer

HyperParameterOptimizer.get_optimizer_top_experiments

get_time_limit

get_top_experiments

get_top_experiments_details

is_active

is_running

reached_time_limit

set_default_job_class

set_report_period

set_time_limit

start

start_locally

stop

wait