OptimizerOptuna

class automation.optuna.OptimizerOptuna(base_task_id, hyper_parameters, objective_metric, execution_queue, num_concurrent_workers, max_iteration_per_job, total_max_jobs, pool_period_min=2.0, min_iteration_per_job=None, time_limit_per_job=None, compute_time_limit=None, optuna_sampler=None, optuna_pruner=None, continue_previous_study=None, **optuna_kwargs)

Initialize an Optuna search strategy optimizer. Optuna performs robust and efficient hyperparameter optimization at scale. A specific hyperparameter pruning strategy can be selected via the optuna_sampler and optuna_pruner arguments.

Parameters
- base_task_id (str) – Task ID to be used as template experiment to optimize.
- hyper_parameters (Sequence[Parameter]) – List of parameter objects to optimize over.
- objective_metric (Objective) – Objective metric to maximize / minimize.
- execution_queue (str) – Execution queue to use for launching Tasks (experiments).
- num_concurrent_workers (int) – Maximum number of concurrent running Tasks (machines).
- max_iteration_per_job (Optional[int]) – Number of iterations per job. iterations are the reported iterations for the specified objective, not the maximum reported iteration of the Task.
- total_max_jobs (Optional[int]) – Total maximum jobs for the optimization process. Must be provided in order to calculate the total budget for the optimization process. The total budget is measured in iterations (see above) and will be set to max_iteration_per_job \* total_max_jobs. This means more than total_max_jobs could be created, as long as the cumulative iterations (summed over all created jobs) do not exceed max_iteration_per_job \* total_max_jobs.
- pool_period_min (float) – Time between two consecutive pools (minutes).
- min_iteration_per_job (Optional[int]) – Minimum number of iterations (of the Objective metric) per single job, before early stopping the job.
- time_limit_per_job (Optional[float]) – Maximum execution time per single job in minutes. When the time limit is exceeded, the job is aborted.
- compute_time_limit (Optional[float]) – Maximum compute time in minutes. When the time limit is exceeded, all jobs are aborted.
- optuna_sampler (Optional[Any]) – Optuna sampler instance (for example, an instance of optuna.samplers.TPESampler) used to control how new configurations are proposed. If None (default), Optuna’s own default sampler is used.
- optuna_pruner (Optional[Any]) – Optuna pruner instance (for example, an instance of optuna.pruners.MedianPruner) used to decide whether to prune (early-stop) an unpromising trial. If None (default), Optuna’s own default pruner is used.
- optuna_kwargs (Any) – Arguments passed directly to the Optuna object.
- continue_previous_study (Optional [ Study ] ) –

create_job

create_job()

Helper function; implementation is optional. Used by the default process_step implementation. Create a new job if needed and return it. Return None if no job needs to be created.

Return type

Optional[ClearmlJob]
Returns

A newly created ClearmlJob object, or None if no job was created.

get_created_jobs_ids

get_created_jobs_ids()

Return a dict of Task IDs created by this optimizer so far, including completed and running jobs. The values of the returned dict are the parameters used in the specific job.

Return type

Mapping[str, dict]
Returns

A dict of Task IDs as keys, and their parameters dict as values.

get_created_jobs_tasks

get_created_jobs_tasks()

Return a dict of Task IDs created by this optimizer so far. The values of the returned dict are the ClearmlJob objects.

Return type

Mapping[str, dict]
Returns

A dict of Task IDs as keys, and their ClearmlJob objects as values.

get_objective_metric

get_objective_metric()

Return the metric title/series pair(s) of the objective.

Return type

Union[Tuple[str, str], List[Tuple[str, str]]]
Returns

A single (title, series) tuple for a single-objective optimization, or a list of (title, series) tuples for a multi-objective optimization.

get_running_jobs

get_running_jobs()

Return the current running ClearmlJob objects.

Return type

Sequence[ClearmlJob]
Returns

List of ClearmlJob objects.

get_top_experiments

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

Parameters

top_k (int) – The number of Tasks (experiments) to return.
Return type

Sequence[Task]
Returns

A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_top_experiments_details

get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)

Return a list of dictionaries of the top performing experiments. Example: [{'task_id': TASK_ID, 'metrics': SCALAR_METRIC_DICT, 'hyper_parameters': HYPER_PARAMETERS},]

Order is based on the controller Objective object.

Parameters
- top_k (int) – The number of Tasks (experiments) to return.
- all_metrics (bool) – If False (default), only the objective metric is included in the metrics dictionary. If True, all scalar metrics of the experiment are included.
- all_hyper_parameters (bool) – If True, return all the hyperparameters from all the sections. If False (default), return only the hyperparameters that are part of the optimization search space.
- only_completed (bool) – If True, return only completed Tasks. Default: False.
Return type

Sequence[Union[str, dict]]

Returns

A list of dictionaries ({task_id: '', hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example with all_metrics=False:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    },
]

Example with all_metrics=True:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    },
]

get_top_experiments_id_metrics_pair

get_top_experiments_id_metrics_pair(top_k, all_metrics=False, only_completed=False)

Return a list of pairs (Task ID, scalar metric dict) of the top performing experiments. Order is based on the controller Objective object.

Parameters
- top_k (int) – The number of Tasks (experiments) to return.
- all_metrics (bool) – If False (default), only the objective metric is included in the metrics dictionary. If True, all scalar metrics of the experiment are included.
- only_completed (bool) – If True, return only completed Tasks. Default: False.
Return type

Sequence[Union[str, dict]]

Returns

A list of pairs (Task ID, metric values dict), ordered by performance, where index 0 is the best performing Task. Example with all_metrics=False:

[
    ('0593b76dc7234c65a13a301f731958fa',
        {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    ),
]

Example with all_metrics=True:

[
    ('0593b76dc7234c65a13a301f731958fa',
        {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    ),
]

helper_create_job

helper_create_job(base_task_id, parameter_override=None, task_overrides=None, tags=None, parent=None, **kwargs)

Create a job using the specified arguments. See ClearmlJob for details.

Parameters
- base_task_id (str) – Task ID to clone from.
- parameter_override (Optional[Mapping[str, str]]) – Dictionary of parameter names and values to override on the cloned Task.
- task_overrides (Optional[Mapping[str, str]]) – Task-object-specific overrides. For example: {'script.version_num': None, 'script.branch': 'main'}.
- tags (Optional[Sequence[str]]) – Additional tags to add to the newly created Task.
- parent (Optional[str]) – Parent Task ID for the newly created Task. If not specified, defaults to the optimizer’s default parent Task (see set_job_default_parent).
- kwargs (Any) – Additional arguments passed directly to the Job class constructor (ClearmlJob by default; see set_job_class).
Return type

ClearmlJob
Returns

A newly created Job instance.

monitor_job

monitor_job(job)

Helper function; implementation is optional. Used by the default process_step implementation. Check if the job needs to be aborted or has already completed.

If this returns False, the job was aborted / completed, and should be taken off the current job list.

If there is a budget limitation, this call should update self.budget.compute_time.update / self.budget.iterations.update.

Parameters

job (ClearmlJob) – A ClearmlJob object to monitor.
Return type

bool
Returns

False, if the job is no longer relevant.

process_step

process_step()

Helper function; implementation is optional. This is the main optimization loop used by the default start implementation, called from the daemon thread that start creates.

Call monitor_job on every ClearmlJob in jobs:
- Check the performance or elapsed time, and then decide whether to kill the jobs.
Call create_job
- Check if spare job slots exist, and if they do, create a new job based on previous tested experiments.

Return type

bool
Returns

True, if continuing the optimization. False, if stopping immediately.

set_job_class

set_job_class(job_class)

Set the class to use for the helper_create_job function.

Parameters

job_class (ClearmlJob) – The Job class to use.
Return type

None

set_job_default_parent

set_job_default_parent(job_parent_task_id, project_name=None)

Set the default parent for all Jobs created by the helper_create_job method.

Parameters
- job_parent_task_id (Optional[str]) – The parent Task ID.
- project_name (Optional[str]) – If specified, create the jobs in the specified project.
Return type

None

set_job_naming_scheme

set_job_naming_scheme(naming_function)

Set the function used to name a newly created job.

Parameters

naming_function (Optional[Callable[[str, dict], str]]) – Callable function for naming a newly created job. Use the following format:
```
naming_function(base_task_name, argument_dict) -&gt; str
```
Return type

None

set_optimizer_task

set_optimizer_task(task)

Set the optimizer task object to be used to store/generate reports on the optimization process. Usually this is the current task of this process.

Parameters

task (Task) – The optimizer’s current Task.
Return type

None

start

start()

Start the optimizer controller’s function loop. If the calling process is stopped, the controller will stop as well.

important

This function returns only after the optimization is completed or stop is called.

Return type

None

stop

stop()

Stop the current running optimization loop. Should be called from a different thread than the one running start.

Return type

None

create_job​

get_created_jobs_ids​

get_created_jobs_tasks​

get_objective_metric​

get_running_jobs​

get_top_experiments​

get_top_experiments_details​

get_top_experiments_id_metrics_pair​

helper_create_job​

monitor_job​

process_step​

set_job_class​

set_job_default_parent​

set_job_naming_scheme​

set_optimizer_task​

start​

stop​