OptimizerBOHB

class automation.hpbandster.OptimizerBOHB()

Initialize a BOHB search strategy optimizer BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.

info

For reference: @InProceedings{falkner-icml-18,

title = {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale}, author = {Falkner, Stefan and Klein, Aaron and Hutter, Frank}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1436–1445}, year = {2018},

}

Parameters
- base_task_id (str ) – Task ID (str)
- hyper_parameters (list ) – list of Parameter objects to optimize over
- objective_metric (Objective ) – Objective metric to maximize / minimize
- execution_queue (str ) – execution queue to use for launching Tasks (experiments).
- num_concurrent_workers (int ) – Limit number of concurrent running Tasks (machines)
- min_iteration_per_job (int ) – minimum number of iterations for a job to run. ‘iterations’ are the reported iterations for the specified objective, not the maximum reported iteration of the Task.
- max_iteration_per_job (int ) – number of iteration per job ‘iterations’ are the reported iterations for the specified objective, not the maximum reported iteration of the Task.
- total_max_jobs (int ) – total maximum job for the optimization process. Must be provided in order to calculate the total budget for the optimization process. The total budget is measured by “iterations” (see above) and will be set to max_iteration_per_job * total_max_jobs This means more than total_max_jobs could be created, as long as the cumulative iterations (summed over all created jobs) will not exceed max_iteration_per_job * total_max_jobs
- pool_period_min (float ) – time in minutes between two consecutive pools
- time_limit_per_job (float ) – Optional, maximum execution time per single job in minutes, when time limit is exceeded job is aborted
- compute_time_limit (float ) – The maximum compute time in minutes. When time limit is exceeded, all jobs aborted. (Optional)
- local_port (int ) – default port 9090 tcp, this is a must for the BOHB workers to communicate, even locally.
- bohb_kwargs (Any) – arguments passed directly to the BOHB object

create_job

create_job()

Abstract helper function. Implementation is not required. Default use in process_step default implementation Create a new job if needed. return the newly created job. If no job needs to be created, return None.

Return type
Optional[ClearmlJob]
Returns
A Newly created ClearmlJob object, or None if no ClearmlJob created.

get_created_jobs_ids

get_created_jobs_ids()

Return a Task IDs dict created by this optimizer until now, including completed and running jobs. The values of the returned dict are the parameters used in the specific job

Return type
Mapping[str, dict]
Returns
dict of task IDs (str) as keys, and their parameters dict as values.

get_created_jobs_tasks

get_created_jobs_tasks()

Return a Task IDs dict created by this optimizer until now. The values of the returned dict are the ClearmlJob.

Return type
Mapping[str, dict]
Returns
dict of task IDs (str) as keys, and their ClearmlJob as values.

get_objective_metric

get_objective_metric()

Return the metric title, series pair of the objective.

Return type
Union[Tuple[str, str], List[Tuple[str, str]]]
Returns
(title, series)

get_random_seed

static get_random_seed()

Get the global seed for all hyperparameter strategy random number sampling.

Return type
int
Returns
The random seed.

get_running_jobs

get_running_jobs()

Return the current running ClearmlJob.

Return type
Sequence[ClearmlJob]
Returns
List of ClearmlJob objects.

get_top_experiments

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

Parameters
top_k (int ) – The number of Tasks (experiments) to return.
Return type
Sequence[Task]
Returns
A list of Task objects, ordered by performance, where index 0 is the best performing Task.

get_top_experiments_details

get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)

Return a list of dictionaries of the top performing experiments. Example: [{'task_id': Task-ID, 'metrics': scalar-metric-dict, 'hyper_parameters': Hyper-Parameters},]

Order is based on the controller Objective object.

Parameters
- top_k (int ) – The number of Tasks (experiments) to return.
- all_metrics (bool ) – Default False, only return the objective metric on the metrics dictionary. If True, return all scalar metrics of the experiment
- all_hyper_parameters (bool ) – Default False. If True, return all the hyperparameters from all the sections.
- only_completed (bool ) – return only completed Tasks. Default False.
Return type
Sequence[(str, dict)]

Returns

A list of dictionaries ({task_id: ‘’, hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example w/ all_metrics=False:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    },
]

Example w/ all_metrics=True:

[
    {
        task_id: '0593b76dc7234c65a13a301f731958fa',
        hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
        metrics: {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    },
]

get_top_experiments_id_metrics_pair

get_top_experiments_id_metrics_pair(top_k, all_metrics=False, only_completed=False)

Return a list of pairs (Task ID, scalar metric dict) of the top performing experiments. Order is based on the controller Objective object.

Parameters
- top_k (int ) – The number of Tasks (experiments) to return.
- all_metrics (bool ) – Default False, only return the objective metric on the metrics dictionary. If True, return all scalar metrics of the experiment
- only_completed (bool ) – return only completed Tasks. Default False.
Return type
Sequence[(str, dict)]
Returns
A list of pairs (Task ID, metric values dict), ordered by performance,

where index 0 is the best performing Task. Example w/ all_metrics=False:

[
    ('0593b76dc7234c65a13a301f731958fa',
        {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
        }
    ),
]

Example w/ all_metrics=True:

[
    ('0593b76dc7234c65a13a301f731958fa',
        {
            'accuracy per class/cat': {
                'metric': 'accuracy per class',
                'variant': 'cat',
                'value': 0.119,
                'min_value': 0.119,
                'max_value': 0.782
            },
            'accuracy per class/deer': {
                'metric': 'accuracy per class',
                'variant': 'deer',
                'value': 0.219,
                'min_value': 0.219,
                'max_value': 0.282
            },
        }
    ),
]

helper_create_job

helper_create_job(base_task_id, parameter_override=None, task_overrides=None, tags=None, parent=None, kwargs)**

Create a Job using the specified arguments, ClearmlJob for details.

Return type
ClearmlJob
Returns
A newly created Job instance.
Parameters
- base_task_id (str ) –
- parameter_override (Optional [ Mapping [ str , str ] ] ) –
- task_overrides (Optional [ Mapping [ str , str ] ] ) –
- tags (Optional [ Sequence [ str ] ] ) –
- parent (Optional [ str ] ) –
- kwargs (Any ) –

monitor_job

monitor_job(job)

Helper function, Implementation is not required. Default use in process_step default implementation. Check if the job needs to be aborted or already completed.

If returns False, the job was aborted / completed, and should be taken off the current job list.

If there is a budget limitation, this call should update self.budget.compute_time.update / self.budget.iterations.update

Parameters
job (ClearmlJob ) – A ClearmlJob object to monitor.
Return type
bool
Returns
False, if the job is no longer relevant.

process_step

process_step()

Abstract helper function. Implementation is not required. Default use in start default implementation Main optimization loop, called from the daemon thread created by start.

Call monitor job on every ClearmlJob in jobs:
- Check the performance or elapsed time, and then decide whether to kill the jobs.
Call create_job:
- Check if spare job slots exist, and if they do call create a new job based on previous tested experiments.

Return type
bool
Returns
True, if continue the optimization. False, if immediately stop.

set_job_class

set_job_class(job_class)

Set the class to use for the helper_create_job function.

Parameters
job_class (ClearmlJob ) – The Job Class type.
Return type
()

set_job_default_parent

set_job_default_parent(job_parent_task_id, project_name=None)

Set the default parent for all Jobs created by the helper_create_job method.

Parameters
- job_parent_task_id (str ) – The parent Task ID.
- project_name (str ) – If specified, create the jobs in the specified project
Return type
()

set_job_naming_scheme

set_job_naming_scheme(naming_function)

Set the function used to name a newly created job.

Parameters
naming_function (callable ) – Callable function for naming a newly created job. Use the following format:
```
naming_functor(base_task_name, argument_dict) -&gt; str
```
Return type
()

set_optimization_args

set_optimization_args(eta=3, min_budget=None, max_budget=None, min_points_in_model=None, top_n_percent=15, num_samples=None, random_fraction=0.3333333333333333, bandwidth_factor=3, min_bandwidth=0.001)

Defaults copied from BOHB constructor, see details in BOHB.init

BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.

info

For reference: @InProceedings{falkner-icml-18,

}

Parameters
- eta (float ) – float (3) In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them ‘advances’ to the next round. Must be greater or equal to 2.
- min_budget (Optional [ float ] ) – float (0.01) The smallest budget to consider. Needs to be positive!
- max_budget (Optional [ float ] ) – float (1) The largest budget to consider. Needs to be larger than min_budget! The budgets will be geometrically distributed $a^2 + b^2 = c^2 /sim /eta^k$ for $k/in [0, 1, ... , num/_subsets - 1]$.
- min_points_in_model (Optional [ int ] ) – int (None) number of observations to start building a KDE. Default ‘None’ means dim+1, the bare minimum.
- top_n_percent (Optional [ int ] ) – int (15) percentage ( between 1 and 99, default 15) of the observations that are considered good.
- num_samples (Optional [ int ] ) – int (64) number of samples to optimize EI (default 64)
- random_fraction (Optional [ float ] ) – float (1/3.) fraction of purely random configurations that are sampled from the prior without the model.
- bandwidth_factor (Optional [ float ] ) – float (3.) to encourage diversity, the points proposed to optimize EI, are sampled from a ‘widened’ KDE where the bandwidth is multiplied by this factor (default: 3)
- min_bandwidth (Optional [ float ] ) – float (1e-3) to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (Default: 1e-3) is used instead of zero.
Return type
()

set_optimizer_task

set_optimizer_task(task)

Set the optimizer task object to be used to store/generate reports on the optimization process. Usually this is the current task of this process.

Parameters
task (Task ) – The optimizer`s current Task.
Return type
()

set_random_seed

static set_random_seed(seed=1337)

Set global seed for all hyperparameter strategy random number sampling.

Parameters
seed (int ) – The random seed.
Return type
()

start

start()

Start the Optimizer controller function loop() If the calling process is stopped, the controller will stop as well.

info

This function returns only after optimization is completed or stop was called.

Return type
()

stop

stop()

Stop the current running optimization loop, Called from a different thread than the start.

Return type
()

class automation.hpbandster.OptimizerBOHB()​

create_job​

get_created_jobs_ids​

get_created_jobs_tasks​

get_objective_metric​

get_random_seed​

get_running_jobs​

get_top_experiments​

get_top_experiments_details​

get_top_experiments_id_metrics_pair​

helper_create_job​

monitor_job​

process_step​

set_job_class​

set_job_default_parent​

set_job_naming_scheme​

set_optimization_args​

set_optimizer_task​

set_random_seed​

start​

stop​

class automation.hpbandster.OptimizerBOHB()

create_job

get_created_jobs_ids

get_created_jobs_tasks

get_objective_metric

get_random_seed

get_running_jobs

get_top_experiments

get_top_experiments_details

get_top_experiments_id_metrics_pair

helper_create_job

monitor_job

process_step

set_job_class

set_job_default_parent

set_job_naming_scheme

set_optimization_args

set_optimizer_task

set_random_seed

start

stop