OptimizerBOHB
class automation.hpbandster.OptimizerBOHB(base_task_id, hyper_parameters, objective_metric, execution_queue, num_concurrent_workers, min_iteration_per_job, max_iteration_per_job, total_max_jobs, pool_period_min=2.0, time_limit_per_job=None, compute_time_limit=None, local_port=9090, **bohb_kwargs)
Initialize a BOHB search strategy optimizer. BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.
For reference:
@InProceedings{falkner-icml-18,
title = {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale},
author = {Falkner, Stefan and Klein, Aaron and Hutter, Frank},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {1436--1445},
year = {2018},
}
-
Parameters
-
base_task_id (
str) – Task ID to be used as a template task to optimize. -
hyper_parameters (
Sequence[Parameter]) – List of Parameter objects to optimize over. -
objective_metric (
Objective) – Objective metric to maximize / minimize. -
execution_queue (
str) – Execution queue to use for launching Tasks (experiments). -
num_concurrent_workers (
int) – Maximum number of concurrent running Tasks (machines). -
min_iteration_per_job (
Optional[int]) – Minimum number of iterations for a job to run.iterationsare the reported iterations for the specified objective, not the maximum reported iteration of the Task. -
max_iteration_per_job (
Optional[int]) – Maximum number of iterations per job.iterationsare the reported iterations for the specified objective, not the maximum reported iteration of the Task. -
total_max_jobs (
Optional[int]) – Total maximum jobs for the optimization process. Must be provided in order to calculate the total budget for the optimization process The total budget is measured initerations(see above) and will be set tomax_iteration_per_job \* total_max_jobs. This means more thantotal_max_jobscould be created, as long as the cumulative iterations (summed over all created jobs) do not exceedmax_iteration_per_job \* total_max_jobs. -
pool_period_min (
float) – Time in minutes between two consecutive pools. -
time_limit_per_job (
Optional[float]) – Maximum execution time per single job in minutes. When the time limit is exceeded, the job is aborted. -
compute_time_limit (
Optional[float]) – Maximum compute time in minutes. When the time limit is exceeded, all jobs are aborted. -
local_port (
int) – Default port is9090(tcp). Required for the BOHB workers to communicate, even locally. -
bohb_kwargs (
Any) – Arguments passed directly to the BOHB object.
-
create_job
create_job()
Helper function; implementation is optional. Used by the default process_step implementation.
Create a new job if needed and return it. Return None if no job needs to be created.
-
Return type
Optional[ClearmlJob] -
Returns
A newly created
ClearmlJobobject, orNoneif no job was created.
get_created_jobs_ids
get_created_jobs_ids()
Return a dict of Task IDs created by this optimizer so far, including completed and running jobs. The values of the returned dict are the parameters used in the specific job.
-
Return type
Mapping[str,dict] -
Returns
A dict of Task IDs as keys, and their parameters dict as values.
get_created_jobs_tasks
get_created_jobs_tasks()
Return a dict of Task IDs created by this optimizer so far.
The values of the returned dict are the ClearmlJob objects.
-
Return type
Mapping[str,dict] -
Returns
A dict of Task IDs as keys, and their
ClearmlJobobjects as values.
get_objective_metric
get_objective_metric()
Return the metric title/series pair(s) of the objective.
-
Return type
Union[Tuple[str,str],List[Tuple[str,str]]] -
Returns
A single
(title, series)tuple for a single-objective optimization, or a list of(title, series)tuples for a multi-objective optimization.
get_random_seed
static get_random_seed()
Get the global seed for all hyperparameter strategy random number sampling.
-
Return type
int -
Returns
The random seed.
get_running_jobs
get_running_jobs()
Return the current running ClearmlJob objects.
-
Return type
Sequence[ClearmlJob] -
Returns
List of
ClearmlJobobjects.
get_top_experiments
get_top_experiments(top_k)
Return a list of Tasks of the top performing experiments, based on the controller Objective object.
-
Parameters
top_k (
int) – The number of Tasks (experiments) to return. -
Return type
Sequence[Task] -
Returns
A list of Task objects, ordered by performance, where index 0 is the best performing Task.
get_top_experiments_details
get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)
Return a list of dictionaries of the top performing experiments.
Example: [{'task_id': TASK_ID, 'metrics': SCALAR_METRIC_DICT, 'hyper_parameters': HYPER_PARAMETERS},]
Order is based on the controller Objective object.
-
Parameters
-
top_k (
int) – The number of Tasks (experiments) to return. -
all_metrics (
bool) – IfFalse(default), only the objective metric is included in the metrics dictionary. IfTrue, all scalar metrics of the experiment are included. -
all_hyper_parameters (
bool) – IfTrue, return all the hyperparameters from all the sections. IfFalse(default), return only the hyperparameters that are part of the optimization search space. -
only_completed (
bool) – IfTrue, return only completed Tasks. Default:False.
-
-
Return type
Sequence[Union[str,dict]] -
Returns
A list of dictionaries
({task_id: '', hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example withall_metrics=False:[{task_id: '0593b76dc7234c65a13a301f731958fa',hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},metrics: {'accuracy per class/cat': {'metric': 'accuracy per class','variant': 'cat','value': 0.119,'min_value': 0.119,'max_value': 0.782},}},]Example with
all_metrics=True:[{task_id: '0593b76dc7234c65a13a301f731958fa',hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},metrics: {'accuracy per class/cat': {'metric': 'accuracy per class','variant': 'cat','value': 0.119,'min_value': 0.119,'max_value': 0.782},'accuracy per class/deer': {'metric': 'accuracy per class','variant': 'deer','value': 0.219,'min_value': 0.219,'max_value': 0.282},}},]
get_top_experiments_id_metrics_pair
get_top_experiments_id_metrics_pair(top_k, all_metrics=False, only_completed=False)
Return a list of pairs (Task ID, scalar metric dict) of the top performing experiments.
Order is based on the controller Objective object.
-
Parameters
-
top_k (
int) – The number of Tasks (experiments) to return. -
all_metrics (
bool) – IfFalse(default), only the objective metric is included in the metrics dictionary. IfTrue, all scalar metrics of the experiment are included. -
only_completed (
bool) – IfTrue, return only completed Tasks. Default:False.
-
-
Return type
Sequence[Union[str,dict]] -
Returns
A list of pairs (Task ID, metric values dict), ordered by performance, where index 0 is the best performing Task. Example with
all_metrics=False:[('0593b76dc7234c65a13a301f731958fa',{'accuracy per class/cat': {'metric': 'accuracy per class','variant': 'cat','value': 0.119,'min_value': 0.119,'max_value': 0.782},}),]Example with
all_metrics=True:[('0593b76dc7234c65a13a301f731958fa',{'accuracy per class/cat': {'metric': 'accuracy per class','variant': 'cat','value': 0.119,'min_value': 0.119,'max_value': 0.782},'accuracy per class/deer': {'metric': 'accuracy per class','variant': 'deer','value': 0.219,'min_value': 0.219,'max_value': 0.282},}),]
helper_create_job
helper_create_job(base_task_id, parameter_override=None, task_overrides=None, tags=None, parent=None, **kwargs)
Create a job using the specified arguments. See ClearmlJob for details.
-
Parameters
-
base_task_id (
str) – Task ID to clone from. -
parameter_override (
Optional[Mapping[str,str]]) – Dictionary of parameter names and values to override on the cloned Task. -
task_overrides (
Optional[Mapping[str,str]]) – Task-object-specific overrides. For example:{'script.version_num': None, 'script.branch': 'main'}. -
tags (
Optional[Sequence[str]]) – Additional tags to add to the newly created Task. -
parent (
Optional[str]) – Parent Task ID for the newly created Task. If not specified, defaults to the optimizer’s default parent Task (seeset_job_default_parent). -
kwargs (
Any) – Additional arguments passed directly to the Job class constructor (ClearmlJobby default; seeset_job_class).
-
-
Return type
-
Returns
A newly created Job instance.
monitor_job
monitor_job(job)
Helper function; implementation is optional. Used by the default process_step implementation.
Check if the job needs to be aborted or has already completed.
If this returns False, the job was aborted / completed, and should be taken off the current job list.
If there is a budget limitation, this call should update
self.budget.compute_time.update / self.budget.iterations.update.
-
Parameters
job (
ClearmlJob) – AClearmlJobobject to monitor. -
Return type
bool -
Returns
False, if the job is no longer relevant.
process_step
process_step()
Helper function; implementation is optional. This is the main optimization loop used by the default
start implementation, called from the daemon thread that start creates.
-
Call
monitor_jobon everyClearmlJobin jobs:- Check the performance or elapsed time, and then decide whether to kill the jobs.
-
Call
create_job- Check if spare job slots exist, and if they do, create a new job based on previous tested experiments.
-
Return type
bool -
Returns
True, if continuing the optimization.False, if stopping immediately.
set_job_class
set_job_class(job_class)
Set the class to use for the helper_create_job function.
-
Parameters
job_class (
ClearmlJob) – The Job class to use. -
Return type
None
set_job_default_parent
set_job_default_parent(job_parent_task_id, project_name=None)
Set the default parent for all Jobs created by the helper_create_job method.
-
Parameters
-
job_parent_task_id (
Optional[str]) – The parent Task ID. -
project_name (
Optional[str]) – If specified, create the jobs in the specified project.
-
-
Return type
None
set_job_naming_scheme
set_job_naming_scheme(naming_function)
Set the function used to name a newly created job.
-
Parameters
naming_function (
Optional[Callable[[str,dict],str]]) – Callable function for naming a newly created job. Use the following format:naming_function(base_task_name, argument_dict) -> str -
Return type
None
set_optimization_args
set_optimization_args(eta=3, min_budget=None, max_budget=None, min_points_in_model=None, top_n_percent=15, num_samples=None, random_fraction=0.3333333333333333, bandwidth_factor=3, min_bandwidth=0.001)
The defaults are copied from the BOHB constructor; see BOHB.__init__ for details.
BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.
For reference:
@InProceedings{falkner-icml-18,
title = {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale},
author = {Falkner, Stefan and Klein, Aaron and Hutter, Frank},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {1436--1445},
year = {2018},
}
-
Parameters
-
eta (
float) – In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of1/etaof them advances to the next round. Must be greater than or equal to2. Default:3. -
min_budget (
Optional[float]) – Smallest budget to consider. Needs to be positive. If not specified, BOHB’s own default (0.01) is used. -
max_budget (
Optional[float]) – Largest budget to consider. Needs to be larger thanmin_budget. If not specified, BOHB’s own default (1) is used. The budgets are geometrically distributed across the brackets, spaced by a factor ofeta. -
min_points_in_model (
Optional[int]) – Number of observations needed to start building a KDE. IfNone(default), usedim + 1, the bare minimum. -
top_n_percent (
Optional[int]) – Percentage (between1and99) of the observations that are considered good. Default:15. -
num_samples (
Optional[int]) – Number of samples used to optimize EI. Default:64. -
random_fraction (
Optional[float]) – Fraction of purely random configurations sampled from the prior, without the model. Default:1/3. -
bandwidth_factor (
Optional[float]) – To encourage diversity, the points proposed to optimize EI are sampled from a widened KDE, where the bandwidth is multiplied by this factor. Default:3. -
min_bandwidth (
Optional[float]) – To keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth is used instead of zero. Default:1e-3.
-
-
Return type
None
set_optimizer_task
set_optimizer_task(task)
Set the optimizer task object to be used to store/generate reports on the optimization process. Usually this is the current task of this process.
-
Parameters
task (
Task) – The optimizer’s current Task. -
Return type
None
set_random_seed
static set_random_seed(seed=1337)
Set global seed for all hyperparameter strategy random number sampling.
-
Parameters
seed (int ) – The random seed.
-
Return type
None
start
start()
Start the optimizer controller’s function loop. If the calling process is stopped, the controller will stop as well.
This function returns only after the optimization is completed or stop is called.
-
Return type
None
stop
stop()
Stop the current running optimization loop. Should be called from a different thread than the one
running start.
-
Return type
None