Skip to main content

OptimizerBOHB

class automation.hpbandster.OptimizerBOHB(base_task_id, hyper_parameters, objective_metric, execution_queue, num_concurrent_workers, min_iteration_per_job, max_iteration_per_job, total_max_jobs, pool_period_min=2.0, time_limit_per_job=None, compute_time_limit=None, local_port=9090, **bohb_kwargs)​

Initialize a BOHB search strategy optimizer. BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.

For reference:
@InProceedings{falkner-icml-18,
title = {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale},
author = {Falkner, Stefan and Klein, Aaron and Hutter, Frank},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {1436--1445},
year = {2018},
}
  • Parameters

    • base_task_id (str) – Task ID to be used as a template task to optimize.

    • hyper_parameters (Sequence[Parameter]) – List of Parameter objects to optimize over.

    • objective_metric (Objective) – Objective metric to maximize / minimize.

    • execution_queue (str) – Execution queue to use for launching Tasks (experiments).

    • num_concurrent_workers (int) – Maximum number of concurrent running Tasks (machines).

    • min_iteration_per_job (Optional[int]) – Minimum number of iterations for a job to run. iterations are the reported iterations for the specified objective, not the maximum reported iteration of the Task.

    • max_iteration_per_job (Optional[int]) – Maximum number of iterations per job. iterations are the reported iterations for the specified objective, not the maximum reported iteration of the Task.

    • total_max_jobs (Optional[int]) – Total maximum jobs for the optimization process. Must be provided in order to calculate the total budget for the optimization process The total budget is measured in iterations (see above) and will be set to max_iteration_per_job \* total_max_jobs. This means more than total_max_jobs could be created, as long as the cumulative iterations (summed over all created jobs) do not exceed max_iteration_per_job \* total_max_jobs.

    • pool_period_min (float) – Time in minutes between two consecutive pools.

    • time_limit_per_job (Optional[float]) – Maximum execution time per single job in minutes. When the time limit is exceeded, the job is aborted.

    • compute_time_limit (Optional[float]) – Maximum compute time in minutes. When the time limit is exceeded, all jobs are aborted.

    • local_port (int) – Default port is 9090 (tcp). Required for the BOHB workers to communicate, even locally.

    • bohb_kwargs (Any) – Arguments passed directly to the BOHB object.


create_job​

create_job()

Helper function; implementation is optional. Used by the default process_step implementation. Create a new job if needed and return it. Return None if no job needs to be created.

  • Return type

    Optional[ClearmlJob]

  • Returns

    A newly created ClearmlJob object, or None if no job was created.


get_created_jobs_ids​

get_created_jobs_ids()

Return a dict of Task IDs created by this optimizer so far, including completed and running jobs. The values of the returned dict are the parameters used in the specific job.

  • Return type

    Mapping[str, dict]

  • Returns

    A dict of Task IDs as keys, and their parameters dict as values.


get_created_jobs_tasks​

get_created_jobs_tasks()

Return a dict of Task IDs created by this optimizer so far. The values of the returned dict are the ClearmlJob objects.

  • Return type

    Mapping[str, dict]

  • Returns

    A dict of Task IDs as keys, and their ClearmlJob objects as values.


get_objective_metric​

get_objective_metric()

Return the metric title/series pair(s) of the objective.

  • Return type

    Union[Tuple[str, str], List[Tuple[str, str]]]

  • Returns

    A single (title, series) tuple for a single-objective optimization, or a list of (title, series) tuples for a multi-objective optimization.


get_random_seed​

static get_random_seed()

Get the global seed for all hyperparameter strategy random number sampling.

  • Return type

    int

  • Returns

    The random seed.


get_running_jobs​

get_running_jobs()

Return the current running ClearmlJob objects.

  • Return type

    Sequence[ClearmlJob]

  • Returns

    List of ClearmlJob objects.


get_top_experiments​

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

  • Parameters

    top_k (int) – The number of Tasks (experiments) to return.

  • Return type

    Sequence[Task]

  • Returns

    A list of Task objects, ordered by performance, where index 0 is the best performing Task.


get_top_experiments_details​

get_top_experiments_details(top_k, all_metrics=False, all_hyper_parameters=False, only_completed=False)

Return a list of dictionaries of the top performing experiments. Example: [{'task_id': TASK_ID, 'metrics': SCALAR_METRIC_DICT, 'hyper_parameters': HYPER_PARAMETERS},]

Order is based on the controller Objective object.

  • Parameters

    • top_k (int) – The number of Tasks (experiments) to return.

    • all_metrics (bool) – If False (default), only the objective metric is included in the metrics dictionary. If True, all scalar metrics of the experiment are included.

    • all_hyper_parameters (bool) – If True, return all the hyperparameters from all the sections. If False (default), return only the hyperparameters that are part of the optimization search space.

    • only_completed (bool) – If True, return only completed Tasks. Default: False.

  • Return type

    Sequence[Union[str, dict]]

  • Returns

    A list of dictionaries ({task_id: '', hyper_parameters: {}, metrics: {}}), ordered by performance, where index 0 is the best performing Task. Example with all_metrics=False:

    [
    {
    task_id: '0593b76dc7234c65a13a301f731958fa',
    hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
    metrics: {
    'accuracy per class/cat': {
    'metric': 'accuracy per class',
    'variant': 'cat',
    'value': 0.119,
    'min_value': 0.119,
    'max_value': 0.782
    },
    }
    },
    ]

    Example with all_metrics=True:

    [
    {
    task_id: '0593b76dc7234c65a13a301f731958fa',
    hyper_parameters: {'General/lr': '0.03', 'General/batch_size': '32'},
    metrics: {
    'accuracy per class/cat': {
    'metric': 'accuracy per class',
    'variant': 'cat',
    'value': 0.119,
    'min_value': 0.119,
    'max_value': 0.782
    },
    'accuracy per class/deer': {
    'metric': 'accuracy per class',
    'variant': 'deer',
    'value': 0.219,
    'min_value': 0.219,
    'max_value': 0.282
    },
    }
    },
    ]

get_top_experiments_id_metrics_pair​

get_top_experiments_id_metrics_pair(top_k, all_metrics=False, only_completed=False)

Return a list of pairs (Task ID, scalar metric dict) of the top performing experiments. Order is based on the controller Objective object.

  • Parameters

    • top_k (int) – The number of Tasks (experiments) to return.

    • all_metrics (bool) – If False (default), only the objective metric is included in the metrics dictionary. If True, all scalar metrics of the experiment are included.

    • only_completed (bool) – If True, return only completed Tasks. Default: False.

  • Return type

    Sequence[Union[str, dict]]

  • Returns

    A list of pairs (Task ID, metric values dict), ordered by performance, where index 0 is the best performing Task. Example with all_metrics=False:

    [
    ('0593b76dc7234c65a13a301f731958fa',
    {
    'accuracy per class/cat': {
    'metric': 'accuracy per class',
    'variant': 'cat',
    'value': 0.119,
    'min_value': 0.119,
    'max_value': 0.782
    },
    }
    ),
    ]

    Example with all_metrics=True:

    [
    ('0593b76dc7234c65a13a301f731958fa',
    {
    'accuracy per class/cat': {
    'metric': 'accuracy per class',
    'variant': 'cat',
    'value': 0.119,
    'min_value': 0.119,
    'max_value': 0.782
    },
    'accuracy per class/deer': {
    'metric': 'accuracy per class',
    'variant': 'deer',
    'value': 0.219,
    'min_value': 0.219,
    'max_value': 0.282
    },
    }
    ),
    ]

helper_create_job​

helper_create_job(base_task_id, parameter_override=None, task_overrides=None, tags=None, parent=None, **kwargs)

Create a job using the specified arguments. See ClearmlJob for details.

  • Parameters

    • base_task_id (str) – Task ID to clone from.

    • parameter_override (Optional[Mapping[str, str]]) – Dictionary of parameter names and values to override on the cloned Task.

    • task_overrides (Optional[Mapping[str, str]]) – Task-object-specific overrides. For example: {'script.version_num': None, 'script.branch': 'main'}.

    • tags (Optional[Sequence[str]]) – Additional tags to add to the newly created Task.

    • parent (Optional[str]) – Parent Task ID for the newly created Task. If not specified, defaults to the optimizer’s default parent Task (see set_job_default_parent).

    • kwargs (Any) – Additional arguments passed directly to the Job class constructor (ClearmlJob by default; see set_job_class).

  • Return type

    ClearmlJob

  • Returns

    A newly created Job instance.


monitor_job​

monitor_job(job)

Helper function; implementation is optional. Used by the default process_step implementation. Check if the job needs to be aborted or has already completed.

If this returns False, the job was aborted / completed, and should be taken off the current job list.

If there is a budget limitation, this call should update self.budget.compute_time.update / self.budget.iterations.update.

  • Parameters

    job (ClearmlJob) – A ClearmlJob object to monitor.

  • Return type

    bool

  • Returns

    False, if the job is no longer relevant.


process_step​

process_step()

Helper function; implementation is optional. This is the main optimization loop used by the default start implementation, called from the daemon thread that start creates.

  • Call monitor_job on every ClearmlJob in jobs:

    • Check the performance or elapsed time, and then decide whether to kill the jobs.
  • Call create_job

    • Check if spare job slots exist, and if they do, create a new job based on previous tested experiments.
  • Return type

    bool

  • Returns

    True, if continuing the optimization. False, if stopping immediately.


set_job_class​

set_job_class(job_class)

Set the class to use for the helper_create_job function.

  • Parameters

    job_class (ClearmlJob) – The Job class to use.

  • Return type

    None


set_job_default_parent​

set_job_default_parent(job_parent_task_id, project_name=None)

Set the default parent for all Jobs created by the helper_create_job method.

  • Parameters

    • job_parent_task_id (Optional[str]) – The parent Task ID.

    • project_name (Optional[str]) – If specified, create the jobs in the specified project.

  • Return type

    None


set_job_naming_scheme​

set_job_naming_scheme(naming_function)

Set the function used to name a newly created job.

  • Parameters

    naming_function (Optional[Callable[[str, dict], str]]) – Callable function for naming a newly created job. Use the following format:

    naming_function(base_task_name, argument_dict) -> str
  • Return type

    None


set_optimization_args​

set_optimization_args(eta=3, min_budget=None, max_budget=None, min_points_in_model=None, top_n_percent=15, num_samples=None, random_fraction=0.3333333333333333, bandwidth_factor=3, min_bandwidth=0.001)

The defaults are copied from the BOHB constructor; see BOHB.__init__ for details.

BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. Instead of sampling new configurations at random, BOHB uses kernel density estimators to select promising candidates.

For reference:
@InProceedings{falkner-icml-18,
title = {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale},
author = {Falkner, Stefan and Klein, Aaron and Hutter, Frank},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
pages = {1436--1445},
year = {2018},
}
  • Parameters

    • eta (float) – In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them advances to the next round. Must be greater than or equal to 2. Default: 3.

    • min_budget (Optional[float]) – Smallest budget to consider. Needs to be positive. If not specified, BOHB’s own default (0.01) is used.

    • max_budget (Optional[float]) – Largest budget to consider. Needs to be larger than min_budget. If not specified, BOHB’s own default (1) is used. The budgets are geometrically distributed across the brackets, spaced by a factor of eta.

    • min_points_in_model (Optional[int]) – Number of observations needed to start building a KDE. If None (default), use dim + 1, the bare minimum.

    • top_n_percent (Optional[int]) – Percentage (between 1 and 99) of the observations that are considered good. Default: 15.

    • num_samples (Optional[int]) – Number of samples used to optimize EI. Default: 64.

    • random_fraction (Optional[float]) – Fraction of purely random configurations sampled from the prior, without the model. Default: 1/3.

    • bandwidth_factor (Optional[float]) – To encourage diversity, the points proposed to optimize EI are sampled from a widened KDE, where the bandwidth is multiplied by this factor. Default: 3.

    • min_bandwidth (Optional[float]) – To keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth is used instead of zero. Default: 1e-3.

  • Return type

    None


set_optimizer_task​

set_optimizer_task(task)

Set the optimizer task object to be used to store/generate reports on the optimization process. Usually this is the current task of this process.

  • Parameters

    task (Task) – The optimizer’s current Task.

  • Return type

    None


set_random_seed​

static set_random_seed(seed=1337)

Set global seed for all hyperparameter strategy random number sampling.

  • Parameters

    seed (int ) – The random seed.

  • Return type

    None


start​

start()

Start the optimizer controller’s function loop. If the calling process is stopped, the controller will stop as well.

important

This function returns only after the optimization is completed or stop is called.

  • Return type

    None


stop​

stop()

Stop the current running optimization loop. Should be called from a different thread than the one running start.

  • Return type

    None