Skip to main content

HyperParameterOptimizer

class automation.HyperParameterOptimizer()#

Hyper-parameter search controller. Clones the base experiment, changes arguments and tries to maximize/minimize the defined objective.

Create a new hyper-parameter controller. The newly created object will launch and monitor the new experiments.

  • Parameters

    • base_task_id (str ) โ€“ The Task ID to be used as template experiment to optimize.

    • hyper_parameters (list ) โ€“ The list of Parameter objects to optimize over.

    • objective_metric_title (str ) โ€“ The Objective metric title to maximize / minimize (for example, validation).

    • objective_metric_series (str ) โ€“ The Objective metric series to maximize / minimize (for example, loss).

    • objective_metric_sign (str ) โ€“ The objective to maximize / minimize.

      The values are:

      * `min` - Minimize the last reported value for the specified title/series scalar.
      * `max` - Maximize the last reported value for the specified title/series scalar.
      * `min_global` - Minimize the min value of *all* reported values for the specific title/series scalar.
      * `max_global` - Maximize the max value of *all* reported values for the specific title/series scalar.
    • optimizer_class (class.SearchStrategy ) โ€“ The SearchStrategy optimizer to use for the hyper-parameter search

    • max_number_of_concurrent_tasks (int ) โ€“ The maximum number of concurrent Tasks (experiments) running at the same time.

    • execution_queue (str ) โ€“ The execution queue to use for launching Tasks (experiments).

    • optimization_time_limit (float ) โ€“ The maximum time (minutes) for the entire optimization process. The default is None, indicating no time limit.

    • compute_time_limit (float ) โ€“ The maximum compute time in minutes. When time limit is exceeded, all jobs aborted. (Optional)

    • auto_connect_task (bool ) โ€“ Store optimization arguments and configuration in the Task

      The values are:

      * `True` - The optimization argument and configuration will be stored in the Task. All arguments will

      be under the hyper-parameter section opt, and the optimization hyper_parameters space will stored in the Task configuration object section.

      * `False` - Do not store with Task.
      * `Task` - A specific Task object to connect the optimization process with.
    • always_create_task (bool ) โ€“ Always create a new Task

      The values are:

      * `True` - No current Task initialized. Create a new task named `optimization` in the `base_task_id`

      project.

      * `False` - Use the `task.Task.current_task` (if exists) to report statistics.
    • spawn_project (str ) โ€“ If project name is specified, create all optimization Jobs (Tasks) in the specified project instead of the original base_task_id project.

    • save_top_k_tasks_only (int ) โ€“ If specified and above 0, keep only the top_k performing Tasks, and archive the rest of the created Tasks. Default: -1 keep everything, nothing will be archived.

    • optimizer_kwargs (**** ) โ€“ Arguments passed directly to the optimizer constructor.

      Example:

      :linenos:
      :caption: Example
      from clearml import Task
      from clearml.automation import UniformParameterRange, DiscreteParameterRange
      from clearml.automation import GridSearch, RandomSearch, HyperParameterOptimizer
      task = Task.init('examples', 'HyperParameterOptimizer example')
      an_optimizer = HyperParameterOptimizer(
      base_task_id='fa30fa45d95d4927b87c323b5b04dc44',
      hyper_parameters=[
      UniformParameterRange('lr', min_value=0.01, max_value=0.3, step_size=0.05),
      DiscreteParameterRange('network', values=['ResNet18', 'ResNet50', 'ResNet101']),
      ],
      objective_metric_title='title',
      objective_metric_series='series',
      objective_metric_sign='min',
      max_number_of_concurrent_tasks=5,
      optimizer_class=RandomSearch,
      execution_queue='workers', time_limit_per_job=120, pool_period_min=0.2)
      # This will automatically create and print the optimizer new task id
      # for later use. if a Task was already created, it will use it.
      an_optimizer.set_time_limit(in_minutes=10.)
      an_optimizer.start()
      # we can create a pooling loop if we like
      while not an_optimizer.reached_time_limit():
      top_exp = an_optimizer.get_top_experiments(top_k=3)
      print(top_exp)
      # wait until optimization completed or timed-out
      an_optimizer.wait()
      # make sure we stop all jobs
      an_optimizer.stop()

elapsed#

elapsed()

Return minutes elapsed from controller stating time stamp.

  • Return type

    float

  • Returns

    The minutes from controller start time. A negative value means the process has not started yet.


get_active_experiments#

get_active_experiments()

Return a list of Tasks of the current active experiments.

  • Return type

    Sequence[Task]

  • Returns

    A list of Task objects, representing the current active experiments.


get_num_active_experiments#

get_num_active_experiments()

Return the number of current active experiments.

  • Return type

    int

  • Returns

    The number of active experiments.


get_optimizer#

get_optimizer()

Return the currently used optimizer object.

  • Return type

    SearchStrategy

  • Returns

    The SearchStrategy object used.


HyperParameterOptimizer.get_optimizer_top_experiments#

classmethod get_optimizer_top_experiments(objective_metric_title, objective_metric_series, objective_metric_sign, optimizer_task_id, top_k)

Return a list of Tasks of the top performing experiments for a specific HyperParameter Optimization session (i.e. Task ID), based on the title/series objective.

  • Parameters

    • objective_metric_title (str ) โ€“ The Objective metric title to maximize / minimize (for example, validation).

    • objective_metric_series (str ) โ€“ The Objective metric series to maximize / minimize (for example, loss).

    • objective_metric_sign (str ) โ€“ The objective to maximize / minimize.

      The values are:

      * `min` - Minimize the last reported value for the specified title/series scalar.
      * `max` - Maximize the last reported value for the specified title/series scalar.
      * `min_global` - Minimize the min value of *all* reported values for the specific title/series scalar.
      * `max_global` - Maximize the max value of *all* reported values for the specific title/series scalar.
    • optimizer_task_id (str ) โ€“ Parent optimizer Task ID

    • top_k (int) โ€“ The number of Tasks (experiments) to return.

  • Return type

    Sequence[Task]

  • Returns

    A list of Task objects, ordered by performance, where index 0 is the best performing Task.


get_time_limit#

get_time_limit()

Return the controller optimization time limit.

  • Return type

    datetime

  • Returns

    The absolute datetime limit of the controller optimization process.


get_top_experiments#

get_top_experiments(top_k)

Return a list of Tasks of the top performing experiments, based on the controller Objective object.

  • Parameters

    top_k (int ) โ€“ The number of Tasks (experiments) to return.

  • Return type

    Sequence[Task]

  • Returns

    A list of Task objects, ordered by performance, where index 0 is the best performing Task.


is_active#

is_active()

Is the optimization procedure active (still running)

The values are:

  • True - The optimization procedure is active (still running).

  • False - The optimization procedure is not active (not still running).

info

If the daemon thread has not yet started, is_active returns True.

  • Return type

    bool

  • Returns

    A boolean indicating whether the optimization procedure is active (still running) or stopped.


is_running#

is_running()

Is the optimization controller is running

The values are:

  • True - The optimization procedure is running.

  • False - The optimization procedure is running.

  • Return type

    bool

  • Returns

    A boolean indicating whether the optimization procedure is active (still running) or stopped.


reached_time_limit#

reached_time_limit()

Did the optimizer reach the time limit

The values are:

  • True - The time limit passed.

  • False - The time limit did not pass.

This method returns immediately, it does not wait for the optimizer.

  • Return type

    bool

  • Returns

    True, if optimizer is running and we passed the time limit, otherwise returns False.


set_default_job_class#

set_default_job_class(job_class)

Set the Job class to use when the optimizer spawns new Jobs.

  • Parameters

    job_class (ClearmlJob ) โ€“ The Job Class type.

  • Return type

    ()


set_report_period#

set_report_period(report_period_minutes)

Set reporting period for the accumulated objective report (minutes). This report is sent on the Optimizer Task, and collects the Objective metric from all running jobs.

  • Parameters

    report_period_minutes (float ) โ€“ The reporting period (minutes). The default is once every 10 minutes.

  • Return type

    ()


set_time_limit#

set_time_limit(in_minutes=None, specific_time=None)

Set a time limit for the HyperParameterOptimizer controller. If we reached the time limit, stop the optimization process. If specific_time is provided, use it; otherwise, use the in_minutes.

  • Parameters

    • in_minutes (float ) โ€“ The maximum processing time from current time (minutes).

    • specific_time (datetime ) โ€“ The specific date/time limit.

  • Return type

    ()


start#

start(job_complete_callback=None)

Start the HyperParameterOptimizer controller. If the calling process is stopped, then the controller stops as well.

  • Parameters

    job_complete_callback (Callable ) โ€“ Callback function, called when a job is completed.

    def job_complete_callback(
    job_id, # type: str
    objective_value, # type: float
    objective_iteration, # type: int
    job_parameters, # type: dict
    top_performance_job_id # type: str
    ):
    pass
  • Return type

    bool

  • Returns

    True, if the controller started. False, if the controller did not start.


stop#

stop(timeout=None, wait_for_reporter=True)

Stop the HyperParameterOptimizer controller and the optimization thread.

  • Parameters

    • timeout (float ) โ€“ Wait timeout for the optimization thread to exit (minutes). The default is None, indicating do not wait terminate immediately.

    • wait_for_reporter (Optional [ bool ] ) โ€“ Wait for reporter to flush data.

  • Return type

    ()


wait#

wait(timeout=None)

Wait for the optimizer to finish.

info

This method does not stop the optimizer. Call stop to terminate the optimizer.

  • Parameters

    timeout (float ) โ€“ The timeout to wait for the optimization to complete (minutes). If None, then wait until we reached the timeout, or optimization completed.

  • Return type

    bool

  • Returns

    True, if the optimization finished. False, if the optimization timed out.