Hyperparameters are the configuration options given for a script. ClearML logs hyperparameters used in experiments from multiple different sources.
In ClearML, parameters are split into 3 sections:
- User Properties - Modifiable section that can be edited post execution.
- Hyperparameters - Individual parameters for configuration
- Configuration Objects - Usually configuration files (Json \ YAML) or python objects.
These sections are further broken down into sub-sections (General \ Args \ TF_Define) for convenience.
Parameters passed to experiments, using Python's built-in argparser module, are automatically captured by ClearML, so no code changes are needed.
Users can directly connect objects, such as dictionaries or even custom classes, to Tasks. All class members will be automatically fetched and logged by ClearML.
- Connecting a class:
- Connecting a dictionary:
User properties are an editable key / value store, which enables adding information to an experiment, making it easier to search / filter. User properties are like parameters that can also be added after a Task's execution, which can also be added to an experiment table (i.e. customize columns).
The above example will connect to the Task a user property named "backbone", with the description "network type", and the value of "great".
Relying on environment variables makes an experiment not fully reproducible, since ClearML Agent can't reproduce them at runtime.
Environment variables can be logged by modifying the clearml.conf file. Modify the log_os_environments parameter specifying parameters to log.
log_os_environments: ["AWS_*", "CUDA_VERSION"]
It's also possible to specify environment variables using the CLEARML_LOG_ENVIRONMENT variable.
The CLEARML_LOG_ENVIRONMENT always overrides the clearml.conf file.
ClearML automatically captures TFDefine files, which are used as configuration files for Tensorflow.
Hydra is a module developed by FaceBook AI Research to manage experiments' parameters. Hydra offers the best of both worlds, managing configurations with files while making parameters overridable at runtime.
ClearML logs the Omegaconf which holds all the configuration files, as well as overridden values.
Check out the example code, which demonstrates the creation of a configuration object to which configuration values can be added and overridden using the command line.
Configuration objects are dictionaries or configuration files connected to the Task. Unlike Hyperparameters, these are saved as a whole and not divided into individual parameters.
To connect a configuration dictionary:
To connect a configuration file:
Configuration objects can be split into categories in the Configuration section. The "name" argument, is the name of the section that the object will go into. If a section name is not specified, the default section is General.
See here for a detailed example.
In addition to connecting a dictionary or a class to log hyperparameters, users can also use the
to define parameters manually. Parameters are inputted as dictionaries.
Additionally, parameters can be categorized, and each category appears in its own section in the hyperparameter tab of the web UI.
Specify a section by putting its name before the parameter, for example
'Args/epochs': 'value' - 'epochs' will go into the
'Args' section. If a section isn't specified, the parameter will go into the General section by default.
set_parameter method will set a single parameter.
The set_parameters method will override any parameters already logged.
To update the parameters in an experiment, use the
set_parameters_as_dict method . Arguments and values are inputted as a dictionary.
Like in the
set_parameters method, the dictionary can be nested, so the parameter's section can be specified.
To get all Task's parameters, use the
get_parameters() method, which will return a dictionary with the parameters, including