Skip to main content

PyTorch Ignite Integration

Integrate ClearML into code using ignite. Use ignite's ClearMLLogger, and the handlers that can be attached to it. See ignite's handler.

note

If you are not already using ClearML, see our Getting Started.

Ignite ClearMLLogger#

Integrate ClearML with the following steps:

  1. Create an Ignite ClearMLLogger object.

  2. When the code runs, it connects to the ClearML backend, and creates a Task (experiment) in ClearML.

    from ignite.contrib.handlers.clearml_logger import *
    clearml_logger = ClearMLLogger(project_name="examples", task_name="ignite")
  3. Later in the code, attach any of the ClearML handlers to the ClearMLLogger object.

    For example, attach the OutputHandler and log training loss at each iteration:

    clearml_logger.attach(trainer,
    log_handler=OutputHandler(tag="training",
    output_transform=lambda loss: {"loss": loss}),
    event_name=Events.ITERATION_COMPLETED)

ClearMLLogger Parameters#

The following are the ClearMLLogger method parameters:

  • project_name (optional[str]) – The name of the project in which the experiment will be created. If the project does not exist, it is created. If project_name is None, the repository name becomes the project name.

  • task_name (optional[str]) – The name of Task (experiment). If task_name is None, the Python experiment script’s file name becomes the Task name.

  • task_type (optional[str]) – The name of the experiment.

    The task_type values include:

    • TaskTypes.training (default)
    • TaskTypes.train
    • TaskTypes.testing
    • TaskTypes.inference
  • report_freq (optional[int]) – The histogram processing frequency (handles histogram values every X calls to the handler). Affects GradsHistHandler and WeightsHistHandler. Default value is 100.

  • histogram_update_freq_multiplier (optional[int]) – The histogram report frequency (report first X histograms and once every X reports afterwards). Default value is 10.

  • histogram_granularity (optional[int]): Optional. Histogram sampling granularity. Default is 50.

Logging#

Ignite Engine Output and / or Metrics#

To log scalars, Ignite engine's output and / or metrics, use the OutputHandler.

  • Log training loss at each iteration:
# Attach the logger to the trainer to log training loss at each iteration
clearml_logger.attach(trainer,
log_handler=OutputHandler(tag="training",
output_transform=lambda loss: {"loss": loss}),
event_name=Events.ITERATION_COMPLETED)
  • Log metrics for training:
# Attach the logger to the evaluator on the training dataset and log NLL, Accuracy metrics after each epoch
# We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer` instead of `train_evaluator`.
clearml_logger.attach(train_evaluator,
log_handler=OutputHandler(tag="training",
metric_names=["nll", "accuracy"],
global_step_transform=global_step_from_engine(trainer)),
event_name=Events.EPOCH_COMPLETED)
  • Log metrics for validation:
# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch of the
# `trainer` instead of `evaluator`.
clearml_logger.attach(evaluator,
log_handler=OutputHandler(tag="validation",
metric_names=["nll", "accuracy"],
global_step_transform=global_step_from_engine(trainer)),
event_name=Events.EPOCH_COMPLETED)

Optimizer Parameters#

To log optimizer parameters, use OptimizerParamsHandler:

# Attach the logger to the trainer to log optimizer's parameters, e.g., learning rate at each iteration
clearml_logger.attach(trainer,
log_handler=OptimizerParamsHandler(optimizer),
event_name=Events.ITERATION_STARTED)

Model Weights#

To log model weights as scalars, use WeightsScalarHandler:

# Attach the logger to the trainer to log model's weights norm after each iteration
clearml_logger.attach(trainer,
log_handler=WeightsScalarHandler(model, reduction=torch.norm),
event_name=Events.ITERATION_COMPLETED)

To log model weights as histograms, use WeightsHistHandler:

# Attach the logger to the trainer to log model's weights norm after each iteration
clearml_logger.attach(trainer,
log_handler=WeightsHistHandler(model),
event_name=Events.ITERATION_COMPLETED)

Model Snapshots#

To save input snapshots as ClearML artifacts, use ClearMLSaver:

to_save = {"model": model}
handler = Checkpoint(to_save, ClearMLSaver(clearml_logger), n_saved=1,
score_function=lambda e: 123, score_name="acc",
filename_prefix="best",
global_step_transform=global_step_from_engine(trainer))
validation_evaluator.add_event_handler(Events.EVENT_COMPLETED, handler)

Visualizing Experiment Results#

When the code with an ignite ClearMLLogger object and attached handlers runs, the experiment results can be visualized in the ClearML Web UI.

The ignite repository contains an MNIST ClearMLLogger example, mnist_with_clearml_logger.py.

Run this code and visualize the experiment results in the ClearML Web UI.

Scalars#

View the scalars, including training and validation metrics, in the experiment's page in the ClearML Web UI, under RESULTS > SCALARS.

image image

Model Snapshots#

To save model snapshots, use ClearMLServer.

handler = Checkpoint(
{"model": model},
ClearMLSaver(clearml_logger, dirname="~/.clearml/cache/"),
n_saved=1,
score_function=lambda e: 123,
score_name="acc",
filename_prefix="best",
global_step_transform=global_step_from_engine(trainer),
)

View saved snapshots in the ARTIFACTS tab.

image

To view the model, in the ARTIFACTS tab, click the model name (or download it).

image