ClearML is able to interface with the most popular storage solutions in the market for storing model checkpoints, artifacts and charts.
Supported storage mediums include:
Once uploading an object to a storage medium, each machine that uses the object must have access to it.
Configuration for storage is done by editing the clearml.conf.
Modify these parts of the clearml.conf file and add the key, secret, and region of the s3 bucket. It's possible to also give access to specific s3 buckets.
ClearML also supports MinIO by adding this configuration:
To configure Azure blob storage specify the account name and key.
To configure Google Storage, specify the project and the path to the credentials json file. It's also possible to specify credentials for a specific bucket.
ClearML Offers a package to manage downloading, uploading and caching of content directly from code.
To upload a file using storage manager, just run the following line specifying the path to a local file or folder, and the remote destination.
To download files into cache, run the following line, specifying the remote destination's URL.
Zip and tar.gz files will be automatically extracted to cache. This can be controlled with the
It's possible to control the maximum cache size by limiting the number of files it stores.
This is done by calling the
ClearML also manages a cache of all downloaded content so nothing is duplicated, and code won't need to download the same piece twice!
Configure cache location by modifying the clearml.conf file:
By default, all artifacts (Models \ Artifacts \ Datasets) are automatically downloaded to the cache before they're used.
Some storage mediums (NFS \ Local storage) allows for direct access, which means that the code would work with the object where it's originally stored and not downloaded to cache first.
To enable direct access, specify the urls to access directly.