ClearML Enterprise's Datasets and Dataset versions provide the internal data structure and functionality for the following purposes:
- Connecting source data to the ClearML Enterprise platform
- Using ClearML Enterprise's GIT-like Dataset versioning
- Integrating the powerful features of Dataviews with an experiment
- Annotating images and videos
Datasets consist of versions with SingleFrames and / or FrameGroups. Each Dataset can contain multiple versions, where each version can have multiple children that inherit their parent's SingleFrames and / or FrameGroups. This inheritance includes the frame metadata and data connecting the source data to the ClearML Enterprise platform, as well as the other metadata and data.
These parent-child version relationships can be represented as version trees with a root-level parent. A Dataset can contain one or more trees.
Mask-labels can be defined globally, for a DatasetVersion, which will be applied to all masks in that version.
Dataset versions can have either Draft or Published status.
A Draft version is editable, so frames can be added to and deleted and / or modified from the Dataset.
A Published version is read-only, which ensures reproducible experiments and preserves a version of a Dataset. Child versions can only be created from Published versions. To create a child of a Draft Dataset version, it must be published first.
ClearML Enterprise provides Example Datasets, available to in the ClearML Enterprise platform, with frames already built, and ready for your experimentation. Find these example Datasets in the ClearML Enterprise WebApp (UI). They appear with an "Example" banner in the WebApp (UI).
Dataset.create method to create a Dataset. It will
contain an empty version named
Or, use the
To raise a
ValueError exception if the Dataset exists, specify the
raise_if_exists parameters as
- Or with
Additionally, create a Dataset with tags and a description.
To get the current Dataset, use the
Dataset.delete method to delete a Dataset.
Delete an empty Dataset (no versions).
Delete a Dataset containing only versions whose status is Draft.
Delete a Dataset even if it contains versions whose status is Published.
Dataset versioning refers to the group of ClearML Enterprise SDK and WebApp (UI) features for creating, modifying, and deleting Dataset versions.
ClearML Enterprise supports simple and sophisticated Dataset versioning, including simple version structures and advanced version structures.
In a simple version structure, a parent can have one and only one child, and the last child in the Dataset versions tree must be a Draft. This simple structure allows working with a single set of versions of a Dataset. Create children and publish versions to preserve data history. Each version whose status is Published in a simple version structure is referred to as a snapshot.
In an advanced version structure, at least one parent has more than one child (this can include more than one parent version at the root level), or the last child in the Dataset versions tree is Published.
Creating a version in a simple version structure may convert it to an advanced structure. This happens when creating a Dataset version that yields a parent with two children, or when publishing the last child version.
Manage Dataset versioning using the DatasetVersion class in the ClearML Enterprise SDK.
If the Dataset contains only one version whose status is Draft, snapshots of the current version can be created. When creating a snapshot, the current version becomes the snapshot (it keeps the same version ID), and the newly created version (with its new version ID) becomes the current version.
To create a snapshot, use the
In the simple version structure, ClearML Enterprise supports two methods for snapshot naming:
Timestamp naming - If only the Dataset name or ID is provided, the snapshot is named
snapshotwith a timestamp appended.
The timestamp format is ISO 8601 (
YYYY-MM-DDTHH:mm:ss.SSSSSS). For example,
Example:from allegroai import DatasetVersionmyDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset')
After the statement above runs, the previous current version keeps its existing version ID, and it becomes a snapshot named
snapshotwith a timestamp appended. The newly created version with a new version ID becomes the current version, and its name is
User-specified snapshot naming - If the
publish_nameparameter is provided, it will be the name of the snapshot name.
Example:myDataset = DatasetVersion.create_snapshot(dataset_name='MyDataset', publish_name='NewSnapshotName')
After the above statement runs, the previous current version keeps its existing version ID and becomes a snapshot named
NewSnapshotName. The newly created version (with a new version ID) becomes the current version, and its name is
In the simple version structure, ClearML Enterprise supports two methods for current version naming:
- Default naming - If the
child_nameparameter is not provided,
Currentis the current version name.
- User-specified current version naming - If the
child_nameparameter is provided, that child name becomes the current version name.
For example, after the following statement runs, the previous current version keeps its existing version ID and becomes
a snapshot named
snapshot with the timestamp appended.
The newly created version (with a new version ID) is the current version, and its name is
Add a metadata dictionary and / or comment to a snapshot.
Create a new version from any version whose status is Published.
To create a new version, call the
method, and provide:
- Either the Dataset name or ID
- The parent version name or ID from which the child inherits frames
- The new version's name.
For example, create a new version named
NewChildVersion from the existing version
where the new version inherits the frames of the existing version. If
NewChildVersion already exists,
it is returned.
To raise a ValueError exception if
NewChildVersion exists, set
Create a new version at the root-level. This is a version without a parent, and it contains no frames.
Getting a list of all versions
Getting a list of all published versions
Getting a list of all drafts versions
Getting the current version
If more than one version exists, ClearML Enterprise outputs a warning.
Getting a specific version
Delete versions which are status Draft using the
Publish (make read-only) versions which are status Draft using the
method. This includes the current version, if the Dataset is in the simple version structure.
In order to visualize masks in a dataset version, the mask values need to be mapped to their labels. Mask-label mapping is stored in a version's metadata.
To define the DatasetVersion level mask-label mapping, use the
method, and input a dictionary of RGB-value tuple keys and label-list values.
The mask values and labels are stored as a property in a dataset version's metadata.
This should return a dictionary of the version's masks and labels, which should look something like this: