This example shows how to use the clearml-data folder sync function.
clearml-data folder sync mode is useful for cases when users have a single point of truth (i.e. a folder) that updates
from time to time. When the point of truth is updated, users can call
clearml-data sync and the
changes (file addition, modification, or removal) will be reflected in ClearML.
First, make sure that you have cloned the clearml repository. This contains all the needed files.
- Open terminal and change directory to the cloned repository's examples folder
Create a dataset and sync the
data_samples folder from the repo to ClearML
As can be seen, the
clearml-data sync command creates the dataset, then uploads the files, and closes the dataset.
Now we'll modify the folder:
- Add another line to one of the files in the
- Add a file to the sample_data folder.
echo "data data data" > data_samples/new_data.txt(this will create the file
new_data.txtand put it in the
We'll repeat the process of creating a new dataset with the previous one as its parent, and syncing the folder.
We can see that 2 files were added or modified, just as we expected!