Upgrading Server from v0.15 or Older to ClearML Server
updating to ClearML Server v1.2+
If you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be compatible with the new version, after performing the migration instructions below.
In v0.16, the Elasticsearch subsystem of Trains Server was upgraded from version 5.6 to version 7.6. This change necessitates the migration of the database contents to accommodate the change in index structure across the different versions.
This page provides the instructions to carry out the migration process. Follow this process if using Trains Server version 0.15 or older and are upgrading to ClearML Server.
The migration process makes use of a script that automatically performs the following:
- Backs up the existing Trains Server Elasticsearch data.
- Launches a pair of Elasticsearch 5 and Elasticsearch 7 migration containers.
- Copies the Elasticsearch indices using the migration containers.
- Terminates the migration containers.
- Renames the original data directory to avoid accidental reuse.
warning
Once the migration process completes successfully, the data is no longer accessible to the older version of Trains Server, and ClearML Server needs to be installed.
#
Prerequisites- Read/write permissions for the default Trains Server data directory
/opt/clearml/data
and its subdirectories, or, if this default directory is not used, the permissions for the directory and subdirectories that are used. - A minimum of 8 GB system RAM.
- Minimum free disk space of at least 30% plus two times the size of the data.
- Python version >=2.7 or >=3.6, and Python accessible from the command-line as
python
#
Migrating the DataTo migrate the data:
If the Trains Server is up, shut it down:
Linux and macOS
Windows
Kubernetes
Kubernetes using Helm
For Kubernetes and Kubernetes using Helm, connect to the node in the Kubernetes cluster labeled
app=trains
.Download the migration package archive.
If the file needs to be downloaded manually, use this direct link: trains-server-0.16.0-migration.zip.
Extract the archive.
Migrate the data.
Linux, macOS, and Windows - if managing own containers.
Run the migration script. If elevated privileges are used to run Docker (
sudo
in Linux, or admin in Windows), then use elevated privileges to run the migration script.The following optional command line parameters can be used to control the execution of the migration script:
<source_path>
- The path to the Elasticsearch data directory in the current Trains Server deployment.
If not specified, uses the default value of/opt/trains/data/elastic
(orc:\opt\trains\data\elastic
in Windows)<target_path>
- The path to the Elasticsearch data directory in the current Trains Server deployment.
If not specified, uses the default value of/opt/trains/data/elastic_7
(orc:\opt\trains\data\elastic_7
in Windows)no-backup
- Skip creating a backup of the existing Elasticsearch data directory before performing the migration.
If not specified, takes on the default value ofFalse
(Performs backup)parallel
- Copy several indices in parallel to utilize more CPU cores. If not specified, parallel indexing is turned off.
Kubernetes
Clone the
trains-server-k8s
repository and change to the newtrains-server-k8s/upgrade-elastic
directory:Create the
upgrade-elastic
namespace and deployments:Wait for the job to be completed. To check if it's completed, run:
Kubernetes using Helm
Add the
clearml-server
repository to Helm client.Confirm the
clearml-server
repository is now in the Helm client.The
helm search
results must includeallegroai/upgrade-elastic-helm
.Install
upgrade-elastic-helm
on the cluster:An upgrade-elastic
namespace
is created in the cluster, and the upgrade is deployed in it.Wait for the job to complete. To check if it completed, execute the following command:
#
Verifying the Data MigrationUpon successful completion, the migration script renames the original Trains Server directory, which contains the now migrated data, and prints a completion message:
All console output during the execution of the migration script is saved to a log file in the directory where the migration script executes:
If the migration script does not complete successfully, the migration script prints the error.
important
For help in resolving migration issues, check the ClearML Slack Channel, GitHub Issues, and the ClearML Server sections of the FAQ.
#
Upgrading to ClearML Server v.1.2 or NewerIf you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be compatible with the new version. See instructions here. Otherwise, continue the instructions below.
#
Completing the InstallationAfter verifying the data migration completed successfully, continue upgrading your server: