Skip to main content

Kubernetes

This guide provides step-by-step instructions for installing the ClearML Enterprise Server (control-plane) in a Kubernetes cluster.

The ClearML Enterprise Server includes the ClearML apiserver, fileserver, and webserver components. The package also includes MongoDB, ElasticSearch, and Redis as Helm dependencies.

Upgrading an Existing Installation

Upgrading an existing deployment requires additional steps, depending on your current chart version. If both apply, complete them in order:

  1. From chart version 10.11.6 and below - starting in chart version 10.11.7, ClearML is transitioning to a new mongodb chart source (mckMongodb). Follow the MongoDB chart Migration Guide to ensure data consistency and compatibility with future chart versions.
  2. To chart version 11.0.0 and greater - chart version 11.0.0 (ClearML Server v3.29) introduces a breaking change: server containers now run rootless by default. Follow the required upgrade steps to re-own the fileserver's existing data before deploying the new chart version.

Prerequisites​

To deploy a ClearML Server, ensure the following components and configurations are in place:

  • Kubernetes Cluster: A standard Kubernetes cluster is recommended for optimal GPU support.
  • CLI Tools: kubectl and helm must be installed and configured.
  • Ingress Controller: An Ingress controller (e.g., nginx-ingress) is required. If exposing services externally, configure a LoadBalancer-capable solution (e.g. MetalLB).
  • Server and workers that communicate on HTTP/S (ports 80 and 443). Additionally, the TCP session feature requires a range of ports for TCP traffic based on your configuration (see AI App Gateway installation).
  • DNS Configuration: A domain with subdomain support is required, ideally with trusted TLS certificates. All entries must be resolvable by the Ingress controller. Example subdomains:
    • Server:
      • api.<BASE_DOMAIN>
      • app.<BASE_DOMAIN>
      • files.<BASE_DOMAIN>
    • Worker:
      • router.<BASE_DOMAIN>
      • tcp-router.<BASE_DOMAIN> (optional, for TCP sessions)
  • Storage: A configured StorageClass and an accessible storage backend.
  • A DockerHub token to access the OCI enterprise Helm charts and Docker images (<CLEARML_DOCKERHUB_TOKEN>)

For optimal performance, a Kubernetes cluster with at least 3 nodes is recommended, each provisioned with:

  • 8 vCPUs
  • 32 GB RAM
  • 500 GB storage

Installation​

Log into the ClearML OCI Registry​

Login to the ClearML OCI registry:

helm registry login docker.io --username allegroaienterprise --password <CLEARML_DOCKERHUB_TOKEN>

Prepare Values​

Create a clearml-values.override.yaml file with the following content:

note

In the following configuration, replace the <BASE_DOMAIN> placeholders with a valid domain that will have records pointing to the cluster's Ingress Controller. This will be the base domain for reaching your ClearML installation.

imageCredentials:
  password: "<CLEARML_DOCKERHUB_TOKEN>"
clearml:
  cookieDomain: "<BASE_DOMAIN>"
apiserver:
  ingress:
    enabled: true
    hostName: "api.<BASE_DOMAIN>"
  service:
    type: ClusterIP
fileserver:
  ingress:
    enabled: true
    hostName: "files.<BASE_DOMAIN>"
  service:
    type: ClusterIP
webserver:
  ingress:
    enabled: true
    hostName: "app.<BASE_DOMAIN>"
  service:
    type: ClusterIP
clearmlApplications:
  enabled: true

Special Clean installation overrides

Add the following settings in your clearml-values.override.yaml file to use the new (10.11.7) bundled MongoDB dependency (mckMongodb) instead of the legacy MongoDB chart:

mongodb:
  enabled: false
mckMongodb:
  enabled: true
  migrated: true

Install the Chart​

Install the ClearML Enterprise Helm chart using the previous values override file.

helm upgrade -i -n clearml clearml-enterprise oci://docker.io/clearml/clearml-enterprise --create-namespace -f clearml-values.override.yaml

Additional Configuration Options​

note

You can view the full set of available and documented values of the chart by running the following command:

helm show readme oci://docker.io/clearml/clearml-enterprise
# or
helm show values oci://docker.io/clearml/clearml-enterprise

Default Secret Values​

For improved security, all the internal credentials are auto-generated randomly and stored in a Secret in Kubernetes.

If you need to define your own credentials to be used instead, replace the default key and secret values in clearml-values.override.yaml.

clearml:
  # Replace the following values to use custom internal credentials.
  apiserverKey: "<generated_key>"
  apiserverSecret: "<generated_key>"
  fileserverKey: "<generated_key>"
  fileserverSecret: "<generated_key>"
  secureAuthTokenSecret: "<generated_key>"
  testUserKey: "<generated_key>"
  testUserSecret: "<generated_key>"

In a shell, if openssl is installed, you can use this simple command to generate random strings suitable as keys and secrets:

openssl rand -hex 16

Fixed Users​

Enable and configure simple login with username and password in clearml-values.override.yaml. This is useful for simple PoC installations. This is an optional step in case the SSO (Identity provider) configuration is not performed.

Please note that this setup is not ideal for multi-tenant setups as fixed users will only be associated with the default tenant.

apiserver:
  additionalConfigs:
    apiserver.conf: |
      auth {
        fixed_users {
          enabled: true
          pass_hashed: false
          users: [
            {
              username: "my_user"
              password: "my_password"
              name: "My User"
              admin: true
            },
          ]
        }
      }

Internal Database Authentication​

important

Internal Database Authentication is supported starting from clearml-enterprise Helm chart version 10.9.0.

By default, the ClearML Enterprise Helm chart deploys its internal DB components (MongoDB, ElasticSearch, and Redis) through dependent charts configured for open authentication. You can enable authentication for these internal databases to improve security.

This following procedure applies specifically to dependency databases deployed as part of the clearml-enterprise Helm chart (see here for additional database configuration options).

To enable authentication, update the control-plane clearml-values.override.yaml file.

For example, the following configuration:

  • Enables username/password authentication on all bundled databases
  • Ensures ClearML services use the configured credentials automatically
# Enable Redis Auth
redis:
  auth:
    enabled: true
    password: "MyStrongPassword123"
# Enable MongoDB Auth
mongodb:
  enabled: true
  auth:
    enabled: true
    rootUser: root
    rootPassword: "MyStrongPassword123"
# Enable ElasticSearch Auth
elasticsearch:
  extraEnvs:
    - name: bootstrap.memory_lock
      value: "false"
    - name: cluster.routing.allocation.node_initial_primaries_recoveries
      value: "500"
    - name: cluster.routing.allocation.disk.watermark.low
      value: "500mb"
    - name: cluster.routing.allocation.disk.watermark.high
      value: "500mb"
    - name: cluster.routing.allocation.disk.watermark.flood_stage
      value: "500mb"
    - name: http.compression_level
      value: "7"
    - name: reindex.remote.whitelist
      value: '"*.*"'
  esConfig: {}
  secret:
    enabled: true
    password: "MyStrongPassword123"

Using External Databases with ClearML​

The ClearML Enterprise Server can be configured to use external services for the ElasticSearch, MongoDB, and Redis databases, instead of those included in the server bundle. Note that if you use external database instances, ClearML will not manage the database's lifecycle or version. For MongoDB specifically, ClearML disables internal version checks (CLEARML__apiserver__mongo__ensure_db_version_on_startup=false).

As a result, you are fully responsible for:

  • Provisioning and maintaining the database instance
  • Applying security updates and version upgrades
  • Ensuring availability, backups, and disaster recovery

Before upgrading ClearML, always consult the ClearML release notes and documentation to verify which DB versions are supported. Running unsupported versions may cause ClearML to fail or behave unexpectedly.

To connect external databases, configure the externalServices section in the clearml-values.override.yaml file:

externalServices:
  # Existing ElasticSearch connectionstring if elasticsearch.enabled is false
  elasticsearchConnectionString: "[{\"host\":\"es_hostname1\",\"port\":9200},{\"host\":\"es_hostname2\",\"port\":9200},{\"host\":\"es_hostname3\",\"port\":9200}]"
  # Existing MongoDB connection string for BACKEND to use if mongodb.enabled is false
  mongodbConnectionStringAuth: "mongodb://mongodb_hostname:27017/auth"
  # Existing MongoDB connection string for AUTH to use if mongodb.enabled is false
  mongodbConnectionStringBackend: "mongodb://mongodb_hostnamehostname:27017/backend"
  # Existing Redis Hostname to use if redis.enabled is false
  redisHost: "redis_hostname"
  # Existing Redis Port to use if redis.enabled is false
  redisPort: 6379

Using External Storage Instead of the Fileserver​

The ClearML Enterprise Server can be configured to use external object storage (e.g. AWS S3, MinIO, GCS, Azure Blob) instead of the bundled file server. In this setup, artifact and model URLs reference the external storage directly. The ClearML control plane, the ClearML Enterprise Agent, and other ClearML clients must all be configured to use the same new external storage URL.

Control Plane Configuration​

In the server's clearml-values.override.yaml, disable the bundled fileserver and set the external storage URL references used by the control plane components:

fileserver:
  enabled: false

clearmlApplications:
  fileServerUrlReferenceOverride: "<EXTERNAL_STORAGE_URL>"

webserver:
  displayedServerURLs:
    apiserver: "<APISERVER_URL>"
    fileserver: "<EXTERNAL_STORAGE_URL>"
  extraEnvs:
    - name: WEBSERVER__fileBaseUrl
      value: "<EXTERNAL_STORAGE_URL>"

Agent Configuration​

In the agent's clearml-agent-values.override.yaml, point the agent's fileserver reference to the same external storage URL:

agentk8sglue:
  fileServerUrlReference: "<EXTERNAL_STORAGE_URL>"

Client Configuration​

To ensure ClearML clients (SDK and Agent-launched tasks) upload artifacts, models, and debug samples to the correct location, configure the credentials and default output URI. The recommended approach is via an Administrator Vault, though credentials can also be set directly in the clearml.conf file on each client machine.

For example, the following configures credentials and sets a default output URI for an S3-compatible endpoint:

sdk {
    aws {
        s3 {
            credentials: [
                {
                    host: "s3://<EXTERNAL_STORAGE_URL>"
                    key: "<EXTERNAL_STORAGE_KEY>"
                    secret: "<EXTERNAL_STORAGE_SECRET>"
                    multipart: false
                    secure: true
                    verify: false
                }
            ]
        }
    }
}

sdk.development.default_output_uri = "s3://<EXTERNAL_STORAGE_URL>"

For configuration examples covering other storage backends (Azure, GCS, non-AWS S3 endpoints, and more), see Storage.

Monitoring​

Monitoring your ClearML deployment is recommended to ensure service availability and detect performance or resource issues early. For monitoring guidelines and recommended metrics, see Monitoring.

Next Steps​

After installing the ClearML Enterprise Server, you can enhance your deployment by enabling optional services that extend ClearML's capabilities for scheduling, interactivity, authentication, and more.

Applications Installation​

ClearML Applications are plugins that extend the functionality of the ClearML Enterprise Server. They enable users to manage ML workloads and automate recurring workflows--no code required

Applications are installed on top of the ClearML Server and are provided by the ClearML team.

For more information, see the Application Installation guide.

ClearML Enterprise Agent​

The ClearML Enterprise Agent enables scheduling and execution of distributed workloads (Tasks) on your Kubernetes cluster.

See the ClearML Agent installation guide.

AI Application Gateway​

The AI App Gateway enables secure, authenticated access to ClearML application endpoints such as model serving or IDE workloads based on ClearML user permissions. It routes HTTPS traffic from users to running pods on the cluster.

See the AI Application Gateway installation guide.

Advanced Options​

GPU Operator​

Deploy the NVIDIA GPU Operator in order to use Nvidia GPUs in ClearML.

See the GPU Operator Basic Deployment guide.

Fractional GPU Support​

Available GPU fractioning methods:

  • ClearML Dynamic MIG Orchestrator (CDMO): Manage GPU fractions using NVIDIA MIGs. See the CDMO guide
  • ClearML Fractional GPU Injector (CFGI): Use fractional (non-MIG) GPU slices for efficient resource sharing. See the CFGI guide
  • Mixed Deployments: Deploy both CDMO and CFGI in clusters with diverse GPU types. Use the NVIDIA GPU Operator to handle mixed hardware setups. See the CDMO and CFGI guide.

Multi-Tenant Setup​

Run multiple isolated tenants on a single ClearML Server deployment, each with its own configuration and user namespaces.

See the Multi-Tenant Service guide.

SSO (Identity Provider) Setup​

Integrate identity providers to enable SSO login for ClearML Enterprise users.

See the SSO Setup guide.

ClearML Custom Event Monitoring​

ClearML Enterprise supports sending custom events to selected Kafka topics.

See the Custom Event guide.

ClearML Presign Service​

The ClearML Presign Service securely generates pre-signed storage URLs for authenticated users.

See the ClearML Presign Service guide.

Install with a Non-Root User​

In some Helm charts, you will find a values file called values-enterprise-non-root-privileged.yaml to be used for a non-root installation.

These values are for Enterprise versions only, and they need to be adapted to specific infrastructure needs. The containerSecurityContext is related to the Kubernetes distribution used/configuration and will need to be customized accordingly.