Skip to main content

Additional Features

This page covers additional ClearML Serving features, including manual model registration and deployment, automatic model updates, canary deployments, and performance monitoring.

Registering and Deploying New Models Manually​

You can upload an existing model file into the model repository via the ClearML RestAPI, the Python interface, or with the clearml-serving CLI.

  1. Upload the model file to the clearml-server file storage and register it with the --path parameter pointing to a local model file:

    clearml-serving --id <service_id> model upload --name "manual sklearn model" --project "serving examples" --framework "scikitlearn" --path ./sklearn-model.pkl

    This creates a new model named manual sklearn model in the serving examples project. The CLI output prints the new model’s UID, which you will use to register a new endpoint.

    In the ClearML web UI, the new model is listed under the Models tab of its project. You can also download the model file itself directly from the web UI.

  2. Register a new endpoint with the new model:

    clearml-serving --id <service_id> model add --engine sklearn --endpoint "test_model_sklearn" --preprocess "examples/sklearn/preprocess.py" --model-id <newly_created_model_id_here>
Model Storage

You can also provide a different storage destination for the model, such as S3/GS/Azure, by using --destination. For example:

  • S3: s3://bucket/folder
  • Non-AWS S3-like services (such as MinIO): s3://host_addr:port/bucket. Note that port specification is required.
  • Google Cloud Storage: gs://bucket-name/folder
  • Azure Storage: azure://<account name>.blob.core.windows.net/path/to/file

There is no need to provide a unique path to the destination argument, the location of the model will be a unique path based on the serving service ID and the model name.

Automatic Model Deployment​

The ClearML Serving Service supports automatic model deployment and upgrades. When automatic model deployment is enabled, new model versions are automatically deployed when you publish or tag a new model in the ClearML model repository. This simplifies CI/CD workflows, as a single API automatically deploys (or removes) a model from the Serving Service.

Automatic Model Deployment Example​

  1. Configure the model auto-update on the Serving Service (for more details, see clearml-serving model auto-update subcommand):

    clearml-serving --id <service_id> model auto-update --engine sklearn --endpoint "test_model_sklearn_auto" --preprocess "preprocess.py" --name "train sklearn model" --project "serving examples" --max-versions 2`

  2. Deploy the inference container (if not already deployed)

  3. Publish a new model in the model repository in one of the following ways:

    • Go to the "serving examples" project in the ClearML web UI, click on the Models Tab, search for "train sklearn model" right-click and select "Publish"

    • Use the RestAPI (see details)

    • Use Python interface:

      from clearml import Model
      Model(model_id="unique_model_id_here").publish()

    The new model is available on a new endpoint version (1).

  4. Test the new model endpoint:

    curl -X POST "http://127.0.0.1:8080/serve/test_model_sklearn_auto/1" -H "accept: application/json" -H "Content-Type: application/json" -d '{"x0": 1, "x1": 2}'

Canary Endpoint Setup​

Canary endpoint deployment adds a new endpoint where the actual request is sent to a preconfigured set of endpoints with pre-provided distribution. For example, create a new endpoint "test_model_sklearn_canary" and provide a list of endpoints and probabilities (weights).

clearml-serving --id <service_id> model canary --endpoint "test_model_sklearn_canary" --weights 0.1 0.9 --input-endpoints test_model_sklearn/2 test_model_sklearn/1

This means that any request coming to /test_model_sklearn_canary/ will be routed with probability of 90% to /test_model_sklearn/1/ and with probability of 10% to /test_model_sklearn/2/.

note

As with any other Serving Service configuration, you can configure the Canary endpoint while the inference containers are already running and deployed. They will update in their next update cycle (default: once every 5 minutes).

You can also prepare a "fixed" canary endpoint, always splitting the load between the last two deployed models:

clearml-serving --id <service_id> model canary --endpoint "test_model_sklearn_canary" --weights 0.1 0.9 --input-endpoints-prefix test_model_sklearn/

This means that you have two model inference endpoints: /test_model_sklearn/1/ and /test_model_sklearn/2/. The 10% probability (weight 0.1) will match the last (order by version number) endpoint, i.e. /test_model_sklearn/2/, and the 90% will match /test_model_sklearn/2/. When you add a new model endpoint version, e.g. /test_model_sklearn/3/, the canary distribution will automatically match the 90% probability to /test_model_sklearn/2/ and the 10% to the new endpoint /test_model_sklearn/3/.

Example:

  1. Add two endpoints:

    • Endpoint 1:

      clearml-serving --id <service_id> model add --engine sklearn --endpoint "test_model_sklearn" --preprocess "examples/sklearn/preprocess.py" --name "train sklearn model" --version 1 --project "serving examples"

    • Endpoint 2:

      clearml-serving --id <service_id> model add --engine sklearn --endpoint "test_model_sklearn" --preprocess "examples/sklearn/preprocess.py" --name "train sklearn model" --version 2 --project "serving examples"

  2. Add Canary endpoint:

    clearml-serving --id <service_id> model canary --endpoint "test_model_sklearn_canary" --weights 0.1 0.9 --input-endpoints test_model_sklearn/2 test_model_sklearn/1

  3. Test Canary endpoint:

    curl -X POST "http://127.0.0.1:8080/serve/test_model" -H "accept: application/json" -H "Content-Type: application/json" -d '{"x0": 1, "x1": 2}'`

Model Monitoring and Performance Metrics​

Grafana Screenshot

ClearML serving instances send serving statistics (count/latency) automatically to Prometheus, and Grafana can be used to visualize and create live dashboards.

The default docker-compose installation is preconfigured with Prometheus and Grafana. Notice that by default data of both containers is not persistent. To add persistence, adding a volume mount is recommended.

You can also add many custom metrics on the input/predictions of your models. Once a model endpoint is registered, adding custom metrics can be done using the CLI.

For example, assume the mock scikit-learn model is deployed on endpoint test_model_sklearn, you can log the requests inputs and outputs (see examples/sklearn/preprocess.py example):

clearml-serving --id <serving_service_id_here> metrics add --endpoint test_model_sklearn --variable-scalar
x0=0,0.1,0.5,1,10 x1=0,0.1,0.5,1,10 y=0,0.1,0.5,0.75,1

This will create a distribution histogram (buckets specified via a list of less-equal values after = sign), that you will be able to visualize on Grafana.

time-series values

You can also log time-series values with --variable-value x2 or discrete results (e.g. classifications strings) with --variable-enum animal=cat,dog,sheep. Additional custom variables can be added in the preprocess and postprocess with a call to collect_custom_statistics_fn({'new_var': 1.337}). See preprocess_template.py.

With the new metrics logged, you can create a visualization dashboard over the latency of the calls, and the output distribution.

Grafana Model Performance Example​

  1. Browse to http://localhost:3000
  2. Login with: admin/admin
  3. Create a new dashboard
  4. Select Prometheus as data source
  5. Add a query: 100 * increase(test_model_sklearn:_latency_bucket[1m]) / increase(test_model_sklearn:_latency_sum[1m])
  6. Change type to heatmap, and select on the right hand-side under "Data Format" select "Time series buckets". You now have the latency distribution, over time.
  7. Repeat the same process for x0, the query would be 100 * increase(test_model_sklearn:x0_bucket[1m]) / increase(test_model_sklearn:x0_sum[1m])
note

If not specified all serving requests will be logged, which can be changed with the CLEARML_DEFAULT_METRIC_LOG_FREQ environment variable. For example CLEARML_DEFAULT_METRIC_LOG_FREQ=0.2 means only 20% of all requests will be logged. You can also specify per-endpoint log frequency with the clearml-serving CLI. See clearml-serving metrics.

Examples​

See examples of ClearML Serving with other supported frameworks: