Resource Configuration

Enterprise Feature

Resource Configuration is available under the ClearML Enterprise plan.

Administrators can define Resource Policies to implement resource quotas and reservations for different user groups to prioritize workload usage across available resources.

Under the Resource Configuration section, administrators define the available resources and the way in which they will be allocated to different workloads.

The Resource Configuration settings page shows the currently provisioned configuration: the defined resource pools, resource profiles, and the resource allocation architecture.

Resource Pools

A resource pool is an aggregation of resources available for use, such as a Kubernetes cluster or a GPU SuperPOD. Administrators specify the total number of resources available in each pool. The resource policy manager ensures workload assignment up to the available number of resources.

Administrators control the execution priority within a pool across the resource profiles making use of it (e.g. if jobs of profile A and jobs of profile B currently need to run in a pool, allocate resources for profile A jobs first or vice versa).

The resource pool cards are displayed on the top of the Resource Configuration settings page. Each card displays the following information:

• Pool name
• Number of resources currently in use out of the total available resources
• Execution Priority - List of linked profiles in order of execution priority.

Assigning Resources to Pools

To utilize a resource pool for task execution, ClearML agents are assigned to the pool servicing its specific queues. Tasks assigned by the resource policy manager to that pool will be sent to these queues for execution.

To configure an agent for a resource pool:

1. Open the resource pool’s information panel by clicking on the pool card. This displays:

   • Resource profiles linked to the pool
   • Resource allocation per task
   • Correlated Queue information

   

2. Copy the ID for the queue you want the agent to service.

3. Configure an agent to listen to the queue. For example, on Kubernetes:

   agentk8sglue:  
     queues:  
       <pool_queue_id>:   # Paste the Queue ID from the Resource Pool card here  
         templateOverrides:   
          resources:  
            limits:  
              nvidia.com/gpu: 2
   

   important

   The Resource Manager performs logical scheduling and accounting only. It does not control the actual hardware allocated to a task.

   You must explicitly configure the ClearML Agent (for example, by setting GPU limits in templateOverrides) to match the resource profile. Otherwise, tasks may be scheduled correctly but still run with incorrect resource allocation.

Tasks assigned to the resource profile connected to this pool will now be executed by your agent..

Resource Profiles

Resource profiles represent the resource consumption requirements of jobs, such as the number of GPUs needed. They are the interface that administrators use to provide users with access to the available resource pools based on their job resource requirements via Resource Policies.

A resource profile can have a fixed resource allotment (i.e. all jobs running through the profile will be assigned that number of resources) or let tasks specify their own resource requirements.

Administrators can control the resource pool allocation precedence within a profile (e.g. only run jobs on pool B if pool A cannot currently satisfy the its resource requirements).

Administrators can control the queuing priority within a profile across resource policies making use of it (e.g. if the R&D team and DevOps team both have pending jobs - run the R&D team's jobs first or vice versa).

The resource profile cards are displayed on the bottom of the Resource Configuration settings page. Each card displays the following information:

• Profile name
• - Number of resources allocated to jobs in this profile
• List of pool links
• - Number of currently pending jobs
• - Number of currently running jobs
• Number of resource policies. Click to open resource policy list and to order queuing priority.

Resource Allotment

Resource allotment is the number of resources allocated to each job running under a resource profile. The profile can have a fixed or dynamic resource allotment.

Fixed Requirement

Each job uses the fixed number of resources you set in Resource Allotment (for example, 4 GPUs per job).

This is best suited for workloads where all jobs have similar resource requirements.

Dynamic Resource Capacity

Dynamic Resource Capacity allows each job to define its own resource requirement, through a task parameter. This is useful when jobs within the same profile require different amounts of resources, avoiding the need to create multiple similar profiles.

When a job is run under a dynamic capacity resource profile, the resource manager will look for the specified parameter for the job’s resource requirement. If no such value is found, the task will be considered as requiring the profiles default resource allotment

For more information see instructions for creating a resource profile.

Example Workflow

You have GPUs spread across a local H100 and additional bare metal servers, as well as on AWS (managed by an autoscaler). Assume that currently most of your resources are already assigned to jobs, and only 16 resources are available: 8 in the H100 resource pool and 8 in the Bare Metal pool:

Teams' jobs have varying resource requirements of 0.5, 2, 4, and 8 GPUs. Resource profiles are defined to reflect these:

The different jobs will be routed to different resource pools by connecting the profiles to the resource pools. Jobs enqueued through the profiles will be run in the pools where there are available resources in order of their priority. For example, the H100 pool will run jobs with the following precedence: 2 GPU jobs first, then 4 GPU ones, then 8 GPU, and lastly 0.5 GPU.

Resource policies are implemented for two teams:

• Dev team
• Research Team

Each team has a resource policy configured with 8 reserved resources and a 16 resource limit. Both teams make use of the 4xGPU profile (i.e. each job running through this profile requires 4 resources).

The Dev team is prioritized over the Research team by placing it higher in the Resource Profile's Policies Priority list:

Both the Dev team and the Research team enqueue four 4-resource jobs each: Dev team jobs will be allocated resources first. The 4xGPU resource profile is connected to two resource pools: Bare Metal Low END GPUs (with the 4 GPU Low End link) and H100 Half a Superpod (with the 4 GPU H100 link).

Resources are assigned from the Bare Metal pool first (precedence set on the resource profile card):

If the first pool cannot currently satisfy the profile’s resource requirements, resources are assigned from the next listed pool. Let's look at the first pool in the image below. Notice that the pool has 8 available resources, therefore it can run two 4-resource jobs.

Since the Bare Metal pool does not have any more available resources, additional jobs will be assigned resources from the next pool that the Resource Profile is connected to. The H100 pool has 8 available resources. There are still 2 jobs pending from the Dev team requiring 8 resources in total, and 4 jobs from the Research team requiring 16 resources in total. In order to honor the Research team’s resource reservation, its first two jobs will be assigned the required 8 resources from the H100 pool.

All available resources having been assigned - 2 jobs of each team will remain pending until some of the currently running jobs finish and resources become available.

Applying Resource Configuration

Administrators can globally activate/deactivate resource policy management. To enable the currently provisioned configuration, click on the Enable resource management toggle. Enabling resource management will service the policy queues according to the provisioned resource profile and pool assignments. Disabling the resource management will stop serving the policy queues. Tasks on these queues will remain pending until resource policy management is reenabled.

Administrators can add, edit, delete, and connect resource pools and profiles in the Resource Configuration settings page.

To make any change (create, delete, or modify a component) to the resource configuration, follow the following steps:

1. Click Open Editor to go into Editing mode
2. After making the desired changes you have the following options:
   • Save - Save the changes you made. These changes will not be applied until you click on Provision
   • Validate - Check whether the saved configuration is valid and can be provisioned. If the configuration is valid, a confirmation message is displayed. If the configuration is invalid, validation fails and displays the reason (e.g., resource profiles that are not linked to any resource pool). Validation does not apply any changes. This option is available only after the configuration has been saved.
   • Provision - Apply the resource policy’s saved changes
   • Reset Configuration - Set the editor to the currently provisioned values. This will delete any unprovisioned changes (both saved and unsaved)
3. Click Exit to leave Editor mode. The page will show the provisioned configuration. Unprovisioned saved changes will still be available in Editor mode.

Resource Pool

To create a resource pool:

1. Click + Add Pool
2. In the Create Pool modal, input:
   • Name - The resource pool’s name. This will appear in the Pool’s information card in the Resource Configuration settings page
   • Number of Resources - Number of resources available in this pool
   • Description - Optional free form text for additional descriptive information
3. Click Create

Sizing Guidance

The "Number of Resources" value should match the physical capacity of the underlying cluster, expressed in whole GPUs. For example, a cluster of 2 servers with 8 GPUs each should be configured as a pool of 16, regardless of whether jobs will consume whole GPUs, fractional GPUs or MIG slices. Compute resources draw down from the pool are controlled by the profile's Resource Allotment (Resource Profiles) e.g. 1 for a full GPU, 0.5 for a half, or approximately 0.143 (1/7) for a 1g.10gb MIG slice. Keeping the pool sized to physical quantities ensures the accounting layer stays aligned with real hardware across all consumption patterns.

To modify a resource pool

1. Click on the relevant resource pool card > click Edit
2. In the Edit Pool modal, change the pool’s name, number of resources, or description
3. Click Save

You can also change the Execution Priority of the linked resource profiles. Click and drag the profile connection anchor to change its position in the order of priority.

Resource Profile

To create a resource profile:

1. Click + Add Profile

2. In the Create Profile modal, input:

   • Name - The resource profile’s name. This will appear in the profile’s information card in the Resource Configuration settings page
   • Resource Allotment - Number of resources allocated to each job running in this profile
   • Dynamic Resource Capacity - Allow each job to define its own resource requirement through a task parameter (for more information, see Dynamic Resource Capacity). When enabled, the following options are available:

     • Task Configuration Parameter - The task parameter used to retrieve the job’s resource usage at runtime. Supported configuration parameters:

       • runtime.*
       • hyperparams.*
       • configuration.*

       For example, runtime.gpus_request, hyperparams.Args.cost, or configuration.test.

     • Default Resource Allotment - The resource requirement for jobs with no valid value under the specified parameter

3. Click Create

To modify a resource profile:

1. Click on the relevant resource profile card > click Edit
2. In the Edit Profile modal, change the pool's name, number of resources, or description
3. Click Save

To control which pool's resources will be assigned first, click and drag the pool connection anchor to change its position in order of priority.

You can also change the Execution Priority of the resource policies making use of this profile. Open the policy list, then click the policy anchor and drag the policy to change its position in order of priority.

To delete a resource profile:

1. Click on the relevant resource pool card
2. Click Delete

Connecting Profiles to Pools

Connect a resource profile to a resource pool to allow jobs assigned to the profile to make use of the pool’s resources.

To connect a profile to a pool:

1. Click Open Editor
2. Drag the of the relevant profile to the resource pool you want to connect the profile to. This opens the Connect Profile modal
3. In the Connect Profile modal, input a name for this connection. This connection name will appear on the profile card

The settings page will show a line linking the profile and the pool cards. The linked profile appears on the pool card, showing its place in the order of execution. To change the profile's priority placement, drag its connection anchor to a new position.

To disconnect a profile from a pool:

1. Click Open Editor
2. On the relevant profile card, hover over connection name and click X

Jobs assigned to this resource profile will no longer be able to utilize the pool’s resources.