By Adam Wolf and Damian Erangey
This blog covers the topic of Automation Security with ClearML as it relates to enforcing AI infrastructure policies within an organization. It accompanies our Enterprise AI Infrastructure Security YouTube series. Watch the corresponding video below.
Automated Workloads Need Identities Too
The previous entries in this series have covered the first two layers of ClearML’s enterprise security model: identity providers and access rules for human users, and configuration vaults for governing the runtime environment. Those mechanisms work because tasks are tied to identities: the platform knows who submitted something and applies the right configuration accordingly.
But in most production AI environments, the majority of work isn’t submitted by humans at all. Training agents execute jobs overnight. CI/CD pipelines deploy validated models. Schedulers trigger retraining on a cadence. Autoscalers spin up workers in response to queue depth. All of these automated workloads require authentication with the ClearML server, and they must have identities that possess the same governance properties as those of human users. That is what service accounts are for.
What Is a Service Account?
A service account is a ClearML identity designed for automation. It authenticates with API credentials (an access key and secret key), with no SSO, no browser login, and no web UI access. It is purpose-built for machines: agents, pipelines, schedulers, and autoscalers.
Critically, everything configured in the previous videos applies to service accounts too. They can be members of user groups, governed by access rules, and have administrator vaults applied to them. The governance model is identical to that of a human user; it’s just the authentication mechanism that differs.
Service accounts are created under Settings → Users & Groups in the ClearML web UI. When you create one, the platform generates an access key and secret key formatted as a ready-to-copy configuration block. This is shown once and cannot be retrieved later, so store it immediately in your secrets manager of choice (Kubernetes Secrets, GitHub Secrets, Azure Key Vault). Once the service account is added to a user group, any access rules and administrator vaults scoped to that group apply automatically.
One Account Per Purpose
The easiest mistake to make with service accounts is creating one account with broad access and using it for everything. It works – until you need to rotate credentials or revoke access, at which point you’re taking down your entire automation layer at once.
Give each identity only the access it needs. One service account per purpose, not one account for all purposes.
The better model is one account per purpose, scoped to exactly what that automation needs:
- Development agent — service account scoped to dev projects and dev queues only
- Production agent — separate account, separate scope, separate credentials
- CI/CD pipeline — its own account for model deployment workflows
- Scheduled retraining — its own account with access to the right data and storage paths
Because service accounts inherit access rules and vaults from their group membership, scoping is mostly handled by which group you add the account to. Create the service account, put it in the right group, and the governance follows automatically. If you need to rotate credentials for the production agent, you do it there; the development agent keeps running.
Impersonation: Least-Privilege Execution
When an agent runs tasks without impersonation, it executes everything under its own identity. That means the agent needs access to every resource any user might submit to, which pushes it toward broad, often over-provisioned access.
Impersonation solves this. When enabled on a service account, the agent executes each task under the identity of whoever submitted it. Bob’s task runs with Bob’s permissions, Bob’s vault config, and Bob’s group memberships. The agent authenticates as itself but assumes the task owner’s identity at execution time.
In the ClearML Kubernetes agent, impersonation is enabled by setting agentk8sglue.useOwnerToken: true in the Helm chart values. For agents run via the CLI, the equivalent flag is –use-owner-token. The service account must have the “Allow impersonation” toggle enabled when created; if it isn’t, running an agent with –use-owner-token will cause the agent to fail.
You can see impersonation working in task logs: the agent identity (e.g. k8s:generic-agent) appears alongside log entries like “Loaded group vault for user [Bob’s ID]” and “Loaded user vault for user [Bob’s ID].” The agent is running the task, but it’s loading Bob’s vaults, not its own. That’s what triggers the administrator vault applied to Bob’s group, and why the governance from the previous video flowed through correctly even though an agent was executing the task.
Without impersonation, none of that vault governance applies to the task. The agent runs with its own config, and the per-user policy layer is bypassed entirely.
When to Run as the Service Account Itself
Impersonation isn’t the right model for every automated workload. Some workflows are system-level operations that shouldn’t be tied to a user’s identity at all:
- A CI/CD pipeline deploying validated models to production: that’s a system operation with its own scoped credentials, not a user’s task
- A scheduled retraining job that runs overnight: the scheduler owns the task, with its own vault config pointing to the right data and storage
- A pipeline controller orchestrating a multi-step workflow: it needs its own identity and access, not whoever happened to kick it off
The rule of thumb: if the agent is executing user-submitted tasks, use impersonation to keep execution least-privileged per user. If the service account is running its own predefined workloads, run as itself with its own scoped vault config and access rules.
Credential Management
Service account credentials are long-lived by nature, which makes a rotation routine important. A few practices worth following:
- Store credentials in a secrets manager: Kubernetes Secrets, GitHub Secrets, Azure Key Vault. Don’t leave them in plaintext config files or environment variables on worker nodes.
- Rotate on a schedule: new credentials can be generated per service account at any time from the Users & Groups settings page. Old credentials can be revoked independently without affecting others.
- Delete unused accounts immediately: deleting a service account revokes all of its credentials instantly. Any agents using those credentials will fail, which is the correct behavior for a decommissioned account.
ClearML redacts sensitive values in task logs by default; credentials injected by a vault at runtime appear as **** in the output, so even when configuration is applied dynamically, credentials don’t surface in plain text.
Where This Fits in the Security Model
Service accounts are the bridge between ClearML’s user-facing governance and the automated infrastructure that does most of the actual work. The identity provider integration controls who can log in. Access rules control what those identities can reach. Administrator vaults govern the runtime environment. Service accounts extend all three to your agents, pipelines, and schedulers; the same controls, applied consistently, whether the task was submitted by a human or a machine.
Here is what this may look like in practice for IT Admins:
Scenario 1 – The Offboarding Risk
A senior ML engineer leaves the company. They have been running overnight training jobs using shared agent credentials that half the team depends on. With Service accounts scoped per purpose, you delete their personal access and rotate the relevant account in one place, so the production pipeline keeps running and nothing else is touched. No scrambling to find every place those credentials were used and no risk that a departing employee’s access lingers in a pipeline somewhere.
Scenario 2 – The Audit Request
Security asks which container image ran the model that went into production last week, and under whose credentials. With impersonation enabled, every task carries the submitting user’s identity through execution (container version, vault config, and user are all logged automatically). You have a clean, complete trail without anyone having to reconstruct it after the fact.
Closing
Most security models are designed around human users and break down when automation enters the picture. ClearML treats service accounts as first-class identities: same group membership, same access rules, same vault governance. Add impersonation and you get least-privilege execution without needing to give your agents broad access to every resource on the platform.
One account per purpose. Credentials in a secrets manager. Impersonation enabled where appropriate. That’s a production-grade automation security posture, and it follows directly from the governance model already in place for your human users.
Find the videos in our Enterprise AI Security series on YouTube here. Get in touch if you would like to discuss how ClearML can support your organization’s AI infrastructure policy requirements.