By Adam Wolf
Zero trust is an architectural principle, not a product. It means assuming breach, verifying every connection explicitly, and granting the minimum access required for each interaction. This post covers how those principles apply to Kubernetes AI infrastructure and specifically how ClearML’s security model slots into each layer: network segmentation, workload identity, access controls, and audit logging. Kubernetes AI infrastructure and where ClearML fits into the model.
This is the fourth and final post in our four-part series on Kubernetes Security for Enterprise AI Environments. The series covers tenant isolation, secrets and credential management, GPU resource governance, and production model serving security.
What Zero Trust Actually Means in Practice
The phrase “zero trust” has been stretched by vendors to describe a wide range of products and approaches. At its core, zero trust is a security model articulated in NIST Special Publication 800-207 that rests on three principles: verify explicitly (authenticate and authorize every request, using all available data points), use least-privilege access (grant only the permissions required for the specific task), and assume breach (design systems so that a compromised component cannot move laterally to compromise others).
In a traditional perimeter-based security model, the assumption is that traffic inside the network boundary is trustworthy. Once a workload is inside the cluster, it can reach any other workload by default. Zero trust rejects this assumption. In a zero-trust Kubernetes environment, a pod should only be able to communicate with the specific services it explicitly needs, all traffic should be authenticated and encrypted, and access should be re-verified continuously rather than assumed based on network location.
This is not a theoretical ideal. It is an increasingly practical requirement for organizations handling regulated data, operating multi-tenant AI platforms, or meeting the demands of modern security frameworks like SOC 2 Type II, ISO 27001, or HIPAA. This series has covered tenant isolation (Blog 1), secrets management (Blog 2), and resource governance (Blog 3). This final post covers the network and identity layer that ties them together.
The Default Kubernetes Network Model (and Why It’s Permissive)
By default, Kubernetes allows all pods to communicate with all other pods across all namespaces. There is no built-in network segmentation. A pod in the team-research namespace can make direct TCP connections to services in the production-inference namespace, to the metadata API on cloud providers, and to any endpoint reachable from the node’s network interface.
This default is intentional: Kubernetes is a general-purpose container orchestration platform, and a restrictive default would break many legitimate use cases. But for a shared AI platform with multiple teams and data governance requirements, the permissive default is a liability.
The specific risks
Without network segmentation in a Kubernetes AI cluster: a compromised training job can probe and connect to internal inference APIs, databases, and secrets management systems; data exfiltration can occur over the cluster network to any endpoint a workload can reach; lateral movement after a container escape or credential theft is unrestricted; and cloud provider instance metadata endpoints (which often provide credential access) are reachable by default from any pod unless explicitly blocked.
Kubernetes NetworkPolicy: The Foundation
NetworkPolicy resources define rules for which pods can communicate with which other pods and external endpoints. They are the primary mechanism for implementing network segmentation in Kubernetes.
A NetworkPolicy is enforced by the cluster’s CNI (Container Network Interface) plugin. Not all CNI plugins support NetworkPolicy: if you’re using Flannel without an additional plugin, NetworkPolicy resources exist in the API but have no effect. CNI plugins that enforce NetworkPolicy include Calico, Cilium, Weave Net, and Antrea. Verify your CNI supports enforcement before relying on NetworkPolicy for security.
Start with Default Deny
The correct starting posture is a default-deny policy in every namespace: all ingress and egress is blocked unless explicitly permitted. Apply this first, then add specific allow rules for the traffic that each workload legitimately requires.
With a default-deny policy applied, no pod in the namespace can receive or initiate any network connection until an explicit allow rule is added. The Kubernetes NetworkPolicy documentation covers the full syntax. Then add specific egress rules for the traffic each workload legitimately requires, which typically would be the ClearML server namespace, the object storage endpoint on port 443, and DNS resolution via kube-system.
Important: block cloud metadata endpoints
Cloud provider metadata services (169.254.169.254 on AWS, GCP, and Azure) provide instance credentials and configuration to workloads by default. In multi-tenant AI clusters, all workloads should explicitly block egress to this endpoint unless they have a specific, audited need for it. Add an egress rule that denies traffic to 169.254.169.254/32 in your default policies.
NetworkPolicy Limitations
NetworkPolicy enforces connectivity rules, but it does not encrypt traffic or verify the identity of the connecting workload. A NetworkPolicy allows or denies based on pod labels and namespace selectors, which are less robust identity signals than cryptographic certificates. If a workload is compromised and its labels are known, NetworkPolicy alone cannot prevent impersonation. For stronger guarantees, mTLS is required.
Mutual TLS (mTLS): Encrypting and Authenticating All Traffic
Mutual TLS (mTLS) is the mechanism by which each side of a connection presents a certificate and verifies the other side’s certificate before communication proceeds. In a Kubernetes context, this means every pod presents a cryptographic identity, and traffic between pods is both encrypted and authenticated at the transport layer, regardless of what the application layer does.
mTLS addresses the gaps that NetworkPolicy leaves: even if a workload can reach another based on network policy, it cannot successfully complete a connection unless it presents a valid certificate. This provides workload identity verification, not just network location.
Service Meshes: Implementing mTLS at Scale
Implementing mTLS manually across every service in a cluster is impractical. Service meshes automate this by injecting a sidecar proxy into each pod that handles certificate management, mTLS establishment, and traffic encryption transparently, without requiring changes to application code.
The two most widely deployed service meshes for Kubernetes are Istio and Linkerd. Both support automatic mTLS between all pods in the mesh, certificate rotation, and policy enforcement based on service identity. Istio is more feature-rich with a steeper operational learning curve; Linkerd is lighter-weight and simpler to operate.
In Istio’s security model, mTLS can be enforced in strict mode (all traffic must be mTLS; plaintext is rejected) or permissive mode (mTLS is preferred but plaintext is allowed). For a security-conscious AI platform, strict mode is the correct posture for all inter-service traffic.
Enabling strict mode across a namespace requires a single PeerAuthentication resource scoped to that namespace. The Istio documentation covers the full configuration. For a security-conscious AI platform, strict mode is the correct posture for all inter-service traffic.
TLS for ClearML Server Communications
ClearML’s own communication model includes several connection paths that should be TLS-secured in a production deployment:
- Browser to ClearML web server: The ClearML UI should be served over HTTPS with a valid certificate. This is handled through an Ingress controller with TLS termination in Kubernetes deployments.
- ClearML Agent to ClearML API server: Agents poll the API server for tasks and report results. This communication should use HTTPS with certificate verification enabled.
- ClearML SDK to API server: Experiment tracking calls from training code use the ClearML SDK, which connects to the API server. In production, this should use verified TLS, not self-signed certificates accepted without validation.
- ClearML file server: Artifact storage (model files, dataset snapshots, plot data) transits through the ClearML file server. This connection should also be TLS-secured.
The ClearML server security documentation covers TLS configuration for each of these components. In Kubernetes deployments via the ClearML Helm chart, TLS is configured through Ingress annotations and certificate management (typically cert-manager with Let’s Encrypt or a private CA).
Network controls define the perimeter of what can connect to what. But perimeter is not identity; a NetworkPolicy allows or denies based on pod labels and namespace selectors, not cryptographic proof of who is making the request. mTLS adds workload identity at the transport layer. Neither, however, determines what an authenticated connection is allowed to do once it arrives.
That’s where identity and access controls take over. In a standard Kubernetes deployment, RBAC handles this at the API level: ClusterRoles and RoleBindings govern which identities can perform which operations on which resources. But Kubernetes RBAC operates only on Kubernetes API calls; it does not govern access to ClearML experiments, model artifacts, queues, or inference endpoints. For AI infrastructure, you need a second access control layer that operates at the workload level. This is what ClearML’s authentication and access model provides.
Identity and Access: ClearML’s Model on Top of Kubernetes
Network security controls prevent unauthorized connections. Identity and access controls determine what authenticated connections can do. In a zero-trust model, both layers are necessary.
SSO Integration
ClearML Enterprise supports Single Sign-On (SSO) via SAML and OAuth/OIDC, as well as LDAP, integrating with identity providers including Okta, Azure Active Directory, Amazon Cognito, Google, Keycloak, Ping Identity, and Microsoft AD. This is important for two zero-trust reasons.
First, SSO removes the need for locally managed credentials in ClearML, centralizing authentication in a system that already has MFA enforcement, session management, and user lifecycle management. When an employee leaves the organization and their IdP account is deprovisioned, their ClearML access is revoked automatically.
Second, SSO enables consistent identity across systems. The identity used to authenticate to ClearML is the same organizational identity used for cloud provider access, code repositories, and other enterprise systems, making it possible to correlate activity across platforms during security investigations.
Role-Based Access Within ClearML
ClearML controls what authenticated users can do within the platform through user groups and access rules. Pre-configured groups include Admins (full platform access), Queue Admins (queue management), and Users (all workspace members including service accounts). Beyond these groups, administrators define access rules that grant specific user groups either Read Only or Read & Modify permissions to specific resources (individual projects, queues, datasets, models, or tasks) rather than applying a flat role across all resources. In a zero-trust model, this user access model should be configured with least-privilege: users have access to the projects they work on, not to all projects in the organization; administrators are a small, audited group; read-only access is the default for external reviewers or stakeholders.
Service Credentials and API Keys
ClearML uses API key/secret pairs for SDK and Agent authentication. These credentials should be treated with the same rigor as any other service credential: stored in a secrets management system (not in environment variables or config files), rotated on a defined schedule, scoped to the minimum access required, and revoked immediately when a service is decommissioned.
In Kubernetes deployments, ClearML Agent credentials should be stored as Kubernetes Secrets with etcd encryption at rest enabled (covered in Blog 2 of this series), or retrieved from a secrets management system via the Secrets Store CSI Driver at pod startup.
Audit Logging: The Zero-Trust Verification Layer
Zero trust requires continuous verification, which means logging. Without an audit trail, you cannot verify that access policies are working as intended, detect anomalous behavior, or investigate incidents after the fact. A complete audit logging posture for a Kubernetes AI platform includes several layers.
Kubernetes Audit Logging
Kubernetes audit logs record every API server request: who made it, from what IP, which resource was accessed, and whether it succeeded. For AI infrastructure, configure an audit policy that captures at minimum: all Secret accesses (read, create, update, delete), all pod creation and deletion events, all RBAC changes, and all authentication events. These events form the foundation of your security investigation capability.
The Kubernetes audit logging documentation covers the full policy configuration syntax. The key events to capture for AI infrastructure are: all Secret accesses (read, create, update, delete), pod creation and deletion, RBAC changes (ClusterRoles, ClusterRoleBindings, RoleBindings), and authentication events. These four categories form the foundation of your security investigation capability.
ClearML Activity Logging
At the application layer, ClearML maintains its own audit trail of user activity within the platform: experiment creation and deletion, model deployments, vault access, user and group changes, and API key operations. In ClearML Enterprise, these logs are accessible to administrators for review and correlation with Kubernetes-level audit events.
Putting the Full Picture Together
Zero trust for AI infrastructure is not a single product or configuration. It’s an architectural approach built from several complementary controls, each addressing a different attack vector. This series has covered the full stack.
Tenant isolation
Virtual clusters via k3k give each team their own API server. Compromise in one tenant cannot reach another. (Blog 1)
Secrets management
ClearML Vaults injects credentials at runtime. Workloads never own their credentials. (Blog 2)
Resource governance
Resource Policies and Platform Management Center ensure no workload exhausts shared capacity or runs unattributed. (Blog 3)
Network segmentation
NetworkPolicy with default-deny limits blast radius. No pod can reach what it hasn’t been explicitly permitted to reach.
mTLS
Every connection between services is encrypted and carries cryptographic workload identity. Impersonation requires a valid certificate.
Identity and SSO
Centralized identity with MFA means every human and service action is attributable to a verified identity
Zero Trust Readiness Checklist for Kubernetes AI Infrastructure
- NetworkPolicy default-deny applied in all namespaces with explicit allow rules for required traffic
- Cloud metadata endpoint (169.254.169.254) blocked via egress NetworkPolicy
- CNI plugin confirmed to enforce NetworkPolicy (Calico, Cilium, Antrea, or Weave)
- mTLS enforced in strict mode for inter-service traffic via service mesh or application-layer TLS
- ClearML server components served over verified TLS (not self-signed certificates ignored by SDK/Agents)
- etcd encryption at rest enabled for Kubernetes Secrets
- ClearML Agent API credentials stored in secrets management, not environment variables
- SSO enabled with MFA enforced at the IdP level
- ClearML user roles configured with least-privilege scoping per project
- Kubernetes audit logging capturing Secret access, pod lifecycle, and RBAC changes
- Virtual cluster isolation (k3k) for teams with different data governance requirements
- ClearML Resource Policies enforcing GPU quotas and concurrent experiment limits
This series has moved through four layers of the same problem: who can access what, how credentials are protected, who gets how much compute, and whether the network itself enforces any of it. None of these layers is sufficient on its own. RBAC without network segmentation leaves lateral movement open. Network segmentation without identity means a label is your trust signal. Identity without audit logging means you’ll never know if any of it worked. The controls compound. That’s the point.
Get in touch if you would like to discuss how ClearML can support your organization’s AI security requirements.