As compute gets increasingly powerful, the fact of the matter is: most AI workloads do not require the entire capacity of a single GPU. Computing power required across the model development lifecycle looks like a normal bell curve – with some compute required for data processing and ingestion, maximum firepower for model training and fine-tuning, and stepped-down requirements for ongoing inference.
Despite that reality, organizations will continue making significant investments in compute because they cannot fully optimize existing resources. Splitting and slicing GPUs is one way to tackle the issue of getting more from each chip (if they know how to do it). By harnessing the power of fractional GPUs, AI infrastructure managers can pre-determine partition sizes for their teams and require stakeholders to use the slice best-suited for the size of their job. With the right setup and process, teams can improve workload throughput up to a factor of 10, significantly speeding up the timeline for model development and reducing time to production. But while fractional GPUs are a fantastic solution for teams with models in different stages of development or production, when managing multiple teams using the same shared infrastructure, you need a more macro-level control mechanism.
For controlling the compute access of multiple business units or departments across a large, scaled-up infrastructure, the answer is multi-tenancy.
In computing, multi-tenancy is a software architecture where a single instance of an application (or in this case, a computing resource) serves multiple individual user groups, often referred to as tenants. It offers a way to share resources efficiently amongst those users.
Multi-tenancy in Computing
Conceptually, multi-tenancy in computing is rather similar to multi-tenancy in real estate. For properties with individual living spaces and shared services (for example, mail delivery), each individual unit (or tenant) has their own private space, isolated and protected from other units. Services to manage the building or infrastructure can be done for all tenants or individually.
For computing, multi-tenancy follows the same principles. Despite sharing the same resources, tenants are siloed and access only what they have been allocated. As more teams, departments, and business units begin experimenting with Generative AI, demand for computing power is increasing and becoming more challenging to satisfy, especially if the computing resources are centrally managed. Allocating who gets what, controlling user access, and enabling secure parallel workloads that don’t conflict with other users’ workloads is part and parcel to implementing multi-tenancy.
What Is Secure Multi-tenancy?
Since multi-tenancy is implemented with efficiency and security in mind, complete network separation is a vital aspect. Each tenant should operate within their own distinct network, safe from the potential threats of other tenants. By isolating each tenant on their own network, organizations can guarantee the utmost privacy and data integrity, allowing tenants to fully utilize their allocated GPU resources without any concerns of unauthorized access or interference. With complete network separation, multi-tenancy GPU clusters offer an unrivaled level of control and protection, giving tenants and the infrastructure owner peace of mind.
The Benefits of Multi-tenancy
In addition to its expanded capability that satisfies more AI stakeholders, a multi-tenant structure allows infrastructure leaders to operate more efficiently through dynamic resource management. Without multi-tenancy, IT teams must resort to using static, hard-set rules to allocate resources, which is cumbersome to update and risks under-serving AI stakeholders. It’s hugely inefficient. In that scenario, some teams will be sitting and waiting for compute even if other teams have allocations which are sitting idle. Multi-tenancy solves these issues dynamically and ensures:
- Cost Effectiveness and Transparency – Investing in cloud compute, whether on-prem or cloud, is a significant line item in the IT budget. Sharing existing compute amongst more users is an easy way to stretch resources without additional investment. Granular usage-level reporting can also show how each tenant consumes compute, data, and microservices, making it easier for organizations to do internal billing (chargebacks) to their AI stakeholders.
- Higher Utilization – Despite most organizations believing that they need more computing firepower, in a recent survey of 1,000 IT leaders, 68% of respondents estimated GPU utilization to be under 70% (or even under 50%!) during peak periods. Consider the lost opportunity for more AI development, not to mention the inefficient ROI. To maximize the utilization of those GPUs, infrastructure engineers should layer in fractional GPU capabilities to right-size computing power for AI workloads in addition to multi-tenancy. Through well-setup partitions, it’s possible to do “smaller jobs” such as data pre-processing or inference on the same chip.
- Data Security – Multi-tenancy ensures no data leakages between tenants. No one wants R&D’s models made visible to Sales (except the sales team, of course). Multi-tenancy is another mechanism to secure data by creating silos for isolated computing and data storage connections.
- Scalability – When your infrastructure is already set up for sharing, it is easy to shift resources around because you can see who is fully using their compute allocation versus teams that do not. The elasticity of your infrastructure makes it simple to “rack and stack.” Adding more clusters or additional cloud providers to increase computing power overall is an easier task and will not affect users with system downtime.
Not all Multi-tenancy is Created Equal
True multi-tenancy controls access to each tenant’s own isolated network, storage, and compute allocations. It’s important to ensure tenants are secured and cannot access the control plane that maintains the silos.
Critical factors for successful multi-tenancy include:
- Strong security with granular permissions management (preferably Role-based Access Control).
- Resource allocation policy management that makes it easy for infrastructure leaders to control access to resources using rules, hierarchies, and logic.
- User-level reporting for understanding real-time usage and accurate billing (for organizations that issue chargebacks or invoices to their tenants).
- Tools for improving GPU utilization to help with maximizing your existing infrastructure investment and control costs.
- Flexibility to ensure your organization is not locked in to buying infrastructure from a single vendor or cloud provider forever.
- Complete network separation so each tenant truly sits on their own network, isolated and unable to access any other tenant’s network.
Conclusion
Infrastructure leaders embarking on the multi-tenancy journey for their AI infrastructure have many considerations and challenges for security and resource optimization, but successful implementation will ultimately increase utilization and ROI. For large organizations with centrally-managed compute, it’s a no-brainer.
ClearML supports secure multi-tenancy which creates isolated networks for tenants to run their AI workloads, protected from other tenants. Infrastructure leaders and IT teams can easily manage each tenant’s computing resource allocations, run real-time usage reports, and implement additional security features, such as role-based access control (RBAC), SSO, and LDAP integration.
To learn more about how ClearML supports multi-tenancy with enterprise-grade security, easy-to-manage resource allocation policies, and features that can boost AI throughput by 10x, please request a demo to speak to someone on our sales team.