By Adam Wolf
Most enterprises sized their AI infrastructure with a playbook written for training. However, training is no longer the typical workload. Inference now eats up roughly two-thirds of all AI compute, and it is changing shape fast enough that the rules of thumb from 18 months ago just do not hold. Our view at ClearML is pretty simple: when the workload shifts this much, the platform underneath it has to shift with it. This piece walks through what has actually changed about inference, why training-shaped capacity plans fall apart, and how we think about the platform layer that makes the new shape of inference workable.
The capacity question has changed
Walk into a CIO conversation about AI infrastructure in 2024 and the discussion was usually about training: How many H100s needed to fine-tune a frontier model. How long the run would take. How the cluster would bisect under multi-node communication. Inference was the afterthought; the part you sized at the end based on rough QPS estimates against a model you already had.
Walk into the same conversation in 2026 and the proportions have flipped. Bloomberg Intelligence data places inference at roughly two-thirds of all AI compute today, and analysts project that share to keep widening. AWS’s backlog has reached $244 billion, up 40% year-over-year. Oracle’s stands at $523 billion. Microsoft has $80 billion in unfulfilled Azure orders because GPUs sit waiting for data center power. Most of that demand is inference
The capex numbers tell the same story. The four largest hyperscalers committed roughly $725 billion in combined capex for 2026, with around 75% of that going into AI-specific infrastructure. Microsoft CFO Amy Hood attributed $25 billion of Microsoft’s increase specifically to memory and chip cost pressure, which is the part of the bill of materials that goes up most when you are building inference fleets, not training clusters. Custom ASICs are taking a growing share of inference, and some analysts project a forward-looking estimate that some caution may prove aggressive. The aggregate market is growing fast enough that NVIDIA’s absolute volume still rises. The change that matters is structural: inference is now an architecturally distinct workload that the entire industry is rebuilding around.
The implication for enterprise capacity planning is direct. The math you used to size a training cluster does not produce the right answer for an inference fleet. The mistake we see most often at ClearML is treating inference like a smaller, simpler version of training. It is not, and the platforms that were designed around training-shaped assumptions show that limitation immediately when the inference load arrives.
Why inference math is not training math
Training is a batch workload with predictable resource needs. You know the model size, you know the dataset, you know the global batch size, and you know roughly how long the run will take. Capacity planning for training is mostly the question of how many GPUs run for how long.
Inference is a continuous-flow workload with stochastic resource needs. You do not know what the next request will look like. You do not know whether it will be a 200-token chat reply or a 50,000-token document summarization. You do not know whether the user will engage a reasoning mode that generates ten times more tokens than the same prompt would have produced last year. You do not know the concurrent load five minutes from now. The capacity question is not how many GPUs you need for the job. It is how many GPUs you need to keep latency under your service-level objective (SLO) at the concurrency you expect, including the worst hour of the worst day.
Three things make that math harder in 2026 than it was in 2024, and each one shows up in ClearML deployments as a distinct operational challenge.
Reasoning model token economics. Models like OpenAI’s o-series, DeepSeek’s R1, and the reasoning modes in frontier models from Anthropic and Google generate hidden reasoning tokens that never appear in the final response but consume GPU compute and KV cache memory like any other token. Reasoning models routinely generate 10 to 100 times more tokens per query than non-reasoning models, and the accuracy-versus-reasoning-compute relationship follows a log-scale curve, which creates real product pressure to let reasoning models think longer. A capacity plan that assumes 500 average output tokens per query is going to be wrong by orders of magnitude when a reasoning mode gets switched on, and the teams running on ClearML who have caught this earliest are the ones who instrumented per-tenant compute consumption before they let the feature into production.
KV cache dominance at long context. For each token generated, modern transformer inference appends key and value tensors at every attention layer, and those tensors stay in GPU memory for the lifetime of the request. At short contexts (under 4K tokens), the KV cache is negligible. At 32K to 128K tokens, it becomes the dominant memory consumer on the GPU. For Llama-3-8B in half precision, the KV cache requires about 0.5 MB per token. At one million tokens of context, you need roughly 125 GB of GPU memory just for the cache, before model weights, before activations. Long contexts and reasoning chains do not just stress compute. They stress memory in ways that capacity-planning spreadsheets built around model weights miss entirely, and that ClearML’s continuous monitoring surfaces alongside throughput so the team sees memory saturation as it builds rather than as the page does.
Heterogeneous phase requirements. Modern inference has two computationally distinct phases. Prefill processes the prompt in parallel and is compute-bound. It benefits from peak-FLOPS hardware: H100, B200, B300. Decode generates output tokens one at a time autoregressively and is memory-bandwidth-bound. It benefits from HBM bandwidth and capacity: H200, B200, B300, AMD Instinct MI300X and MI350X. Running both phases on the same GPU produces resource contention that caps throughput. Disaggregating them onto separate GPU pools, which is the architecture now shipping in vLLM v0.8+ (via NixlConnector), NVIDIA Dynamo 1.0, and the llm-d open-source reference stack, can deliver 1.4 to 2.3 times TTFT speedup and 35 to 46% cost reduction in some published configurations. It also means your GPU fleet is no longer homogeneous, which is exactly the orchestration problem ClearML’s silicon-agnostic platform is built to solve.
The new architectural fact: prefill and decode are different workloads
The disaggregation pattern is the most important architectural change in enterprise inference since 2024. Production-grade inference stacks (vLLM, llm-d, NVIDIA Dynamo, SGLang, LMDeploy) have all converged on the idea that prefill and decode should be treated as separate workloads with separate GPU pools that scale independently. ClearML’s position is that the inference runtime is the right place to optimize the inside of the container, and the platform layer is the right place to orchestrate across the pools. Trying to do both in the same component is what produced the operational debt many enterprises are now unwinding.
For capacity planning, that means the question of how many GPUs you need for a model is now actually two questions. How many prefill GPUs do you need to keep time-to-first-token under your SLO at your expected prompt-length distribution? And how many decode GPUs do you need to keep inter-token latency under your SLO at your expected concurrency and output-length distribution? The two answers are not the same number, and the ratio between them depends on your workload. Chatbot-style applications with short prompts and verbose responses tend to need decode-heavy fleets. Long-context summarization workloads tilt toward prefill. Reasoning-heavy applications with extensive chain-of-thought generation push the decode side significantly further.
It also means that the right GPU model may differ between the two pools. A common pattern in 2026 is to mix H100s in the prefill pool with H200s in the decode pool, or to use AMD MI300X-class accelerators in the decode pool specifically because of their HBM3e capacity. Heterogeneous GPU fleets are no longer an edge case. They are an emerging best practice. ClearML’s control plane was designed for this case from the start, with workload routing, scheduling, and quota enforcement that work across mixed silicon, mixed clouds, and mixed deployment topologies.
This has consequences for procurement, scheduling, and operations that most capacity-planning processes have not absorbed yet.
GPU memory is no longer a fixed ceiling
Training-era capacity math assumes a GPU’s high-bandwidth memory (HBM) is a hard ceiling: if the model and its working set do not fit, the job does not run. Inference is loosening that assumption. Modern accelerators and inference runtimes can now move part of the working set, especially the KV cache, out of GPU memory and into the much larger pool of CPU host memory, then bring it back as needed.
This happens at two levels. In hardware, unified-memory designs such as NVIDIA’s Grace Hopper and Grace Blackwell connect the CPU and GPU over a coherent NVLink-C2C link so the two share one memory space and the GPU can reach into CPU memory when its own runs short. In software, inference engines do the equivalent through CPU swapping: vLLM can offload KV cache blocks to CPU memory, and NVIDIA Dynamo tiers the KV cache across GPU, CPU, and storage.
Those hardware and runtime capabilities are real, but they have limits. Hardware-unified memory depends on specific, expensive silicon like Grace Hopper or Grace Blackwell, not the discrete GPUs most fleets actually run, and the runtime methods in vLLM and Dynamo work inside a single serving engine aimed mainly at fitting one model’s working set. Neither addresses the problem most teams hit in production: more models than they have GPUs to keep loaded. ClearML’s Unified Memory Technology targets exactly that, using software, on the GPUs you already have, whether NVIDIA or AMD. Instead of swapping cache blocks for one model, it parks whole models, weights and caches, in CPU memory and pulls the requested one into VRAM on demand, automatically and with no manual setup, so a single GPU keeps many models hot and ready with near-zero switching latency. Put simply, the hardware and runtime approaches help one model fit a constrained GPU; ClearML’s feature lets one GPU serve many models, which is where the consolidation and cost savings come from.The capacity-planning consequence is that effective memory per GPU is no longer just its HBM. A decode pool serving long-context requests can hold more concurrent sessions than its raw HBM would suggest, at the cost of some added latency when blocks move between tiers, and that tradeoff belongs in the workload model rather than being discovered in production. For the platform layer, the takeaway is that GPU memory can no longer be treated as a single fixed number per card. The workload model, quotas, and scheduling that ClearML enforces have to reflect effective capacity, including memory that tiers out to the host, not raw HBM alone.
What capacity planning should actually look like
A serviceable capacity plan for production inference in 2026 needs five things that a training-shaped plan does not, and each one is something ClearML is built to support directly.
A workload model that distinguishes phase. You should be able to articulate the expected prompt length distribution, the expected output length distribution, the expected concurrent request rate, and the expected reasoning-mode usage as a function of time of day. The KV cache footprint and the prefill-to-decode ratio fall out of that workload model. Without it, you are essentially guessing. ClearML’s observability surface, which captures request-level metrics across deployed inference endpoints, is what most platform teams use to build that model from real traffic rather than estimates.
GPU heterogeneity by design. The cluster should be planned to support multiple GPU SKUs serving different phases of the same model, with the scheduling and routing logic to land prefill requests on the prefill pool and decode requests on the decode pool. ClearML orchestrates across heterogeneous GPU pools natively, including the mixed NVIDIA and AMD Instinct deployments that several customers now run in production.
Reasoning-aware sizing. If your application supports reasoning modes, even optionally, the capacity plan needs to account for the long tail of high-token responses. The right way to do this is to build the workload model with an explicit reasoning percentage and a token-budget distribution for reasoning responses, then size against the 95th-percentile total token consumption per request rather than the mean. ClearML’s quota and fair-share scheduling lets you enforce that sizing in production, so a reasoning-heavy team cannot starve a latency-sensitive one even when both are running on the same shared fleet.
Fractional GPU allocation. Not every model needs a full H200. Smaller models, distilled variants, and inference services with low QPS can share a GPU through partitioning. ClearML supports fractional GPU allocation across both NVIDIA MIG and AMD’s compute and memory partitioning, which we validated at 8× MI300X scale on ROCm 7.0.1 using CPX compute partitioning and NPS4 memory-locality partitioning. Fractional GPU support is the difference between a 60% utilized inference fleet and an 85% utilized one. At the prices the inference market is operating at, the gap pays for itself many times over.
Continuous observability tied to capacity decisions. Inference fleets do not have a finish line. They run continuously, they degrade gracefully under load, and they need capacity decisions made in production, not at procurement time. ClearML’s monitoring stack surfaces time-to-first-token, inter-token latency, KV cache utilization, prefill and decode queue depths, and per-tenant compute consumption in a single dashboard, with the alerts and policies in place to drive capacity actions rather than just generating reports.
How ClearML approaches inference capacity
ClearML’s view is that inference is not a separate concern from the rest of the AI platform. It is the workload that ends up consuming the largest share of compute, and it needs to be governed by the same control plane that handles training, fine-tuning, experimentation, and data pipelines. Four capabilities sit at the center of how we approach it.
One control plane for training and inference. The same orchestration layer that runs your training jobs runs your inference services. The same quotas, the same identity model, the same scheduling policies, the same observability. Teams that built their inference stack on a separate platform from their training stack end up reconciling two operating models, two cost ledgers, and two on-call rotations. ClearML’s position is that training and inference are phases of the same workload lifecycle and belong in the same control plane.
Quotas, fair-share scheduling, and multi-tenancy. Inference fleets in 2026 are not single-tenant. Multiple teams share the same GPUs, often with very different latency requirements and very different token economics. ClearML’s multi-tenant compute governance lets you enforce the workload model you sized against. One team’s reasoning workload cannot starve another team’s chatbot. A burst of long-context summarization cannot push a real-time agent past its SLO. The fairness machinery is the same machinery that already governs training jobs and notebook sessions, applied to inference endpoints.
Fractional GPU allocation across NVIDIA and AMD. ClearML is silicon-agnostic by design. The platform manages fractional GPU allocation across NVIDIA accelerators using MIG and across the AMD Instinct family using AMD’s compute and memory partitioning, with validated support for ROCm 7. For a heterogeneous inference fleet, that means a single scheduling layer covers both halves of the cluster, including the case where prefill runs on one silicon vendor and decode runs on another.
The App Gateway for inference endpoints. ClearML’s App Gateway is where deployed inference services meet the applications consuming them. Managed networking, identity-aware access control, authentication tied to your existing identity provider, per-tenant autoscaling, and per-tenant isolation all live at the gateway layer. The inference runtime underneath can be vLLM, NVIDIA Dynamo, AMD AIMs, NVIDIA NIMs, a custom container, or any combination. The gateway abstracts those choices behind a consistent operational surface, so the production controls for your inference services do not have to change every time the inference runtime evolves.
None of that replaces the workload modeling, the GPU procurement decisions, or the SLO definitions that your team has to do. It connects them. A capacity plan you cannot enforce is a forecast, and a forecast you cannot measure against is a wish.
The capacity question worth asking
The right capacity question for a 2026 inference fleet is not “how many GPUs do we need?” It is “how do we keep our P95 time-to-first-token under 500 milliseconds and our P95 inter-token latency under 100 milliseconds at the concurrency we expect, including reasoning workload, including long-context workload, including the worst hour of the worst day, while keeping utilization high enough to justify the spend?”
That question has a different answer than the training-shaped one, and the answer depends on workload shape, hardware mix, software stack, and operational discipline in ways that do not show up in a procurement spreadsheet. ClearML’s view is that the platform layer is where these dimensions come together and where the capacity question actually gets answered, not in a spreadsheet and not at procurement time but continuously, in production, against the workload you actually have rather than the one you forecast.
The enterprises that figure this out early are the ones whose AI applications scale gracefully into production. The enterprises that do not will discover, the way several public benchmarks have already shown, that an inference fleet sized like a training cluster is the most expensive way to deliver mediocre latency.
If you would like to discuss how ClearML fits into your inference capacity planning, get in touch.