By Adam Wolf
Two years ago, GenAI in production usually meant a single LLM serving a single endpoint. In 2026, it usually means much more. The applications shipping in front of users today are compound AI systems: orchestrated pipelines of retrievers, embedders, dialogue models, classifiers, code interpreters, SQL executors, and tools, with a single user request fanning out to several model calls across the stack. The infrastructure patterns that worked for single-model serving do not work for these systems, and the gap is now well-documented in production literature. This piece walks through what changes, what breaks, and what compound AI systems now require from the infrastructure layer beneath them.
The shift from models to compound systems
The vocabulary for this shift was crystallized in a February 2024 Berkeley AI Research blog post by Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. Their argument was that “state-of-the-art AI results are increasingly obtained by compound systems with multiple components, not just monolithic models.” The piece cited Google’s AlphaCode 2, which set state-of-the-art programming results by generating up to one million possible solutions for a task and then filtering the set, as the canonical example of system-level engineering producing gains a single model could not match alone.
Two years on, the production data has caught up with the thesis. An April 2026 production deployment study from Salesforce covering 12+ months of operating compound AI inference at enterprise scale (8,000 users, 722,000 daily LLM inferences on average, 1.4 million on peak business days, 136 billion tokens processed in March 2026 alone, and 8.7x year-over-year request volume growth) characterizes the infrastructure layer that compound AI systems require and documents the patterns that emerge in production. The patterns the Salesforce team identified map closely to what teams running compound AI systems in industry after industry are now hitting in their own deployments.
Four of those patterns matter most to anyone building or operating the infrastructure underneath. They are the ones single-model serving did not surface, and they are the ones a chatbot-shaped mental model will get wrong.
Pattern 1: Fan-out amplification
A single user request to a compound AI system does not produce a single model call. It produces several.
A typical agent query at Salesforce, per the production study, fans out to two to four model invocations. The same paper notes that across compound AI systems more broadly, a single user request commonly triggers three to five distinct model calls. The mix is workload-dependent: an embedding model for retrieval, a dialogue LLM for generation, a classifier for intent or safety, sometimes a code interpreter or a SQL executor, and occasionally a vision model. From the user’s perspective there is one request. From the infrastructure’s perspective there are several, arriving simultaneously at different model backends.
The implication for capacity and scaling is direct. If your platform scales models based on the aggregate user request rate, the scaling decisions for individual models will be wrong. The embedding model that runs on every request needs to scale at the full user-request rate. The dialogue LLM that runs on 70% of requests needs to scale at 70% of that rate. The SQL executor that runs on 25% of requests needs to scale at a quarter of that rate. Scaling them all together produces over-provisioning of the rarely-invoked components and under-provisioning of the always-invoked ones. The Salesforce data showed this clearly under a 10× traffic spike: embeddings scaled 10× because every request needed them, the dialogue LLM scaled 6 to 7×, and a SQL executor scaled 2 to 3× because it was conditionally invoked. Uniform scaling would have over-provisioned the SQL executor by 3 to 5× while potentially under-provisioning embeddings. Per-model invocation tracking, rather than aggregate user-request tracking, is what gets that right.
Pattern 2: Heterogeneous latency profiles and resource contention
The components in a compound system do not share latency characteristics. An embedding model may respond in 50 milliseconds. A dialogue LLM may take three to five seconds. A classifier may finish in 20 milliseconds. A code interpreter may run for tens of seconds. When these models share infrastructure, the slow ones can starve the fast ones of resources, and the fast ones can be blocked behind the slow ones in shared queues.
The Salesforce paper documents a specific manifestation of this: a fast embedding model competing for GPU resources with a slow dialogue LLM in a shared queue. Without latency-class-aware routing, the embedding model’s 50-millisecond response time gets multiplied by however long the LLM takes to clear the queue ahead of it. A compound system request that should have started its first model call in 50 milliseconds instead waits seconds for upstream resources to free up. The fast-path operation becomes the bottleneck, not because it is slow, but because it is blocked.
The fix is request priority queuing aware of compound system invocation patterns and, at the deployment level, the freedom to assign different models to different deployment modes. The dialogue LLM with steady high QPS and strict latency requirements might run on dedicated capacity. The embedding model with high volume but fast cold start might run serverless. The SQL executor invoked conditionally might run serverless with no provisioned concurrency. The Salesforce team reported that this kind of mixed-mode deployment reduced total inference cost by an additional 15 to 20% compared to pure serverless or pure dedicated strategies, because each model used the mode that fit its traffic pattern.
Pattern 3: Cascading cold starts
The most counterintuitive infrastructure pattern in compound AI systems is what happens at cold start.
After an idle period, the first user request triggers cold starts across multiple models simultaneously. The intuition is that the effective cold-start latency is the maximum of the individual cold-start times. The reality, in production, is worse. Dependency chains in the compound system mean that some cold starts cannot begin until upstream models produce results. The embedding model has to finish before the dialogue LLM can generate a context-aware response. The serial dependency turns a 150-second maximum cold start into a 180-second effective cold start.
The Salesforce paper measured this directly: three models with cold-start times of 30 seconds (embedding), 150 seconds (dialogue LLM), and 20 seconds (classifier), with dependency relationships between them, produced an effective compound cold-start latency of about 180 seconds. Not 150 seconds. Cold starts compound multiplicatively through dependency graphs, not additively.
The mitigation strategy that the Salesforce team documents has three tiers. Coordinated pre-warming, which proactively warms downstream models when any model in a registered pipeline is first accessed, reduced compound cold-start latency by 65% in their measurements (from ~180 seconds to ~65 seconds). Tiered provisioned concurrency, which maintains warm capacity only on the critical-path model (typically the LLM with the longest cold start), provisions only the bottleneck model at roughly 20% of the cost of provisioning all models while eliminating roughly 70% of the user-perceived cold-start delay. Predictive warming from traffic signals, which pre-loads models before anticipated demand, eliminated more than 90% of cold starts during peak hours for the Salesforce Agentforce workload.
The principle is the same across all three: cold-start mitigation in a compound system is a pipeline-level concern, not a per-model concern. The model that warms the cheapest is not always the model that matters most for user-perceived latency.
Pattern 4: Multi-component overhead and the cost of serial dispatch
The fourth pattern is the one that determines whether a compound system is operationally viable at all. When the multiple model calls in a compound pipeline are dispatched in parallel, the coordination overhead is small. When they are dispatched serially, it is the dominant cost.
Salesforce’s measurements show fan-out overhead of 45 to 80 milliseconds when models are dispatched in parallel through their architecture, representing less than 2% overhead relative to a total agent response time of five to eight seconds. The same set of models dispatched sequentially (as the legacy architecture required) added 1.5 to 3 seconds of serial waiting. For pipelines with three or more model calls, that serial waiting time made interactive response times infeasible. The infrastructure architecture choice (parallel dispatch versus serial) was the difference between a compound system that worked in production and one that did not.
The lesson is that parallel dispatch and event-driven orchestration are not optimizations. They are preconditions. A compound system that has to wait for each model call to complete before initiating the next one is not going to clear interactive SLAs at three-plus components. The orchestration layer above the inference layer has to be able to fan out concurrent invocations, aggregate responses asynchronously, and tolerate per-component failures without cascading. The inference layer underneath has to be able to serve those concurrent invocations without blocking them on each other.
What this means for the infrastructure underneath
The four patterns above point at a coherent shape for the infrastructure layer that compound AI systems require. It is not a list of point optimizations. It is a set of architectural commitments.
First, independent per-model scaling, driven by per-model invocation rates rather than aggregate user-request rates. This is the foundation that makes everything else work, because without it the heterogeneous scaling problem produces either over-provisioning or starvation regardless of any other optimization.
Second, mixed-mode deployment, with the ability to assign each model in a compound pipeline to the serving mode (dedicated, serverless, hybrid) that fits its individual traffic pattern. A pipeline whose every model runs in the same mode is paying a tax somewhere, either in cold-start latency for the high-QPS components or in dedicated cost for the rarely-invoked ones.
Third, pipeline-aware orchestration with parallel dispatch and circuit breakers. A compound system that fans out three model calls and aggregates their responses should not have a serial waiting profile. A compound system that has one component fail should degrade gracefully, not cascade.
Fourth, coordinated cold-start mitigation, with knowledge of which models belong to which pipelines and the ability to warm them together when one is first accessed. The dependency graph that produces multiplicative cold starts is also what enables coordinated pre-warming to be effective.
Fifth, pipeline-level observability that captures not just per-model latency, throughput, and error rate, but the decomposition of agent-level response time across the components that produced it. The Salesforce team explicitly called out that model-level observability is insufficient for compound systems: a model operating within its individual SLA can still cause agent-level SLA breaches when combined with other models, and finding the critical-path bottleneck requires observability above the model layer, not just at it.
Where ClearML’s GenAI App Engine fits
ClearML’s GenAI App Engine is the inference infrastructure layer for compound AI systems. It does not replace the orchestration layer above it (frameworks like DSPy, LangGraph, or custom agent runtimes), and it does not replace the application logic that defines what the compound system actually does. What it provides is the inference backend that compound systems depend on, designed around the patterns above.
Multi-engine model serving. The GenAI App Engine deploys LLMs through serving engines, including vLLM, Triton, and Llama.cpp, with a one-click deployment path from a custom or fine-tuned model on Hugging Face to a secure API endpoint. The compound system above can route to different models on different engines without the orchestration layer needing to know which engine is serving which call.
Dynamic traffic routing and per-endpoint autoscaling. ClearML’s dynamic traffic routing manages load balancing and compute allocation per endpoint, scaling horizontally as demand on individual models changes. This is the independent per-model scaling that the fan-out and heterogeneous-scaling patterns require, with each model in a compound pipeline scaling at the rate its own invocations actually demand.
Unified memory technology for idle models. The GenAI App Engine’s unified memory technology holds idle models in active CPU memory rather than scaling them fully cold, conserving GPU power for active models while keeping warm-up costs lower than a full cold start. For compound systems where the cold-start cascade is dominated by the slowest component, this changes the math on what “cold” actually costs.
Endpoint monitoring across the surface that compound systems live on. The platform monitors all AI API traffic, with per-endpoint request volume, latency, memory usage, and resource utilization across CPU, GPU, I/O, and network. That is the data surface a compound system needs to do pipeline-level decomposition, attribute critical-path delays to specific components, and answer the “which model is the bottleneck” question that single-model observability cannot.
RBAC, authentication, and AI agent visibility. Compound systems do not run as anonymous services; they run in enterprise environments where identity, access control, and audit matter. ClearML’s App Gateway exposes deployed endpoints with secure access control and tenant-aware scoping. Deployed AI agents are tracked through the same surface, with usage and performance visible alongside the model endpoints they depend on.
What ClearML’s position adds up to is the inference plane underneath the compound system: the layer that handles serving, scaling, monitoring, and access control for the individual model components, leaving the orchestration framework above to handle the pipeline logic, the parallel dispatch, and the application-specific behavior. The two layers are designed to be separable, because that is what production compound AI architectures have converged on.
The infrastructure question worth asking
Most enterprises running GenAI in production today are running compound AI systems, whether or not they call them that. A RAG-backed chatbot is a compound system. An agent that calls tools is a compound system. A summarization pipeline with a guardrail model is a compound system. The mental model of “the AI workload” being a single LLM serving a single endpoint is, in most production environments, no longer accurate.
The infrastructure question worth asking, then, is not whether your platform can serve a model. It is whether your platform can serve the compound system that is actually running on top of it. Can it scale each component independently based on its own invocation rate? Can it dispatch parallel calls without serial waiting? Can it survive a cold start in one component without breaking the SLA on the pipeline? Can it decompose pipeline-level latency back to the component that caused it? Can it support per-component A/B testing without disrupting the rest of the pipeline?
If the answer to any of those questions is no, you do not have compound AI infrastructure. You have model-serving infrastructure with compound AI workloads running on top of it, paying the translation cost in latency, cost, and reliability that the literature now documents at scale. ClearML’s view is that the infrastructure plane underneath compound AI systems deserves to be designed for what is actually running on it, and the GenAI App Engine is built around that premise.
If you would like to discuss how ClearML’s GenAI App Engine fits into your compound AI infrastructure, get in touch.