Google’s Test Says Inference Capacity Can Be Pooled Across Regions

A large AI allocation no longer has to arrive in one region before it becomes useful. Google Cloud’s latest GKE Inference Gateway results make that case with operating evidence: in a three-region test, a distributed serving deployment reached 8,457 tokens per second, while Google reports routing overhead of less than 1% relative to a direct call to a local cluster.

That does not establish a universal performance result. It is a reported test of one unspecified mixture-of-experts model, served through SGLang, on Google Cloud’s own infrastructure. But it changes a practical planning assumption. Teams facing fragmented accelerator availability can evaluate capacity across regions as one prospective serving fleet, rather than treating every regional allocation below their target size as operationally insufficient.

What changed: evidence, not the product’s availability

On September 21, Google published results from a multi-region GKE deployment spanning 17,000 compute nodes in the United States and Europe. The disclosed throughput experiment placed three GKE clusters in us-east5, us-west8 and europe-west4 behind one global virtual IP. Google said the system served an unspecified mixture-of-experts model with SGLang.

The gateway itself is not new. Google announced the multi-cluster GKE Inference Gateway in preview on March 17. Nor does the September post announce a new general-availability tier. The material update is measured operating evidence about how the architecture behaved when traffic was spread across three regions.

Google reported 2,898 tokens per second with one cluster, 6,380 with two, and 8,457 with three. Reported success rates were 99.87%, 99.95% and 99.90% respectively. The company also reported that gateway-routed traffic achieved 99.5% of the throughput of a direct local-cluster call—its basis for saying routing overhead was below 1% in this test.

“The relevant planning unit is no longer necessarily the largest regional allocation. It can be the reachable fleet, provided routing understands the condition of model servers.

The important mechanism is model-state-aware routing

Conventional load balancing can distribute requests by a simple pattern such as round robin. That is inadequate for inference serving when nominally identical endpoints carry very different amounts of active model state. A request may be cheap to continue in the region that already holds relevant state and expensive to place elsewhere; a server can be reachable while nearing a condition that will impair useful work.

Google’s routing layer consumed live KV-cache-utilization telemetry from model servers. In the reported configuration, traffic began spilling into another region when the primary region crossed a configured 40% KV-cache-utilization threshold. This is the architectural distinction that matters. The gateway is not merely selecting a geographic endpoint. It is making a traffic decision from a signal tied to the working state of the serving system.

Google’s documentation describes the broader pattern as a single private endpoint that pools accelerator capacity across regions and routes overflow when the preferred region reaches capacity. The September test supplies a concrete example of the pattern at scale. It is not an argument that latency, cost or reliability will be equivalent across every geography and workload.

What enterprise infrastructure teams should decide now

The immediate decision is not whether every inference workload should become cross-region. It is whether a large single-region allocation remains a prerequisite for starting a production service. For capacity-constrained teams, the answer may increasingly be no—but only after testing the actual workload across the proposed regional topology.

• Treat capacity discovery as fleet design. Inventory usable allocations by region, then ask whether their combined capacity meets the service target. Do not discard smaller allocations automatically because none is sufficient alone.
• Make serving telemetry part of routing design. KV-cache pressure is the signal disclosed in Google’s test; other model servers and workloads may require different signals. The principle is to route on conditions that affect useful inference work, not merely on endpoint reachability.
• Set a preferred-region and overflow policy deliberately. A global address does not remove data-location requirements, latency objectives or regional failure boundaries. Those constraints must determine which regions are eligible before capacity pressure determines where traffic goes.
• Run a workload-specific comparison. Measure throughput, success behavior and the user-relevant latency profile for direct local serving and for the proposed multi-region path. Google’s less-than-1% overhead applies to its reported test, not automatically to another model, framework, network path or deployment.

What the results do not show

The figures should be read with discipline. Google does not disclose the node allocation behind the three-cluster throughput table, the accelerator count or models, or the identity and configuration of the mixture-of-experts model. The 17,000-node deployment description does not mean every node participated simultaneously in that table. The test was not multicloud: all disclosed clusters ran on GKE in Google Cloud regions.

Nor are the success rates an availability SLA, and the test does not demonstrate dollar savings or a general utilization benchmark. The throughput progression is strong operating evidence for this architecture under these conditions. It is not a promise of linear scaling, sub-1% overhead, or equivalent service quality for a different model and network path.

From regional quota to serving topology

This development is separate from the question of extracting more inference from a fixed electrical power budget. The constraint being addressed here is geographically fragmented accelerator capacity. The proposed answer is to pool it through a shared endpoint and route requests using signals from the model-serving layer.

That is a consequential shift in capacity planning. Procurement and infrastructure teams have often had to wait for one region to provide enough accelerators to support a service. Google’s results suggest a different route: accept capacity where it is available, assemble a regional fleet, and determine through measurement whether state-aware routing produces a service that meets the required experience and operating boundaries.

The right conclusion is measured rather than categorical. Cross-region inference is not automatically a substitute for local capacity. It is now a serious design option for teams whose allocation is scattered—and it should be evaluated as a serving architecture, not dismissed as an emergency overflow arrangement.

Source: Google Cloud, “GPU and TPU utilization with multi-cluster GKE Inference Gateway,” published September 21, 2026; Google Cloud documentation on elastic cross-region high availability and multi-cluster inference gateway.