700 Tokens Per Second: Google’s TPU Push Challenges Nvidia’s Dominance

The latest benchmark figures have emerged, showing a tuned inference engine achieving over 700 tokens per second on Google’s seventh-generation Ironwood TPU. This performance metric, generated by Inferact using a 2.8-trillion-parameter model, stands in stark contrast to the 452 tokens per second recorded on Nvidia’s GB200 hardware under similar conditions. The result highlights the raw processing power now available on non-Nvidia silicon, marking a critical data point in the ongoing competition for AI supremacy.

These numbers are the technical backbone of a broader strategic shift announced this week. Google and Meta revealed that SGLang, a popular inference engine, now runs natively on TPUs through a new PyTorch integration called TorchTPU. Previously, running these workloads required reliance on Google’s JAX framework, a hurdle that often deterred major users. The new software stack allows PyTorch workloads to deploy directly on TPUs without significant code rewrites. Meta’s PyTorch-TPU lead, Zongwei Zhou, confirmed that SGLang-TorchTPU is already running in production at Meta, citing a "close three-way collaboration" between the companies and the SGLang team.

This development is not merely a software update; it represents a fundamental change in how major tech firms source their AI compute. Meta’s director of PyTorch engineering, Jana van Greunen, noted that the company "cannot be constrained by a single hardware architecture." By adopting TorchTPU, Meta gains a "remarkably mature, PyTorch-native serving stack" that decouples its software from any single chip vendor. This move follows years of tension, as Meta previously canceled TPU contracts in 2023 after using the older PyTorch/XLA bridge. However, recent reports suggest Meta has since signed a multi-year rental agreement with Google Cloud, with purchases potentially beginning in 2027.

The timing of the announcement underscores the competitive pressure on Nvidia. It arrived just one day after Amazon open-sourced TorchNeuron, its own native PyTorch backend for Trainium chips. Both moves appear designed to loosen Nvidia’s software lock-in, which has long been a key barrier to entry for competitors. By ensuring that the most popular AI framework, PyTorch, runs efficiently on rival hardware, Google and Amazon are lowering the switching costs for large-scale AI operators.

The performance gains extend beyond single-user speed. Inferact reported that its kernels for specific models deliver roughly 1.4 to 2 times the throughput of the Nvidia baseline at small batch sizes, even without speculative decoding. This efficiency gain is crucial for data centers where power and cooling are major constraints. The partnership between Google, Meta, and RadixArk, the startup behind SGLang, aims to solidify this advantage by adding support for new models like DeepSeek V4.1 Flash.

Looking ahead, the ecosystem plans to expand rapidly. RadixArk is working toward "Day 0" support for new open models by the first quarter of 2027, ensuring that TPU users can deploy the latest AI architectures immediately upon release. Google, Meta, and RadixArk are scheduled to present further details at the SGLang Summit in San Francisco this November. For the Great Lakes and Ohio Valley regions, where major cloud infrastructure and tech hubs are located, this shift suggests a more diversified and competitive market for AI compute resources in the coming years.

The move by Meta to run production workloads on Google’s TPUs via PyTorch is a significant milestone. It validates the TPU as a credible alternative to Nvidia’s GPUs for large-scale AI inference. With performance benchmarks now favoring the new stack in specific scenarios, other tech giants may follow suit, further eroding Nvidia’s market dominance and reshaping the supply chain for AI hardware.