Cloudflare's LLM Leap: Faster, Bigger AI

Unlocking Extra-Large LLM Performance

Cloudflare's blog post offers a compelling deep dive into the technical underpinnings of running exceptionally large language models (LLMs) efficiently. The emphasis on disaggregated prefill and decode stages is a standout innovation, directly addressing GPU underutilization by allowing independent tuning and scaling of compute-bound prefill and memory-bound decode. This architectural shift, coupled with advanced load balancing techniques like token-aware routing, demonstrates a sophisticated approach to maximizing hardware efficiency. The introduction of their proprietary Infire inference engine, written in Rust and optimized for multi-GPU parallelism and reduced memory overhead, is another significant development. By enabling models like Kimi K2.5 to run on fewer, more powerful GPUs with substantial KV cache remaining, Cloudflare is lowering the barrier to entry for deploying massive models.

However, while the technical details are impressive, the article could benefit from more explicit quantitative comparisons against other leading inference solutions beyond vLLM. While they mention Infire's advantages in memory overhead and cold-start times, a more direct benchmark against frameworks like TensorRT-LLM or Triton Inference Server would solidify their claims. Furthermore, the reliance on client-side x-session-affinity headers for prompt caching, while incentivized, introduces a dependency on developer adoption. A more robust server-side caching mechanism or intelligent client routing would further enhance reliability and ease of use. The article also hints at the complexity of the load balancer, but a deeper explanation of its fault tolerance and scaling mechanisms would be valuable for understanding the robustness of the disaggregated architecture.

Key Points

• Cloudflare is significantly enhancing its Workers AI platform to host and run extra-large language models (LLMs) efficiently.
• Key innovations include Prefill Decode (PD) disaggregation, separating compute-bound prefill from memory-bound decode for better GPU utilization.
• Token-aware load balancing is employed to manage the disaggregated prefill and decode stages effectively.
• Prompt caching, incentivized via x-session-affinity headers and discounted tokens, is crucial for agentic use cases to reduce recomputation.
• KV-cache optimization leverages Moonshot AI's Mooncake Transfer Engine and Store for high-performance data transfer and extended cache storage (including NVMe), enabling efficient multi-GPU and multi-node cache sharing.
• Speculative decoding, using a smaller draft model, is implemented to accelerate token generation while maintaining quality, particularly beneficial for structured outputs and tool calls in agents.
• Cloudflare's proprietary Infire inference engine, written in Rust, offers multi-GPU support (pipeline, tensor, expert parallelism), lower memory overhead, and faster cold-starts, outperforming solutions like vLLM for large models.
• These optimizations allow for higher tokens-per-second throughput, enable running large models on less hardware, and improve overall inference performance and cost-efficiency.

---

📖 Source: Building the foundation for running extra-large language models