KV Cache Leak in GLM-5.3: Cloudflare's SGLang Fix

Unpacking the KV Cache Vulnerability

The Cloudflare blog post provides a transparent and technically rich account of a critical security vulnerability in their GLM-5.3 Workers AI model, stemming from a misconfiguration of the SGLang inference engine and a specific bug in its disaggregated prefill/decode mode. The key insight is how a seemingly minor mismatch between the KV cache pool size and the model's context window, combined with an edge case in SGLang's request handling, led to cross-request data leakage. The remediation strategy, involving both configuration adjustments and a patched inference engine, demonstrates a robust approach to addressing such issues. The proactive collaboration with the SGLang upstream maintainers and the detailed timeline of the incident and resolution are commendable, highlighting a mature incident response process.

The article's strength lies in its detailed technical explanation of the bug's mechanics, particularly the interaction between prefill and decode nodes and the flawed finished() check in SGLang. This level of transparency is invaluable for other operators of large language models. The prompt exceeding the KV cache pool size was marked for rejection but incorrectly proceeded through the pipeline, leading the decode node to use stale KV cache data from previous requests. The solution involved not only increasing the KV cache pool size to accommodate the full context but also patching SGLang to ensure rejected requests are properly aborted before any KV cache transfer. The emphasis on ongoing hardening and the commitment to upstream contributions are also positive indicators of responsible AI deployment.

However, a notable limitation is the reliance on specific dates in the future (September 2026) for the incident. While this might be a stylistic choice for the article's context or a placeholder, it detracts from the immediate relevance and believability of a real-time security report. For a technical audience, concrete, past dates would lend more credibility. Additionally, while the article mentions that only GLM-5.3 was affected, a brief explanation of why other models were immune (e.g., their KV pools were already sufficiently sized) would further strengthen the narrative and provide clearer lessons learned. The potential for future, similar bugs related to KV cache memory safety is acknowledged, underscoring the inherent complexity and the need for continuous vigilance in managing these stateful AI components.

Key Points

• A security vulnerability in Cloudflare's GLM-5.3 Workers AI model exposed cross-request KV cache data to a small number of customers.
• The issue stemmed from a misconfiguration where the KV cache pool size (681,280 tokens) was smaller than the model's context window (1,048,576 tokens) and a bug in SGLang's handling of over-length requests in disaggregated mode.
• When a prompt exceeded the KV cache pool, SGLang incorrectly marked it as rejected but proceeded with KV cache transfer, leading the decode node to use stale data from previous requests.
• Cloudflare remediated the issue by increasing the KV cache pool size to accommodate the full context and patching SGLang to safely handle rejected requests.
• The company collaborated with SGLang maintainers for upstream fixes and implemented further hardening measures across all its SGLang deployments.
• The incident highlights the critical importance of precise configuration and robust error handling in AI inference engines, especially in disaggregated architectures.

---

📖 Source: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang