RAG Reliability: Lessons from Production AI Search

Production RAG: Beyond the LLM

Lan Chu's presentation at QCon London 2026 offers a crucial, ground-level perspective on building production-ready Retrieval-Augmented Generation (RAG) systems. The core takeaway – that indexing and retrieval are the primary failure points, not the LLM itself – is a vital realization for any team venturing into RAG. Her detailed breakdown of challenges, from complex document parsing requiring visual-language models to the nuanced art of chunking and the necessity of temporal scoring and routing layers, provides concrete solutions derived from hard-won experience. The emphasis on evaluating with real user queries and tracking specific failure modes is particularly noteworthy, shifting focus from theoretical benchmarks to practical, user-centric performance. This approach is essential for moving RAG from experimental stages to reliable enterprise applications.

However, while Chu highlights the complexity introduced by agentic architectures, the article could delve deeper into the trade-offs and specific patterns for managing this complexity. The description of adding temporal scoring and a routing layer, while effective, hints at a growing architecture that can quickly become difficult to manage and debug. The article implicitly suggests that for robust systems, a more sophisticated orchestration layer is needed, which could be a point of further exploration. Furthermore, while the use of visual-language models for parsing is innovative, the operational overhead and cost implications of such a hybrid approach in a large-scale enterprise setting warrant more discussion. The article provides excellent tactical advice, but a broader strategic overview of balancing complexity, cost, and performance in evolving RAG systems would enhance its value.

Key Points

• Production RAG systems often fail due to indexing and retrieval issues, not the language model itself.
• Accurate document parsing is critical; complex layouts (tables, infographics) require more than plain text conversion, potentially needing visual-language models.
• Chunking strategies significantly impact accuracy and cost; testing on real datasets is essential, with section-based chunking showing promise.
• Standard vector similarity retrieval can miss context; incorporating temporal scoring and routing layers can enhance relevance and system functionality.
• Robust evaluation requires datasets from real user queries, tracking specific failure modes (routing, temporal errors), and using statistical methods for verification.
• Agentic architectures offer enhanced capabilities but increase complexity; structured evaluation frameworks are crucial for reliability.

---

📖 Source: QCon London 2026: Reliable Retrieval for Production AI Systems