AI Agents Tackle SQL: Claude Opus 5.5, GPT-6 & More Tested

Agentic Analytics: The New Frontier

The ClickHouse Blog's follow-up to their Agentic Analytics Benchmark offers a valuable, in-depth look at the performance of several leading LLMs in generating SQL for complex analytical queries. The meticulous methodology, including a fixed dataset, judge panel, and turn budget, provides a robust framework for comparison. The article highlights key shifts, such as Claude Sonnet 5.5's significant improvement in efficiency and its enhanced ability to navigate dimensional CRM layers, while also noting that GPT-6 variants, despite being new, do not surpass GPT-5.5 at default reasoning effort. The emphasis on the 'planning' aspect as the primary failure mode for these agents is a critical takeaway, suggesting that while SQL execution is largely solved, the strategic formulation of queries remains a significant challenge. The transparency in releasing the benchmark harness is commendable and encourages community participation and independent verification.

However, a limitation lies in the 'default reasoning effort' caveat for GPT-6. Without a sweep across various effort levels, a full comparison of GPT-6's potential against its predecessors and other models remains incomplete. The article also mentions that cost data for GPT-6 and Opus 5.5 is unreliable due to gateway limitations, which hinders a complete cost-benefit analysis for these specific models. While the benchmark is designed to be repeatable, the 'frozen' nature of the board means that new model releases will always lag behind the cutting edge. The article correctly advises users to run the benchmark on their own data, acknowledging that warehouse-specific discriminators can significantly impact performance, but this also means the published results are indicative rather than definitive for all use cases. The inclusion of LibreChat scoring adds an interesting dimension, showing a slight advantage over direct loop execution, suggesting the value of agentic frameworks.

Key Points

• Claude Sonnet 5.5 shows significant efficiency gains, completing all questions within the budget and reducing cost by ~30% compared to Sonnet 5.
• GPT-6 family (Astra, Luna, Sol) performs in a tight band (68.7-69.7%), falling slightly below GPT-5.5 and not surpassing their predecessor at default effort.
• Planning remains the primary failure mode for AI agents in analytics, with incorrect query plans being the largest category of errors.
• Claude Sonnet 5.5 demonstrates a notable improvement in handling dimensional CRM layers, scoring 73.5% on these specific questions compared to Sonnet 5's 62.7%.
• The benchmark emphasizes the importance of running tests on one's own data, as warehouse-specific discriminators significantly influence model performance.

---

📖 Source: The Agentic Analytics Benchmark, round two: GPT-6, Claude Opus 5.5, Claude Sonnet 5.5