AlloyDB: LLMs Inside DBs, No Cloud Calls Needed

Database-Native LLM Inference

Google's AlloyDB AI functions, particularly the introduction of proxy models, represent a compelling architectural shift in database-LLM integration. The core innovation lies in transforming the database from a passive client making per-row external LLM calls into an active participant that can locally infer LLM behavior. This 'distillation-at-query-time' approach, where a lightweight local model learns from a frontier LLM, has the potential to drastically reduce latency and cost, especially for high-volume operations. The reported throughput improvements (up to 23,000x with proxy models) are eye-catching and suggest a fundamental re-architecting of how AI capabilities are embedded within data management. The ability to execute semantic filtering and other LLM-driven operations directly within SQL queries, rather than relying on application-level logic or cumbersome APIs, significantly simplifies development and enhances performance. This move positions AlloyDB as a strong contender for organizations looking to leverage AI within their data workflows without incurring the prohibitive costs and latency associated with constant external LLM calls.

Key Points

• AlloyDB introduces AI functions that allow LLM calls directly within SQL queries.
• Two key acceleration techniques are smart batching and optimized proxy models.
• Proxy models enable local inference within the database by training lightweight models from frontier LLMs, reducing latency and cost.
• Reported performance gains are substantial, with proxy models achieving up to 23,000x throughput improvement and 6,000x cost reduction in internal testing.
• This architecture shifts the LLM role from a runtime dependency to a teacher for local inference.
• Smart batching groups multiple rows for single LLM calls, improving throughput by up to 2,400x.
• New AI functions include ai.summarize, ai.agg_summarize, and ai.analyze_sentiment.
• The optimized proxy model for ai.if is currently in preview, while smart batching is GA for ai.if and ai.rank.
• The release includes a managed MCP server for easier AI agent integration.

---

📖 Source: AlloyDB Ships Proxy Models That Replace LLM Calls with Local Inference Inside the Database