DoorDash's SafeChat: AI Moderation at Scale

Hybrid AI for Scalable Safety

DoorDash's SafeChat presentation offers a compelling case study in building a robust, scalable AI moderation system for a high-volume marketplace. The core innovation lies in the hybrid architecture, moving away from an LLM-only approach to a tiered system. This intelligently leverages fast, cheap internal models for initial filtering of obvious cases, reserving expensive LLM calls for more nuanced decisions. The emphasis on using LLMs for multi-axis scoring rather than simple binary classification is particularly insightful, enabling graduated actions based on severity and offering greater flexibility for evolving safety policies. The shift from a specific product ('SafeChat') to a content-agnostic platform underscores a mature understanding of architectural patterns and reusability. This approach is highly relevant for any organization dealing with user-generated content at scale, aiming to balance safety, cost, and user experience.

However, a few considerations warrant attention. While the presentation highlights the benefits of the hybrid model, the initial 'boring step' of data understanding and labeling, though crucial, can be a significant undertaking and a bottleneck for many organizations. The article mentions using a free moderation API for initial data collection, but the specifics of its integration and limitations are not detailed. Furthermore, the 'content-agnostic' platform, while powerful, still requires domain-specific meaning and business logic to be provided by client teams. This implies a need for internal expertise and careful configuration to ensure effective moderation across diverse use cases. The scalability of the internal ML platform and the LLM gateway itself would also be critical factors for sustained performance. The success hinges on the ability to maintain and update both internal models and external integrations as the threat landscape and LLM capabilities evolve.

This architectural pattern is highly beneficial for companies operating large-scale marketplaces, social platforms, or any service with significant user-generated content, especially where real-time moderation is critical. Developers and architects can draw direct inspiration for building cost-effective and efficient AI moderation pipelines. The insights into data-driven architecture design, the strategic use of LLMs, and the platformization of AI solutions are transferable to a wide range of AI applications beyond just content moderation, such as fraud detection, recommendation systems, and customer support automation. The emphasis on graduated actions based on LLM scoring is a particularly valuable takeaway for enhancing the precision and fairness of AI decision-making.

Key Points

• DoorDash implemented a hybrid AI moderation system (SafeChat) to handle millions of daily messages efficiently.
• The architecture uses a fast, cheap internal ML model for initial filtering of obvious cases.
• LLMs are reserved for complex cases, scoring messages across multiple axes (e.g., threatening, profane) instead of a simple safe/unsafe binary.
• This multi-axis scoring enables graduated actions based on severity, leading to a ~50% reduction in safety incidents.
• The core architectural pattern (cheap then expensive layer with graduated actions) was productized into a content-agnostic platform for broader use within DoorDash.
• The platform supports internal models, external vendor models, and external LLM prompts via an LLM gateway.

---

📖 Source: Presentation: SafeChat: Building AI-Powered Safety Systems at Scale in a Real-Time Marketplace