Datadog & ClickHouse: Full-Fidelity Observability Data

Bridging Observability Gaps

The announcement of the partnership between Datadog and ClickHouse marks a significant step forward in addressing the escalating challenge of managing and analyzing vast volumes of telemetry data in modern, distributed, and AI-driven environments. The core innovation lies in enabling users to retain full-fidelity logs in ClickHouse for cost-effective, long-term storage and analysis, while seamlessly querying and investigating these logs directly within the familiar Datadog Log Explorer interface. This eliminates the historical trade-off between data retention and accessibility, a critical pain point for observability teams struggling with rising costs and the need for comprehensive historical data for deep troubleshooting and compliance. The integration leverages Datadog Observability Pipelines, supporting open standards like OpenTelemetry and OCSF, to route processed and enriched logs directly to ClickHouse. The federated search capability is particularly noteworthy, allowing engineers to avoid data duplication and maintain their existing workflows, thereby reducing operational overhead and enhancing efficiency. This approach acknowledges the distinct strengths of each platform: Datadog excels in providing a rich user experience for real-time investigation and collection, while ClickHouse offers a performant and cost-effective solution for massive-scale data storage and analytics. The partnership's focus on 'full-fidelity data' is a powerful differentiator, especially as AI applications and agentic workflows generate increasingly complex and voluminous telemetry, making sampling or filtering a less viable option for critical investigations.

However, while the preview availability is promising, the true impact will depend on the maturity and performance of the federated search in production environments. Potential limitations could include query latency for very large datasets or complex queries that might not perfectly match Datadog's native query engine performance. Furthermore, the operational complexity of managing a hybrid storage and querying strategy, even with this integration, might still pose a challenge for some organizations. Ensuring seamless data synchronization and robust error handling between Datadog and ClickHouse will be crucial. Organizations that would benefit most are those already operating at significant scale with high telemetry volumes, particularly those in sectors heavily reliant on AI, machine learning, and complex microservices architectures, where detailed historical data is paramount for debugging, security investigations, and performance optimization. This includes companies like OpenAI, DoorDash, Anthropic, and Shopify, who are already identified as ClickHouse users and are likely facing these exact challenges. The technical implications point towards a more flexible and cost-optimized observability architecture, allowing for greater control over data lifecycle management without sacrificing analytical capabilities or user experience. This could pave the way for more sophisticated anomaly detection and predictive analysis powered by complete historical datasets.

Key Points

• Datadog and ClickHouse partner to enable full-fidelity log retention and analysis.
• Users can route logs directly to ClickHouse via Datadog Observability Pipelines.
• Federated log search allows querying ClickHouse data directly from Datadog Log Explorer.
• Eliminates the trade-off between data retention cost and accessibility for observability data.
• Leverages open standards like OpenTelemetry and OCSF for flexible telemetry routing.
• Aims to reduce operational overhead by avoiding data duplication.

---

📖 Source: Datadog and ClickHouse partner to bring full-fidelity data to modern observability