Datadog & ClickHouse: Full-Fidelity Observability Data Unleashed

Bridging Observability Gaps

The partnership between Datadog and ClickHouse marks a significant advancement in the observability landscape, directly addressing the long-standing tension between retaining comprehensive telemetry data and making it easily accessible within existing operational tools. By enabling direct routing of logs to ClickHouse via Datadog Observability Pipelines and subsequently allowing federated search from Datadog Log Explorer, the solution empowers organizations to maintain full-fidelity data without the prohibitive costs or complexity of traditional approaches. This is particularly relevant in the current era of increasingly distributed systems, AI applications, and agent-based workflows, where the sheer volume of telemetry data can overwhelm standard retention policies and budgeting. The ability to leverage ClickHouse's cost-effective, high-performance storage for long-term retention while seamlessly querying this data within the familiar Datadog interface represents a powerful synergy, eliminating the need for data duplication and complex data movement.

While the benefits are substantial, a key consideration will be the operational overhead of managing two distinct, albeit integrated, platforms. Ensuring seamless updates, consistent data schema management between Datadog's processing and ClickHouse's ingestion, and robust security across both environments will be crucial. Furthermore, the 'Preview' status suggests that this integration is still evolving, and organizations adopting it early should be prepared for potential changes and the need for continuous adaptation. The success of this partnership will also hinge on the performance and scalability of the federated search mechanism, especially when querying vast datasets stored in ClickHouse. Organizations need to carefully evaluate their specific data volume, query patterns, and latency requirements to ensure this solution meets their needs effectively. However, for many, the promise of uncompromised data retention and accessible historical analysis within a unified workflow is a compelling proposition.

Key Points

• Partnership between Datadog and ClickHouse aims to solve the trade-off between retaining full-fidelity telemetry data and its accessibility.
• New feature allows routing logs directly to ClickHouse via Datadog Observability Pipelines.
• Federated search enables querying ClickHouse-stored logs directly from Datadog Log Explorer without data duplication.
• Benefits include longer data retention, cost-efficiency, and a unified user experience for troubleshooting and analysis.
• Leverages ClickHouse's scalability and cost-effectiveness for long-term storage, combined with Datadog's familiar interface and investigative capabilities.
• Preview feature is now available for private preview access.

---

📖 Source: Datadog と ClickHouse が提携、最新のオブザーバビリティにフルフィデリティ（全量）なデータをもたらす