ClickHouse's Failover Pipeline for OTel

Beyond the Default: ClickHouse's OTel Ingestion

ClickHouse's blog post details a robust and pragmatic approach to solving the critical challenge of reliable OpenTelemetry (OTel) ingestion at massive scale. The evolution from a simple agent-gateway model to a sophisticated S3-backed failover mechanism highlights the complexities of real-world observability pipelines. The key insight is the recognition that standard solutions, while suitable for 99% of use cases, falter under the extreme demands of high-volume, unpredictable telemetry. Their adoption of blob storage as a cost-effective, durable overflow buffer, coupled with a priority-based failover connector, is a particularly noteworthy innovation. This strategy cleverly sidesteps the operational overhead of deploying and managing a separate streaming system like Kafka while effectively addressing backpressure and durability concerns. The emphasis on avoiding head-of-line blocking and managing costs by only incurring high PUT costs during actual failovers demonstrates a deep understanding of both technical requirements and economic realities.

However, a potential limitation lies in the complexity introduced by the multi-stage failover and catch-up mechanism. While effective, this architecture requires careful configuration and monitoring. The reliance on SQS for event notifications introduces another component into the pipeline, and while it's a managed service, it still represents a point of potential failure or degradation. The article mentions that full blob store outages are rare, but the strategy primarily addresses throttling and partial disruptions. A complete S3 outage would indeed be catastrophic, and while they acknowledge this, the chosen mitigation of distributing keys across a wide keyspace might not fully prevent issues if the event notification system itself is impacted or if ingress to S3 is completely blocked. The article also implicitly assumes a certain level of expertise within the team to implement and maintain such a custom solution, which might be a barrier for smaller organizations. Nevertheless, for organizations grappling with similar scale and reliability issues in their OTel ingestion, this detailed architectural breakdown offers invaluable lessons and a compelling blueprint.

Key Points

• ClickHouse ingests 50 million OTel events per second, storing 177 PiB of uncompressed data.
• The initial agent-gateway architecture failed due to backpressure from ClickHouse.
• Disk-based write-ahead logs (WALs) in collectors proved insufficient due to immutability, performance degradation with backlog growth, and FIFO ordering leading to fresh data delays.
• Kafka was considered but rejected due to operational overhead of managing a new tier-zero service.
• The final architecture uses S3 as a cost-effective, durable overflow buffer with a priority-based failover connector.
• Blob storage PUT costs are managed by only incurring them during actual ClickHouse unavailability.
• Event notifications (e.g., SQS) trigger a separate catch-up receiver to write data from S3 back to ClickHouse.
• The failover connector requires disabling the sending queue on the ClickHouse exporter to ensure timely backpressure signals.
• The catch-up receiver scales independently and can scale down to zero when the SQS queue is empty.

---

📖 Source: Ensuring reliable OpenTelemetry ingestion at scale