ClickStack's 5x Speed-Up for ClickHouse Observability

Benchmarking Observability Performance

The ClickHouse blog post effectively articulates the journey to optimizing ClickStack's observability schemas, highlighting a benchmark-driven approach that led to a substantial 5x reduction in query latency. The key insights revolve around the iterative refinement of data storage and indexing strategies, specifically mentioning primary key redesign, text indexes, query rewrites, and materialized views. The article's strength lies in its transparency about the inherited default schema's limitations and the systematic process of identifying common query patterns through detailed analysis. The introduction of ClickCannon as a benchmarking tool is a significant takeaway, demonstrating a commitment to data-driven optimization rather than speculative changes. The detailed breakdown of schema optimizations, including the original schema and the categories of improvements, provides valuable technical depth for database professionals. The emphasis on maintaining a balanced operational profile (ingestion, storage, query performance) is crucial and often overlooked in pure performance tuning exercises. The article's success in demonstrating real-world impact through realistic production workloads and the utilization of ClickHouse Cloud warehouses lends it significant credibility.

However, while the article showcases impressive results, some limitations and areas for further exploration exist. The specific trade-offs made during schema optimization are mentioned but not deeply detailed. For instance, how much did ingestion throughput or storage footprint change to achieve the query latency improvements? Understanding these trade-offs is vital for users to assess applicability to their own environments, especially those with different cost sensitivities or ingestion volumes. Additionally, while the article focuses on the logs schema, observability workloads often involve traces and metrics as well. A more holistic view of how these different data types were optimized or how their optimizations interact could provide a more complete picture. The article could also benefit from a more direct comparison with other observability solutions that might leverage similar database technologies, even if indirectly, to contextualize ClickStack's advancements within the broader market landscape. Finally, the continued evolution of the dataset and insights from anonymized platform telemetry, while good practice, leaves room for speculation about the representativeness for users with highly divergent data characteristics.

Key Points

• ClickStack achieved a 5x reduction in query latency for ClickHouse observability through a benchmark-driven schema optimization process.
• Key optimization techniques included primary key redesign, text indexes, query rewrites, materialized views, and leveraging new ClickHouse features.
• A systematic approach involved identifying common user query patterns and rigorously testing changes using ClickCannon and a formal testing framework.
• The optimization process balanced query performance with ingestion and storage efficiency, aiming for an overall improved operational profile.
• Materialized views for common Kubernetes attributes and efficient indexing of map keys/values and log bodies were crucial for performance gains.

---

📖 Source: How we made ClickStack 5x faster for ClickHouse observability