HighLevel's Data Transformation with ClickHouse Cloud

Unifying Data: HighLevel's Architecture

The HighLevel case study offers valuable insights into migrating from a fragmented data infrastructure to a unified ClickHouse Cloud solution. The primary innovation lies in the successful consolidation of various data stores (MySQL, Elasticsearch, Firestore, and a document database) onto a single platform, resulting in substantial storage reductions (up to 88%), dramatic query latency improvements (sub-200ms from 6 seconds), and simplified scaling. The emphasis on immutability and the strategic use of MergeTree and ReplacingMergeTree tables, along with materialized views, demonstrates a deep understanding of ClickHouse's strengths. A noteworthy aspect is the client-side observability layer, which provides crucial visibility into query performance across microservices. However, the article could have benefited from a more detailed exploration of the migration process itself, including the challenges faced, the specific query patterns optimized, and any potential trade-offs. While cost savings are mentioned, a more thorough cost analysis, comparing the old and new architectures, would have strengthened the case. Furthermore, a deeper dive into the specific query optimization techniques used for each use case, beyond the high-level descriptions, would have been beneficial for readers looking to replicate similar results. Finally, while the article highlights the benefits, it also subtly acknowledges the learning curve involved in adopting ClickHouse, and this could be expanded to provide more concrete guidance.

This article is excellent for those looking to improve performance and cost for large datasets, especially those using ClickHouse or considering it. The case study is valuable for database architects, data engineers, and DevOps professionals. The emphasis on performance, scalability, and simplified operational overhead should resonate with those dealing with similar challenges. The insights into schema design and client-side observability are particularly relevant. However, the article doesn't cover some of the more advanced features of ClickHouse, like geo-spatial or full-text search, so it would be less useful to those looking for a solution that covers all of those features. The article also does not discuss the nuances of data pipelines or any of the complexities that come with real-time data ingestion.

Key Points

• HighLevel migrated from MySQL, Elasticsearch, Firestore, and a document database to ClickHouse Cloud for improved performance and scalability.
• The migration resulted in significant storage reductions (up to 88%) and dramatically reduced query latency (sub-200ms).
• The architecture leverages MergeTree and ReplacingMergeTree tables, materialized views, and client-side observability for optimal performance.
• Key use cases include lead activity, workflow logs, notifications, and agency billing dashboards.

---

📖 Source: How HighLevel rebuilt its data platform for speed, scale, and simplicity on ClickHouse Cloud