iSAM Funds' 10,000x Data Leap with ClickHouse Cloud

Scaling Options Data: A ClickHouse Case Study

The article effectively highlights ClickHouse Cloud's strengths in handling large-scale time-series data for financial research, particularly its impressive ingestion rates and compression ratios compared to Postgres. The narrative around simplifying the data pipeline by eliminating rollups and materialized views due to ClickHouse's query performance is a compelling argument for its adoption. The emphasis on ClickHouse Cloud as a turnkey solution, allowing iSAM to focus on innovation rather than infrastructure management, is a key takeaway for businesses seeking to accelerate development cycles. The ability to integrate with existing AWS infrastructure and enterprise authentication further solidifies its appeal for enterprise adoption.

However, the article could benefit from a more detailed exploration of the '10,000x data problem'. While the fan-out multiplier is mentioned, quantifying the actual data volume (e.g., TBs, PBs) and the specific characteristics of the options data (e.g., tick frequency, data schema complexity) would provide a clearer picture of the scale. Additionally, while the performance gains are impressive, providing benchmarks for specific query types and latencies would offer more concrete technical insights. The article also touches on the uneven distribution of market data activity, which makes simple partitioning ineffective, but a deeper dive into how ClickHouse's architecture specifically addresses this challenge, beyond its general performance, would be valuable. Finally, while the benefits of ClickHouse Cloud are emphasized, a brief mention of potential trade-offs or considerations for organizations with existing on-premises ClickHouse expertise might offer a more balanced perspective.

Key Points

• iSAM Funds leveraged ClickHouse Cloud to build an options research platform, handling a 10,000x data multiplier.
• Migration from Postgres to ClickHouse Cloud resulted in 35-50x data compression and a massive increase in ingestion speed (8,000 to 800,000 rows/sec per core).
• ClickHouse's columnar storage and vectorized execution enabled fast, on-demand queries, allowing the elimination of complex pre-computed rollups and materialized views.
• ClickHouse Cloud provided a turnkey solution, allowing iSAM to focus on innovation rather than infrastructure management.
• The platform integrates with iSAM's existing AWS footprint and enterprise authentication.
• ClickHouse Cloud's scalability allows iSAM to dynamically adjust resources for peak loads, offering a cost-efficient approach to hardware utilization.
• The speed of data exploration directly translates to faster hypothesis testing and a reduced time-to-market for research ideas, which is crucial in the context of AI development.

---

📖 Source: How iSAM Funds built an options research platform for a 10,000x data problem with ClickHouse Cloud