Netflix's Cold Storage: Cassandra & S3 for Time-Series

Beyond Hot Data: Tiered Storage for Scale

Netflix's approach to tiered time-series data storage is a masterclass in pragmatic engineering, demonstrating a deep understanding of cost-efficiency at petabyte scale. The evolution from a Kafka/Flink streaming solution to directly leveraging Cassandra backups in S3 is particularly noteworthy. This second approach significantly simplifies the architecture by eliminating an entire streaming pipeline, reducing operational overhead and data transfer costs. The introduction of ReadOnlyCompactionStrategy (ROCS) and pluggable Memtables to read directly from S3 backups is a clever utilization of existing Cassandra features and a testament to the platform's extensibility. The meticulous detail on how backups are structured and how Cassandra initializes the 'backup memtable' showcases a robust and resilient design.

However, a key concern with the Cassandra-native cold tier lies in the potential for increased CPU utilization on read paths, as acknowledged by the authors. Serving data from S3 is inherently more computationally intensive than local disk reads. While optimizations like larger compression chunks and disabling speculative retries are implemented, the ultimate impact on cluster performance and the need for potentially more powerful nodes for cold reads warrants careful monitoring. Furthermore, the reliance on specific Cassandra versions (4.1+) for pluggable Memtables could be a barrier for organizations not yet on the latest releases. The complexity of managing the backup manifests and ensuring their integrity across the distributed Cassandra ring, while addressed, remains a critical operational aspect that requires robust automation and monitoring.

The primary beneficiaries of this work are organizations dealing with massive volumes of time-series data where cost-efficiency is paramount. This includes companies in IoT, financial services, observability platforms, and any domain requiring long-term historical data retention. The solutions offer a clear path to drastically reducing storage costs without sacrificing accessibility for infrequent queries. For developers and architects facing similar scaling challenges, this article provides concrete blueprints and innovative techniques. The insights into optimizing S3 reads, managing immutable data, and the detailed explanation of ROCS offer valuable lessons applicable beyond Netflix's specific stack. The ability to maintain a consistent API for users across hot and cold tiers is a significant win for user experience and simplifies application development.

Key Points

• Netflix implemented a tiered storage strategy for petabyte-scale time-series data to reduce costs.
• Solution 1: Kafka/Flink streams data to S3, with a daily compaction job to optimize Parquet files for cold reads.
• Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables (via ReadOnlyCompactionStrategy - ROCS) to read data directly from S3, eliminating the need for a separate streaming pipeline.
• ROCS optimizes SSTable alignment with token ranges, reducing read fan-out and improving latency.
• Pluggable Memtables allow Cassandra to directly query data from S3 backups, significantly reducing storage footprint on hot nodes.
• The system automates the lifecycle transitions of time slices from hot to cold storage.

---

📖 Source: TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads