6 articles
Bridging Batch and Streaming Gaps The article excels at detailing the practical challenges and pragmatic solutions encountered when evolving a batch-oriented data pipeline towards a more continuous, f...
Alps Wang
Taming Spark's Memory Beast Pinterest's approach to reducing Spark OOM failures by 96% is a compelling case study in practical, large-scale data engineering. The key insight is the multi-pronged strat...
Alps Wang
AI-Powered Spark: A Performance Leap The article presents a compelling case for using reinforcement learning (RL) to optimize Apache Spark configurations. The core innovation lies in the agent's abili...
Alps Wang
Moka: A Modern Data Platform The Pinterest Moka project is a compelling case study in modernizing big data infrastructure. The move to Kubernetes on Amazon EKS, coupled with Apache Spark, offers signi...
Alps Wang
Data Quality: The Financial Backbone The Agoda case study provides a compelling example of how to tackle data quality issues in a large enterprise environment. The move to a centralized, Spark-based p...
Alps Wang
Polars: A Data Engineer's New Best Friend? The key insight is the significant performance and cost gains achieved by Decathlon by switching from Apache Spark to Polars for smaller datasets. This is no...
Alps Wang