Slack Ditches SSH for REST in EMR, Modernizing 700+ Jobs

Decoupling Execution from Infrastructure

Slack's migration away from SSH-based job execution in their EMR pipelines represents a mature engineering decision, moving from a simple, albeit brittle, approach to a more robust, scalable, and observable architecture. The key insight is the fundamental shift in how jobs are managed: from direct, connection-dependent execution to a server-side, lifecycle-driven model facilitated by a RESTful orchestration layer (Quarry). This not only enhances security by eliminating direct production cluster access and simplifying key management but also dramatically improves reliability. Jobs are no longer susceptible to silent failures due to transient network issues, as the tracking and cancellation are managed centrally. The adoption of YARN's Distributed Shell for arbitrary shell commands is a clever solution to bridge the gap for legacy or specialized workloads that don't fit neatly into structured APIs like Livy or HiveServer2. This migration offers a blueprint for any organization grappling with the operational complexities and security risks inherent in direct SSH access for distributed systems.

Key Points

• Slack replaced SSH-based job execution with a REST-driven orchestration layer (Quarry) for its Amazon EMR pipelines.
• This migration involved over 700 Airflow operators and aimed to improve security, reliability, and observability.
• The new architecture decouples job execution from client connectivity, enabling server-side lifecycle management and centralized control.
• Apache Hadoop YARN's Distributed Shell was utilized to support arbitrary shell command workloads.
• The migration was executed incrementally across multiple environments and data regions, highlighting the importance of phased rollouts and cross-team coordination.

---

📖 Source: Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to Rest-Based Architecture