ClickHouse: Unlocking 32ms Latency Secrets

Decoding TCP and Protocol for Millisecond Gains

The article provides an excellent deep dive into a specific, albeit niche, performance bottleneck within the ClickHouse Native protocol, particularly concerning idle TCP connections and the interaction with the Linux TCP stack. The authors meticulously reproduce the issue, offering a clear cause-and-effect explanation that bridges application-level protocol design with low-level network behavior. The solution, application-level batching of metadata packets, is elegant and directly addresses the root cause by reducing the number of TCP segments that can overwhelm a reset congestion window. This approach is highly commendable for its focus on optimizing the existing protocol rather than resorting to more drastic changes. The detailed reproduction steps and validation using tshark are invaluable for anyone needing to troubleshoot similar network-related latency issues.

However, the article's primary limitation is its specificity to the ClickHouse Native protocol and the particular scenario of idle connections combined with TCP_NODELAY and a reset congestion window. While the underlying TCP principles are universal, the direct applicability of the solution is confined to ClickHouse users employing the Native protocol. Furthermore, the article briefly touches upon TCP_CORK as an alternative but dismisses it due to kernel specificity and control limitations. While valid concerns, a more in-depth comparison or discussion of its trade-offs against application-level batching for this specific problem might have enriched the analysis. The workaround of disabling tcp_slow_start_after_idle is mentioned as a server-side configuration, but its implications and potential downsides (e.g., impact on other connections) are not explored. The focus on a specific 32ms latency might also, for some readers, feel a bit too granular, although it serves as an effective case study for a larger problem of network-aware application design.

Key Points

• Idle TCP connections in the ClickHouse Native protocol can experience significant tail latency due to the Linux TCP stack resetting the congestion window (cwnd) after an idle period.
• When TCP_NODELAY is enabled, multiple small metadata packets (Logs, Progress, ProfileEvents) sent by ClickHouse after a query can exceed the reset, small cwnd, forcing an additional round trip.
• The issue was reproduced by simulating a 32ms RTT network, where idle connections led to an extra ~32ms of tail latency per query.
• The solution involves application-level batching of metadata packets within the ClickHouse Native protocol to reduce the number of TCP segments sent, thus fitting within the reduced cwnd and avoiding extra round trips.
• This batching significantly reduces the number of TCP packets, effectively eliminating the observed tail latency and improving connection efficiency after idle periods.

---

📖 Source: Chasing that 32 ms tail latency in ClickHouse