WalShadow: Sub-Second Postgres WAL to ClickHouse Sync

Reimagining Postgres-ClickHouse Replication

The introduction of WalShadow by ClickHouse marks a significant advancement in the realm of database replication, particularly for scenarios involving PostgreSQL and ClickHouse. The core innovation lies in its direct consumption of physical Write-Ahead Logs (WAL) rather than relying on logical replication. This approach bypasses the overhead and potential complexities associated with logical decoding, such as managing replication slots and dealing with specific decoding plugins. The reported sub-second latency and impressive throughput (289K rows/sec) are particularly noteworthy, positioning WalShadow as a compelling solution for real-time analytics. The ability to handle schema evolution, including ADD COLUMN, RENAME COLUMN, and DROP COLUMN, directly from physical WAL is a substantial operational advantage over many existing logical replication-based methods that can struggle with such changes. Furthermore, the architecture's parallel processing of decoding and insertion, along with the use of ClickHouse-native blocks, suggests a highly optimized pipeline designed to maximize performance and minimize resource consumption on the source PostgreSQL instance.

However, a key limitation of this approach, as acknowledged by the article, is its reliance on direct access to physical WAL. This is often not exposed by managed PostgreSQL services, creating a barrier to adoption for users of these platforms. While ClickHouse Managed Postgres offers a native integration, this restricts WalShadow's applicability to environments where physical WAL access is feasible or where users opt for ClickHouse's managed offering. Another potential concern, though addressed by the _lsn mechanism, is the out-of-order arrival of blocks in ClickHouse due to parallel processing. While the article states correctness is preserved, rigorous testing across diverse workloads and failure scenarios would be crucial to fully validate this claim. The comparison with PeerDB, while effective in demonstrating WalShadow's superiority in specific benchmarks, might not cover all edge cases or the full spectrum of operational considerations that might favor a logical replication-based approach in certain contexts (e.g., fine-grained filtering at the source). The success of WalShadow will ultimately depend on its robustness, ease of configuration, and long-term maintainability in production environments, especially concerning complex schema migrations and high-volume transactional workloads.

Key Points

• WalShadow enables sub-second replication from PostgreSQL to ClickHouse by directly consuming physical WAL.
• It bypasses logical replication, eliminating replication slots and reducing overhead on the source PostgreSQL.
• Achieves ~200ms commit-to-visible latency and sustains 289K rows/sec throughput in benchmarks.
• Supports full replication lifecycle including initial load, continuous replication, and schema evolution (ADD COLUMN, RENAME, DROP).
• Architecture uses parallel Rust-based decoders and ClickHouse-native blocks for efficient processing.
• Out-of-order block arrival is handled by _lsn for correctness, with schema changes and truncations acting as barriers.

---

📖 Source: Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL