kubenetmon: Unlocking Cloud Network Cost Visibility

Demystifying Cloud Network Spend

ClickHouse's open-sourcing of kubenetmon is a commendable move, directly tackling the often-neglected but significant cost of cloud network traffic. The detailed explanation of their goals – forensics, attribution, and metering – highlights a sophisticated understanding of the problem. Their systematic evaluation of existing solutions, from Cilium Hubble to CSP flow logs and cost-metering tools, demonstrates a thorough R&D process. The chosen approach, leveraging Linux's conntrack with accounting enabled, is pragmatic and leverages existing kernel features. The article effectively communicates the technical challenges and the rationale behind their in-house solution, making it valuable for anyone managing complex cloud infrastructure. The focus on L3/L4 connection records and workload attribution provides actionable insights for cost optimization and operational understanding. The open-sourcing aspect significantly amplifies its impact, allowing the community to benefit and potentially contribute.

However, while the solution is elegant in its use of conntrack, potential limitations warrant consideration. The article acknowledges the risk of conntrack table overfilling, though it mitigates this by deploying kubenetmon in less connection-dense clusters. This implies that in highly saturated environments, careful capacity planning and tuning would still be crucial. Furthermore, the article mentions that short-lived connections might be missed, and while deemed minimal, this data loss could be significant for certain use cases or for precise metering. The attribution mechanism, while described as being solved, is only partially detailed in the provided excerpt, leaving room for further exploration of its robustness and scalability, especially in multi-tenant environments with dynamic workload scaling. The reliance on scraping conntrack periodically also introduces a granularity limit to the forensic data, which might not be sufficient for extremely high-frequency analysis.

Despite these points, the overall contribution is substantial. By providing a concrete, open-source tool for monitoring and attributing network data transfer costs, ClickHouse empowers other organizations to gain much-needed visibility. This is particularly beneficial for multi-cloud, multi-region deployments, where network costs can quickly become a major, unpredictable expense. The tool's focus on L3/L4 connections and workload attribution makes it distinct from purely FinOps tools, offering a deeper technical understanding of network usage. Developers and SREs responsible for cloud infrastructure and cost management will find kubenetmon immediately useful for diagnosing network-related issues and optimizing spending. The potential for broader adoption and community contributions through open-sourcing makes this a significant release.

Key Points

• ClickHouse has open-sourced kubenetmon, a tool designed to monitor and attribute data transfer costs in cloud environments.
• The tool addresses the often-neglected but significant cost of network traffic, which frequently ranks third in cloud bills.
• kubenetmon's primary goals are forensics (detailed connection data over time), attribution (linking connections to workloads/AZs/pods), and metering (cost accounting by workload for specific transfer categories).
• Existing solutions like Cilium Hubble, CSP flow logs, and FinOps tools were evaluated but found insufficient for ClickHouse's specific needs regarding volumetric data with context, custom annotations, and detailed connection behavior analysis.
• The solution leverages Linux's conntrack with the accounting feature enabled to track byte and packet counts for L3/L4 connections.
• kubenetmon scrapes conntrack entries periodically, atomically resetting counters to measure throughput in the interval between scrapes.
• The tool aims to provide granular visibility into network usage, enabling better cost management and operational troubleshooting for complex, multi-cloud infrastructures.

---

📖 Source: Open sourcing kubenetmon: how we monitor data transfer in ClickHouse Cloud