ClickPipes Now Fully Managed via Terraform & OpenAPI

IaC for Data Ingestion

The general availability of Terraform and OpenAPI support for ClickPipes marks a crucial step for ClickHouse Cloud in embracing modern DevOps practices. By enabling infrastructure-as-code (IaC) for data ingestion, ClickHouse is addressing a key pain point for enterprises moving beyond manual UI-driven configurations. The comprehensive coverage, including CDC connectors for popular databases like PostgreSQL, MySQL, and MongoDB, as well as support for cloud storage like S3, GCS, BigQuery, and Azure Blob Storage, significantly enhances the platform's appeal to developers and SREs. The ability to manage ClickPipes programmatically allows for version control, automated deployments, and seamless integration into existing CI/CD pipelines, ultimately boosting reliability and reducing operational overhead. This move democratizes advanced data ingestion capabilities, making them accessible to a wider audience accustomed to declarative configuration and automated provisioning.

However, the article does highlight a few limitations that warrant attention. The absence of SSH tunneling support for ClickPipes is a notable omission, as it's a common requirement for secure connections to on-premises or private network data sources. While ClickHouse acknowledges this and is working on it, it means that for some users, a full IaC workflow for all ClickPipes might still require a manual step or workaround. Additionally, the specific note about needing to set stopped = true for Postgres CDC ClickPipes during import and then applying again to resume suggests a potential edge case in the import process that could lead to unexpected downtime or require careful planning. The reliance on API keys for authentication, while standard, also necessitates robust credential management practices to ensure security, a topic that could benefit from more in-depth guidance within the article itself. Despite these points, the overall direction is positive, demonstrating a commitment to API-first design and developer experience.

Key Points

• ClickHouse Cloud's ClickPipes data ingestion platform is now generally available with full Terraform and OpenAPI support.
• This enables infrastructure-as-code (IaC) management for ClickPipes, allowing for automated provisioning, version control, and integration into CI/CD pipelines.
• Full connector coverage is now available, including CDC for PostgreSQL, MySQL, and MongoDB, along with BigQuery, Azure Blob Storage, S3, and GCS connectors.
• The release aims to bridge the gap between UI-based onboarding and production-ready automated workflows.
• Support for creating and managing reverse private endpoints is also included.
• A known limitation is the current lack of SSH tunneling configuration, though it is under development.
• The article provides guides for both creating new ClickPipes and importing existing ones into Terraform state.
• ClickHouse emphasizes treating Terraform and OpenAPI as first-class interfaces for ClickPipes moving forward.

---

📖 Source: Terraform & OpenAPI for ClickPipes is now Generally Available