How can I use API to manage clusters on ClickHouse Cloud?
\nWe will use Terraform to configure our infra and ClickHouse Provider
\nSteps:
\n1). Create an API Key on Cloud.\nFollow the docs here - https://clickhouse.com/docs/cloud/manage/openapi
\nSave the creds locally.
\n2). Install Terraform using - https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
\nYou can use Homebrew package manager if you're on Mac.
\n3). Create a directory anywhere you like:
\n4). Create 2 files: main.tf and secret.tfvars
Copy the following:
\nmain.tf file would be:
You can replace your own parameters like service name, region etc.. in the resources section above.
\nsecret.tfvars is where you'll put all the API Key related info that you downloaded earlier. The idea behind this file is that all your secret credentials will be hidden from the main config file.
It would be something like (replace these parameters):
\n5). Run terraform init from this directory
Expected output:
\n6). Run terraform apply -var-file=secret.tfvars command.
Something like:
\nType yes and hit enter
Side note: Notice it says password = (sensitive value) above.\nThis is because we set sensitive = true for the password in the main.tf file.
7). It will take a couple of mins to create the service but eventually it should come up like:
\n8). Check Cloud Console, you should be able to see the service created.
\n9). To clean up/destroy the service again, run terraform destroy -var-file=secret.tfvars
Something like:
\nType yes and hit enter
\n10).
\nAnd it should be gone from the Cloud Console.
\nMore details about the Cloud API can be found here - https://clickhouse.com/docs/cloud/manage/api/api-overview
This guide explains how to expose an Amazon MSK cluster to ClickPipes with AWS PrivateLink VPC endpoint services.
\nUse this approach when MSK multi-VPC connectivity is not available or not sufficient, for example:
\nFor same-region standard MSK clusters, MSK multi-VPC connectivity is usually simpler and is the recommended option.
\nThe same PrivateLink pattern can also be used for self-managed or on-premises Kafka clusters that are exposed through AWS, but the examples in this article use MSK.
\nKafka clients first connect to bootstrap brokers, fetch cluster metadata, and then connect directly to the broker hostnames advertised in that metadata.
\nFor that reason, a single load balancer in front of all brokers is not sufficient for a reliable MSK PrivateLink setup. Instead, create one PrivateLink path per broker:
\nEach broker gets its own:
\nThe NLB should use TCP passthrough. This allows the Kafka client to keep using the original MSK broker hostname for TLS SNI and certificate validation.
\narn:aws:iam::072088201116:root.Custom private DNS name field is not available when creating a reverse private endpoint.Use the AWS CLI to list MSK broker nodes:
\nRecord each broker hostname, private IP address, and Kafka port. For MSK IAM authentication, the port is usually 9098.
The ClickPipe will use these broker hostnames as its Kafka brokers. Do not use the MSK bootstrap broker string for this UI-based setup.
\nRepeat these steps for each MSK broker.
\nCreate an internal Network Load Balancer in the MSK VPC. The NLB should use private subnets and must not be internet-facing.
\nIf a security group is attached to the NLB, disable Enforce Security Group Inbound Rules on Private Link Traffic. This is required to allow traffic from the ClickPipes VPC endpoint to reach the NLB.
Create a target group with:
\nIP.TCP.9098 for MSK IAM.Use TCP health checks on the traffic port.
\nCreate an NLB listener on the broker Kafka port and forward traffic to the broker-specific target group.
\nCreate a VPC endpoint service backed by the broker-specific NLB.
\nAdd the ClickPipes AWS principal to the allowed principals:
\nIf the MSK cluster and ClickPipes are in different AWS regions, enable cross-region access by adding the ClickPipes region to the endpoint service supported regions.
\nRecord the endpoint service name. It will look similar to:
\nIn the ClickHouse Cloud console, create one reverse private endpoint for each broker endpoint service:
\nReverse private endpoint.VPC endpoint service as the VPC endpoint type.Custom private DNS name, enter the original MSK broker hostname for that broker.Repeat the process until every broker hostname has a matching RPE.
\nFor custom private DNS behavior, naming rules, and API/Terraform options, see\nCustom private DNS.
\nYou can also create the reverse private endpoints with OpenAPI or Terraform.
\nFor OpenAPI, create one reverse private endpoint per broker with type set to VPC_ENDPOINT_SERVICE, the broker-specific VPC endpoint service name, and one custom private DNS mapping for the broker hostname:
For Terraform, use clickhouse_clickpipes_reverse_private_endpoint to create the endpoint and clickhouse_clickpipes_reverse_private_endpoint_custom_private_dns to configure the broker hostname mapping.
Follow the ClickPipes AWS PrivateLink guide\nto create the ClickPipe with reverse private endpoints.
\nWhen creating the Kafka ClickPipe:
\nThe port depends on your MSK listener. For example, IAM authentication commonly uses 9098, SASL/SCRAM commonly uses 9096, and TLS commonly uses 9094. Confirm the correct port from your MSK cluster configuration or bootstrap broker output.
Example broker list:
\nEach hostname in the broker list must match the Custom private DNS name configured on one of the RPEs.
The aws-msk-vpc-endpoint-service Terraform module shows how to automate this setup. It creates broker-specific NLBs, target groups, VPC endpoint services, ClickPipes RPEs, and custom private DNS mappings.
\nThe Terraform module can manage multiple custom private DNS mappings per RPE, including both bootstrap and broker hostnames. This article focuses on the manual ClickPipes UI workflow, where each RPE uses one broker hostname as its custom private DNS name.
This guide explains how to connect ClickPipes to an existing Confluent Cloud Kafka cluster through private networking.
\nUse this guide for:
\nThis guide assumes the Confluent Cloud Kafka cluster, topic, and Kafka credentials already exist.\nIt focuses on private connectivity and ClickPipes reverse private endpoints (RPEs).
\nYou cannot currently complete this Confluent Cloud setup with only the ClickHouse Cloud console.
Confluent Cloud requires multiple custom private DNS mappings for a single reverse private endpoint.\nThe ClickHouse Cloud console currently supports only one custom private DNS name per RPE.
Use the ClickHouse Cloud OpenAPI\nor Terraform to create or update RPE custom DNS mappings.
Kafka clients connect to a bootstrap hostname, fetch metadata, and then connect to the broker hostnames advertised by Confluent Cloud.\nFor private Confluent Cloud networking, those hostnames must resolve inside the ClickPipes network to the private endpoint created by ClickPipes.
\nClickPipes uses:
\nUse this section for Confluent Cloud Dedicated clusters connected through a Confluent private network.
\nFrom the Confluent Cloud network attached to your Dedicated cluster, collect:
\n{dns_domain}.The zone values are part of the DNS names that ClickPipes must resolve.\nFor AWS they are availability zone IDs such as use1-az1.\nFor GCP they are zones such as us-central1-a.
Add a PrivateLink or PSC access entry in Confluent Cloud for the ClickPipes consumer identity:
\n072088201116.clickpipes-production.This authorization is required before the ClickPipes RPE can connect to the Confluent Cloud private network.
\nCreate the RPEs with OpenAPI\nor Terraform.
\nWith OpenAPI, use customPrivateDnsMappings in the create request.\nIf you update an existing RPE with the update operation,\nremember that customPrivateDnsMappings is a full replacement list.
Create one VPC_ENDPOINT_SERVICE RPE that points to the Confluent Cloud PrivateLink endpoint service.
Add all of these custom private DNS mappings to the same RPE:
\n*.{dns_domain}*.{zone}.{dns_domain} for each Confluent network zoneExample OpenAPI request body:
\nCreate one GCP_PSC_SERVICE_ATTACHMENT RPE per Confluent Cloud zone.\nEach RPE points to that zone's PSC service attachment.
Add *.{zone}.{dns_domain} to the matching zone RPE.\nAdd *.{dns_domain} to one of the RPEs only.
Example first-zone OpenAPI request body:
\nExample additional-zone OpenAPI request body:
\nUse this section for Confluent Cloud serverless clusters that use Confluent private ingress gateways and access points.
\nCreate the private ingress gateway in Confluent Cloud for the same cloud and region as the Confluent Kafka cluster:
\nRecord the gateway target that ClickPipes must connect to:
\nCreate one ClickPipes RPE for the Confluent ingress gateway.
\nFor AWS, create a VPC_ENDPOINT_SERVICE RPE:
For GCP, create a GCP_PSC_SERVICE_ATTACHMENT RPE:
Wait until the RPE response includes an endpointId.\nYou need this ID to create the Confluent access point.
Create a Confluent Cloud ingress access point for the ClickPipes endpoint:
\nendpointId as the VPC endpoint ID.endpointId as the PSC connection ID.After the access point is created, collect:
\n{access_point_dns_domain}.Patch the ClickPipes RPE with both custom private DNS mappings:
\n*.{region}.{cloud}.accesspoint.glb.confluent.cloud*.{access_point_dns_domain}Example OpenAPI update request body:
\nThe *.{region}.{cloud}.accesspoint.glb.confluent.cloud mapping is broad and is unique per cloud and region.\nBecause custom private DNS names must be unique across all RPEs in a ClickHouse service,\nyou cannot currently create multiple Confluent ingress access points in the same cloud and region for the same ClickHouse service.
With Terraform, use:
\nclickhouse_clickpipes_reverse_private_endpoint to create RPEs.clickhouse_clickpipes_reverse_private_endpoint_custom_private_dns to manage the full set of custom private DNS mappings for each RPE.The ClickPipes Terraform modules also provide complete Confluent examples:
\nconfluent-dedicated for Confluent Cloud Dedicated private networking.confluent-serverless for Confluent Cloud serverless private ingress.Use the same DNS mapping patterns from this guide.\nIn Terraform, field names use snake_case, for example vpc_endpoint_service_name, gcp_service_attachment, and private_dns_name.
After the RPEs are ready:
\nSASL_SSL:// prefix.PLAIN authentication with the Confluent Cloud Kafka API key and secret.If you use a Confluent Schema Registry over private networking, configure the schema registry URL in the ClickPipe and ensure its hostname resolves through the same RPE custom private DNS setup.\nFor more details, see Schema registries for Kafka ClickPipe.
\nClickHouse Cloud does not support multi-statement transactions in the traditional RDBMS sense.\nThis creates two common challenges:
\nClickHouse guarantees atomicity at the single-insert, single-partition level: if an INSERT succeeds, all rows in that block are visible; if it fails, none are. However, there is no built-in mechanism to atomically commit data across multiple inserts or multiple tables.
The partition manipulation commands (MOVE PARTITION TO TABLE, REPLACE PARTITION, ATTACH PARTITION FROM) operate at the metadata level when the source and destination tables share the same storage policy.
This means they execute almost instantaneously regardless of data size, making them ideal building blocks for atomic-swap patterns.
\nInstead of inserting directly into production tables and attempting to clean up on failure, use dedicated staging tables as a landing zone. After validating the data, use partition-level operations to atomically promote the data into production.
\nUse the same schema, partition key, ORDER BY, and storage policy as the production table.
Run your insert against the staging table instead of the production table.
Truncate the staging table and run the load again. No production data has been affected.
To move the data into production and remove it from the staging table, use MOVE PARTITION.
Alternatively, copy the data into an existing partition in production with ATTACH PARTITION.
Both operations are metadata-level changes on the same storage policy and complete almost instantaneously.
Once all partitions have been moved, truncate the staging table to leave it empty for the next load.
The same pattern solves pipelines where two or more tables must all be loaded before any of them becomes visible to analysts. Load each table's data into its own staging table and validate all of them, then run the partition moves together. Because each move is a near-instant metadata operation, the window in which the tables disagree shrinks from the duration of the full load to the time it takes to swap partitions.
\nFor MOVE PARTITION TO TABLE, REPLACE PARTITION, and ATTACH PARTITION FROM, the source and destination tables must have:
ORDER BY key, and primary keyYou can run the example below interactively using fiddle:
\nThe JSON column type is production-ready from ClickHouse 25.3+. Earlier versions are not recommended for production use.
Your data arrives as JSON. ClickHouse gives you several ways to store it, from fully typed columns to a raw String. The right choice depends on how predictable your schema is and whether you need field-level queries.
\nScope: This page covers schema design decisions for storing JSON data. It does not cover JSON input/output formats, JSON functions, or query syntax. For background on the JSON column type itself, see Use JSON where appropriate.
\nAssumes: Familiarity with ClickHouse table creation, MergeTree basics, and column type syntax.
\nMap instead of JSONDon't confuse the JSON format with the JSON column type. You can insert JSON-formatted data (via JSONEachRow, etc.) into typed columns without using the JSON column type at all. The decision here is about column types, not input formats.
When to use: The JSON structure is fully known at design time. Fields and types don't change between records. Even complex nested structures (arrays of objects, nested maps) can be expressed with Array, Tuple, and Nested types.
Trade-offs: Schema changes require ALTER TABLE. Unexpected fields are silently dropped on insert unless the schema is updated.
Setup
Verification
Watch out for
JSONEachRow and the JSON contains fields not in the schema, ClickHouse drops them silently by default. Set input_format_skip_unknown_fields to 0 if you want errors instead.When to use: A core set of fields is stable (timestamps, IDs, status codes), but part of the payload is dynamic. Think user-defined attributes, tags, metadata, or extension fields that vary across records.
\nTrade-offs: Full performance on typed columns, flexibility on the JSON column. The JSON column still carries insert overhead and storage cost for its dynamic portion.
\nSetup
Verification
Watch out for
SKIP or SKIP REGEXP for paths you never query (debug metadata, internal tracing IDs) to save storage and reduce subcolumn count.max_dynamic_paths proportional to the number of distinct paths you actually query. The default (1024) works for most cases. Lower it if your dynamic section is narrow.max_dynamic_paths above 10,000. High values increase resource consumption and reduce efficiency.Keys with dots (e.g., http.status_code) are treated as nested paths by default, so {\"http.status_code\": 200} is stored the same as {\"http\": {\"status_code\": 200}}. This is common with OTel attributes. Use type hints to control how dotted paths are stored, or enable json_type_escape_dots_in_keys (25.8+).
When to use: The structure is genuinely unpredictable, with keys that appear and disappear across records. User-generated schemas, plugin systems, or data-lake ingestion where you don't control the upstream schema.
\nTrade-offs: Slower inserts than typed columns. Slower full-object reads than String. Storage overhead from subcolumn management. Works well for field-level queries on specific paths.
\nSetup
Use JSONAsObject format when inserting whole JSON documents into a JSON column. It treats each input line as a complete JSON object mapped to the column.
Verification
Watch out for
score arrives as \"10\" (string) in one record and 10 (integer) in another, the path gets a discriminator column and queries become slower. Add hints for paths with known types.max_dynamic_paths, overflow values move to a shared data structure with reduced query performance. Monitor with JSONDynamicPaths() and keep the limit below 10,000.max_dynamic_types (default 32) distinct data types. If a single path exceeds this, extra types fall back to shared variant storage. This rarely matters unless your data has highly inconsistent types for the same field.When to use: JSON documents are stored and retrieved whole, then passed through to an application, archived, or forwarded downstream. No field-level filtering or aggregation inside ClickHouse.
\nTrade-offs: Fastest inserts and simplest schema. No field-level queries without runtime parsing (JSONExtract family), which is slow at scale.
Setup
Verification
Watch out for
JSONExtract functions parse the string on every query. Acceptable for ad-hoc exploration, not for production dashboards or high-QPS workloads.ZSTD) on the String column if the JSON payloads are large — it compresses well.| Dimension | Typed columns | Hybrid | Native JSON | String |
|---|---|---|---|---|
| Insert throughput | Fastest | Fast | Moderate | Fastest |
| Field-level queries | Fastest | Fast (typed); good (hinted JSON) | Good (hinted); slower (dynamic) | Slow (runtime parsing) |
| Full-object reads | Fast | Moderate | Slow | Fastest |
| Storage efficiency | Best | Good | Moderate | Good (compresses well) |
| Schema flexibility | None (ALTER TABLE) | Partial (rigid core, flexible tail) | Full | Full |
| Complexity | Low | Medium | Medium–High | Low |
If your dynamic fields are homogeneous key-value pairs — all values share the same type — Map(String, T) is simpler and more efficient than a JSON column. Common examples: string tags (Map(String, String)), numeric metrics (Map(String, Float64)), or feature flags (Map(String, Bool)).
Map supports key-level filtering (tags['env'] = 'prod'), is cheaper to store than JSON, and avoids the subcolumn overhead of the JSON type. Note that key lookups scan the map linearly by default — fine for small tag sets, but for maps with 100+ keys consider with_buckets serialization. Use JSON when values have mixed types or the structure has nesting — use Map when it's flat key-value pairs with a uniform value type.
Starting February 27, 2026, ClickHouse Cloud will begin collecting crash reports when server crashes occur. This is part of our commitment to improving service reliability for all our users.
\nCrash report collection helps us:
\nWhen a crash occurs, crash reports capture a snapshot of the server's state at that moment. We want to be transparent that this may include sensitive information that was in memory, such as:
\nWe take the security of this data seriously:
\nSome services are automatically excluded from crash report collection:
\nIf you'd prefer not to participate in crash report collection, you can easily opt out:
\nIn addition to the service setting, we also support configuring preferences for the organization. If the setting is disabled, i.e., opt out of the crash report collection, all existing and new services will be opted out automatically. If you set the preference to enabled i.e. opt in, you can selectively disable at the service level to override and opt out.
ClickHouse uses two independent concepts for organizing data within parts:
\nPart types determine how column data is organized within a data part.
\n| Type | Description | Best for |
|---|---|---|
| Wide | Each column is stored in separate file(s), each with its own marks file | Queries selecting a subset of columns |
| Compact | All columns are stored in a single file with a single marks file | Ingestion performance and queries needing all columns |
The part type is controlled by the following table settings:
\nmin_bytes_for_wide_part: Minimum bytes for a part to use Wide formatmin_rows_for_wide_part: Minimum rows for a part to use Wide formatIf either the number of bytes or rows in a data part is less than the corresponding setting value, the part is stored in Compact format; otherwise, it uses Wide format.
Compact parts:
\nWide parts:
\nCompact-to-wide merges are slower than wide-to-wide merges because ClickHouse uses vertical merge algorithms for compact-to-wide conversions, while using horizontal algorithms for compact-to-compact merges. If you want to force the use of wide parts regardless of size, you can set min_bytes_for_wide_part=0.
This can be useful in the following scenarios:
\nsystem.metric_log and system.text_log that are consuming excessive memory during mergesIt's worth noting that while setting this to 0 can improve certain performance characteristics, it may generate more GET requests to S3 storage due to the wide parts format.
\nColumn size statistics are not calculated for compact parts, which can affect monitoring and optimization efforts. When querying system.parts_columns, compact parts show 0 for column_data_compressed_bytes and column_data_uncompressed_bytes. For more information see \"Find counts and sizes of wide or compact parts\".
Storage formats determine how a part's files are physically stored on disk.
\n| Format | Description | Availability |
|---|---|---|
| Full | Each file is stored individually in the part directory | Open source and Cloud |
| Packed | All files are bundled into a single archive file | ClickHouse Cloud only |
In Full storage, each part consists of multiple separate files stored individually on disk. This is the default and only option in open source ClickHouse.
\nIn Packed storage, all part files are bundled into a single archive. This significantly reduces the number of file operations, which is critical for remote storage like S3 where each request has associated latency and cost.
\nBenefits of Packed storage:
\nIn ClickHouse Cloud, a part uses Packed storage if any of these conditions are true:
\nmin_bytes_for_full_part_storagemin_rows_for_full_part_storagemin_level_for_full_part_storageWhile the settings min_bytes_for_full_part_storage, min_rows_for_full_part_storage, and min_level_for_full_part_storage are defined in open source ClickHouse, they have no effect because the Packed storage implementation is only available in ClickHouse Cloud.
The min_level_for_full_part_storage setting can be used to optimize both performance and costs in ClickHouse Cloud environments, particularly for tables that receive continuous inserts.\nThe setting is available starting in ClickHouse version 25.10 and works in conjunction with min_level_for_wide_part to provide comprehensive control over part storage strategies.
Changing it from the default value (0) could be considered in the following use cases:
\nRecommended configuration:
\nSet min_level_for_full_part_storage = 2 for cloud deployments.\nThis ensures that:
Avoid this setting for tables that receive very large but infrequent writes where insufficient merges occur, as large initial writes may benefit from full part storage immediately.
These two concepts are orthogonal and you can have any combination, depending on if you are using ClickHouse Cloud or Open Source ClickHouse:
\n| Combination | Use case |
|---|---|
| Wide + Full | Large parts on local storage (default in open source) |
| Wide + Packed | Large parts on cloud storage |
| Compact + Full | Small parts on local storage |
| Compact + Packed | Small parts on cloud storage |
You can inspect the part type and part storage type of existing parts using the system.parts table:
You will see something like this:
\nThe following query can be used to count the number of parts by type:
\nAn example response is shown below:
\nRun the query below to query part size for wide and compact parts:
\nAn example response is shown below:
\nNote that columns sizes are not calculated in compact parts and they therefore show as 0 above.
Prerequisites:
\nopentelemetry_span_log system table is enabledIf you have not made any modifications to the opentelemetry_span_log section of config.xml you can skip this step.
Open your default ClickHouse config.xml file and find the following section:
Make sure that it is not commented out, or else you won't be able to see system.opentelemetry_span_log in the following steps.\nThis can also be the case if your ClickHouse server is not using the default configuration file.
Check your server logs for something like:
In standard installations this file is located at /etc/clickhouse-server/config.xml
With your ClickHouse server running, open ClickHouse client and enable trace collection using the following query:
You should now see the opentelemetry_span_log system table if you run:
Next run:
This sets the probability that the ClickHouse can start a trace for executed queries, where\n1 means that the trace is enabled for all executed queries.
Run the following dummy query, or the query you are interested in tracing:
Copy the query id:
Run the following query, substituting in the query ID you obtained in the previous step:
This will write the trace to a file named trace.json.\nBy default, this file is created in the current working directory from which you are running the clickhouse-client or clickhouse-local tool.
Use the hosted trace visualizer at https://trace-visualizer.clickhouse.com/.\nLoad the trace.json file from the previous step to visualize the trace.
We recommend Grafana for visualizing and exploring trace data using the official ClickHouse plugin.\nThe plugin has been enhanced to allow visualization of traces using the Trace Panel.\nThis is supported as both a visualization and as a component in Explore.
Follow the steps described in \"Using Grafana and ClickHouse for Observability\"\nto set up Grafana with the ClickHouse plugin.
From the Explore tab you can then run the following query, replacing the trace_id with your own:
Make sure to set Query type to Traces:
Click \"Run Query\" and inspect the trace diagram:
This guide assumes that the <path> parameter in your config.xml file is set to:
If you have configured a different data path, replace all instances of /var/lib/clickhouse in the below commands with the actual value of your <path> setting.
Copy the contents of the access folder which contains local users from the healthy replica:
/var/lib/clickhouse/store/.. which contains all the table DDLs.This command ensures that both the metadata files, and the symlink architecture are preserved in the backup.
backup.tar file to the faulty replica.To trigger automatic data synchronization from other replicas, create the following flag: