Monitoring ClickPipes

In addition to in-console monitoring, ClickPipes exposes metrics to a Prometheus-compatible endpoint for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., Grafana, Datadog).

Metric labels

ClickPipes metrics use the standard service-level labels (clickhouse_org, clickhouse_service, clickhouse_service_name) plus the following ClickPipes-specific labels:

Label	Description
`clickpipe_id`	The unique identifier of the ClickPipe
`clickpipe_name`	The name of the ClickPipe
`clickpipe_source`	The source type (e.g., `kafka`, `postgres`, `s3`)
`clickpipe_state`	The state of the ClickPipe (e.g., `Stopped`, `Provisioning`, `Running`)

Sample response

The following shows what ClickPipes metrics look like in a Prometheus scrape response for a Kafka ClickPipe. The set of exposed metrics varies by pipe type — see Available metrics for which metrics apply to which pipe types:

# HELP ClickPipes_Info Always equal to 1. Label "clickpipe_state" contains the current state of the pipe: Stopped/Provisioning/Running/Paused/Failed
# TYPE ClickPipes_Info gauge
ClickPipes_Info{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent",clickpipe_state="Running"} 1

# HELP ClickPipes_FetchedEvents_Total Total number of records fetched from the source.
# TYPE ClickPipes_FetchedEvents_Total counter
ClickPipes_FetchedEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5535376

# HELP ClickPipes_SentEvents_Total Total number of records sent to ClickHouse
# TYPE ClickPipes_SentEvents_Total counter
ClickPipes_SentEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5534250

# HELP ClickPipes_FetchedBytes_Total Total uncompressed bytes fetched from the source.
# TYPE ClickPipes_FetchedBytes_Total counter
ClickPipes_FetchedBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 873286202

# HELP ClickPipes_SentBytes_Total Total uncompressed bytes sent to ClickHouse.
# TYPE ClickPipes_SentBytes_Total counter
ClickPipes_SentBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 477187967

# HELP ClickPipes_Errors_Total Total errors ingesting data.
# TYPE ClickPipes_Errors_Total counter
ClickPipes_Errors_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 0

# HELP ClickPipes_Latency Time in milliseconds between when the record was produced to when it was written to ClickHouse.
# TYPE ClickPipes_Latency gauge
ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 340

Available metrics

Not every metric is emitted by every pipe type. In particular, CDC ClickPipes expose a different subset of metrics than streaming and object storage ClickPipes.

Streaming, object storage — Kafka, Kinesis, Pub/Sub, S3, and GCS
CDC — Postgres, MySQL, MongoDB and BigQuery
All — every pipe type

Data transfer

Metric	Type	Available for	Description
`ClickPipes_FetchedBytes_Total`	Counter	Streaming, object storage	Total uncompressed bytes fetched from the source.
`ClickPipes_FetchedBytesCompressed_Total`	Counter	All	Total bytes fetched from the source, as read over the wire. If the source data is uncompressed, equivalent to `ClickPipes_FetchedBytes_Total`. For CDC ClickPipes, this excludes bytes fetched from initial load, which is emitted separately to `ClickPipes_FetchedBytesInitialLoad_Total`.
`ClickPipes_FetchedBytesInitialLoad_Total`	Counter	CDC	Total bytes fetched during the initial load phase, including resyncs.
`ClickPipes_SentBytes_Total`	Counter	Streaming, object storage	Total uncompressed bytes sent to ClickHouse.
`ClickPipes_SentBytesCompressed_Total`	Counter	Streaming, object storage	Total compressed bytes sent to ClickHouse.

Events and records

Metric	Type	Available for	Description
`ClickPipes_FetchedEvents_Total`	Counter	Streaming, object storage	Total number of records fetched from the source.
`ClickPipes_SentEvents_Total`	Counter	Streaming, object storage	Total number of records sent to ClickHouse.
`ClickPipes_FetchedEvents_PerTable`	Gauge	CDC	Number of records fetched per destination table and event type. Uses additional `destination_table` and `event_type` labels.

Errors

Metric	Type	Available for	Description
`ClickPipes_Errors_Total`	Counter	All	Total errors encountered during ingestion.

Latency

Metric	Type	Available for	Description
`ClickPipes_Latency`	Gauge (avg)	All	Time in milliseconds between when a record was produced at the source and when it was written to ClickHouse.
`ClickPipes_SourceReplicationLatency_MiB`	Gauge (avg)	CDC (Postgres only)	Replication slot lag at the source in MiB. Only emitted by Postgres ClickPipes.
`ClickPipes_CDC_SourceLatency`	Gauge (avg)	CDC	Lag in milliseconds from a source event timestamp to ClickPipes processing.
`ClickPipes_CDC_DestinationLatency`	Gauge (avg)	CDC	Lag in milliseconds from receiving an event in a ClickPipe to writing it into a ClickHouse table.

Resource utilization

ClickPipes_Replica_* metrics report resource utilization for streaming and object storage ClickPipes. Each ClickPipe has dedicated compute, so these values reflect utilization for individual ClickPipes.

Metric	Type	Available for	Description
`ClickPipes_Replica_CPUUsage`	Gauge (avg)	Streaming, object storage	Average CPU usage of the ingestion replicas, in millicores.
`ClickPipes_Replica_CPULimit`	Gauge (last)	Streaming, object storage	Configured CPU limit of the ingestion replicas, in millicores.
`ClickPipes_Replica_MemoryUsage`	Gauge (avg)	Streaming, object storage	Average memory usage of the ingestion replicas, in bytes.
`ClickPipes_Replica_MemoryLimit`	Gauge (last)	Streaming, object storage	Configured memory limit of the ingestion replicas, in bytes.

ClickPipes_CDC_* metrics report resource utilization for CDC ClickPipes. All CDC ClickPipes in the same service share compute, so these values reflect the combined usage of every CDC ClickPipe in the service.

Metric	Type	Available for	Description
`ClickPipes_CDC_CPUUsage`	Gauge (avg)	CDC	Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service.
`ClickPipes_CDC_CPULimit`	Gauge (last)	CDC	Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service.
`ClickPipes_CDC_MemoryUsage`	Gauge (avg)	CDC	Average memory usage, in bytes. Shared across all CDC ClickPipes in the service.
`ClickPipes_CDC_MemoryLimit`	Gauge (last)	CDC	Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service.
`ClickPipes_CDC_NetworkReceiveBytes`	Gauge	CDC	Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service.

State and progress

Metric	Type	Available for	Description
`ClickPipes_Info`	Gauge	All	Always equal to `1`. Use the `clickpipe_state` label to filter or alert on pipe state.
`ClickPipes_LastFetchedBatchId`	Gauge	CDC	ID of the last batch fetched from the source.
`ClickPipes_LastSentBatchId`	Gauge	CDC	ID of the last batch sent to ClickHouse.

Prometheus endpoint — Endpoint reference, authentication, and setup for external tools (e.g., Grafana, Datadog)
Error reporting — Where record and system errors are stored