Skip to content
ClickHouse Docs
ClickHouse DocsClickHouse Docs
49.6kGet Started

Integrating Confluent platform with ClickHouse

View as MarkdownEdit this pageReport an issueSet up docs MCP

Prerequisites

We assume you’re familiar with:

The official Kafka connector from ClickHouse with Confluent Platform

Installing on Confluent platform

This is meant to be a quick guide to get you started with the ClickHouse Sink Connector on Confluent Platform. For more details, please refer to the official Confluent documentation.

Create a Topic

Creating a topic on Confluent Platform is fairly simple, and there are detailed instructions here.

Important notes

  • The Kafka topic name must be the same as the ClickHouse table name. The way to tweak this is by using a transformer (for example ExtractTopic).
  • More partitions doesn’t always mean more performance - see our upcoming guide for more details and performance tips.

Install connector

You can download the connector from our repository - please feel free to submit comments and issues there as well!

Navigate to “Connector Plugins” -> “Add plugin” and using the following settings:

'Connector Class' - 'com.clickhouse.kafka.connect.ClickHouseSinkConnector'
'Connector type' - Sink
'Sensitive properties' - 'password'. This will ensure entries of the ClickHouse password are masked during configuration.

Example:

Confluent Platform UI showing settings for adding a custom ClickHouse connector

Gather your connection details

To connect to ClickHouse with HTTP(S) you need this information:

Parameter(s) Description
HOST and PORT Typically, the port is 8443 when using TLS or 8123 when not using TLS.
DATABASE NAME Out of the box, there is a database named default, use the name of the database that you want to connect to.
USERNAME and PASSWORD Out of the box, the username is default. Use the username appropriate for your use case.

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console. Select a service and click Connect:

ClickHouse Cloud service connect button

Choose HTTPS. Connection details are displayed in an example curl command.

ClickHouse Cloud HTTPS connection details

If you’re using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

Configure the connector

Navigate to Connectors -> Add Connector and use the following settings (note that the values are examples only):

{
  "database": "<DATABASE_NAME>",
  "errors.retry.timeout": "30",
  "exactlyOnce": "false",
  "schemas.enable": "false",
  "hostname": "<CLICKHOUSE_HOSTNAME>",
  "password": "<SAMPLE_PASSWORD>",
  "port": "8443",
  "ssl": "true",
  "topics": "<TOPIC_NAME>",
  "username": "<SAMPLE_USERNAME>",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": "false"
}

Specify the connection endpoints

You need to specify the allow-list of endpoints that the connector can access. You must use a fully-qualified domain name (FQDN) when adding the networking egress endpoints. Example: u57swl97we.eu-west-1.aws.clickhouse.com:8443

Read the documentation.

You should be all set!

Known limitations

Navigation