Wiki Admin

Location Connection for Apache Kafka - DSS 6 | Data Source Solutions Documentation

Export as HTML Export as PDF Export as DOCX
Edit

Documentation: Location Connection for Apache Kafka - DSS 6 | Data Source Solutions Documentation

Location Connection for Apache Kafka

This section describes the connection details required for creating Kafka location in Data Source Solutions DSS. For information about the pre-requisites, access privileges, and other configuration requirements for creating Kafka location, see Apache Kafka Requirements.

DSS uses librdkafka (C library which talks Kafka's protocol) to connect to Kafka.

If the managed secrets feature is enabled, option USE TOKEN INSTEAD is displayed in the fields designated for entering secrets.

Field Description Equivalent Location Property
BROKER(S) AND PORT(S)

Hostname or IP address of the Kafka broker server(s) along with the TCP port that the Kafka server uses to listen for client connections. The default port is 9092.

DSS supports connecting to more than one Kafka broker servers. Click to add more Kafka brokers.

When connecting to Kafka clusters in Confluent Cloud, use the Bootstrap server from the Cluster settings section of the Confluent Cloud web interface.
Example: pkc_lgwgm.eastus2.azure.confluent.cloud

When connecting to Azure Event Hubs, use the endpoint for your namespace.
Example: demohub.servicebus.windows.net
The port for Azure Event Hubs is always 9093.		 Kafka_Brokers
AUTHENTICATION METHOD

Authentication method for connecting DSS to Kafka server (BROKER).

Available options are:

  • No Authentication
  • User Name and Password
  • Kerberos

On Linux, to use either of the Kafka authentication methods (User Name and Password or Kerberos), DSS requires the library libsasl2.so.2 to be installed. For more information, see section Installation Dependencies in Apache Kafka Requirements.

When connecting to Kafka clusters in Confluent Cloud, set the authentication mode to User Name and Password.

 Kafka_Authentication_Method
USER

Username for connecting DSS to the Kafka server.

This field is enabled only if AUTHENTICATION METHOD is set to User Name and Password.

When connecting to Kafka clusters in Confluent Cloud, use the API access key obtained from the Confluent Cloud web interface.

When connecting to Azure Event Hubs, use $ConnectionString. 		Stream_User
PASSWORD

Password of the USER.

When connecting to Kafka clusters in Confluent Cloud, use the API access secret obtained from the Confluent Cloud web interface.

When connecting to Azure Event Hubs, use the following pattern for the password (this should be a one line without spaces):

Endpoint=endpoint; SharedAccessKeyName=keyName; SharedAccessKey=accessKey

where:

  • endpoint is the namespace URL, which can be copied from the Azure Shared Access Policy tab on the left panel in the Azure portal.
  • keyName is RootManageSharedAccessKey.
  • accessKey is a security key associated with the Shared Access Policy and used to access your event hub.

Example (this should be a one line without spaces):

Endpoint=sb://demohub.servicebus.windows.net/; SharedAccessKeyName=RootManageSharedAccessKey; SharedAccessKey=xxxxxxxxxxxxx;

Stream_Password
SERVICE NAME

Kerberos Service Principal Name (SPN) of the Kafka server.

This field is enabled only if AUTHENTICATION METHOD is set to Kerberos.		 Kafka_Kerberos_Service
CLIENT PRINCIPAL

Full Kerberos principal of the client connecting to the Kafka server. This is required only on Linux/Unix.

This field is enabled only if AUTHENTICATION METHOD is set to Kerberos.		 Kafka_Kerberos_Client_Principal
CLIENT KEYTAB

Directory path where the Kerberos keytab file containing key for the CLIENT PRINCIPAL is located.

This field is enabled only if AUTHENTICATION METHOD is set to Kerberos.		 Kafka_Kerberos_Keytab
DEFAULT TOPIC
 Kafka topic to which the messages are written.
You can use strings/text or expressions as Kafka topic name. Following are the expressions to substitute capture location or table or schema name as topic name:
  • {dss_cap_loc} - for capture location name.
  • {dss_tbl_name} - for current table name. This is only allowed if the channel is defined with tables.
  • {dss_schema}- for schema name of the table. This is only allowed if the channel contains tables that have action TableProperties with parameter Schema=my_schema explicitly defined for these tables on the target file location.
The Kafka topics should either exist already in the Kafka broker, or it should be configured to auto-create Kafka topics when DSS sends a message.

When connecting to Azure Event Hubs, use your Event Hub name (e.g. myeventhub). This is optional.

 Kafka_Default_Topic

Advanced Settings

Field Description Equivalent Location Property
Enable SSL

Enable/disable (one way) SSL. If enabled, DSS authenticates the Kafka server by validating the SSL certificate shared by the Kafka server.

When connecting to Azure Event Hubs, SSL must be enabled if you have chosen User Name and Password in the AUTHENTICATION METHOD field.
CLIENT PUBLIC CERTIFICATE Directory path where the .pem file containing the client's SSL public certificate is located. Stream_Client_Public_Certificate
CLIENT PRIVATE KEY Directory path where the .pem file containing the client's SSL private key is located. Stream_Client_Private_Key
CLIENT PRIVATE KEY PASSWORD Password of the private key file that is specified in CLIENT PRIVATE KEY. Stream_Client_Private_Key_Password
TRUSTED CA CERTIFICATE
 When connecting to Kafka clusters in Confluent Cloud or Azure Event Hubs, a Broker CA must be specified here. DSS supplies the CA in DSS_HOME/etc/cert/ca-bundle.crt. Stream_Public_Certificate