all-things-risingwave

How to correctly configure Kafka source with SASL authentication in RisingWave?

I'm trying to set up a Kafka source in RisingWave with SASL authentication, but I'm encountering a GroupAuthorizationFailed exception. I've confirmed that the credentials are correct. Here's the configuration I'm using:

with (
   connector = 'kafka',
   topic = 'xxxxxxx',
   properties.group.id = 'xxxxxxxxxxxxx',
   properties.bootstrap.server = 'alikafka-pre-cn-xxxxxxxxxx-1-vpc.alikafka.aliyuncs.com:9094',
   scan.startup.mode = 'latest',
   enable.auto.commit = 'true',
   properties.security.protocol = 'SASL_PLAINTEXT',
   properties.sasl.mechanism = 'SCRAM-SHA-256',
   properties.sasl.username = 'xxxxxxxxx',
   properties.sasl.password = 'xxxxxxxxxxxxxxx',
   properties.sasl.jaas.config = 'org.apache.kafka.common.security.scram.ScramLoginModule required username="xxxxxxx" password="xxxxxxxxxxxxx";',
   properties.session.timeout.ms = '240000',
   properties.request.timeout.ms = '240000'
) FORMAT PLAIN ENCODE JSON;

Could the issue be related to the configuration format or is there something else I'm missing?

Mi

Michael Li

Asked on Jan 03, 2024

It seems like the configuration for SASL authentication might not be set correctly for Alibaba Kafka in RisingWave. Here's a configuration that was suggested to work:

create source s with (
  connector = 'kafka',
  topic = 'test',
  properties.bootstrap.server = 'endpoint',
  properties.security.protocol = 'SASL_SSL',
  properties.sasl.mechanism = 'PLAIN',
  properties.sasl.username = '<user>',
  properties.sasl.password = '<password>',
  properties.ssl.ca.location = '/Users/wutao/only-4096-ca-cert'
) row format json;

Additionally, it was mentioned that the properties.sasl.jaas.config is not supported by RisingWave. If you are using a VPC endpoint to access Kafka, you should set the mechanism to 'PLAIN'. Also, ensure that your API key has access to the topic. If the issue persists, consider testing with a direct Rust implementation using rust-rdkafka to verify if the problem is specific to Alibaba Kafka or RisingWave.

Jan 04, 2024Edited by