all-things-risingwave

How to Resolve Kafka Connection Error in Docker Compose Configuration?

I'm encountering a Kafka connection error while trying to create a source using Docker Compose. The error message is internal error: Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure). I've provided my Docker Compose configuration file below. Can someone help me identify what might be causing this issue and how to fix it?

version: "3"
name: rise
services:
  # ... (other services)

  kafka:
    image: confluentinc/cp-kafka:7.0.1
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: <PLAINTEXT://kafka:9092>
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_COMPRESSION_TYPE: lz4
      KAFKA_LOG_RETENTION_MS: 31104000000
    depends_on:
      zookeeper: { condition: service_healthy }
    healthcheck:
      {
        test: nc -z localhost 9092,
        interval: 5s,
        start_period: 120s
      }

  # ... (other services)

Additionally, here's the CREATE SOURCE SQL statement I used:

create materialized source block (
        # ... (other fields)
)
with (
        'connector' = 'kafka',
        'kafka.topic' = 'blockchain',
        'kafka.brokers' = '127.0.0.1:9092',
        'kafka.scan.startup.mode'='latest'
)
row format json;
ji

jie

Asked on Jun 01, 2022

The issue seems to be related to the Kafka broker address specified in the CREATE SOURCE statement. In a Docker Compose environment, services are typically accessible to each other using the service name as the hostname. Therefore, instead of using 127.0.0.1:9092 for the kafka.brokers property, you should use kafka:9092, which is the service name and port defined in your Docker Compose file. Here's the corrected part of your CREATE SOURCE statement:

create materialized source block (
        # ... (other fields)
)
with (
        'connector' = 'kafka',
        'kafka.topic' = 'blockchain',
        'kafka.brokers' = 'kafka:9092',
        'kafka.scan.startup.mode'='latest'
)
row format json;

Try updating the statement and running it again to see if it resolves the connection error.

Jun 01, 2022Edited by