How to Resolve Kafka Connection Error in Docker Compose Configuration?
I'm encountering a Kafka connection error while trying to create a source using Docker Compose. The error message is internal error: Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure)
. I've provided my Docker Compose configuration file below. Can someone help me identify what might be causing this issue and how to fix it?
version: "3"
name: rise
services:
# ... (other services)
kafka:
image: confluentinc/cp-kafka:7.0.1
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: <PLAINTEXT://kafka:9092>
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_COMPRESSION_TYPE: lz4
KAFKA_LOG_RETENTION_MS: 31104000000
depends_on:
zookeeper: { condition: service_healthy }
healthcheck:
{
test: nc -z localhost 9092,
interval: 5s,
start_period: 120s
}
# ... (other services)
Additionally, here's the CREATE SOURCE SQL statement I used:
create materialized source block (
# ... (other fields)
)
with (
'connector' = 'kafka',
'kafka.topic' = 'blockchain',
'kafka.brokers' = '127.0.0.1:9092',
'kafka.scan.startup.mode'='latest'
)
row format json;
jie
Asked on Jun 01, 2022
The issue seems to be related to the Kafka broker address specified in the CREATE SOURCE statement. In a Docker Compose environment, services are typically accessible to each other using the service name as the hostname. Therefore, instead of using 127.0.0.1:9092
for the kafka.brokers
property, you should use kafka:9092
, which is the service name and port defined in your Docker Compose file. Here's the corrected part of your CREATE SOURCE statement:
create materialized source block (
# ... (other fields)
)
with (
'connector' = 'kafka',
'kafka.topic' = 'blockchain',
'kafka.brokers' = 'kafka:9092',
'kafka.scan.startup.mode'='latest'
)
row format json;
Try updating the statement and running it again to see if it resolves the connection error.