RisingWave Community - Why am I getting a 'BrokerTransportFailure' error when connecting RisingWave to Kafka using Docker?

I'm trying to connect RisingWave to Kafka using Docker, but I'm encountering a 'BrokerTransportFailure' error. I've tried using both ports 9092 and 29092, but neither seems to work. Here's the error message I'm getting:

SQL Error [XX000]: ERROR: ExecuteError: internal error: connector error: failed to fetch metadata from kafka (localhost:29092), error: Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure)

I've confirmed that Kafka is working from Python outside the container:

from kafka import KafkaAdminClient

admin = KafkaAdminClient(bootstrap_servers='localhost:29092')
print(admin.list_topics())

> ['test']

I'm using the RisingWave docker-compose file from their GitHub repository. Any ideas on what might be causing this issue and how to resolve it?

It seems like there's a network configuration issue between RisingWave and Kafka within the Docker environment. When running services in Docker, especially when they are in different containers, you need to ensure that they can communicate with each other using the correct network settings. Here's what you can try:

Make sure that the Kafka service is accessible from within the RisingWave container. You can use the service name defined in the Docker Compose file (e.g., message_queue) along with the internal port (e.g., 29092) when configuring RisingWave to connect to Kafka.
Check the Kafka container's health status to ensure it's running properly.
Review the RisingWave and Kafka logs for any connection errors or issues.
If you're running commands from the host machine, use localhost with the external port mapped by Docker (e.g., 9092). If the command is run within the Docker network, use the service name and internal port.
Refer to the Kafka listeners configuration and ensure that the Kafka broker is configured to accept connections from both inside and outside the Docker network as needed.
Verify that the RisingWave Docker Compose configuration includes the correct Kafka bootstrap server address.

If you continue to face issues, you might want to check out resources like Confluent's blog posts on Kafka client connection issues for more detailed troubleshooting steps.