Why are no records appearing in Kafka from my RisingWave sink, and where can I find error logs?
I've created a sink in RisingWave to write data to a Kafka topic, but I'm not seeing any records in my broker. I've checked the compute node logs for errors around the time my sink returned, but I haven't found anything. I also don't have Grafana/Prometheus set up for monitoring. Can someone help me understand how sinks are created in RisingWave and where I might find logs for potential errors?
Dominic Lindsay
Asked on Oct 03, 2023
I've checked the compute node (CN) logs and didn't find any errors. I also looked into the meta-node logs. I'm not running Grafana/Prometheus, so I don't have sink throughput metrics. I've confirmed that my sink-related internal tables exist and that my query returns results when run manually. However, the CREATE SINK command took about 5 days to return, which is unusual as it should return immediately. I suspect there might be an issue with the topic partition configuration in Kafka, as RisingWave's Kafka client might be writing all records to a single partition due to the lack of a specified key value. I'm considering whether RisingWave supports Avro for Kafka sinks and how to specify key values for Kafka records.