Does RisingWave Work Correctly with Compacted Kafka Topics Using Debezium?
I'm using Kafka Connect's Debezium to write to compacted Kafka topics and considering using RisingWave for stream processing. I want to know if RisingWave can handle scenarios where the 'before' record in a Kafka message has been compacted out and removed. Specifically, if I have a Kafka topic with compaction enabled and some messages are compacted, will RisingWave still work correctly when I point a new source at this topic?
James Johnston (he/him/his)
Asked on Dec 19, 2023
RisingWave can handle compacted Kafka topics correctly. It omits the 'before' field when processing UPDATE and INSERT operations, focusing only on the 'after' value, regardless of what the old value is. There was a bug related to this issue, but it has been fixed. So, even if some rows get compacted out, RisingWave will still function properly with the remaining messages.