I'm experiencing issues with order preservation in my APPEND ONLY table that has a Kafka source in RisingWave. Despite having enable.idempotence set on the Kafka producer side, the values don't seem to append in the order they are sent. This is surprising since I'm using a single topic, single partition queue. I've tried different versions of RisingWave but the issue persists. Is there a specific configuration I need to set in RisingWave to maintain the order of incoming messages?
John Garland
Asked on Oct 22, 2023
To maintain the order of messages in RisingWave when using a Kafka source, you should set the streaming_parallelism session variable to 1. This ensures that RisingWave doesn't parallelize operations, which can preserve the order of the input. However, it's important to note that the _row_id generated by RisingWave is not strictly monotonic, so you might still see differences in order when querying based on _row_id. If you're still experiencing out-of-order issues after setting streaming_parallelism, it could be related to the backfilling process during the creation of materialized views or sinks. To avoid this, you can build your materialized views on an empty Kafka topic first and then send data to the topic after all views are built. Once the CREATE MATERIALIZED VIEW command returns, it means the views have been created successfully.
