RisingWave Community - How does a Kafka source in RisingWave deal with updated proto files?

troubleshooting

How does a Kafka source in RisingWave deal with updated proto files?

Suppose I have an updated set of proto files - how does a kafka source in RisingWave deal with that? If I update the descriptors on S3 that is referenced in the RW source, will the source automatically use the new proto definitions? Or do I have to stop the source and remake it? And if I do that, will it continue at the same partition and offset that it left off, or does it start at a new place because it thinks it's a brand new source?

Ne

Neil

Asked on Dec 15, 2023

Currently, RisingWave does not support schema evolution for Source/Sink.
A Kafka source in RisingWave would not read the updated descriptor files from S3 again while it is running.
If a recovery happens or the source is recreated, it will use the new schema to parse incoming messages.
A recovered source is expected to continue where it left off, while a recreated source would work independently and may lead to schema mismatch issues.

Dec 15, 2023Edited by