troubleshooting
Can I use emit-on-close logic on streams with a Debezium source and a watermark?
I am trying to use emit-on-close logic to capture 'windows of updates' to a table with a Debezium source. I want to add a watermark to the source schema to achieve this. However, it seems that custom schemas cannot be specified for a Debezium Avro source, and watermarks can only be added to sources. Does this mean I cannot use emit-on-close logic on streams with a Debezium source and a watermark?
Do
Dominic Lindsay
Asked on Sep 08, 2023
- Debezium sources do not support custom schemas or adding watermarks directly.
- Watermarks can only be added to sources, not to Debezium Avro sources.
- To achieve emit-on-close logic with Debezium, consider creating an append-only table in RisingWave with a Debezium format upstream where a watermark can be defined.
- A workaround is to create a source using plain JSON format to decode the Debezium JSON stream, which can be viewed as append-only if the upstream contains only inserts.
Nov 23, 2023Edited by