I'm trying to use watermarks with a Kafka source and Protobuf format in RisingWave, but I'm encountering an error. It seems that watermarks are defined in the user-defined schema, but I can't define it when using Protobuf. Here's the error message I'm getting:
ERROR: ExecuteError: Protocol error: User-defined schema is not allowed with FORMAT PLAIN ENCODE PROTOBUF
Is there a way to use watermarks with Protobuf, or is this feature not supported yet?
Pierre Nowak
Asked on Aug 21, 2023
Currently, there seems to be a limitation in RisingWave that prevents the use of watermarks with the Protobuf format. However, a workaround was suggested by using a direct subtraction on the timestamp without casting it to a different type:
WATERMARK FOR timestamp AS timestamp - 5000000000
This approach should work as long as the type of the timestamp
field in the Protobuf definition is compatible with the expected type for the watermark expression. The RisingWave team has acknowledged the issue and mentioned that a fix will be included in the next release. In the meantime, you might need to use JSON format if you require watermarks, or wait for the upcoming update that addresses this bug.