troubleshooting

Can I use watermarks with Kafka source and Protobuf format in RisingWave?

I'm trying to use watermarks with a Kafka source and Protobuf format in RisingWave, but I'm encountering an error. It seems that watermarks are defined in the user-defined schema, but I can't define it when using Protobuf. Here's the error message I'm getting:

ERROR: ExecuteError: Protocol error: User-defined schema is not allowed with FORMAT PLAIN ENCODE PROTOBUF

Is there a way to use watermarks with Protobuf, or is this feature not supported yet?

Pi

Pierre Nowak

Asked on Aug 21, 2023

Currently, there seems to be a limitation in RisingWave that prevents the use of watermarks with the Protobuf format. However, a workaround was suggested by using a direct subtraction on the timestamp without casting it to a different type:

WATERMARK FOR timestamp AS timestamp - 5000000000

This approach should work as long as the type of the timestamp field in the Protobuf definition is compatible with the expected type for the watermark expression. The RisingWave team has acknowledged the issue and mentioned that a fix will be included in the next release. In the meantime, you might need to use JSON format if you require watermarks, or wait for the upcoming update that addresses this bug.

Aug 31, 2023Edited by