Neil is looking for a way to collect messages from a source for a specific duration before sending them to a sink in Kafka. He tried using WATERMARK
but didn't get the desired result. He also mentioned the use case of sinking data to Iceberg and wanting to optimize the table in between batches.
Neil
Asked on Dec 11, 2023
WATERMARK
but it did not produce the expected outcome.checkpoint_frequency
to a larger value, is recommended.