I'm working with RisingWave and I need to track the progress of changes from the source (like PostgreSQL or Kafka) through multiple chained views and sinks. I want to ensure that all views have processed the changes and that my sinks are in sync with my input change. Is there a way to track this progress, perhaps with an 'id of progress' or similar mechanism?
Charlie
Asked on Jul 25, 2023
The discussion suggests using a FLUSH
command to ensure all records inserted before the command are viewed/sinked. However, this may not be suitable for high-frequency updates. Another approach is to use a 'neutral' record with a timestamp to simulate progress tracking. Additionally, exposing Kafka consumer offsets or using a custom timestamp forwarded to all sinks could be considered. The idea of progress messages in outputs and a configurable interval for these messages was also discussed. Ultimately, the suggestion is to use a timestamp column in the upstream table to report progress, which RisingWave can use to refresh metadata and indicate the processing status of the data.