How can I determine when a RisingWave table is caught up with a Postgres source?
I'm using RisingWave with a postgres-cdc source and I need to know when the RisingWave table is caught up with the source so I can start querying it. The documentation I found only tells me when the DDL is complete, not when the data is caught up. Is there a way to tell when the initial load is done?
Rick Otten
Asked on Sep 19, 2023
To check the consumption progress of a RisingWave table with a postgres-cdc source, you need to find the internal table of the source operator and check the start_offset field to compare with the most recent write offset in the source partitions. This process involves running SHOW INTERNAL TABLES LIKE '__internal_{table_name}%'; and select * from {internal_table_name};. Additionally, you can extract the lsn value from the json in RisingWave and compare it with the lsn on the replication slot in PostgreSQL. A script that checks this every few seconds can be written, and a business rule can be established to determine when the table is "close enough" to start querying.