troubleshooting

How to sync records from the start of binlog using Debezium MySQL connector?

I am trying to fetch data from the start of the binlog using the Debezium MySQL connector. I have tried setting debezium.snapshot.mode = 'never', but no data is getting synced. Modifying to debezium.snapshot.mode = 'schema_only' works but only gives data from when the connector was started. I want to sync records from the start of the binlog available. How can I achieve this with the Debezium MySQL connector?

Su

Sumit Singh

Asked on Jan 15, 2024

  1. The never mode in debezium.snapshot.mode does not sync data as expected because it reads from the beginning of the binlog only upon first startup with a logical server name. If the binlog does not contain the entire history of the database, no data will be synced.

  2. To sync records from the start of the binlog, you can consider using a combination of settings. One approach is to set debezium.snapshot.mode = 'schema_only' and debezium.snapshot.locking.mode = 'none'. This will stream binlog data from when the connector was started.

  3. However, syncing data from the start of the binlog may not be directly supported by the Debezium MySQL connector. Enabling snapshots may be necessary to ensure all required data is captured, even though it may load historical data.

  4. Consider the trade-off between historical data and the specific data you need for your materialized view. Enabling snapshots may provide a more comprehensive solution for syncing data, even if it includes historical records that you do not need for your daily sales view.

Jan 19, 2024Edited by