Can I Stream Only Changed Fields from Materialized Views to Kafka?
I need to trigger actions in an internal system based on changes to specific fields in a materialized view (MV). Is it possible to send only the names of the updated MV and the list of changed fields to a Kafka topic? Ideally, I would also like to know the list of changed rows. Can this be done using the debezium format, and does it include before
and after
sections for only the changed fields, or for all fields?
Nizar Hejazi
Asked on May 11, 2023
It seems that the debezium format includes before
and after
sections for all fields, not just the changed ones. This could make it challenging to trigger actions based on changed fields alone without comparing the before
and after
data. However, there are a couple of workarounds:
- Write customized Single Message Transform (SMT) code and deploy it to Kafka Connect. This approach might be trivial.
- Use RisingWave to read and consume the debezium topic again and create materialized views that filter only the changed fields.
Both options might raise scalability concerns, especially if you want to avoid maintaining custom transforms or materialized views for a large number of MVs. The debezium format is unlikely to change unless Debezium itself updates its protocol. Fortunately, Debezium has a feature on their roadmap that might address this issue in the future (DBZ-5283).
As for the upsert format supported by RisingWave, it emits a message with the primary key and the changed columns for updates, and just the primary key for deletes. However, it does not seem to support streaming only the changed fields without including the primary key.