I'm using the ByLogicalTableRouter transformer of Debezium to make the tenant column visible to my RisingWave sources. However, the tenant key is only visible in the raw message's orderingKey when using Google PubSub, but not in the message.data field. I've tried various configurations and transformations, but I'm unable to make the tenant key visible in the way that I need for RisingWave. Here's the configuration I'm using:
debzium.transforms=Reroute,copyKeyToHeaders,moveHeadersToValue
debzium.transforms.copyKeyToHeaders.type=org.apache.kafka.connect.transforms.HeaderFrom$Key
debzium.transforms.copyKeyToHeaders.fields=tenant
debzium.transforms.copyKeyToHeaders.headers=tenant
debzium.transforms.copyKeyToHeaders.operation=copyAny idea how I could achieve this?
Florian Klein
Asked on Jan 22, 2024
I'm getting an error potentially related to the absence of the tenant key in the schema. I'm considering writing my own Debezium alternative in Rust to bypass message queues and directly write INSERT .. VALUES () statements to RisingWave. I'm curious about the scalability of this approach and if RisingWave could make the PubSub orderingKey visible as normal columns, which would solve my problem.