Can RisingWave control Kafka partitioning logic to match a specific schema?
I need to sink changes from RisingWave to a Kafka topic that has a specific partitioning schema. Is there any way to control RisingWave's write to Kafka partitioning logic to match this schema? For example, the ultimate sink is Pinot which can benefit from data partitioning by a column or a set of columns to optimize query routing as described here: Pinot Routing Optimization.
Nizar Hejazi
Asked on May 11, 2023
Currently, RisingWave does not support custom partitioning logic for Kafka sinks. The partition key in Kafka is typically the primary key, but in some cases, like with Pinot, there might be a need to partition by a non-primary key. While RisingWave plans to evaluate and potentially implement this feature, a workaround might involve using Kafka Connect Single Message Transform (SMT) to repartition the data. However, this requires managing additional configurations and might introduce complexity in debugging and automation.