Snowshoe logo
all-things-risingwave

Can I use the Kafka source connector to analyze data in a similar fashion as streaming into PostGres with generic tables?

Georg Boegerl is currently analyzing data in Kafka by streaming it into PostGres with generic tables. He creates a table for every topic with columns: kafka_key VARCHAR, kafka_ts TIMESTAMP, kafka_payload JSONB, kafka_headers JSONB. He is wondering if he can achieve a similar setup using the Kafka source connector.

Ge

Georg Boegerl

Asked on Nov 20, 2023

  • The Kafka source connector does not natively support creating generic tables with specific columns like in PostGres.
  • However, you can create a source like normal and use a view to convert the payload to JSONB and kafka_key.
  • The support for kafka_ts and kafka_headers is not available yet, but it is actively being worked on.
  • If you need to read kafka key and value, you can refer to the upcoming release v1.6.0 of RisingWave, which will include this feature.
  • The official documentation for this feature will be available in v1.7.0.
  • You can refer to an example in the RisingWave repository for including the key in the source connector: Example Link
Jan 04, 2024Edited by