I'm testing RisingWave and trying to get the consumer group to display in my RP console with the following configuration:
CREATE SOURCE IF NOT EXISTS rp_shares_cairo
WITH (
connector='kafka',
topic='shares_data',
properties.group.id='rwo_group',
properties.client.id='rwo_client',
properties.bootstrap.server='x.x.x.x:9092',
properties.security.protocol='SASL_SSL',
properties.sasl.mechanism='SCRAM-SHA-256',
properties.sasl.username='xxx',
properties.sasl.password='xxx',
)
FORMAT PLAIN ENCODE PROTOBUF (
message = 'pool.v1beta1.Data',
schema.location = '<https://b2.workers.dev/share/data.pb>'
);However, the consumer groups show up without the name I expect, making it hard to track lag. I'm looking for guidance on how to properly track consumer lag in this situation. Thanks in advance!
Eddie Wang
Asked on Nov 02, 2023
It seems that RisingWave has a design choice where consumer group IDs are not exposed to the user, and this is not a bug but an intentional feature. To track consumer lag, RisingWave provides lag metrics in their own dashboard, which can be accessed through Grafana. Every query in RisingWave creates a new consumer group, and cleaning up these groups would require using the admin API, which is not within the scope of a streaming database. RisingWave is working on displaying lag size in their Grafana dashboard to help users understand and monitor lag times.