I've created a RisingWave table from a non-persistent Pulsar topic that uses Snappy compression for the data, which is encoded in Avro schema. However, I'm not receiving any messages when I try to consume data from the table or any materialized view created from it. The error logs suggest issues with the Pulsar connection and complaints about Snappy compression. Do I need to use a persistent topic instead, and does RisingWave support Snappy compression? Here's the table creation statement and the error logs:
CREATE TABLE IF NOT EXISTS fibersensordata
WITH (
connector='pulsar',
topic='<non-persistent://unifiyadkinville/ifs/fibersensordata>',
service.url='<pulsar://10.16.2.9:6650/',scan.startup.mode='latest>',
) FORMAT PLAIN ENCODE AVRO (
message = 'message',
schema.location = '<https://ifs-pulsar-schema.s3.ca-central-1.amazonaws.com/fibersensordata.avsc>'
);
Error Log from the compute pod:
2024-03-27T17:29:43.065043546Z WARN actor{otel.name="Actor 12" actor_id=12 prev_epoch=6181708139986944 curr_epoch=6181708205522944}:executor{otel.name="Source C00002715 (actor 12)"}:build:new: pulsar::retry_op: Retry #0 -> connecting consumer 926 using connection 882cbdd5-1170-4e6c-ac74-6fb3aa34fb60 to broker <pulsar://10.16.2.9:6650/> to topic <non-persistent://unifiyadkinville/ifs/fibersensordata>
...
2024-03-27T17:29:45.446268131Z WARN actor{otel.name="Actor 12" actor_id=12 prev_epoch=6181708270993408 curr_epoch=6181708336594944}:executor{otel.name="Source C00002715 (actor 12)"}: risingwave_stream::executor::source::source_executor: stream source reader error error=Connector error: consumer error: Decompression error: got a Snappy compressed message but 'snap' cargo feature is deactivated
Atiqul Islam
Asked on Mar 27, 2024