troubleshooting

Can RisingWave Connect to a Non-Persistent Pulsar Topic with Snappy Compression?

I've created a RisingWave table from a non-persistent Pulsar topic that uses Snappy compression for the data, which is encoded in Avro schema. However, I'm not receiving any messages when I try to consume data from the table or any materialized view created from it. The error logs suggest issues with the Pulsar connection and complaints about Snappy compression. Do I need to use a persistent topic instead, and does RisingWave support Snappy compression? Here's the table creation statement and the error logs:

CREATE TABLE IF NOT EXISTS fibersensordata
WITH (
   connector='pulsar',
   topic='<non-persistent://unifiyadkinville/ifs/fibersensordata>',
   service.url='<pulsar://10.16.2.9:6650/',scan.startup.mode='latest>',
) FORMAT PLAIN ENCODE AVRO (
   message = 'message',
   schema.location = '<https://ifs-pulsar-schema.s3.ca-central-1.amazonaws.com/fibersensordata.avsc>'
);

Error Log from the compute pod:

2024-03-27T17:29:43.065043546Z  WARN actor{otel.name="Actor 12" actor_id=12 prev_epoch=6181708139986944 curr_epoch=6181708205522944}:executor{otel.name="Source C00002715 (actor 12)"}:build:new: pulsar::retry_op: Retry #0 -> connecting consumer 926 using connection 882cbdd5-1170-4e6c-ac74-6fb3aa34fb60 to broker <pulsar://10.16.2.9:6650/> to topic <non-persistent://unifiyadkinville/ifs/fibersensordata>
...
2024-03-27T17:29:45.446268131Z  WARN actor{otel.name="Actor 12" actor_id=12 prev_epoch=6181708270993408 curr_epoch=6181708336594944}:executor{otel.name="Source C00002715 (actor 12)"}: risingwave_stream::executor::source::source_executor: stream source reader error error=Connector error: consumer error: Decompression error: got a Snappy compressed message but 'snap' cargo feature is deactivated
At

Atiqul Islam

Asked on Mar 27, 2024

  1. We do not encourage any use of non-persistent pulsar because all in-transit messages are lost if rw restart by accident, and will break the exactly once semantic.
  2. We currently do not support snappy compression. Only lz4 and zstd are supported.
Mar 29, 2024Edited by