Why isn't the Risingwave compactor reducing the size of my Ceph bucket?
I'm using Risingwave and have connected it to a Ceph bucket for storage. However, I've noticed that the bucket size is steadily growing, which suggests that the compactor might not be working as expected. I've created a materialized view to query the last hour of sensor readings, but it doesn't seem to help with the compaction. Here's the SQL for the materialized view:
CREATE materialized VIEW amazing_view as (
select
id,
seconds,
value
from
amazing_source
where to_timestamp(seconds) > now() - interval '1 hour'
);The compactor logs show some information, but I'm not sure how to interpret it or what action to take. Can anyone provide some insights or suggestions on how to address this issue?
Maximilian Bernkopf
Asked on Apr 17, 2024
Based on the discussion, it seems that the compactor is indeed running but might be facing resource constraints, which could be causing the Ceph bucket to grow without proper compaction. The logs indicate that the compactor is working but also show errors related to streaming read bytes and warnings about not enough core parallelism to serve tasks. Upgrading Risingwave to version 1.8.1 and possibly increasing resources allocated to the compactor (more CPU cores or memory) could help resolve the issue. It's also recommended to monitor the relevant Grafana metrics to better understand the compactor's performance and behavior.