How to Resolve RisingWave Consuming Excessive S3 Space?
I'm using RisingWave v:1.5.4 and planning to upgrade to v:1.6 soon. I've noticed that RisingWave is consuming over 2 TB of S3 space, even though the original sourced table and the materialized views are less than 100 GB. The Grafana monitoring indicates that this space is 'not referenced by versions'. I'm looking for an explanation of what this means and how to resolve the issue. Additionally, I'd like to know if there's a way to clean up the non-versioned storage.
Ibrahim
Asked on Jan 30, 2024
The excessive S3 space usage by RisingWave could be due to unvacuumed data. To address this, you can try restarting both the meta node and compactor node as a quick workaround. Before restarting, it's important to share the logs from the compactor node and check the 'Version Checkpoint Latency' in the Grafana panel under 'Hummock Manager'. If you encounter errors related to vacuuming tasks or cluster ID conflicts after a restart, ensure that you're not losing metadata and that your ETCD container is not being reset. To safely restart the cluster without such errors, only restart RisingWave components and avoid restarting the ETCD container. For large-scale operations, consider adjusting the vacuum interval and request rate to S3 to prevent flooding and errors.