troubleshooting
What could be causing the cluster meta nodes to crash while creating SINKs in a RisingWave cluster?
Justin Joseph is facing issues with cluster meta nodes crashing while creating SINKs in a RisingWave cluster. The error message he encounters is risingwave_meta::rpc::election::etcd: client risingwave-meta-0.risingwave-meta:5690 lost leadership. Yuhao Su and Martin provide some insights and suggestions to address the issue.
Ju
Justin Joseph
Asked on Mar 18, 2024
- The cluster meta nodes crashing while creating SINKs in a RisingWave cluster could be due to ETCD's high-demanding requirement for the quality of the disk it is running on.
- ETCD may crash or lose connection with RisingWave due to transient disk latency spikes, leading to cluster instability.
- One possible solution is to increase the memory quota for ETCD and adjust CPU resources if high CPU usage is observed.
- Consider implementing an SQL metadata backend (e.g., PostgreSQL, MySQL) to replace ETCD for better stability under intensive workloads.
- The RisingWave team is working on implementing an SQL metadata backend to address these issues in the upcoming release 1.8.
Mar 19, 2024Edited by