How to resolve the 'waiting for new workers' warning in RisingWave after scaling down nodes?
I'm encountering a WARN bootstrap_recovery{prev_epoch=5817409244561408}:recovery_attempt: risingwave_meta::barrier::recovery: waiting for new workers warning after scaling my RisingWave cluster from 10 nodes to 4, but increasing the CPUs per node from 4 to 10. I attempted to set RW_PARALLELISM but it didn't help because the values.yaml file defines a limit which is used by the chart. How can I resolve this issue?
André Falk
Asked on Jan 23, 2024
The warning you're experiencing is likely due to some streaming jobs that previously required a higher parallelism than what your cluster can now provide after scaling down. To resolve this, you can try dropping the relation that's causing the high parallelism requirement. If you're unable to drop the materialized view normally and the drop command hangs, you may need to restart the meta node to force the catalog to drop. After restarting the meta node, the warning should no longer appear.