RisingWave Community - How does repartitioning work in the system and does it impact data flow between parallel units located on different nodes?

all-things-risingwave

How does repartitioning work in the system and does it impact data flow between parallel units located on different nodes?

I see, so when a new vnode is added to vnode map in meta, you move minimal data both in memory (actor in parallel unit) and in persistent storage to the new vnode. During repartitioning, are you locking the actor in parallel unit and lock that vnode? Will this congest incoming data from actor in upstream parallel unit assuming the downstream and upstream parallel units are located on different node? Is the incoming data buffer in upstream parallel unit?

Go

Gopher

Asked on Dec 30, 2022

Repartitioning in the system involves blocking the whole DAG temporarily.
Previously, there was a design to only block the corresponding vnode, but it required too many alignments in the system.
The current approach involves blocking the entire DAG, but the repartitioning process is generally fast (under 10s).
Most steps during repartitioning are moved to background tasks, with the critical path involving only several RPCs and memory operations.

Jan 03, 2023Edited by