all-things-risingwave

What is the biggest change in the source code that allows one source to consume multiple splits concurrently?

In the discussion thread, Bohan Zhang mentioned a significant change in the source code that enables one source to consume multiple splits concurrently. He highlighted that the source can now spawn connectors into new threads. What is the biggest change in the source code that allows this concurrency?

JJ

JJ

Asked on May 19, 2022

The biggest change in the source code that allows one source to consume multiple splits concurrently is the ability to spawn connectors into new threads. This enables parallel processing of splits, improving efficiency and performance. Here is a simplified example of how connectors can be spawned into new threads in source code:

// Spawning a connector into a new thread
Thread connectorThread = new Thread(() -> {
    // Connector logic here
});
connectorThread.start();
May 19, 2022Edited by