In the discussion thread, Bohan Zhang mentioned a significant change in the source code that enables one source to consume multiple splits concurrently. He highlighted that the source can now spawn connectors into new threads. What is the biggest change in the source code that allows this concurrency?
JJ
Asked on May 19, 2022
The biggest change in the source code that allows one source to consume multiple splits concurrently is the ability to spawn connectors into new threads. This enables parallel processing of splits, improving efficiency and performance. Here is a simplified example of how connectors can be spawned into new threads in source code:
// Spawning a connector into a new thread
Thread connectorThread = new Thread(() -> {
// Connector logic here
});
connectorThread.start();