I'm currently setting up a RisingWave cluster and I've come across Microsoft's documentation that suggests gRPC doesn't work well with L4 load balancers. Since RisingWave cluster nodes communicate using gRPC, I'm concerned about whether I can use L4 load balancers for the cluster components. I'm not using Kubernetes and my setup is on AWS ECS with network load balancers. If I encounter connection issues, should I consider switching to application load balancers?
Rick Otten
Asked on Oct 23, 2023
L4 load balancers balance TCP connections, but gRPC multiplexes calls on a single TCP connection, which L4 load balancers can't distribute effectively. While deploying L4 LBs for cluster nodes won't necessarily break connectivity, it's not ideal. In Kubernetes, L4 LBs aren't needed for internal communications, but since you're using AWS ECS, you might face different challenges. If you're experiencing intermittent network issues, it could be related to the load balancers. AWS recommends using application load balancers with support for HTTP/2 and gRPC. If you switch to HTTPS with an application load balancer, ensure that the gRPC clients can handle HTTPS as well.