RisingWave Community - Why am I getting a 'Scheduler error' when querying a materialized view in RisingWave?

I'm currently using RisingWave version 1.7.0 and encountering an issue when trying to query a materialized view. Creating the materialized view works without any problems, but when I attempt to select from it, I receive a series of errors related to scheduler, gRPC, transport, and connection issues. This problem persists even when querying regular tables. Here's the error message I'm getting:

[2024-03-26 15:23:28] [XX000] ERROR: Failed to execute the statement
[2024-03-26 15:23:28] Caused by these errors (recent errors listed first):
[2024-03-26 15:23:28] 1: Scheduler error
[2024-03-26 15:23:28] 2: gRPC request failed: Unknown error
[2024-03-26 15:23:28] 3: transport error
[2024-03-26 15:23:28] 4: connection error
[2024-03-26 15:23:28] 5: connection reset

The issue occurs when running a simple select query on the materialized view or table:

select * from test_source_mv;

There are no restarts or suspicious log entries in the cluster, and other sources and sinks seem to be working fine. I'm also using the RW Operator for Kubernetes with a standard configuration for compute nodes. Any ideas on what could be causing this issue and how to resolve it?

Based on the error message you're receiving, it seems like there might be an issue with the communication between the different components of your RisingWave cluster, specifically related to the scheduler and gRPC connections. Since you've confirmed that there are no restarts or suspicious log entries, and the issue is not isolated to materialized views but also affects regular tables, it's possible that there could be a network-related problem or a misconfiguration in your cluster setup.

Here are a few steps you can take to troubleshoot the issue:

Check Cluster Health: Ensure that all nodes in the cluster are running and can communicate with each other. Use kubectl get pods to check the status of the pods.
Review Configuration: Double-check the configuration of your RisingWave cluster, especially any network-related settings that could affect node communication.
Examine Logs: Look for any errors or warnings in the logs of the front-end, meta, and compute nodes that might indicate a problem with the scheduler or network connectivity.
Run Diagnostic Queries: If possible, run diagnostic queries or use monitoring tools to check the health of the gRPC connections and scheduler.
Contact Support: If the issue persists, consider reaching out to the RisingWave community or support team with a detailed bug report, including logs from the front-end, meta, and compute nodes.

Remember to replace any sensitive information in the logs before sharing them with others.