Why are RisingWave pods in Kubernetes stuck in CrashLoopBackOff with no logs?
I deployed a RisingWave instance to Kubernetes using the provided command. All images were pulled successfully, but all RisingWave pods (meta, compute, compactor, frontend) are stuck in CrashLoopBackOff and there are no logs available. I tried both RisingWave v1.1.17 and v1.18.0 images, and the issue persists even when using an external MinIO. Here's the command I used for deployment:
kubectl apply -f https://raw.githubusercontent.com/risingwavelabs/risingwave-operator/main/docs/manifests/risingwave/risingwave-etcd-minio.yamlAnd here's the output of kubectl get pods showing the pods in CrashLoopBackOff state:
[root@node-240 risingwave]# kubectl get pods -n risingwave
NAME READY STATUS RESTARTS AGE
risingwave-etcd-0 1/1 Running 0 155m
risingwave-etcd-minio-compactor-69c6476b65-7v4fg 0/1 CrashLoopBackOff 32 (65s ago) 141m
...I also tried to get logs from the crashing pods using kubectl logs pod <pod name> -p but there was no output. I'm running this on rancher rke2.
Lucci
Asked on Apr 26, 2023
It seems like the issue might be related to the meta pods crashing, which causes other pods to exit as well. I checked the logs using kubectl logs pod risingwave-etcd-minio-meta-0 -p but there was no output. I also tried deploying risingwave-in-memory and risingwave-etcd-minio on another Kubernetes cluster and it worked fine, so the problem might be specific to my current environment. I'm considering building from the main branch to see if that resolves the issue.