How to resolve meta-node crashes and warnings in RisingWave using Docker Compose?
I'm running RisingWave with docker-compose, and my meta-node keeps crashing after I started CDC sync. I'm also receiving warnings like failed to get cpu quota in container, use system value instead, err: "not a number". How can I solve these warnings and prevent the meta-node from crashing? Here's a snippet of the warnings I'm seeing:
2023-07-18T13:46:07.489174968Z WARN risingwave_common::util::resource_util::cpu: failed to get cpu quota in container, use system value instead, err: "not a number"
at src/common/src/util/resource_util.rs:217
...And here's part of the crash log:
2023-07-18T13:46:42.979645678Z ERROR risingwave_meta::rpc::election_client: lease 6036944774505287782 keep alive timeout
at src/meta/src/rpc/election_client.rs:158
...Povilas Žvaliauskas
Asked on Jul 18, 2023
For the CPU quota warnings, you can check the cgroup settings within the container using commands like cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us. If it shows -1, it means there's no specific CPU quota set, and you can ignore the warning. The team will improve the logging message to avoid confusion.
Regarding the meta-node crashes, it seems to be an issue with etcd. You can check etcd's health and logs to diagnose the problem. Restarting the Docker Compose services with docker compose restart might help temporarily. If the problem persists, ensure that etcd is mounted on a high-performance disk as per its official requirements. If you're using Docker volumes, they might cause worse etcd performance, so consider writing directly to the disk instead to prevent crashes.