I'm experiencing an issue where inserting new data into a table in RisingWave results in very high memory usage. For example, inserting around 300 million records takes up to 15GB of memory. I've provided the table schema and the Docker Compose configuration for the RisingWave cluster. I'm looking for insights into why this is happening and how to troubleshoot it.
jie
Asked on Feb 03, 2023
After several tests, it was found that there might be a memory leak on the compute node (CN), but the issue could not be reproduced on the frontend (FE). The memory usage of the FE remained low when inserting 300 million records using both psql and the psycopg2 Python driver. It was suggested to provide more information about the operating system, architecture, driver version, and possibly the test script to further investigate the issue. Additionally, it was recommended to test with the latest release of RisingWave to see if the issue persists. The high memory usage on the compute node was attributed to cache utilization rather than a leak, but there was an acknowledgment that cache efficiency could be improved.
