How can I load log files into RisingWave and what are the common sources for log data ingestion?
I am interested in loading log files into RisingWave and exploring the common sources for log data ingestion. I have seen some examples related to Cloudwatch and S3, but I would like to know more about the process and potential sources for log data ingestion into RisingWave.
Mark Needham
Asked on May 02, 2023
-
For loading log files into RisingWave, common sources for log data ingestion include:
- Storing log data in S3 and directly loading data from S3 to RisingWave. RisingWave will monitor the S3 bucket for any new updates and ingest them.
- Streaming logs from Cloudwatch to Lambda Function to Kinesis to RisingWave.
- Periodically exporting logs from Cloudwatch to Lambda Function to S3 to RisingWave.
-
RisingWave can replace Redis as it is a database system that can serve ad-hoc queries. It is a streaming database that can store data and serve queries, unlike streaming processors like Flink and Faust.
-
JSON is a popular format for log entries in log use cases, and RisingWave's materialized view is persistent, ensuring data integrity even in case of node failures.