How to Export RisingWave Metrics to DataDog with Lower Cardinality?
I'm using RisingWave and want to export metrics to DataDog, but I'm concerned about the high cardinality of the metrics, which can increase costs. The metrics have labels like actor_id that seem dynamic and could lead to a large number of unique metric series. I'm looking for a way to export aggregated metrics to avoid high cardinality. Here's an example of the metrics I'm dealing with:
stream_executor_row_count{actor_id="2277",executor_identity="MaterializeExecutor 8E500000003 (actor 2277, operator 3)"} 2803
...
stream_materialize_cache_hit_count{actor_id="2280",table_id="3015"} 230Is there a way to export coarse high-level aggregated metrics to DataDog instead of these high-cardinality metrics?
Nizar Hejazi
Asked on Jul 23, 2023
RisingWave currently reports raw metrics data points to Prometheus, and the aggregation is done at query time. While there isn't a built-in feature to export aggregated metrics with lower cardinality directly to DataDog, you can consider deploying an on-prem Prometheus and Grafana stack to handle the metrics. Since DataDog has built-in support for Prometheus, you may be able to use this feature to export the aggregated metrics from Prometheus to DataDog, thus avoiding high cardinality in your monitoring system.