I'm using RisingWave and want to export metrics to DataDog, but I'm concerned about the high cardinality of the metrics, which can increase costs. The metrics have labels like actor_id
that seem dynamic and could lead to a large number of unique metric series. I'm looking for a way to export aggregated metrics to avoid high cardinality. Here's an example of the metrics I'm dealing with:
stream_executor_row_count{actor_id="2277",executor_identity="MaterializeExecutor 8E500000003 (actor 2277, operator 3)"} 2803
...
stream_materialize_cache_hit_count{actor_id="2280",table_id="3015"} 230
Is there a way to export coarse high-level aggregated metrics to DataDog instead of these high-cardinality metrics?
Nizar Hejazi
Asked on Jul 23, 2023
RisingWave currently reports raw metrics data points to Prometheus, and the aggregation is done at query time. While there isn't a built-in feature to export aggregated metrics with lower cardinality directly to DataDog, you can consider deploying an on-prem Prometheus and Grafana stack to handle the metrics. Since DataDog has built-in support for Prometheus, you may be able to use this feature to export the aggregated metrics from Prometheus to DataDog, thus avoiding high cardinality in your monitoring system.