Live Metrics Displayed for Deleted Pods

[abhipatwari17]

Steps to Reproduce
Enable pod-level metrics in Retina.
Observe metrics for active pods.
Delete some pods.
Continue to observe the metrics display.
Expected Behavior
Metrics data and graphs should only be displayed for active pods. Deleted pods should no longer appear in the live metrics display.

Actual Behavior
Metrics data and graphs are still being displayed for pods that have been deleted.

Environment
Retina version: v0.0.12
Kubernetes version: 1.28
Platform: Grafana ui

Here you can see the metrics for "monitoring-kube-prometheus-operator-9b8fb7d86-fwv6z",But this pod doesn't exist in my cluster.

Additional Information
Any guidance on resolving this issue would be greatly appreciated. If you need further information, please let me know.

Thank you for your assistance.

[rbtr]

I believe this is intended behavior of Prometheus/Grafana - the historical data still exists for those Pods

[vakalapa]

@abhipatwari17 feel free to contribute, there is a way to fix this. Whenever we get a delete notification, we can loop over all the existing timeseries and check if this is used in a dimension and delete that specific time series. This will help in reducing cardinality greatly !