Mimir images built with profile-guided optimization

[LasseHels]

Is your feature request related to a problem?

No!

Describe the solution you'd like

We would like to be able to run Mimir images built with profile-guided optimization (PGO).

Since Mimir runs different components from a single image (grafana/mimir), it might make sense to have different images like:

• grafana/mimir:2.14.0 - Built without PGO.
• grafana/mimir:2.14.0-ingester-pgo - Built with ingester profiles.
• grafana/mimir:2.14.0-distributor-pgo - Built with distributor profiles.
• Etc.

To maintain backwards compatibility, I imagine it should still be possible to run an image optimized for component X as component Y. For example, it should be possible to run grafana/mimir:2.14.0-ingester-pgo with a target=distributor flag. In this case, distributors will be running with an image optimized with ingester profiles. In this case, running a component with the incorrect image is sub-optimal, but will at least still work.

Describe alternatives you've considered

We've considered running custom Mimir images optimized with profiles from our own environment. Not only is this cumbersome and error prone, we also imagine that profiles pulled from Grafana Cloud will be the most accurate (and thus provide the largest performance benefit).

Additional context

Various sources online claim performance benefits in the order of 2-14%. This is a significant performance improvement at a relatively low cost.

[dimitarvdimitrov]

with pyroscope we can get a single pprof file from the whole cell. That will contains nodes from different components. Maybe that's another approach which avoids having multiple images

[LasseHels]

Is there an expected timeline for this feature? I'd love to contribute, but I am unsure of how I can help with this as I am obviously unable to pull profiles from Grafana's production environment 😄.

[narqo]

Various sources online claim performance benefits in the order of 2-14%. This is a significant performance improvement at a relatively low cost.

Last time I asked internally, when people looked at building the binary with pgo, focusing only on ingesters, the experience was that the resulting performance improvements weren't obviously visible, comparing to added complexity. This doesn't mean there is no interest to make this happen, though.

Is there an expected timeline for this feature? I'd love to contribute, but I am unsure of how I can help

Before working on shipping per-target pgo-images it'd be helpful to know about your own experience. That is, if one has an sufficiently large installation of mimir, consider building your own image for one pgo-optimized target. Run this image as an experiment internally, and contribute back your performance results. This will be a huge gain to move this forward.

[LasseHels]

@narqo Roger that. We will look into gauging the performance impact with custom images and report back. Could take a while.