Moving 900+ GB of Prometheus Without a Cutover Weekend

The Prometheus I was responsible for had grown past 900 GB of time series data, and at that size it had every disease that comes with it.

Queries timed out unless you narrowed them down to an hour. Cardinality kept growing, because teams added labels "for debugging" and then sometimes forgot to remove them. On top of that, we were shipping a copy of fractions of our metric data to an external SaaS vendor, just in case, and that bill grew every quarter. And the fleet under all of this ran latency-sensitive trading services, so monitoring itself was not allowed to eat an extra CPU core at a wrong moment.