[Openinfralabs] Prometheus long-term storage

Lars Kellogg-Stedman lars at redhat.com
Fri Apr 3 14:50:30 UTC 2020


On Fri, Apr 03, 2020 at 12:49:44PM +0200, Marcel Hild wrote:
> Also Red Hat uses thanos for ingesting all telemetry data from connected
> OpenShift 4 deployments, so it works at scale.
> 
> I'm happy to contribute working deployment artifacts

That sounds great, and I would be interested in a high level overview
of how you have things set up.

And as long as I have your attention:

We have a simple sandbox set up, and I've noticed recently that the
compactor service is falling over with errors along the lines of:

> error executing compaction: compaction failed: compaction failed
> for group 0 at 2818969819553058366: pre compaction overlap check:
> overlaps found while gathering blocks.

Have you seen that before, and do you know how to deal with it
effectively? We're only collecting data from a single prometheus
instance right now, so it's not like we have an HA pair sending
duplicate data or something.

-- 
Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {irc,twitter,github}
http://blog.oddbit.com/                | N1LKS




More information about the Openinfralabs mailing list