[Openinfralabs] Prometheus long-term storage

[Marcel Hild]

You should have 2 additional components running all the time. A `compact`
component, that, well, compacts. And then a `verify`, which also can repair
blocks.
https://github.com/thanos-io/thanos/blob/master/docs/components/bucket.md

See if that helps


On Fri, Apr 3, 2020 at 4:50 PM Lars Kellogg-Stedman <lars at redhat.com> wrote:

> On Fri, Apr 03, 2020 at 12:49:44PM +0200, Marcel Hild wrote:
> > Also Red Hat uses thanos for ingesting all telemetry data from connected
> > OpenShift 4 deployments, so it works at scale.
> >
> > I'm happy to contribute working deployment artifacts
>
> That sounds great, and I would be interested in a high level overview
> of how you have things set up.
>
> And as long as I have your attention:
>
> We have a simple sandbox set up, and I've noticed recently that the
> compactor service is falling over with errors along the lines of:
>
> > error executing compaction: compaction failed: compaction failed
> > for group 0 at 2818969819553058366: pre compaction overlap check:
> > overlaps found while gathering blocks.
>
> Have you seen that before, and do you know how to deal with it
> effectively? We're only collecting data from a single prometheus
> instance right now, so it's not like we have an HA pair sending
> duplicate data or something.
>
> --
> Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {irc,twitter,github}
> http://blog.oddbit.com/                | N1LKS
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendev.org/pipermail/openinfralabs/attachments/20200403/01055c6d/attachment.html>