Next steps with new review server

[Clark Boylan]

On Thu, Apr 1, 2021, at 8:20 AM, Clark Boylan wrote:
> On Wed, Mar 31, 2021, at 7:27 PM, Ian Wienand wrote:

snip

> > 
> > 2) IPv6 issues
> > 
> > We've seen a couple of cases that are looking increasingly like stray
> > RA's are some how assigning extra addresses, similar to [1].  Our
> > mirror in the same region has managed to acquire 50+ default routes
> > somehow.
> > 
> > It seems like inbound traffic keeps working (why we haven't seen
> > issues with other production servers?).  But I feel like it's a little
> > bit troubling to have undiagnosed before we switch our major service
> > to it.  I'm running some tracing, trying to at least catch a stray RA
> > while the server is quite, in the etherpad.  But suggestions here are
> > welcome.
> 
> Agreed, ideally we would sort this out before any migration completes. 
> I want to say we saw similar with the mirror in vexxhost and the 
> "solution" there was to disable RAs and create a static yaml config for 
> ubuntu using its new network management config file? That seems less 
> than ideal from a cloud perspective as we can't be the only ones 
> noticing this (in fact some of our CI jobs may indicate they suffer 
> from similar causing some jobs to run long when reaching network 
> resources). I know when we brought this up with the mirror mnaser 
> suggested static config was fine, but maybe we need to reinforce that 
> this is problematic as a cloud user and see if we can help debug 
> (network traces seem like a good start there).

I ended up double checking the mirror node and in mirror.ca-ymq-1.vexxhost.opendev.org:/etc/netplan/50-cloud-init.yaml you can see what we did there. Essentially we set dhcpv6 and accept-ra to false then set an address and routes. We should be able to do the same thing with the new review host if we can't figure anything else out.

If we do go this route maybe we should consider updating launch-node to do it for us automatically when launching focal nodes on vexxhost (I don't think bionic does netplan?), or at the very least document this somewhere.

We should also double check that the address and routes are static and can be configured statically like this (the address should not change but I suppose the routes could at some point?). Ideally though we would sort this out properly and avoid these workarounds.

> 
> > 
> > -i
> > 
> > 
> > [1] https://etherpad.opendev.org/p/gerrit-upgrade-2021
> > [2] https://review.opendev.org/c/opendev/system-config/+/775961
> > [3] https://launchpad.net/bugs/1844712
> 
>