From fungi at yuggoth.org Thu Sep 2 23:21:24 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 Sep 2021 23:21:24 +0000 Subject: Mailing lists offline 2021-09-12 for server upgrade Message-ID: <20210902232124.4okak74j2rvhqffo@yuggoth.org> On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Tue Sep 7 15:29:36 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 07 Sep 2021 08:29:36 -0700 Subject: Team meeting agenda for September 7, 2021 Message-ID: We will meet with this agenda September 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210907) *** lists.openstack.org upgrade scheduled for September 12 beginning 15:00UTC ** Improving OpenDev's CD throughput (clarkb 20210907) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210907) *** 33 conflicts remain. Need to investigate and email their users about planned next steps. ** OpenDev logo hosting (clarkb 20210907) *** https://review.opendev.org/c/opendev/system-config/+/805932/ ready to switch to using the assets container for gitea *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. * Open discussion From fungi at yuggoth.org Mon Sep 13 00:45:14 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 13 Sep 2021 00:45:14 +0000 Subject: Mailing lists are back in service again In-Reply-To: <20210902232124.4okak74j2rvhqffo@yuggoth.org> References: <20210902232124.4okak74j2rvhqffo@yuggoth.org> Message-ID: <20210913004514.s5mlzqeo47spvj2f@yuggoth.org> Over the course of the past day, the OpenDev sysadmins performed a series of operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Some unexpected issues arose which we did not encounter in our earlier test upgrades, which caused the outage to run four hours longer than anticipated. Unfortunately, the server began accepting messages while we were still wrestling with a bug which may have caused some queued inbound messages to be silently dropped. If you sent a message to one of the mailing lists on these sites on September 12 between 15:00 and 00:00 UTC and don't see it appear in the list archive, it was likely lost and you should send another copy. All mailing list sites should be back in service as of now, but if you have any questions or notice a problem we've overlooked, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon Sep 13 21:13:03 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 Sep 2021 14:13:03 -0700 Subject: Team Meeting Agenda for September 14, 2021 Message-ID: <494c3d48-973d-41a9-9383-6749a4e49d2c@www.fastmail.com> We will meet September 14, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210914) *** lists.openstack.org is running focal now **** Number of manual edits made to services captured in https://review.opendev.org/c/opendev/system-config/+/808570 **** Ran into issues with Xen being able to boot lz4 compressed kernels **** We decompressed the kernel but that still failed **** We installed grub-xen then set up a chainload situation to a xen specific grub preloader via /boot/grub/menu.lst **** This chainload situation may not need a compressed kernel? If it doesn't this server may be viable long term and we can document things. **** We should consider replacing the server to get away from Xen PV weirdness and to maybe use a floating IP which can be moved between upgraded instances in the future. ** Improving OpenDev's CD throughput (clarkb 20210914) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210914) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210914) *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. ** Working with InMotion Friday to increase IP addr allocation and scale up (clarkb 20210914) *** Will try to debug instance launch failures there prior to Friday. ** Scheduling Gerrit project renames (clarkb 20210914) *** We've got a few of these showing up now. We should look at a calendar and pencil in some plans. * Open discussion From fungi at yuggoth.org Tue Sep 14 16:53:54 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 Sep 2021 16:53:54 +0000 Subject: Mailing lists are back in service again In-Reply-To: <20210913004514.s5mlzqeo47spvj2f@yuggoth.org> References: <20210902232124.4okak74j2rvhqffo@yuggoth.org> <20210913004514.s5mlzqeo47spvj2f@yuggoth.org> Message-ID: <20210914165353.oc52gtx55ujtd5kv@yuggoth.org> On 2021-09-13 00:45:14 +0000 (+0000), Jeremy Stanley wrote: [...] > All mailing list sites should be back in service as of now, but if > you have any questions or notice a problem we've overlooked, feel > free to reply to this message, or find us in the #opendev channel on > the OFTC IRC network. Not sending this to all lists, but we've completed our post-upgrade cleanup tasks, and redeployed configuration with Ansible so things are back in sync between the server and configuration management now. Receipt of this message by the list indicates things are still working after a restart of services on the freshly synced configs. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Tue Sep 21 01:24:40 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 20 Sep 2021 18:24:40 -0700 Subject: Infra Team Meeting Agenda for September 21, 2021 Message-ID: <170cf55d-b2c3-49c4-80be-df1a41d67b2f@www.fastmail.com> We will meet on September 21, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210921) *** lists.openstack.org is running focal now **** Kernel packages have been pinned. If you update the kernel you need to decompress it. **** Fungi is working on a mailman3 spec. We can decide the best route for replacing the server with that context. ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Stack at https://review.opendev.org/c/opendev/system-config/+/809510 migrates services to the new asset locations. ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 ** Gerrit replication "leaks" (clarkb 20210921) *** We've added a 15 minute network timeout to the gerrit config. Need to restart the server to pick that up. ** InMotion cloud has been scaled up (clarkb 20210921) *** We need to work on tuning the over subscription ratios. * Open discussion From cboylan at sapwetik.org Mon Sep 27 23:17:30 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 27 Sep 2021 16:17:30 -0700 Subject: Team Meeting Agenda for September 28, 2021 Message-ID: We will meet on September 28, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Fix for Gerrit logo hosting https://review.opendev.org/c/opendev/system-config/+/811233 ** Gitea 1.15.3 Upgrade (clarkb 20210928) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion