- service-discuss - lists.opendev.org

Team Meeting Agenda for October 12, 2021
by Clark Boylan 11 Oct '21

11 Oct '21

We will meet on October 12, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** This appears to have the votes. Plan to merge it Thursday 23:00UTC ish if no objections show up. ** Mailman 3 spec https://review.opendev.org/810990 *** Can use more reviews, but looks good to clarkb. * Topics ** Improving OpenDev's CD throughput (clarkb 20211012) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211012) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Scheduling Gerrit project renames (clarkb 20211012) *** October 15 at 18:00UTC. *** Now is the time to make sure projects have their changes in order and that we are ready. *** Updating the process to update project metadata on renamed projects ** Upgrading to Gerrit 3.3 (ianw 20211012) *** Upgrade happened over the weekend. Went very smoothly *** clarkb has been using hashtag:gerrit-3.3 to track post upgrade changes that we want to land related to the upgrade *** Are we ready to drop the 3.2 images? https://review.opendev.org/c/opendev/system-config/+/813074 *** Next up 3.4! * Open discussion

1 0

Team Meeting Agenda for October 5, 2021
by Clark Boylan 04 Oct '21

04 Oct '21

Hello, we will meet with this agenda on October 5, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release tomorrow * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to update to propose running node exporter from their shipped binary. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211005) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211005) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Debian Buster to Bullseye Updates (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/809269 Gitea bullseye update *** https://review.opendev.org/c/opendev/system-config/+/809286 Gerrit bullseye update ** Gitea 1.15.3 Upgrade (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Upgrading to Gerrit 3.3 (ianw 20211005) *** 3.4 is a much bigger jump and will need more care. Clarkb suggests we get to 3.3 soon then plan 3.4. ** Scheduling Gerrit project renames (clarkb 20211005) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Team Meeting Agenda for September 28, 2021
by Clark Boylan 27 Sep '21

27 Sep '21

We will meet on September 28, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Fix for Gerrit logo hosting https://review.opendev.org/c/opendev/system-config/+/811233 ** Gitea 1.15.3 Upgrade (clarkb 20210928) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Infra Team Meeting Agenda for September 21, 2021
by Clark Boylan 20 Sep '21

20 Sep '21

We will meet on September 21, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210921) *** lists.openstack.org is running focal now **** Kernel packages have been pinned. If you update the kernel you need to decompress it. **** Fungi is working on a mailman3 spec. We can decide the best route for replacing the server with that context. ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Stack at https://review.opendev.org/c/opendev/system-config/+/809510 migrates services to the new asset locations. ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 ** Gerrit replication "leaks" (clarkb 20210921) *** We've added a 15 minute network timeout to the gerrit config. Need to restart the server to pick that up. ** InMotion cloud has been scaled up (clarkb 20210921) *** We need to work on tuning the over subscription ratios. * Open discussion

1 0

Mailing lists offline 2021-09-12 for server upgrade
by Jeremy Stanley 14 Sep '21

14 Sep '21

On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 2

Team Meeting Agenda for September 14, 2021
by Clark Boylan 13 Sep '21

13 Sep '21

We will meet September 14, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210914) *** lists.openstack.org is running focal now **** Number of manual edits made to services captured in https://review.opendev.org/c/opendev/system-config/+/808570 **** Ran into issues with Xen being able to boot lz4 compressed kernels **** We decompressed the kernel but that still failed **** We installed grub-xen then set up a chainload situation to a xen specific grub preloader via /boot/grub/menu.lst **** This chainload situation may not need a compressed kernel? If it doesn't this server may be viable long term and we can document things. **** We should consider replacing the server to get away from Xen PV weirdness and to maybe use a floating IP which can be moved between upgraded instances in the future. ** Improving OpenDev's CD throughput (clarkb 20210914) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210914) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210914) *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. ** Working with InMotion Friday to increase IP addr allocation and scale up (clarkb 20210914) *** Will try to debug instance launch failures there prior to Friday. ** Scheduling Gerrit project renames (clarkb 20210914) *** We've got a few of these showing up now. We should look at a calendar and pencil in some plans. * Open discussion

1 0

Team meeting agenda for September 7, 2021
by Clark Boylan 07 Sep '21

07 Sep '21

We will meet with this agenda September 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210907) *** lists.openstack.org upgrade scheduled for September 12 beginning 15:00UTC ** Improving OpenDev's CD throughput (clarkb 20210907) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210907) *** 33 conflicts remain. Need to investigate and email their users about planned next steps. ** OpenDev logo hosting (clarkb 20210907) *** https://review.opendev.org/c/opendev/system-config/+/805932/ ready to switch to using the assets container for gitea *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. * Open discussion

1 0

Team Meeting Agenda for August 31, 2021
by Clark Boylan 31 Aug '21

31 Aug '21

Sorry, this is going out a little late this week. We will meet on August 31, 2021 (today) at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Feature Freeze has begun. * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210831) *** lists.kc.io has been upgraded to Focal. Seems to be happy. *** Do resource consumption numbers look different compared to pre upgrade state? *** Time to schedule a lists.o.o upgrade? ** Improving OpenDev's CD throughput (clarkb 20210831) *** Quick check on how things look after our quick and easy updates. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210831) *** Need to delete external ids for the recent batch of acount retirements. ** OpenDev logo hosting (clarkb 20210831) *** https://etherpad.opendev.org/p/logo-locations *** Are we happy with an approach yet? * Open discussion

1 0

Infra Team Meeting Agenda for August 24, 2021
by Clark Boylan 23 Aug '21

23 Aug '21

We will meet on August 24, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Zuul is now on our matrix homeserver at #zuul:opendev.org ** OpenStack Feature Freeze starts next week * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210824) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o *** Will coordinate with Kata to do the upgrade this week hopefully. ** Improving OpenDev's CD throughput (clarkb 20210824) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/805487 Be more specific about when inventory changes force jobs to run. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210824) *** Need to delete external ids for the recent batch of acount retirements. ** Gitea 1.15.0 upgrade (clarkb 20210824) *** This release exists now *** How do we want to handle the opendev logo image hosting for the gerrit theme? * Open discussion

1 0

Team Meeting Agenda for August 17, 2021
by Clark Boylan 16 Aug '21

16 Aug '21

We will meet with this agenda on August 17, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210817) *** Clark was the only nomination. Takes the seat by default. ** Matrix eavesdrop and gerritbot bots (clarkb 20210817) *** Zuul's next steps ** Gitea01 backups to one of two servers are unhappy (clarkb 20210817) *** Reboots didn't help *** On Friday we had a set of servers that backup at the same time with similar issues but to the other backup server. Doesn't seem to be a persistent issue. ** Mailman Ansible and Server Upgrades (clarkb 20210817) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o ** Improving OpenDev's CD throughput (clarkb 20210817) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/804795 Run cloud launcher daily not hourly. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. * Open discussion

1 0