- service-discuss - lists.opendev.org

Team Meeting Agenda for June 28, 2022
by Clark Boylan 27 Jun '22

27 Jun '22

Hello, We will meet with this agenda on June 28, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220628) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto Zuul upgrade and reboots didn't run due to a flock issue. Should be fixed for this next weekend. ** Gerrit 3.5 upgrade (ianw 20220628) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** #link https://review.opendev.org/c/opendev/system-config/+/847035 Removing Gerrit 3.4 and adding Gerrit 3.6 images and testing. ** Improving Grafana management tooling (clarkb 20220628) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://review.opendev.org/q/topic:grafana-screenshots Improved testing of our dashboards to see that they render as expected. ** Run a custom URL shortener service (frickler 20220628) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220628) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** The cause seems to be post-run timeouts during swift log uploads *** Both OSA and tripleo upload quite a number of log files. It could be related to this, but we're yet to find a root cause due to difficulty of debugging things that don't log properly. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion

1 0

Team Meeting Agenda for June 21, 2022
by Clark Boylan 20 Jun '22

20 Jun '22

Hello, We will meet with this agenda on June 21, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220621) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://review.opendev.org/c/opendev/system-config/+/846195 Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220621) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB **** review02's mariadb was upgraded to 10.6 from 10.4 using the process described in https://etherpad.opendev.org/p/gerrit-upgrade-3.5 ** Gerrit 3.5 upgrade (ianw 20220621) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Couple of issues discovered **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16018 **** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** Time to look at removing our 3.4 image builds and adding 3.6 image builds with updated upgrade testing. ** Enable webapp on nodepool launchers? (frickler 20220621) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220621) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team meeting agenda for June 14, 2022
by Clark Boylan 13 Jun '22

13 Jun '22

We will meet on June 14, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220614) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220614) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220614) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 ** Zuul changing default Ansible version to v5 soon (clarkb 20220614) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Enable webapp on nodepool launchers? (frickler 20220611) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220611) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team Meeting Agenda for May 31, 2022
by Clark Boylan 30 May '22

30 May '22

Hello, We will meet on May 31, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Summit happens June 7-9. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220531) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul_… Automated graceful Zuul upgrades and server reboots **** Hit a bug with zuul merger graceful shutdowns **** Hit a bug in zuul's model api 8 transition **** Otherwise seemed to work well. ** Container maintenance (clarkb 20220531) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220531) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Manually triggering periodic jobs (frickler 20220531) *** What is the correct process for manually triggering periodic jobs? ** Zuul changing default Ansible version to v5 soon (clarkb 20220531) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Removing Ethercalc (clarkb 20220531) *** Announced removal date was May 31, 2022 *** Last call before clarkb proceeds with this cleanup ** Do we want to hold a meeting June 7, 2022 (clarkb 20220531) *** clarkb and fungi are unlikely to be able to attend due to the summit acting as a conflict * Open discussion

1 0

Team Meeting Agenda for May 24, 2022
by Clark Boylan 23 May '22

23 May '22

Hello, We will meet with this agenda on May 24, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Summit happens June 7-9. No meeting June 7. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220524) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220524) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220524) *** Last major missing piece is wheel builds (and mirroring?) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842839 *** https://review.opendev.org/c/openstack/project-config/+/842841 ** Gerrit 3.5 upgrade planning (ianw 20220524) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Move requirements propose-updates job to py3.8 + py3.9 (frickler 20220523) *** Initial change: https://review.opendev.org/c/openstack/project-config/+/837815 *** Build results: https://zuul.opendev.org/t/openstack/builds?job_name=propose-updates&projec… *** First attempt at fixing: https://review.opendev.org/c/openstack/project-config/+/842837 *** Second fix: https://review.opendev.org/c/openstack/project-config/+/842853 *** Related question: How to properly trigger manual runs of periodic pipeline? **** Using example from zuul docs is failing, e.g. https://zuul.opendev.org/t/openstack/build/3d28674ac1044c22956c20ed39c82a4a ** Zuul changing default Ansible version to v5 soon (clarkb 20220524) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Playbooks and roles likely need updating. We should look into that for our hosted projects. *** Tenants can override a default to older Ansible if necessary. *** Eventually Ansible will remove old Ansible support. * Open discussion

1 0

Team Meeting Agenda for May 17, 2022
by Clark Boylan 16 May '22

16 May '22

We will meet on May 17, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. ** ClarkB out May 18 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Gerrit 3.5 upgrade planning (ianw 20220517) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Scheduling * Open discussion

1 0

Team Meeting Agenda for May 10, 2022
by Clark Boylan 09 May '22

09 May '22

Hello, We will meet on May 10, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220510) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Automating the build process for our Ubuntu PPA packages (clarkb 20220510) *** ianw is adding more automation to our vhd-util and openafs package builds. This should make it easier for us to update the packages and add new distro releases. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Keycloak container image update (clarkb 20220510) *** Keycloak stopped publishing to Docker Hub. *** Keycloak has two versions of the image. The Legacy wildfly image (what we use) and a non wildfly image we should probably convert to. *** https://review.opendev.org/c/opendev/system-config/+/840529 Pull keycloak from quay.io * Open discussion

1 0

Team Meeting Agenda for May 3, 2022
by Clark Boylan 02 May '22

02 May '22

We will meet with this agenda on May 3, 2022 at 19:00 UTC in @opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220503) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220503) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220503) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220503) *** Ubuntu-ports mirroring is in progress *** Do we need to build our openafs package for jammy (this would be to build wheel mirrors to start, but would potentially be used on zuul executors etc in the future) ** Shutting down Ethercalc (clarkb 20220503) *** The software doesn't seem to be well maintained *** The software lacks history tracking and rollback like etherpad *** Is largely used for PTG and other tools could be used for that (like PTGbot) *** clarkb would like to propose shutting this service down at the end of the month. * Open discussion

1 0

Team Meeting Agenda for April 26, 2022
by Clark Boylan 25 Apr '22

25 Apr '22

Hello, We will meet on April 26, 2022 at 19:00 UTC in #opendev with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220426) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220426) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220426) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220425) *** Repo mirrors https://review.opendev.org/c/opendev/system-config/+/838923 *** DIB testing https://review.opendev.org/c/openstack/diskimage-builder/+/836228 *** Actual build https://review.opendev.org/c/openstack/project-config/+/839057 *** Nodepool release with fixes? *** Drop mirroring for Xenial in order to recover AFS quota? **** We cleaned up a bunch of centos, epel, opensuse, open euler, fedora, etc mirrors to remove unnecessary content which made more than enough room for Jammy. ** Deleting the subunit2sql MySQL Trove instance (clarkb 20220426) *** The subunit2sql workers and the health api service that sat in front of this database are gone now *** Currently using 286GB of disk. Do we need to backup/snapshot this or can we simply delete it and move on. * Open discussion

1 0

Skipping Tomorrow's (April 19, 2022) Team Meeting
by Clark Boylan 18 Apr '22

18 Apr '22

Hello everyone, Due to many of us being out with either holidays or errands I'll go ahead and cancel tomorrow's (April 19, 2022) meeting. If you have something worth discussing feel free to start a thread on this mailing list or join us in #opendev on OFTC and start a conversation there. Clark

1 0