- service-discuss - lists.opendev.org

Team Meeting Agenda for February 13, 2024
by Clark Boylan 13 Feb '24

13 Feb '24

We will meet with this agenda on February 13, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** A new keycloak03 server is up and ready for use. This server is a redeployment of latest keycloak and a real database backend. ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/908328 ** Zuul Circular Dependency Refactor (clarkb 20240213) *** This has been deployed to OpenDev in a prerelease form. If you notice problems please let the Zuul project know *** Zuul has temporarily enabled this functionality within its tenant for testing/dogfooding purposes, but we don't expect OpenDev to enable this generally. * Open discussion

1 0

Team Meeting Agenda for February 6, 2024
by Clark Boylan 06 Feb '24

06 Feb '24

We will meet with this agenda on February 6, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** Keycloak converted its runtime from wildfly to quarkus. This led to legacy wildfly images and modern quarkus images that are not super compatible with each other *** A switch from the legacy wildfly to quarkus image failed because the h2 database isn't very portable between the two. (Different file paths, filenames and db credentials). *** Pivoting to a new latest release deployment using a mariadb. Will require us to rebuild the zuul auth stuff if we can't port the db content across. ** Gitea Upgrade and DB Confg Changes (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit *** https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** No change yet. The changelog indicates this update impacts how plugins are installed. We should make sure the plugins we rely on function in our images builds. * Open discussion

1 0

OpenDev Team Meeting Agenda for January 30, 2024
by Clark Boylan 30 Jan '24

30 Jan '24

We will meet on January 30, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Broken wheel build issues (frickler 20231217) *** https://review.opendev.org/c/openstack/project-config/+/907150 Will fix centos 8 and 9 stream *** CentOS 7 appears to be failing now too. ** Keycloak upgrade struggles (clarkb 20240130) *** Keycloak converted its runtime from wildfly to quarkus. This led to legacy wildfly images and modern quarkus images that are not super compatible with each other *** A switch from the legacy wildfly to quarkus image failed because the h2 database isn't very portable between the two. (Different file paths, filenames and db credentials). *** Pivoting to a new latest release deployment using a mariadb. Will require us to rebuild the zuul auth stuff if we can't port the db content across. ** InMotion/OpenMetal Cloud RabbitMQ Problems (clarkb 20240130) *** RabbitMq crashed and got into an unhappy state. *** Naive restarts were not sufficient, but using some kolla rabbitmq restart made things happy again. * Open discussion

1 0

OpenDev Pre PTG Planning
by Clark Boylan 25 Jan '24

25 Jan '24

The OpenDev team is planning a pre PTG event where the OpenDev team can get together to prioritize its backlog and do some planning for the year. We will meet on February 14 and 15 from 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. An agenda is being built in this etherpad: https://etherpad.opendev.org/p/opendev-preptg-202402. We will use the corresponding meetpad room, https://meetpad.opendev.org/opendev-preptg-202402, for our meetings. Please feel free to add agenda items or add content to existing items. If there are specific topics you are particularly interested in, please note that on the etherpad so that we can do our best to accommodate time zones in our scheduling. This pre PTG is being aimed at existing contributors so that we can sync up on where we are today and what needs to be done tomorrow. If you would like to start contributing to OpenDev or have OpenDev usage questions, please reach out to us directly or join us at the PTG in April. We are happy to help. Finally, to avoid any confusion we do plan to participate in the PTG this April. Our focus at the April PTG will be more towards users and new contributors. This split allows us to dive into the pre PTG agenda earlier in the year, and many of our contributors are busy during the PTG which makes planning time during that week difficult. See you there, Clark

1 0

https://review.opendev.org/SignInFailure,SIGN_IN,Local+signature+verification+failed
by lujie＠ict.ac.cn 23 Jan '24

23 Jan '24

I just want to login https://review.opendev.org/ but redirect to https://review.opendev.org/SignInFailure,SIGN_IN,Local+signature+verificati… response is "Not Found" lujie(a)ict.ac.cn

2 1

Team Meeting Agenda for January 23, 2024
by Clark Boylan 23 Jan '24

23 Jan '24

We will meet with this agenda on January 23, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** Updating Zuul's database server (clarkb 20231121) *** Currently this is an older mysql 5.7 trove instance *** We can move it to a self hosted instance (maybe on a dedicated host?) running out of docker like many of our other services and get it more up to date. *** Are there other services we should consider this for as well? *** Research/Planning questions: https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Broken wheel build issues (frickler 20231217) *** For arm nodes this is due to lacking working openafs packages for centos on arm. This should be resolvable now that we have working arm dib builds again *** There is also a publication issue in choosing the wrong afs volume to vos release that affects x86 and presumably arm once openafs on arm works again. ** OpenDev Pre PTG (clarkb 20240123) *** Looking at try to do two days Wednesday and Thursday sometime in February: **** February 7+8 **** February 14+15 **** February 21+22 *** Have two blocks of time each day one that works better for EU and another for APAC. Probably 14:00-16:00UTC and 22:00-00:00 UTC. *** Still very much open to feedback but I have to start somewhere *** Will still sign up for the proper PTG, but with fewer expectations from admins that need to attend other PTG activities. Will gear the PTG proper towards users and new contributors. * Open discussion

1 0

Team Meeting Agenda for January 16, 2024
by Clark Boylan 16 Jan '24

16 Jan '24

We will meet with this agenda on January 16, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Live will feature OpenDev January 18, 2024. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/905510 ** Python container updates (tonyb 20230718) *** https://review.opendev.org/c/opendev/system-config/+/905018 Drop Bullseye python3.11 images *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** Updating Zuul's database server (clarkb 20231121) *** Currently this is an older mysql 5.7 trove instance *** We can move it to a self hosted instance (maybe on a dedicated host?) running out of docker like many of our other services and get it more up to date. *** Are there other services we should consider this for as well? *** Research/Planning questions: https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade ** EMS discontinuing legacy/consumer hosting plans (fungi 20231219) *** Fungi engaged to renew under the new discounted terms. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Broken wheel build issues (frickler 20231217) *** For arm nodes this is due to lacking working openafs packages for centos on arm. This should be resolvable now that we have working arm dib builds again *** There is also a publication issue in choosing the wrong afs volume to vos release that affects x86 and presumably arm once openafs on arm works again. ** Gitea repo-archives filling server disk (clarkb 20240109) *** https://review.opendev.org/c/opendev/system-config/+/904868 update robots.txt on upstream's suggestion *** The cron job to clear archives weekly is configured but services need to be restarted to pick up the change. Maybe land https://review.opendev.org/c/opendev/system-config/+/905020 first and then restart things. ** OpenDev Service Coordinator Election happening in February (clarkb 20240109) *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Open discussion

1 0

First OpenDev Service Coordinator Election for 2024
by Clark Boylan 13 Jan '24

13 Jan '24

It is that time of year again where we elect an OpenDev Service Coordinator for six months. I have added six months to the dates we used for the last election [0] and came up with this plan for this election. We will have nominations open from February 6, 2024 to February 20, 2024. If necessary, we will have election voting happen from February 21, 2024 to February 28, 2024. All dates and deadlines will use UTC time. Hopefully this gives everyone plenty of time to plan and consider if they are interested. As usual, feel free to reach out to me if you have questions about the dates, process, service coordinator duties, or anything else really. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… Clark

1 0

OpenDev Team Meeting Agenda for January 9, 2024
by Clark Boylan 09 Jan '24

09 Jan '24

We will meet with this agenda on January 9, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Foundation Individual Board member election is happening now. Look for your ballot via email and vote. ** OpenInfra Live will feature OpenDev January 18, 2024. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Python container updates (tonyb 20230718) *** https://review.opendev.org/c/opendev/system-config/+/905018 Drop Bullseye python3.11 images *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** Updating Zuul's database server (clarkb 20231121) *** Currently this is an older mysql 5.7 trove instance *** We can move it to a self hosted instance (maybe on a dedicated host?) running out of docker like many of our other services and get it more up to date. *** Are there other services we should consider this for as well? *** Research/Planning questions: https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade ** EMS discontinuing legacy/consumer hosting plans (fungi 20231219) *** We have until 2024-02-07 to upgrade to a business hosting plan (prepaying a year at 10x the current price) or move elsewhere. ** Followup on 20231216 incident (frickler 20231217) *** Do we want to pin external images like haproxy and only bump them after testing? (Not sure that would've helped for the current issue though) *** Use docker prune less aggressively for easier rollback? **** We do so for some services, like https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles…, might want to duplicate for all containers? Bump the hold time to 7d? *** Add timestamps to zuul_reboot.log? **** https://opendev.org/opendev/system-config/src/branch/master/playbooks/servi… **** Also this is running on Saturdays (weekday: 6), do we want to fix the comment or the dow? *** Do we want to document or implement a procedure for rolling back zuul upgrades? Or do we assume that issues can always be fixed in a forward going way? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** project.zuul with the latest releases is getting close to its tight limit of 1GB (sic), I suggest to simply double that ** Broken wheel build issues (frickler 20231217) *** wheel builds for centos >=8 seem broken, with nobody maintaining these it might be better to drop them? ** Gitea repo-archives filling server disk (clarkb 20240109) *** https://review.opendev.org/c/opendev/system-config/+/904868 update robots.txt on upstream's suggestion *** https://review.opendev.org/c/opendev/system-config/+/904874 Run weekly removal of all cached repo archives ** OpenDev Service Coordinator Election happening soon (clarkb 20240109) * Open discussion

1 0

Team Meeting Agenda for December 19, 2023
by Jeremy Stanley 19 Dec '23

19 Dec '23

We will meet with this agenda on December 19, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** No meetings December 12, 26, and January 2. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Tonyb is adding new mirror servers. *** New mirror servers should be ready. Is it time to update CNAMEs and plan for old server cleanup? ** DIB bionic support (ianw 20231206) *** py36 tox unittesting broken -- proposal to drop https://review.opendev.org/c/openstack/diskimage-builder/+/901093 *** unit testing has been kept this long to ensure in-chroot tools are python3.6 clean -- bionic only platform still supported *** one suggestion is to pause bionic builds - see comment in https://review.opendev.org/c/openstack/project-config/+/901692 and comments in PS1 **** clarkb points out this is painful if cloud providers change, and dib _probably_ won't break the build anyway *** second option is to drop tox 36 testing, but leave bionic test in dib-functests. Probably enough coverage for basic support. Should probably do a release first. ** Python container updates (tonyb 20230718) *** https://review.opendev.org/q/(topic:bookworm-python3.11+OR+hashtag:bookworm… Next round of image rebuilds onto bookworm. *** zuul-operator is the last hold out on python3.10. Working through failures in CI there. **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** Gitea 1.21.1 Upgrade (clarkb 20230926) *** https://review.opendev.org/c/opendev/system-config/+/902490 Configure Gerrit to use new SSH key *** After Gerrit is restarted and using the new key we can remove the old key from Gitea. At that point we should be ready to plan the Gitea upgrade. *** https://review.opendev.org/c/opendev/system-config/+/897679 Upgrade to Gitea 1.21.0 ** Updating Zuul's database server (clarkb 20231121) *** Currently this is an older mysql 5.7 trove instance *** We can move it to a self hosted instance (maybe on a dedicated host?) running out of docker like many of our other services and get it more up to date. *** Are there other services we should consider this for as well? *** Research/Planning questions: https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade ** Annual Report Season (clarkb 20231128) *** OpenDev's 2023 Annual Report Draft will live here: https://etherpad.opendev.org/p/2023-opendev-annual-report ** EMS discontinuing legacy/consumer hosting plans (fungi 20231219) *** We have until 2024-02-07 to upgrade to a business hosting plan (prepaying a year at 10x the current price) or move elsewhere. ** Followup on 20231216 incident (frickler 20231217) *** Do we want to pin external images like haproxy and only bump them after testing? (Not sure that would've helped for the current issue though) *** Use docker prune less aggressively for easier rollback? **** We do so for some services, like https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles…, might want to duplicate for all containers? Bump the hold time to 7d? *** Add timestamps to zuul_reboot.log? **** https://opendev.org/opendev/system-config/src/branch/master/playbooks/servi… **** Also this is running on Saturdays (weekday: 6), do we want to fix the comment or the dow? *** Do we want to document or implement a procedure for rolling back zuul upgrades? Or do we assume that issues can always be fixed in a forward going way? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** project.zuul with the latest releases is getting close to its tight limit of 1GB (sic), I suggest to simply double that ** Broken wheel build issues (frickler 20231217) *** wheel builds for centos >=8 seem broken, with nobody maintaining these it might be better to drop them? * Open discussion ** (tonyb 20231128) [If time permits]. Could we enable via roles/jobs or with an additional nodepool driver. The ability for [OpenStack] project teams to run unit tests with the python images we already build eg https://review.opendev.org/c/opendev/system-config/+/898756. -- Jeremy Stanley

1 0