- service-discuss - lists.opendev.org

Meeting Agenda for February 16, 2021
by Clark Boylan 16 Feb '21

16 Feb '21

Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten.

1 0

Meeting Agenda for February 9, 2021
by Clark Boylan 09 Feb '21

09 Feb '21

We will meet in #opendev-meeting at 19:00 UTC on February 9, 2021 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. * General topics ** OpenAFS cluster status (clarkb 20210209) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210209) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210209) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210209) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** ianw offered to help with the deployment side *** kopecmartin will help with testing the new instance ** Meetpad was not functional late last week. Seems fine now (clarkb 2020210209) * Open discussion

1 0

Meeting Agenda for February 2, 2020
by Clark Boylan 02 Feb '21

02 Feb '21

We will meet February 2, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html Only nomination appears to be from Clark. *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 81 accounts with preferred emails missing external ids have been fixed. **** We have 28 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210202) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210202) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210202) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** Deploy a new refstack.openstack.org server (kopecmartin 20210202) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** Help with deploying the server is needed *** kopecmartin will help with testing the new instance * Open discussion

1 0

Service Coordinator Nominations
by Clark Boylan 31 Jan '21

31 Jan '21

Hello everyone, It has been pointed out that in our governance document we stated we would elect a service coordinator every 6 months. I filled this role at the beginning of May 2020 which means we are overdue for a new election. The beginning of the year is always a bit slow so I figure we can give folks until January 31st at 23:59UTC to nominate themselves. Then if we need an election we can do that during the week from February 1st to February 8 at 23:59UTC. I've now been in this role (either as Service Coordinator or Infra PTL) a while. I think it would excellent to rotate in some new eyeballs and perspectives if possible. If you have interest but also have questions feel free to reach out. In order to avoid this slipping by in the future I'm going to go ahead and write down the dates for the next nomination period and election now: * Nominations July 12, 20201 - August 1, 2021 at 23:59 UTC * Election if necessary August 2 - August 9, 2021 at 23:59 UTC I think I got the math right. If I've got it wrong or you have other concerns feel free to bring them up on this thread and we can sort things out. I'm trying to balance giving people enough time to show interest at the beginning of the year while also not letting this languish for longer than it needs to. Then doing math for what 6 months from now will look like. Clark

1 1

Deploy a new refstack.openstack.org instance
by Martin Kopec 28 Jan '21

28 Jan '21

Hi, I'd like to ask for help with deploying a new refstack.openstack.org instance [1]. I also added this topic to the weekly meeting agenda [2]. I'll help with testing the new instance before we put it to the production. [1] https://review.opendev.org/c/opendev/system-config/+/705258 [2] https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Weekly_Project_In… Thank you for your help. Regards, -- Martin Kopec Software Quality Engineer Red Hat EMEA <https://www.redhat.com>

2 1

Fwd: [All][StoryBoard] Angular.js Alternatives
by Kendall Nelson 27 Jan '21

27 Jan '21

Hello Everyone! The StoryBoard team is looking at alternatives to Angular.js since its going end of life. After some research, we've boiled all the options down to two possibilities: Vue.js or React.js I am diving more deeply into researching those two options this week, but any opinions or feedback on your experiences with either of them would be helpful! Here is the etherpad with our research so far[3]. Feel free to add opinions there or in response to this thread! -Kendall Nelson (diablo_rojo) & The StoryBoard Team [1] https://vuejs.org/ [2] https://reactjs.org/ [3] https://etherpad.opendev.org/p/replace-angularjs-storyboard-research

5 5

Meeting agenda for January 26, 2021
by Clark Boylan 26 Jan '21

26 Jan '21

We will meet on January 26, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html less than a week remaining. *** Gerrit account and group inconsistencies **** ~1 group has a subgroup membership loop. **** We have ~109 accounts with preferred email addresses that don't have a matching external id ***** Some of these are inactive accounts that we can more properly retire, others are accounts that can be set to inactive due to lack of external ids, some have email addresses we might convert to preferred email address, and others will need proper digging. **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210126) *** https://review.opendev.org/c/opendev/system-config/+/771521 properly install new openafs on xenial openafs clients. *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210126) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210126) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** two-review rule impact on low-activity projects (zbr 20210126) *** projects like git-review rely on infra team and delay changes *** Properly setting expectations for what our small team is able to pay attention to *** Communicating the topics that are consuming our time day to day **** We update the priority efforts documentation and possibly link to a storyboard dashboard there that is populated with the current fires. * Open discussion

1 0

Requesting help with OpenStack configuration files
by Bessghaier, Narjes 19 Jan '21

19 Jan '21

Dear OpenStack team, I’m a Ph.D. student in software engineering at the ETS Montreal, University of Quebec working on the quality and configuration of web-based software systems. I’m particularly interested in analyzing configuration files from different OpenStack files. One of the main challenges I am currently facing is the proper identification of configuration files. I’m mostly confused between the python files used for production and the python files used for configuration. I am kindly requesting your precious help with the following questions: 1- How to distinguish between python files used for configuration and python files used for production? It will be very helpful if there are some configuration-based patterns (eg, textual patterns or expressions) that we can find in python files to help us distinguish between source code and configuration files? 2- Certain python files use the oslo_config to access and define configuration options. Could "all" these python files be considered as configuration files? For example, the following python file of the keystone project: keystone/keystone/conf/auth.py, is it considered a source code or a configuration file? 3- Why are there different source code and configuration repositories for OpenStack projects (eg, nova and puppet-nova)? For instance, does the OpenStack-nova service have some configuration files in its repository and have the puppet-nova as a separate configuration repository as well? Thank you very much in advance for your time and your help! Kind regards, Narjes Bessghaier Narjes Bessghaier Ph.D student in Software Engineering École de Technologie Supérieure (ETS)| University of Quebec Montreal, Canada narjes.bessghaier.1(a)ens.etsmtl.ca<mailto:yasmine.makroum.1@ens.etsmtl.ca>

2 1

Team Meeting Agenda for January 19, 2021
by Clark Boylan 19 Jan '21

19 Jan '21

We will meet January 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html *** Zuul results table rendering (ianw 20210105) **** https://review.opendev.org/q/topic:%22gerrit-admin-user%22+status:open *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea 1.13.1 now available **** https://review.opendev.org/c/opendev/system-config/+/769226 * General topics ** OpenAFS cluster status (clarkb 20210119) *** New packages built for CentOS *** Need to sort out openafs-client installation on Debian Buster *** Are we properly installing openafs-client on Xenial? *** https://review.opendev.org/c/opendev/system-config/+/771268 ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210119) *** Enable Xenial -> Bionic/Focal system upgrades ** Discuss infra-core (on behalf of OpenStack) (mnaser) ** two-review rule impact on low-activity projects (zbr 20210115) *** projects like git-review rely on infra team and delay changes *** risks for enabling exceptions? how could we expose these in gerrit? *** apparently that issue is affecting multiple projects, maybe we can think about a generic solution? ** Bup and Borg Backups (clarkb 20210105) * Open discussion

1 0

OpenAFS timestamp rollover issues, discussion and plan
by Ian Wienand 16 Jan '21

16 Jan '21

Hello, We were made aware of an issue relating to handling of a timestamp rollover affecting OpenAFS deployments [1] that in the worst case results in inability of the client and server to communicate. From the OpenAFS 1.8.7 release: It fixes a critical issue in the generation of Rx connection IDs that prevent Rx clients started after 14 Jan 2021 08:25:36 AM UTC from being able to successfully make connections. In addition to cache managers and client utilities, fileservers and database servers are also affected, since they initiate connections to (other) database servers during their normal operation. The issue occurs only at startup, while generating the initial connection ID, so cache managers or servers that were already running at the time in question will not be affected until they restart. The full extent of the issues for our particular circumstances is still somewhat unclear. We run a heterogeneous deployment with all clients under our control using 1.8.6 release, but all servers running 1.6 era packages from Ubuntu Xenial. My current understanding is that since we started our servers and clients before the problem time, we are currently unaffected. When we restart our servers, due to differences in the code with our 1.6 versions, the bug will manifest more as periodic/random-ish I/O failures rather than a complete inability to communicate. We have rebuilt our openafs packages with the required fixes and deployed these to all clients under our control. Thus if any client restarts for whatever reason, they will at least be using fixed code. Some clients we have restarted just to ensure sanity of the new version against the existing severs (so far seems good). All clients have been checked for deployment of these new packages. We do not expect issues, but will monitor this situation. This leaves us with the server side. We have been told there are no fixes for the 1.6 servers planned. We do not have a wonderful answer here, unfortunately. We have planned to move our AFS infrastructure off the Xenial hosts it runs on for some time. These servers are all deployed with legacy puppet, something none of us want to spend significant time hacking on. There is also the small matter that 1.6 to 1.8 upgrades are not terribly well documented, to say the least. While we don't intend on restarting the servers, from time to time the cloud provider needs to migrate things or has other issues that affect uptime; we can't predict this other than it will happen, eventually. It seems our best option here is to take some down-time of the AFS services and perform a manual, in-place upgrade of the existing servers to 1.8 code. We will freeze these from ongoing automated config managment (puppet). This will allow us to deploy and evaluate 1.8 servers and keep with the principle of "change one thing at a time". Once we are in a steady state, we should have enough redundancy that we can replace the servers one-at-a-time with no, or very little, downtime. This gives us time to write, test and review Ansible based deployment as usual. When we do switch in new Focal based servers, we have the advantage we are not also trying to change the AFS version too. In some ways, it works out well (something about lemons and lemonade). We usually try to have a rollback plan with any upgrades. Since 1.6 is not getting updates, and as far as I know we still do not have the technology to move reality backwards in the time continum before the problem rollover time (if I do figure out time travel, I will be sure to pre-respond to this message that it can be ignored) our revert plans seem limited. However, we will be no worse off than if the servers decided to reboot themselves now. In it's defence, this is not new code; 1.8.0 came out early 2018 and we have been using 1.8 clients since we started integrating arm64. The queues are very deep at the moment, so we'd obviously like to minimise downtime. I'd suggest that I could start looking at this around 2021-01-17 21:00UTC, following the plan layed out in [2], which interested parties should definitely review. This should hopefully be a quiet time, and give us a decent runway if we do hit issues. -i [1] https://lists.openafs.org/pipermail/openafs-info/2021-January/043013.html [2] https://etherpad.opendev.org/p/infra-openafs-1.8

3 3