- service-discuss - lists.opendev.org

Testing OpenStackSDK/OSC on real clouds
by Artem Goncharov 13 Oct '22

13 Oct '22

Hi all, Last years of my work on OpenStackSDK and CLI on one hand and being representative of a public cloud on the other hand made clear to me that not a single OpenStack cloud behaves like another one, doesn’t matter what you do and how hard you try. And sadly Interop is not helping here at all. The API calls you use to talk with one public cloud are guaranteed to work slightly differently on another one. This is clearly visible through the Storyboard of SDK, IRC and other offline discussions. Lot of customers are capable to work without issues with SDK/CLI on their cloud, while others report the same things are not working for them. This is a relatively complex discussion and Clark reserved a slot in the TC meeting during PTG https://etherpad.opendev.org/p/tc-2023-1-ptg#L38 <https://etherpad.opendev.org/p/tc-2023-1-ptg#L38> to describe his stake in the problem. For a quite some time I am trying to follow an approach of having possibility to run functional tests of both SDK and CLI on real clouds and not only on devstack (on devstack things are working fine but it means as much as nothing at all in real life). On multiple occurrences I was asking for credentials and I got at the moment credentials for 2 public clouds. This already helped both to find issues in SDK (i.e. RADOSGW != Swift) as well as to point clouds on real issues on their side. It should be actually in the interest of those clouds to have some useful checks that give guarantee to their users SDK and CLI (ideally also Ansible/TF) are working fine. This is even a better form of interoperability check than anything else what we have now. Since people in this ML are involved in maintaining resource donations to OpenDev I would like to ask you to consider allocating cloud credentials for the SDK/CLI team to ensure things are running smoothly for your users (pls do not mix with allocating real resources - we need only cloud credentials to test APIs and there are no static resources required). At the moment I maintain those credentials manually until we finally get post-review pipeline created in OpenStack tenant to fully automate those tests. If you have questions or suggestions please feel free to post here, join SDK/CLI team during PTG on Friday 27.10 13:00-15:00 or contact me directly for clarifications. Thanks a lot for support, Artem

2 1

Team Meeting Agenda for October 11, 2022
by Clark Boylan 11 Oct '22

11 Oct '22

We will meet with this agenda on October 11, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** PTG next week * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221011) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 Disable writing console log files on bridge and static **** Needs a second change pushed to opendev/base-jobs/playbooks/infra-prod/setup-keys.yaml *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221011) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Need to retest launching a Jammy node now that user management is updated. ** Mailman 3 (clarkb 20221011) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Ubuntu 22.04.1 is out for some months now and thus can be considered stable, no need to wait a full year like we did with 20.04 *** Patch to switch base-test job: https://review.opendev.org/c/opendev/base-jobs/+/860686/2 *** Plan for further steps? Timetable, which repos to test, announcements *** Devstack currently inherits the base nodeset (via multinode) without pinning * Open discussion

1 0

Team Meeting Agenda for October 4, 2022
by Clark Boylan 04 Oct '22

04 Oct '22

We will meet on October 4, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221004) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221004) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/860112 Update to launch node to handle jammy hosts ** Mailman 3 (clarkb 20221004) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Gitea connectivity issues (clarkb 20221004) *** Users in certain geographies have reported trouble with Gitea access *** Gitea changed http libraries which broke our ability to trace requests from haproxy -> apache -> gitea **** Partially worked around that by removing X-Forwarded-For headers. * Open discussion

1 0

Team Meeting Agenda for September 27, 2022
by Clark Boylan 27 Sep '22

27 Sep '22

We will meet with this agenda on September 27, 2022 at 19:00 UTC in #opendev-meeting. This agenda is a slightly edited version compared to the wiki as I anticipate we'll have less participation this week. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Mailman 3 (clarkb 20220927) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220927) *** Change has landed. ** Nodepool Builder Disk utilization (clarkb 20220927) *** Have we expanded disk space on existing builders yet? * Open discussion

1 0

Team Meeting Agenda for September 20, 2022
by Clark Boylan 20 Sep '22

20 Sep '22

We will meet with this agenda on September 20, 2022 in #opendev-meeting at 19:00 UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220920) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220920) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220920) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220920) *** Change has landed. ** Fedora 36 Rollout (clarkb 20220920) *** Now not booting reliably. ** Improving Ansible task runtime (clarkb 20220920) *** Ansible tasks can be slow on some test nodes. *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Possibly related to Ansible 5 breaking pipelining in ssh connection specific config *** https://review.opendev.org/c/opendev/system-config/+/857239 ** Nodepool Builder Disk utilization (clarkb 20220920) *** We're using more of the disk now which makes us more prone to filling the disk if something goes wrong. *** Can we trim any images out? *** Do we need more or bigger builders? * Open discussion

1 0

Updates to meetpad's Jitsi Meet configuration
by Jeremy Stanley 19 Sep '22

19 Sep '22

Last week, we refreshed our configuration defaults for the Jitsi Meet services which provide the teleconferencing features of our meetpad.opendev.org service. The primary behavior change you're likely to notice is that meeting room URLs first take you to a pre-join page now. This is a workaround for some browsers, particularly Firefox, treating the audio stream as unsolicited and automatically blocking it. Previously, that was responsible for causing confusing "I can't hear anyone else" problems for many users. If you had trouble with sound on our meetpad service before, please do try it again at your convenience and let us know if it helped (or more importantly, if it didn't). -- Jeremy Stanley

1 1

Team Meeting Agenda for September 13, 2022
by Clark Boylan 13 Sep '22

13 Sep '22

Hello, We will meet with this agenda on September 13, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220913) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220913) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Bulk migration identified a few things that have since been updated. Need another pass to check. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220913) *** https://review.opendev.org/c/opendev/system-config/+/855983 Adds ansible and docker deployment for Jaeger ** Fedora 36 Rollout (clarkb 20220913) *** https://review.opendev.org/c/zuul/nodepool/+/853914 Remove fedora 35 testing from nodepool ** Jitsi Meet Updates for Meetpad (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856553 Update to use colibri websockets and scale out JVBs *** Not sure if the java keystore in use by the JVBs need to share a CA or otherwise be trusted by something else. ** Stability of Zuul reboot playbook (clarkb 20220913) *** New instability identified caused by race between container shutdown and docker exec for graceful command *** https://review.opendev.org/c/opendev/system-config/+/857209 Fix for this race ** Python container image updates (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856537 *** Updates to latest python point releases. *** This pulls in newer glibc allowing us to remove the backport in Zuul images ** Improving Ansible task runtime (clarkb 20220913) *** Ansible tasks can be slow on some test nodes. *** Particularly problematic when loops are used that serially process a large set of items *** https://review.opendev.org/c/zuul/zuul-jobs/+/857228 *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Be on the lookout for problematic tasks and updates to speed them up. * Open discussion

1 0

Team Meeting Agenda for September 6, 2022
by Clark Boylan 06 Sep '22

06 Sep '22

Hello, This agenda is getting sent out a bit late due to the holiday yesterday. We will meet at 19:00 UTC in #opendev-meeting with this agenda today, September 6, 2022: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220906) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220906) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220906) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** Migration testing looking good. Need to test updated database settings for large attachments (including that db backups function) and redirects for old pipermail archives. ** Jaeger tracing server (for Zuul) (corvus 20220906) *** Any changes need review yet? ** Fedora 36 Rollout (clarkb 20220906) *** Status Update ** Jitsi Meet Updates for Meetpad (clarkb 20220906) *** Configs updated to start with landing page for meetings. Should fix Firefox auto mute of auto playing audio *** Discovered that JVBs rely on newer colibri websocket system for scaling out. Our extra JVBs need to be updated to function under this system. ** Stability of Zuul reboot playbook (clarkb 20220906) *** Crashes if unattended-upgrades is running due to conflict over apt lock. *** Potentially crashes if services were already stopped outside of the playbook run. * Open discussion

1 0

Team Meeting Agenda for August 30, 2022
by Clark Boylan 30 Aug '22

30 Aug '22

We will meet with this agenda on August 30, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack feature freeze this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220830) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220830) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220830) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** Working through migration testing now. ** Gerrit 08:00 UTC load issues (clarkb 20220830) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220830) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html ** Fedora 36 Rollout (clarkb 20220830) *** Do we need to wait for OpenStack to release first? *** We'll update the nodeset and most jobs will switch automatically. *** Will we remove Fedora 35 next? * Open discussion

1 0

Team Meeting Agenda for August 23, 2022
by Clark Boylan 23 Aug '22

23 Aug '22

Hello, We will meet with this agenda on August 23, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clark appears to have been the only Service Coordinator nominee. ** OpenStack release is fast approaching. https://releases.openstack.org/zed/schedule.html feature freeze September 2 ish. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220823) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220823) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** This should be ready to go now that 1.17.1 has been released. Just need to decide on a good window of time to deploy it. ** Gerrit 08:00 UTC load issues (clarkb 20220823) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220823) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html *** Implementation should be straightforward; does this warrant an OpenDev spec, or should I just propose the change? * Open discussion

1 0