- service-discuss - lists.opendev.org

New approaches for grafana.opendev.org
by Ian Wienand 01 Jul '22

01 Jul '22

Hello, 65;6800;1c Currently all graphs pushed to grafana.opendev.org are defined in project-config/graphana as YAML files consumed by grafyaml [1] The fundamental tension for grafyaml is that the upstream Grafana project do not document and publish a defined schema for dashboards and their components. grafyaml has a subset of the upstream data model which is incomplete, in some cases buggy -- but also, perhaps most importantly, undocumented [1]. We have over a million data points of interesting information in graphite but I feel there are significant barriers to new and interesting visualisations. With no clear documentation, either upstream or in grafyaml, where is somebody supposed to start? A series of changes have been reviewed and landed through grafyaml that allow it to upload dashboards exported directly from the Grafana UI in its native .json format. I would like to achieve some consensus that we use this feature in the OpenDev environment. I will leave aside the issues with the schema encoded in grafyaml; it's possible this might be fixed. AIUI the main reason for duplicating the schema in grafyaml was that it presented more reviewable YAML files. To this I would say: 1) Layout of the page; i.e. the rows, panels, nesting, etc. My argument here is that reviewers having to build a mental model of what a dashboard will look like -- from either YAML or json -- does not make for thorough reviews, especially if you're not already intimately familiar with the desired output. To this end, I have added a new job "project-config-grafana" which produces an artifact "Screenshots" that loads changed graphs into a Grafana instance and stores actual screenshots loaded from a headless browser. I believe this is a much more effective way to review proposed layout changes for both formats of input. 2) The data graphed. This comes down to the metric selected, and any functions applied. My argument here is that firstly the screenshot is a good way to evaluate this; for example you will see if you've accidentally treated milliseconds as seconds, etc. when the graph axis is wrong. As to the actual data -- ensuring we have the right metric, etc. -- I would say that the "raw" output of the exported graphs just isn't that hard to parse. It is unobfuscated and reads logically. I have proposed some examples: https://review.opendev.org/c/openstack/project-config/+/833213/6/grafana/in… https://review.opendev.org/c/openstack/project-config/+/848212/3/grafana/no… I think you can clearly see the metrics chosen and the functions applied. 3) Generally more confusing. This is true, as the .json file is meant for Grafana to read. However, for better or worse, this is the actual data model of your graph page. To this end, I have proposed documentation and a helper-script to start a Grafana instance in a local container, and load it with the defined dashboards: https://review.opendev.org/c/openstack/project-config/+/833214/ This is useful for interactive editing sessions to develop new dashboards, and if a reviewer wishes to examine a change more closely than the screenshots provided by CI, they can simply pull the change from gerrit and load it into a live instance using this simple method. I think this is a significantly lower barrier to get people developing new and interesting things against the data provided. I'm not proposing any existing graph need change [2] and grafyaml's features to setup datasources and load the graphs are still used. I'm not proposing we remove or even stop any development of the YAML schema if people want to work on that and prefer to keep their graphs that way. I think that there is a great resource here that is underutilised, and my hope is we have a path to greatly reduce the barriers to new contributions. Sorry for the long mail, -i [1] Grafana does a good job of backwards compatibility, so "old" dashboards work in new releases. Hence our extant graphs, though producing output that looks very different from what the UI produces now, generally work. Modulo some bugs where the "update" process doesn't work (thresholds was one I found), deprecations of features that will disappear (c.f. time-series graphs) and just the many panel types that are completely unsupported. [2] Though most extant graphs use deprecated panel types that will have to be updated one day; but that's an issue for another time.

1 0

Team Meeting Agenda for June 28, 2022
by Clark Boylan 28 Jun '22

28 Jun '22

Hello, We will meet with this agenda on June 28, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220628) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto Zuul upgrade and reboots didn't run due to a flock issue. Should be fixed for this next weekend. ** Gerrit 3.5 upgrade (ianw 20220628) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** #link https://review.opendev.org/c/opendev/system-config/+/847035 Removing Gerrit 3.4 and adding Gerrit 3.6 images and testing. ** Improving Grafana management tooling (clarkb 20220628) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://review.opendev.org/q/topic:grafana-screenshots Improved testing of our dashboards to see that they render as expected. ** Run a custom URL shortener service (frickler 20220628) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220628) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** The cause seems to be post-run timeouts during swift log uploads *** Both OSA and tripleo upload quite a number of log files. It could be related to this, but we're yet to find a root cause due to difficulty of debugging things that don't log properly. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion

1 0

Team Meeting Agenda for June 21, 2022
by Clark Boylan 21 Jun '22

21 Jun '22

Hello, We will meet with this agenda on June 21, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220621) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://review.opendev.org/c/opendev/system-config/+/846195 Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220621) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB **** review02's mariadb was upgraded to 10.6 from 10.4 using the process described in https://etherpad.opendev.org/p/gerrit-upgrade-3.5 ** Gerrit 3.5 upgrade (ianw 20220621) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Couple of issues discovered **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16018 **** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** Time to look at removing our 3.4 image builds and adding 3.6 image builds with updated upgrade testing. ** Enable webapp on nodepool launchers? (frickler 20220621) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220621) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team meeting agenda for June 14, 2022
by Clark Boylan 14 Jun '22

14 Jun '22

We will meet on June 14, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220614) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220614) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220614) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 ** Zuul changing default Ansible version to v5 soon (clarkb 20220614) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Enable webapp on nodepool launchers? (frickler 20220611) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220611) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team Meeting Agenda for May 31, 2022
by Clark Boylan 31 May '22

31 May '22

Hello, We will meet on May 31, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Summit happens June 7-9. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220531) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul_… Automated graceful Zuul upgrades and server reboots **** Hit a bug with zuul merger graceful shutdowns **** Hit a bug in zuul's model api 8 transition **** Otherwise seemed to work well. ** Container maintenance (clarkb 20220531) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220531) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Manually triggering periodic jobs (frickler 20220531) *** What is the correct process for manually triggering periodic jobs? ** Zuul changing default Ansible version to v5 soon (clarkb 20220531) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Removing Ethercalc (clarkb 20220531) *** Announced removal date was May 31, 2022 *** Last call before clarkb proceeds with this cleanup ** Do we want to hold a meeting June 7, 2022 (clarkb 20220531) *** clarkb and fungi are unlikely to be able to attend due to the summit acting as a conflict * Open discussion

1 0

Team Meeting Agenda for May 24, 2022
by Clark Boylan 24 May '22

24 May '22

Hello, We will meet with this agenda on May 24, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Summit happens June 7-9. No meeting June 7. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220524) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220524) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220524) *** Last major missing piece is wheel builds (and mirroring?) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842839 *** https://review.opendev.org/c/openstack/project-config/+/842841 ** Gerrit 3.5 upgrade planning (ianw 20220524) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Move requirements propose-updates job to py3.8 + py3.9 (frickler 20220523) *** Initial change: https://review.opendev.org/c/openstack/project-config/+/837815 *** Build results: https://zuul.opendev.org/t/openstack/builds?job_name=propose-updates&projec… *** First attempt at fixing: https://review.opendev.org/c/openstack/project-config/+/842837 *** Second fix: https://review.opendev.org/c/openstack/project-config/+/842853 *** Related question: How to properly trigger manual runs of periodic pipeline? **** Using example from zuul docs is failing, e.g. https://zuul.opendev.org/t/openstack/build/3d28674ac1044c22956c20ed39c82a4a ** Zuul changing default Ansible version to v5 soon (clarkb 20220524) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Playbooks and roles likely need updating. We should look into that for our hosted projects. *** Tenants can override a default to older Ansible if necessary. *** Eventually Ansible will remove old Ansible support. * Open discussion

1 0

Team Meeting Agenda for May 17, 2022
by Clark Boylan 17 May '22

17 May '22

We will meet on May 17, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. ** ClarkB out May 18 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Gerrit 3.5 upgrade planning (ianw 20220517) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Scheduling * Open discussion

1 0

Team Meeting Agenda for May 10, 2022
by Clark Boylan 10 May '22

10 May '22

Hello, We will meet on May 10, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220510) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Automating the build process for our Ubuntu PPA packages (clarkb 20220510) *** ianw is adding more automation to our vhd-util and openafs package builds. This should make it easier for us to update the packages and add new distro releases. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Keycloak container image update (clarkb 20220510) *** Keycloak stopped publishing to Docker Hub. *** Keycloak has two versions of the image. The Legacy wildfly image (what we use) and a non wildfly image we should probably convert to. *** https://review.opendev.org/c/opendev/system-config/+/840529 Pull keycloak from quay.io * Open discussion

1 0

Team Meeting Agenda for May 3, 2022
by Clark Boylan 03 May '22

03 May '22

We will meet with this agenda on May 3, 2022 at 19:00 UTC in @opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220503) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220503) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220503) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220503) *** Ubuntu-ports mirroring is in progress *** Do we need to build our openafs package for jammy (this would be to build wheel mirrors to start, but would potentially be used on zuul executors etc in the future) ** Shutting down Ethercalc (clarkb 20220503) *** The software doesn't seem to be well maintained *** The software lacks history tracking and rollback like etherpad *** Is largely used for PTG and other tools could be used for that (like PTGbot) *** clarkb would like to propose shutting this service down at the end of the month. * Open discussion

1 0

Team Meeting Agenda for April 26, 2022
by Clark Boylan 26 Apr '22

26 Apr '22

Hello, We will meet on April 26, 2022 at 19:00 UTC in #opendev with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220426) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220426) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220426) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220425) *** Repo mirrors https://review.opendev.org/c/opendev/system-config/+/838923 *** DIB testing https://review.opendev.org/c/openstack/diskimage-builder/+/836228 *** Actual build https://review.opendev.org/c/openstack/project-config/+/839057 *** Nodepool release with fixes? *** Drop mirroring for Xenial in order to recover AFS quota? **** We cleaned up a bunch of centos, epel, opensuse, open euler, fedora, etc mirrors to remove unnecessary content which made more than enough room for Jammy. ** Deleting the subunit2sql MySQL Trove instance (clarkb 20220426) *** The subunit2sql workers and the health api service that sat in front of this database are gone now *** Currently using 286GB of disk. Do we need to backup/snapshot this or can we simply delete it and move on. * Open discussion

1 0