April 2019 - Zuul-discuss - lists.opendev.org

Inheritance confusion, docs needed?
by Michael Johnson 06 May '19

06 May '19

As a developer creating Zuul jobs, one of the areas i find confusing is the inheritance of variables. I am not sure if this is an enhancement style request or just a missing section in the documentation. When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". As an example is the opendev Zuul jobs (I think this applies to non-OpenStack jobs as well): When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". host-vars: controller: devstack_localrc: DATABASE_USER: a-user In this example snippet, the variables at the devstack_localrc level are "merged", but the variables at the "DATABASE_USER" level are replaced and not inherited from the parent. Should Zuul be recursively "merging" all the way down or should we document how/which variables are inherited from the parent and which are not? The reason I'm asking is as our job definitions get more complicated (multi-node), I find myself copying variables from the parent jobs that I would really like to just inherit. Michael

2 1

Zuul working group session at OpenInfra Summit
by corvus＠inaugust.com 29 Apr '19

29 Apr '19

Hi, If you're at the OpenInfra Summit and would like to discuss what is or isn't working about sharing jobs between Zuul installations, please join the working group session on Tuesday: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23736/z… -Jim

1 0

Celebrating Zuul's OSF Top Level Project Confirmation
by Clark Boylan 28 Apr '19

28 Apr '19

Zuul was confirmed by the OSF board as a top level OSF project today in Denver. To celebrate we'd like to meetup for arcade games, drinks, and food at FTW Denver Monday night. This is doubling as the OpenDev Infra team gathering, but considering the overlap between Zuul and Infra I figure that won't be an issue. If you'd like to join us we will be gathering at 6:45pm Monday in front of the big blue bear statue at the convention center. If it is raining, we'll likely meet up just inside the convention center building next to the statue to stay dry. Hope to see you there and congratulations Zuul! Clark

1 0

Tagging beta / rc releases
by Paul Belanger 24 Apr '19

24 Apr '19

Greetings, I wanted to raise the topic of maybe using beta / rc tags to publish pre-release versions of zuul / nodepool to pypi. Currently, before each official release we usually select a git SHA to be installed onto zuul.opendev.org, to ensure we can run a scale without any issues. Then a few days later, if things are well we tag a new release or work to address the problem. Then restart the process until things look good. I really like this approach, as currently there is no larger zuul and helps to catch any issues before our users install the released version. However, I would like to propose we make a minor change, is when we select to git SHA, is to also tag it using a pre-release version number (eg: beta / rc). By having it pre-release, (eg: pip install --pre zuul) would allow my current tooling (today we only installed from pypi, not git) to pick up the change and also provide some feedback. Our deployment tooling is setup to only use releases from pypi (wheels) in an effort to try not to carry any patches downstream. We could also switch to pip install via git, however this would add some extra overhead to production servers to compile wheels. - Paul

3 8

Migrating git repos to OpenDev
by corvus＠inaugust.com 21 Apr '19

21 Apr '19

Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org). This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). >From a technical standpoint, it's excellent, but its usability could be improved. Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org), but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org), we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit… [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition

5 6

Post OpenDev Repo Migration Issues
by Clark Boylan 21 Apr '19

21 Apr '19

Hello, There are two quick things we want to point out that we've noticed after the OpenDev Gerrit manuevers. First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship. Second, while we have fairly robust http(s) redirects in place for old urls (including those from cgit to gitea) any ssh repo urls will need to be updated to use the new canonical paths if a repo has moved. We are unable to redirect incoming ssh requests against Gerrit. If you need to know what the canonical path is go to the corresponding https repo url and the resulting destination is the canonical location. Again thank you for your patience, Clark

3 2

Should we drop our custom Ansible plugins?
by corvus＠inaugust.com 15 Apr '19

15 Apr '19

Hi, When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor). To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous. Broadly speaking, there are two classes of actions we might want to block: 1) Actions which may expose private information stored on the executor (including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted). 2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services. We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class. Here's more background on the bubblewrap addition: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-execut… Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here. Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use. Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure. That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility. Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter. Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth. What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins? -Jim

7 19

Upgrading the lists.(openstack|airshipit|starlingx|zuul-ci).org server Friday April 12
by Clark Boylan 12 Apr '19

12 Apr '19

It is that time of the Ubuntu LTS cycle again and we need to upgrade our mailman mailing list server. We'd like to do that this Friday, April 12. We expect the upgrade to begin at about 17:00UTC and result in a 30-45 minute outage. The reason for the extended outage is that we will be upgrading the server in place to preserve its mail reputation. Thankfully email is a persistent system and your clients should queue up email sent until the server is back up again and accepting smtp connections. This means the outage shouldn't be very noticeable. Finally, for list admins, there are new DMARC moderation action settings. We ask that you don't change these settings and instead work with us if you need to address DMARC problems. Our current preference is that we pass email through unmodified so that the signatures still validate. Thank you all for your patience and feel free to ask us questions, Clark

1 1

New Zuul job to replicate a project's git repository to a remote git server
by David Moreau Simard 11 Apr '19

11 Apr '19

Hi, It is now possible for projects to replicate their git repository to a custom location by inheriting from the 'upload-git-mirror' job provided by Zuul. This job wraps around the 'upload-git-mirror' Ansible role that is part of the zuul-jobs library [1]. In order to use this job, you must supply a secret in the following format: === - secret: name: <name of your secret> data: user: <ssh user of the remote git server> host: <address of the remote git server> host_key: <ssh host key of the remote git server> ssh_key: <private key to authenticate with the remote git server> === The 'host_key' parameter can be retrieved from your known_hosts file or with a command like 'ssh-keyscan -H <host>' or 'ssh-keyscan -t rsa <host>'. For example, the 'host_key' when pushing to GitHub would be, on a single line: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== The 'ssh_key' parameter should be encrypted before being committed to the git repository. Zuul provides a tool for easily encrypting files such as SSH private keys and you can find more information about it in the documentation [2]. For example, encrypting a key for the "openstack/ara" project would look like this: === zuul/tools/encrypt_secret.py --infile /home/dmsimard/.ssh/ara_git_key --tenant openstack https://zuul.openstack.org openstack/ara === You would then use the secret in a job inheriting from 'upload-git-mirror' as such: === - job: name: <project>-upload-git-mirror parent: upload-git-mirror description: Mirrors openstack/<project> to neworg/<project> vars: git_mirror_repository: neworg/<project> secrets: - name: git_mirror_credentials secret: <name of your secret> pass-to-parent: true === Finally, the job must be set to run in your project's 'post' pipeline which is triggered every time a new commit is merged to the repository: === - project: check: jobs: # [...] gate: jobs: # [...] post: jobs: - <project>-upload-git-mirror === Note that the replication would only begin *after* the change has merged, meaning that merging the addition of the post job would not trigger the post job itself immediately. The post job will only trigger the next time that a commit is merged. [1]: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-upload-git-mirror [2]: https://zuul-ci.org/docs/zuul/user/encryption.html David Moreau Simard dmsimard = [irc, github, twitter]

1 0

Update on the tenant-scoped admin REST API for Zuul
by Matthieu Huin 11 Apr '19

11 Apr '19

Hello, I'd like to introduce the applications of the spec referred to in the title [1]. I have worked in parallel on a possible implementation of said spec, backend-side [2]. Since it is not necessarily trivial to evaluate or test by oneself, I have also tried to put together a proof of concept for the GUI using a keycloak instance as the JWT provider, with -almost- everything set up together with docker, so that people can play around with the feature. [3] It requires a few manual steps in order to be set up properly though: 1. Build the zuul containers using the code from the patchset: cd zuul/doc/sources/admin/examples && docker-compose up --build 2. log in to the keycloak instance at http://localhost:8282 as admin/admin and create users in the "zuul" realm. The default rules of the PoC expect a user called "venkman", and a user belonging to the "gozerians" group (groups should be created automatically within the zuul realm) 3. The zuul web root page will automatically redirect to keycloak's login page. When back to Zuul, you can check your token and the tenants on which the user is allowed to perform actions, they are logged in the browser's javascript console. If you don't feel like deploying, here is a short screencast of what enqueueing/dequeueing from the GUI could look like: https://softwarefactory-project.io/static/zuul_jwt_demo.mkv Where can you help? The GUI part is in no way production ready, it is tightly coupled to keycloak, and I've started learning React about 2 weeks ago, so the code is terrible and takes a lot of shortcuts. Anybody willing and able to take over is welcome! the spec however is in a state where it is ready for reviews, same for the gerrit topic implementation. Feedback and possibly merging would be greatly appreciated. Cheers, MHU [1] https://review.openstack.org/#/c/562321/ [2] https://review.openstack.org/#/q/topic:zuul_admin_web [3] https://review.openstack.org/#/c/643536/

1 0