- Zuul-discuss - lists.opendev.org

Migrating git repos to OpenDev
by corvus＠inaugust.com 21 Apr '19

21 Apr '19

Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org) This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). >From a technical standpoint, it's excellent, but its usability could be improved. Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org) but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org) we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit… [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition

5 6

Post OpenDev Repo Migration Issues
by Clark Boylan 20 Apr '19

20 Apr '19

Hello, There are two quick things we want to point out that we've noticed after the OpenDev Gerrit manuevers. First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship. Second, while we have fairly robust http(s) redirects in place for old urls (including those from cgit to gitea) any ssh repo urls will need to be updated to use the new canonical paths if a repo has moved. We are unable to redirect incoming ssh requests against Gerrit. If you need to know what the canonical path is go to the corresponding https repo url and the resulting destination is the canonical location. Again thank you for your patience, Clark

3 2

Should we drop our custom Ansible plugins?
by corvus＠inaugust.com 15 Apr '19

15 Apr '19

Hi, When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor). To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous. Broadly speaking, there are two classes of actions we might want to block: 1) Actions which may expose private information stored on the executor (including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted). 2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services. We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class. Here's more background on the bubblewrap addition: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-execut… Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here. Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use. Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure. That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility. Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter. Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth. What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins? -Jim

7 19

Upgrading the lists.(openstack|airshipit|starlingx|zuul-ci).org server Friday April 12
by Clark Boylan 12 Apr '19

12 Apr '19

It is that time of the Ubuntu LTS cycle again and we need to upgrade our mailman mailing list server. We'd like to do that this Friday, April 12. We expect the upgrade to begin at about 17:00UTC and result in a 30-45 minute outage. The reason for the extended outage is that we will be upgrading the server in place to preserve its mail reputation. Thankfully email is a persistent system and your clients should queue up email sent until the server is back up again and accepting smtp connections. This means the outage shouldn't be very noticeable. Finally, for list admins, there are new DMARC moderation action settings. We ask that you don't change these settings and instead work with us if you need to address DMARC problems. Our current preference is that we pass email through unmodified so that the signatures still validate. Thank you all for your patience and feel free to ask us questions, Clark

1 1

New Zuul job to replicate a project's git repository to a remote git server
by David Moreau Simard 11 Apr '19

11 Apr '19

Hi, It is now possible for projects to replicate their git repository to a custom location by inheriting from the 'upload-git-mirror' job provided by Zuul. This job wraps around the 'upload-git-mirror' Ansible role that is part of the zuul-jobs library [1]. In order to use this job, you must supply a secret in the following format: === - secret: name: <name of your secret> data: user: <ssh user of the remote git server> host: <address of the remote git server> host_key: <ssh host key of the remote git server> ssh_key: <private key to authenticate with the remote git server> === The 'host_key' parameter can be retrieved from your known_hosts file or with a command like 'ssh-keyscan -H <host>' or 'ssh-keyscan -t rsa <host>'. For example, the 'host_key' when pushing to GitHub would be, on a single line: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== The 'ssh_key' parameter should be encrypted before being committed to the git repository. Zuul provides a tool for easily encrypting files such as SSH private keys and you can find more information about it in the documentation [2]. For example, encrypting a key for the "openstack/ara" project would look like this: === zuul/tools/encrypt_secret.py --infile /home/dmsimard/.ssh/ara_git_key --tenant openstack https://zuul.openstack.org openstack/ara === You would then use the secret in a job inheriting from 'upload-git-mirror' as such: === - job: name: <project>-upload-git-mirror parent: upload-git-mirror description: Mirrors openstack/<project> to neworg/<project> vars: git_mirror_repository: neworg/<project> secrets: - name: git_mirror_credentials secret: <name of your secret> pass-to-parent: true === Finally, the job must be set to run in your project's 'post' pipeline which is triggered every time a new commit is merged to the repository: === - project: check: jobs: # [...] gate: jobs: # [...] post: jobs: - <project>-upload-git-mirror === Note that the replication would only begin *after* the change has merged, meaning that merging the addition of the post job would not trigger the post job itself immediately. The post job will only trigger the next time that a commit is merged. [1]: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-upload-git-mirror [2]: https://zuul-ci.org/docs/zuul/user/encryption.html David Moreau Simard dmsimard = [irc, github, twitter]

1 0

Update on the tenant-scoped admin REST API for Zuul
by Matthieu Huin 11 Apr '19

11 Apr '19

Hello, I'd like to introduce the applications of the spec referred to in the title [1]. I have worked in parallel on a possible implementation of said spec, backend-side [2]. Since it is not necessarily trivial to evaluate or test by oneself, I have also tried to put together a proof of concept for the GUI using a keycloak instance as the JWT provider, with -almost- everything set up together with docker, so that people can play around with the feature. [3] It requires a few manual steps in order to be set up properly though: 1. Build the zuul containers using the code from the patchset: cd zuul/doc/sources/admin/examples && docker-compose up --build 2. log in to the keycloak instance at http://localhost:8282 as admin/admin and create users in the "zuul" realm. The default rules of the PoC expect a user called "venkman", and a user belonging to the "gozerians" group (groups should be created automatically within the zuul realm) 3. The zuul web root page will automatically redirect to keycloak's login page. When back to Zuul, you can check your token and the tenants on which the user is allowed to perform actions, they are logged in the browser's javascript console. If you don't feel like deploying, here is a short screencast of what enqueueing/dequeueing from the GUI could look like: https://softwarefactory-project.io/static/zuul_jwt_demo.mkv Where can you help? The GUI part is in no way production ready, it is tightly coupled to keycloak, and I've started learning React about 2 weeks ago, so the code is terrible and takes a lot of shortcuts. Anybody willing and able to take over is welcome! the spec however is in a state where it is ready for reviews, same for the gerrit topic implementation. Feedback and possibly merging would be greatly appreciated. Cheers, MHU [1] https://review.openstack.org/#/c/562321/ [2] https://review.openstack.org/#/q/topic:zuul_admin_web [3] https://review.openstack.org/#/c/643536/

1 0

Proposal to create zuul-operator project
by Monty Taylor 08 Apr '19

08 Apr '19

Hey everybody, It seems that kubernetes operators are all the rage these days, and in the last 2 weeks I've had multiple conversations that have all come back around to someone wanting a zuul operator to exist. SO ... I would like to propose that we create a zuul-operator project as part of Zuul. Given our close ties with Ansible, I think we should use ansible-operator [0] to write it. Since we have zuul in production on both vanilla k8s and on OpenShift, I think we should set up gate jobs to make sure it works on both. Once we're happy with it - we should publish it to operatorhub. [1] Does anyone object? It not, I'll get a repo started. Monty [0] https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/… [1] https://www.operatorhub.io/

9 23

New driver for Nodepool
by Alexander Braverman Masis 03 Apr '19

03 Apr '19

Hello, I'm interested in writing a new driver for Nodepool based on Linchpin[1]. Should I contribute it to zuul core as a built in driver, or add a configuration option so the driver can be developed separately and run from out-of-tree? Thanks, [1] https://github.com/CentOS-PaaS-SIG/linchpin -- Alex Braverman Masis

2 1

PTG Team Photos
by Kendall Nelson 28 Mar '19

28 Mar '19

Hello! If your team is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

1 0

Zuul project update 2019-03-25
by corvus＠inaugust.com 25 Mar '19

25 Mar '19

Hi, Since the last update, Tristan has updated the zookeeper auth patch based on recent comments, rebased the zuul-runner patch on the multi-ansible stack, and Investigated how to enable the executor service to use a COE. I expect we'll have a good discussion about that once Tristan's findings are ready. Fabian worked on a Elasticsearch Zuul reporter to enable indexation of build and buildset data. It enable building analytics dashboards on top of Kibana for instance. I reviewed and helped merge Tobias's work on multi-ansible. We upgrade OpenDev's Zuul, released 3.7.0, found some bugs, fixed them, and I plan on releasing 3.7.1 today. Next I plan on reviewing the zuul-runner series (as outlined in the previous update email). The remainder of the backlog from the last update email: 1) The zuul-runner series This has been in development for a long time, and has a significant impact on the executor. Now that the multi-ansible series has merged and it has been rebased, let's focus on this next. 2) The web authentication spec This has been in development for a while, and folks working on it recently signaled it's ready for review. This shouldn't conflict with anything ahead of it, so we can review it as time permits. 3) The web authentication series Once the spec lands, we can review the implementation which has been developing in parallel. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0