- Zuul-discuss - lists.opendev.org

Projects config introspection
by Tristan Cacqueray 11 Dec '18

11 Dec '18

Hi all, I'd like to start the discussion around introspecting projects configs. The goal is to be able to efficiently answer the following questions: * Which projects use a specific job, * Which jobs use a specific nodeset, * Which nodesets or jobs use a specific label, and * Which jobs, labels or nodesets are not used. Putting aside the job variants model (which isn't well exposed in the current REST APIs), the easiest solution would be to add new endpoints, such as /api/job-per-project/{job_name}, or /api/label-per-project/{label_name}. But this feels sub-optimal and we need a new route for each type of query. Another solution would be to add a global toDict() method for the whole Tenant configuration object, which can be exposed at /api/config (or simply dumped using a zuul-scheduler command line argument). This would enable custom search to be done client-side. Another solution would be to implement the query using GraphQL custom resolver as started in https://review.openstack.org/574625 Regards, -Tristan

8 14

Adding an unconditional cleanup job phase
by Brendan Jackman 11 Dec '18

11 Dec '18

Hi all, Looking, for the moment, for lower-hanging fruit in my quest to reduce test resource pressure with Zuul. One unfortunate hack in my use-case is that we currently lock and release test resources which are shared with another system. This is handled by an ansible role that sends API calls to that other system. This means summarily killing a running job can prevent a resource from getting unlocked. To get around this I've had to disable pipeline.dequeue-on-new-patchset. I'd like to add stage where playbooks can be added that still gets run even if the job gets stopped. My guess is that this is also useful to other people who are using the static nodepool driver and would like to unconditionally clean up files left behind by jobs - does that sound correct? It generally seems that there would be value in having and unconditional cleanup stage, anyway. If you agree so far, can we talk semantics & implementation? I think it would be sensible that these playbooks are _never_ killed, but that obviously poses a risk that users can get stuck with an unkillable job running. I guess a simple solution is to have a distinct timeout from the job timeout, which covers just the running of these playbooks. We recommend setting this pretty short (default 30 secs, say), and strictly minimising the logic in these playbooks. If I'm still making sense then from what I've read so far, it seems all the complexity here can just go in AnsibleJob - the gearman communication would already support this. The client would just send the executor:stop TextJob as normal, and then the server can return its response asynchronously when the cleanup playbooks have finished? How does that sound? Cheers, Brendan

2 3

Roadmap for running jobs on OpenShift
by Tristan Cacqueray 10 Dec '18

10 Dec '18

Hello everyone, Please find below a list of topics I think we should address in order to land the OpenShift driver. Kube config integration in zuul-executor ---------------------------------------- As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668 The Openshift drivers --------------------- https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver. Inventory update by job ----------------------- We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092 Roles and parent jobs --------------------- The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community. Define the security context --------------------------- It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe. How many users are interested in these topics and what are your review bandwidths? Can we get an estimated release date for OpenShift users? Regards, -Tristan

7 16

[Newsletter] What's Happening in the Open Infrastructure Community
by Claire Massey 05 Dec '18

05 Dec '18

Hi everyone, This week, the OSF team distributed the first Open Infrastructure Community Newsletter. Our goal with the bi-weekly newsletter is to provide a digest of the latest developments and activities across open infrastructure projects, events and users. This week, the newsletter highlighted the Berlin Summit as well as brief updates from Zuul and other OSF projects - OpenStack, Airship, Kata Containers and StarlingX. You can checkout the full newsletter on Superuser [1], and if you are interested in receiving the upcoming newsletters, you can subscribe here [2]. If you would like to contribute to a future newsletter or have feedback, please reach out to community(a)openstack.org <mailto:community@openstack.org>. Thanks, Claire [1] http://superuser.openstack.org/articles/inside-open-infrastructure-newslett… <http://superuser.openstack.org/articles/inside-open-infrastructure-newslett…> [2] https://www.openstack.org/community/email-signup <https://www.openstack.org/community/email-signup>

1 0

Software Factory 3.2 released
by Tristan Cacqueray 05 Dec '18

05 Dec '18

We are pleased to announce the release of Software Factory version 3.2: https://www.softwarefactory-project.io/releases/3.2 This new release consolidates the 3.1 version with updated packages, minor fixes and some new features such as an improved update mechanism. Software Factory version 3.2 integrates Zuul and Nodepool version 3.3.1. Here is the list of features we picked in the Software Factory version of Zuul and Nodepool that are still under review upstream. We are confident these features will eventually land upstream, but they might evolve by then. They should therefore be considered a "tech preview", although we will do our best to integrate these in the least disruptive way possible: * Improved dashboard: https://review.openstack.org/#/q/topic:react-zuul * Openshift driver: https://review.openstack.org/#/q/topic:container-build-resources * RunC driver: https://review.openstack.org/535556 The Nodepool package also contains some backports to mitigate stability issues found in the (upstream) version 3.3.1: * Set pool for error'ed instances: https://review.openstack.org/621681 * Set type for error'ed instances: https://review.openstack.org/622101 * Log exceptions deleting ZK nodes: https://review.openstack.org/621301 * Fix race when deleting Node znodes: https://review.openstack.org/622403 * Count leaked nodes in unmanaged quota: https://review.openstack.org/621040 * Cleanup routine to delete empty nodes: https://review.openstack.org/622616 Best regards, The Software Factory Team

2 1

Zuul project update 2018-12-03
by corvus＠inaugust.com 03 Dec '18

03 Dec '18

Hi, It's been a while since I've sent one of these -- mostly due to all the conferences and holidays which seemed to all happen at once. There's a lot going on: I'm working on support for dynamically adjusting node request priorities. Because of the current heavy load on OpenStack's Zuul system, this should help us get results faster on changes to Zuul. I also have changes in progress to add support for storing information about artifacts (but not the artifacts themselves) in the SQL database. That will make it easier for jobs to store artifacts in arbitrary locations and have later jobs use those artifacts. In particular, this is in service of the idea of a "promotion" pipeline. Finally, I'll be proposing a formal governance structure for the project this week -- something I've been drafting for a while. Paul has been adding support for executor zones -- a long-standing feature request. This allows admins to locate executors near to the node resources they use. Shrews added support for generic node attributes via config file to nodepool (mainly supporting the executor zone feature). Monty is continuing work on refactoring of nodepool TaskManager across openstacksdk and keystoneauth libraries, and is also working with tristanC towards supporting both statsd and prometheus for metrics collection. There is a new project "promstats" as part of this effort. Tobias has been working on zk caching in nodepool, and per project and tenant usage statistics, and a spec for a scale-out scheduler. Tristan has proposed a roadmap for Openshift support, started discussion about config introspection and worked on a web ui to handle generic API errors. Clint is working on producing reliable unit test(s) for AWS/EC2 driver. Thanks everyone! -Jim

1 0

Road to fully scale out Zuul
by Tobias.Henkel＠bmw.de 02 Dec '18

02 Dec '18

Hi Zuulers, currently the zuul-scheduler is the only component of zuul that is not scale out ready and thus a single point of failure. We always had the goal to make the scheduler scale out ready at some point in time. In order to make this more concrete so we can start to work on this I tried to transform my thoughts on this into a spec: https://review.openstack.org/621479 I'd be interested in what you think about this. Cheers, Tobias

1 0

Testing multiple commits in one job with Gerrit
by Brendan Jackman 28 Nov '18

28 Nov '18

Hi all, I'm gradually setting up Zuul for my org, and basically I have just one impedance mismatch between Zuul's behaviour and my requirements: Due to pressure on our test hardware I don't think running full end-to-end tests on every individual commit will scale. What I'd ultimately like is a check/gate setup where jobs in the check pipeline runs on every single commit, but jobs in the gate pipeline only run once for an entire "stack" of commits, and then that entire stack can be merged at once. IIUC this is basically the behaviour you get already with GitHub, where the zuul notion of "change" maps to a whole Pull Request rather than an individual commit. I started looking into ways something similar can be achieved with Gerrit. After some comments from #zuul and starting at the code a bit this strategy is the one I'm considering: enable one Zuul change to encapsulate multiple Gerrit changes. (Gerrit and Zuul both use the word "change", which works because Gerrit changes and Zuul changes are currently 1:1. That wouldn't be the caes any more with my proposal so I'll always say "Gerrit change" or "Zuul change"). I haven't figured out exactly how the Gerrit changes should be grouped together, and how this should be configured in the Gerrit trigger, but I think the basic idea would be to inject extra events into the Gerrit event stream (somewhere around GerritEventConnector._handleEvent). One example might be that we detect when an entire stack of Gerrit changes has received some common set of approvals, we inject an event called (say) "zuul-change-stack-approved". The user can then have a pipeline.trigger.gerrit entry with this event, and that trigger would create a Zuul change encapsulating the entire stack of Gerrit changes. These wouldn't need to actually be treated in the code like events coming from Gerrit, but this basic idea would allow expressing this behaviour with the existing structure of the Gerrit trigger configuration. I think it would then be a matter of finding the places where the Scheduler and pipeline Managers interact with the Gerrit driver using references to Zuul changes, and ensuring that those parts of the Gerrit driver can handling being passed the new class of Zuul change. I think most of these would involve checking for if the Zuul change is of the new type and then iterating over the constituent Gerrit changes to performe some action against each (e.g. GerritConnection.canMerge, GerritRefFilter.matches, and, with a little more complication, GerritReporter.report). Does that seem like a reasonable approach to the problem? Cheers, Brendan

3 6

Multi-repo merge
by Mike Pagel 27 Nov '18

27 Nov '18

Zuul is great in gating projects consisting of many repositories. It even supports dependencies of changes within a single or across multiple repositories. Our project is making heavy use of this. Every now and then we are facing dependencies that unfortunately are not unidirectional. I know, we should have better thought about clean repo structures in the first place, but one not too artificial example is a situation where repositories are structured to express ownership, visibility and write access. In Github for instance, all this is configured at the repository level. If you like see the sample how such a situation could arise. --- Start of sample --- Assume company A is the owner of repository A, holding operating system and a top-level application framework (the "main" function in C). Further assume company B is the owner of repository B, implementing a module on top of the operating system, and the module is invoked by A's top level application. At the build level the dependencies are clear and well defined, but at the repo level they are bidirectional. Some changes in A and B will depend on each other. It turns out that a clean, backward compatible PR design, leading to unidirectional changes as required by zuul is almost always possible, but not always easy. E.g. think of a change where A replaces a BSP in their operating system, which requires adaptation in B. It might not be possible to temporarily have two BSPs in A, the the change to A will break the build. Nor can B simply switch its usage of APIs, as that, by itself would break the build as well. The only chance to get this merged would be to temporarily disable some checks, allow for the first merge, then reenable and merge the second change. Ugly. --- End of sample --- It would very much simplify our workflows if an "atomic" merge (from zuul's point of view) would be possible. I can think of some ways to sort of realize this with zuul, but none of them is really safe, even if zuul is the only party executing merges to protected branches. What would be your suggestion to accommodate such a requirement? Thanks a lot for any insight and ideas you might have. Best, Mike

6 12

Zuul CI and Gitlab
by Алексей Степаненко 26 Nov '18

26 Nov '18

Hi. At the summit in Berlin was interested in how the Zuul CI system. GitHub support has been announced, but we're using GitLab. Will there be added support for Networks in the future ? -- Aleksei Stepanenko, Selectel Team

1 0

Semaphores and resources
by Tobias.Henkel＠bmw.de 26 Nov '18

26 Nov '18

Hi Zuulers, We just discovered an (in our case) unpleasant behaviour of zuul and semaphores. Scenario: We have a job ‘foo’ with a semaphore of 5. Now the queue grows and we have 30 items with job ‘foo’ in the queue. The current behaviour of zuul is that it first gets the nodes and then locks the semaphore. This means that 5 jobs run and 25 wait for the semaphore but already got and locked their nodes. In case the job is expensive and long running this can become quite a big resource waste problem if we’re in a quota constrained situation. After some discussion on IRC this would be a proposal to solve this problem: We could make the order of ‘request nodes’ and ‘lock semaphore’ configurable per job with a default to the safe behaviour of ‘lock semaphore’ then ‘request nodes’. Jobs would still have the possibility to do the fast path by requesting the nodes first. This makes sense if you have lightweight jobs that should run in a fast consecutive sequence and you know that some waste of resources is ok. What do the other semaphore users out there think about this? Kind regards Tobias

5 7

Dequeuing buildset on first error
by Tobias.Henkel＠bmw.de 19 Nov '18

19 Nov '18

Hi Zuulers, We have several projects that have buildsets with fast (few minutes, e.g. linters) and longer running (> 40 minutes, e.g. big fat c++ compilation) jobs. We often see the case that one of the linter jobs failed and already could give feedback. However zuul waits for the completion of the fat compile job before reporting. Generally this is the desired behaviour because we want to give as much feedback as possible. However some projects may want to decide that quick feedback is more important than the most complete feedback possible. So I would propose to add a ‘report-on-first-failure’ flag to the project-pipeline. The default would be false in order to retain the current behaviour. If the deque happens the aborted jobs will have the ABORTED state so it can be clearly seen in the reporting that these were aborted. Enabling this for certain projects also could help to reduce resource consumption in quota-constrained periods. What do you think about this? Kind regards Tobias

7 6

Slides from Tungsten talk on Berlin Summit
by Jarek Łukow 15 Nov '18

15 Nov '18

Hi all, As promised, I'm sending the link to our slide deck: https://wiki.tungsten.io/download/attachments/1409104/Zuul_as_a_build_syste… Regards, -- Jarosław Łukow DevOps Engineer jaroslaw.lukow(a)codilime.com -- ------------------------------- The information in this email is confidential and may be legally privileged, it may contain information that is confidential in CodiLime Sp. z o. o. It is intended solely for the addressee. Any access to this email by third parties is unauthorized. If you are not the intended recipient of this message, any disclosure, copying, distribution or any action undertaken or neglected in reliance thereon is prohibited and may result in your liability for damages.

2 1

OpenDev, the future of OpenStack Infra
by Clark Boylan 08 Nov '18

08 Nov '18

Hello everyone, Sorry for another cross post so soon. In the land before time we had Stackforge. Stackforge gave non-OpenStack projects a place to live with their own clearly defined "not OpenStack" namespacing. As the wheel of time spun we realized that many Stackforge projects were becoming OpenStack projects and we would have to migrate them. This involved Gerrit downtimes to rename things safely. To ease the pain of this, the TC decided that all projects developed in the OpenStack Infrastructure could live under the OpenStack git namespace to simplify migrations. Unfortunately this had the effect of creating confusion over which projects were officially a part of OpenStack, and whether or not projects that were not OpenStack could use our project hosting. Stackforge lived on under a different name, "unofficial project hosting", but many potential infrastructure users either didn't understand this or didn't want that strong association to OpenStack for their project hosting [0]. Turns out that we want to be able to host OpenStack and non-OpenStack projects together without confusion in a way that makes all of the projects involved happy. In an effort to make this a reality the OpenStack Infra team has been working through a process to rename itself to make it clear that our awesome project infrastructure and open collaboration tooling is community run, not just for OpenStack, but for others that want to be involved. To this end we've acquired the opendev.org domain which will allow us to host services under a neutral name as the OpenDev Infrastructure team. The OpenStack community will continue to be the largest and a primary user for the OpenDev Infrastructure team, but our hope in making our infrastructure services more inclusive is that we'll also attract new contributors, which will ultimately benefit OpenStack and other open infrastructure projects. Our goals for OpenDev are to: * Encourage additional active infrastructure contributors to help us scale. Make it clear that this is community-run tooling & infrastructure and everyone can get involved. * Make open source collaboration tools and project infrastructure more accessible to those that want it. * Have exposure to and dogfooding of OpenStack clouds as viable open source cloud providers. * Enable more projects to take advantage of the OpenStack-pioneered model of development and collaboration, including recommended practices like code review and gating. * Help build relationships with new and adjacent open source projects and create an inclusive space for collaboration and open source development. Much of this is still in the early planning stages. This is the infrastructure team's current thinking on the subject, but understand we have an existing community from which we'd like to see buy-in and involvement. To that end we have tried to compile a list of expected FAQ/Q&A information below, but feel free to followup either on this thread or with myself for anything we haven't considered already. Any transition will be slow and considered so don't expect everything to change overnight. But don't be surprised if you run into some new HTTP redirects as we reorganize the names under which services run. We'll also be sure to keep you informed on any major (and probably minor) transition steps so that they won't surprise you. Thank you, Clark [0] It should be noted that some projects did not mind this and hosted with OpenStack Infra anyway. ARA is an excellent example of this. FAQ * What is OpenDev? OpenDev is community-run tools and infrastructure services for collaboratively developing open source software. The OpenDev infrastruture team is the community of people who operate the infrastructure under the umbrella of the OpenStack Foundation. * What services are you offering? What is the expected timeline? In the near-term we expect to transition simple services like etherpad hosting to the OpenDev domain. It wil take us months and potentially up to a year to transition key infrastructure pieces like Git and Gerrit. Example services managed by the team today include etherpad, wiki, the zuul and nodepool CI system, git and gerrit, and other minor systems like pbx conferencing and survey tools. * Where will these services live? We've acquired opendev.org and are planning to set up DNS hosting very soon. We will post a simple information page and FAQ on the website and build it out as necessary over time. * Why are you changing from the OpenStack infrastructure team to the OpenDev infrastructure team? In the same way we want to signal that our services are not strictly for OpenStack projects, and that not every project using our services is an official part of OpenStack, we want to make it clear that our team also serves this larger community. * Who should use OpenDev services? Does it have to be projects related to OpenStack, or any open source projects? In short, open source contributors who share our community values, especially those who might want to help contribute to improving and maintaining OpenDev infrastructure over time. Projects using OpenDev hosted git and gerrit services should have an OSI-approved license. * Will the OpenStack projects live at git.opendev.org? All projects hosted with OpenDev will live at git.opendev.org. For backwards compatibility reasons, at the very least git.openstack.org will be an alias for git.opendev.org for the forseeable future. The same is true of the other existing whitelabel git domains such as git.starlingx.io and git.zuul-ci.org. Whether or not other 'whitelabel' domains are created is an open question. Given a neutral domain name, the desire for such sites may not seem as necessary. * Does this mean the infrastruture team will be spending less time on OpenStack? OpenStack will continue to be the largest and a primary user for the OpenDev Infrastructure team, and we expect that our work will benefit all users. There will be additional effort required as we transition to the new namespace and reorganize, but over the long term we hope this inclusive approach will help us attract new contributors and ultimately benefit OpenStack. * Are OpenStack cloud test resources the only resources that will be used? At the present time all of the donated resources come to us from a combination of OpenStack Public and Private clouds. Nobody from any of the proprietary clouds has asked to donate resources to us. It is conceivable that the shift to OpenDev could open the door to those cloud providers wanting to donate some cloud resources. Assuming nodepool supports talking to those clouds, it is certainly a possibility, but at the moment it's all speculation. * Is this name associated with the OpenDev Conferences (opendevconf.com) that OpenStack Foundation has previously organized? Yes! They are related, albeit indirectly. To quote from the conference's promotional site, "the focus is on bringing together composable open infrastructure technologies across communities and industries." The possibility of cross-promotional tie-ins could prove synergistic, since the conference and the collaboratory we're building share a lot of similar values and goals, and are ultimately supported by the same donors, community and foundation. * How will OpenDev be governed? Will the OpenStack TC retain oversight over it? The OpenDev governance discussion is just getting started, but like all OSF-supported initiatives, OpenDev follows the Four Opens, so it will ultimately be directly governed by OpenDev contributors. While it won't be under the sole oversight of the OpenStack Technical Committee anymore, OpenDev users (in particular OpenStack) should be represented in the governance model so that they can feed back their requirements to the OpenDev team.

1 0

Community Infrastructure Berlin Summit Onboarding Session
by Clark Boylan 05 Nov '18

05 Nov '18

Hello everyone, My apologies for cross posting but wanted to make sure the various developer groups saw this. Rather than use the Infrastructure Onboarding session in Berlin [0] for infrastructure sysadmin/developer onboarding, I thought we could use the time for user onboarding. We've got quite a few new groups interacting with us recently, and it would probably be useful to have a session on what we do, how people can take advantage of this, and so on. I've been brainstorming ideas on this etherpad [1]. If you think you'll attend the session and find any of these subjects to be useful please +1 them. Also feel free to add additional topics. I expect this will be an informal session that directly targets the interests of those attending. Please do drop by if you have any interest in using this infrastructure at all. This is your chance to better understand Zuul job configuration, the test environments themselves, the metrics and data we collect, and basically anything else related to the community developer infrastructure. [0] https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22950/i… [1] https://etherpad.openstack.org/p/openstack-infra-berlin-onboarding Hope to see you there, Clark

1 0

Node request priorities
by corvus＠inaugust.com 31 Oct '18

31 Oct '18

Hi, In OpenStack, we occasionally have periods where, for various reasons, one project, or a group of projects, uses a sufficiently large number of node resources that it significantly delays results for other projects. We happen to be in one of those periods, so I've had some time to think about ways to mitigate this. First, let me say that one of the potential causes is long and unreliable tests. There are efforts underway to communicate that and hopefully address it. But that topic isn't really for this list. Our concern is, when a system gets into this state, can or should we do anything in Zuul to mitigate it? I would like to avoid (for now) the use of project-based quotas. I could certainly see us adopting tenant quotas in the future, but that doesn't help us in this case where all of the contention is within a single tenant. Within a tenant, project quotas just seem to be too much work. I wouldn't even begin to know how to set them fairly. And what we really want to do is use as many node resources are available, but use them in a fair manner. I think we can accomplish this by augmenting our use of node request priority. Currently, we use a priority queue with very coarse granularity (high, normal, low) set by Zuul according to pipeline. I'd like to adjust the system so that each subsequent node request from a given project has a lower priority. In other words, at any given time, the first change from project A in gate has the highest priority, and the same is true for project B. The second change from A has a slightly lower priority, and so forth. The problem with this is that the node request priorities much change over time as the contents of the pipeline queues change. Once the first change from A merges, the second change from A should now have the highest priority. That makes this incompatible with our current priority queue system. Fortunately, Tobias has been working on adding caching to Nodepool's use of Zookeeper. If we start having node requests cached in-memory in the launchers, the in-memory records can be automatically updated by Zookeeper callbacks. This means that we can encode our additional priority information on an attribute in the Znode, updating it as necessary, and combine it with the already existing information from the priority queue to determine the order in which to fulfill requests. To summarize, I propose the following: * The current priority queue system for node requests will remain in place. I'll call that "precedence" to distinguish it from the new bits. * Add a "priority" field to the node request znodes in Zookeeper. This is an integer which indicates the priority of this request relative to others at the same pipeline precedence level. * Zuul will update the priority field after each time a pipeline queue processor runs, according to internal heuristics. I propose that initially we decrement the priority for each subsequent change from the same project, though we may also add an option to do so by project-group, as determined by membership in a shared change queue. * We add temporary feature flags to Zuul so that we can try out these new features (including different options for how to determine the priority) before committing to future support. Whatever we decide to keep, we may also want to make optional, so these flags may eventually turn into permanent configuration flags. * We implement a TreeCache for node requests in nodepool. * When nodepool begins to iterate over all node requests in order to handle them, it first sorts by precedence, then priority. Items with the same precedence and priority will remain in their chronological order of submission. I think that will help make node distribution more fair, and it builds on top of the current system without changing it so it should be forward/backward compatible, and even optional. -Jim

5 8

Discussion between OSF projects in Berlin
by Thierry Carrez 31 Oct '18

31 Oct '18

Hi! For those who will be in Berlin for the Summit, please mark your calendars for the following Forum session: https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22815/c… This session is an opportunity for the technical leadership of all OSF projects to meet, discuss pain points, compare experiences and explore synergies. We expect to have members of the OpenStack, Kata Containers, StarlingX, Airship, and Zuul communities around. It will be the occasion, among other things, to push for Zuul adoption all around! If you can make it, please join us ! -- Thierry Carrez (ttx)

1 0

Announcing the First Release of StarlingX, an open source edge computing platform
by Ildiko Vancsa 24 Oct '18

24 Oct '18

Hi, We would like to draw your attention to StarlingX[1], a new independent, top-level, open source pilot project that's supported by the OpenStack Foundation. StarlingX joins other pilot projects hosted at OpenStack Foundation[2], including Airship, Kata Containers and Zuul. Today the first release of StarlingX is here! We invite you to participate in getting the word out that the release is ready and that we’re eager to welcome more contributors to this project. Learn more about it: • Read more about the project at starlingx.io • Listen to a recording of the onboarding webinar[3] • On-boarding slide deck[4] • Overview document[5] Some things you can share: • A blog on starlingx.io[6] • Social sharing: Announcements on Twitter[7] Want to get involved in the community? • Mailing Lists[8] • Weekly Calls[9] • Freenode IRC: #starlingx channel[10] Ready to dive into the code? • You can get download the first release at git.starlingx.io • StarlingX Install Guide[11] • StarlingX Developer Guide[12] If you’re at the Berlin Summit November 13-15[13]: Tuesday 11/13 • StarlingX – Project update – 6 months in the life of a new Open Source project with Brent Rowsell & Dean Troyer • StarlingX CI, from zero to Zuul with Hazzim Anaya & Elio Martinez Wednesday 11/14 • Keynote spotlight on the main stage with Ian Jolliffe & Dean Troyer • MVP (Minimum Viable Product) architecture for edge - Forum session • "Ask Me Anything" about StarlingX - Forum session • StarlingX Enhancements for Edge Networking presentation with Kailun Qin, Ruijing Guo & Dan Chen • Project Onboarding session with Greg Waines • Integrating IOT Device Management with the Edge Cloud - Forum session Thursday 11/15 • Containerized Applications' Requirements on Kubernetes Cluster at the Edge - Forum session Check out the materials to learn about the project, try out the software and join the community! We hope to see many of you in Berlin! Ildikó [1] https://www.starlingx.io/ [2] https://www.openstack.org/foundation/ [3] https://www.youtube.com/watch?v=G9uwGnKD6tM&t=232s [4] https://www.starlingx.io/collateral/StarlingX-Onboarding-Deck-Web.pdf [5] https://www.starlingx.io/collateral/StarlingX_OnePager_Web-102318pdf/ [6] https://www.starlingx.io/blog/starlingx-initial-release.html [7] https://twitter.com/starlingx [8] http://lists.starlingx.io/cgi-bin/mailman/listinfo [9] https://wiki.openstack.org/wiki/Starlingx/Meetings [10] https://freenode.net/ [11] https://docs.starlingx.io/installation_guide/index.html [12] https://docs.starlingx.io/developer_guide/index.html [13] https://www.openstack.org/summit/berlin-2018

1 0

Zuul webinar recording
by Jarek Łukow 17 Oct '18

17 Oct '18

Hi, I want to let you know that on Tuesday, me and my friend Lukas conducted an introductory webinar on Zuul. You can find the recording here: https://crowdcast.io/e/zuul (There will be also an upload without the registration wall, once our media team cuts the video a bit for publishing on YouTube). After a short presentation we've made a tutorial/demo that demonstrated integration with GitHub, Zuul status UI, multiproject jobs and "Depends-On" functionality between two projects on GH. >From the technical side, we prepared an all-in-one Zuul install (scheduler, executor, merger, web, Gearman, Zookeeper, nodepool-launcher and Apache for logs) using Docker compose, very similar to the one from https://review.openstack.org/#/c/608344/. Then we connected nodepool to a Vexxhost project and created an GitHub app attached to three repos: config project and two untrusted projects for the demo. The demo includes also the new refreshed UI - which we've built right before going live, with fingers crossed, but it went smoothly ;). We also have the tutorial scenario available here: https://jluk-codi.github.io/zuul-webinar/. Our small Zuul setup serving pull requests for the sample GitHub project will be available through the weekend (we’ll remove it Monday CEST when we get to work). This was our first webinar and a kind of a "test drive", but I expect we'll have more opportunities to help bring some attention to Zuul. Best, Jarek -- ------------------------------- The information in this email is confidential and may be legally privileged, it may contain information that is confidential in CodiLime Sp. z o. o. It is intended solely for the addressee. Any access to this email by third parties is unauthorized. If you are not the intended recipient of this message, any disclosure, copying, distribution or any action undertaken or neglected in reliance thereon is prohibited and may result in your liability for damages.

3 3

Oct 24 OSF Community Mtg: StarlingX First Release
by Claire Massey 15 Oct '18

15 Oct '18

Hey Zuul team, For the upcoming OpenStack Foundation Community Meeting[1] we will focus on the StarlingX project and edge computing. Please join us on *October 24 at 8:00am PDT* for a StarlingX community on-boarding webinar that's scheduled to coincide with the first release for the StarlingX project. Ian Jolliffe and Bruce Jones will lead the project on-boarding presentation followed by Q&A. Please mark your calendars and plan to join us. Meeting link: https://zoom.us/j/112003649. More information and news around the StarlingX first release will be shared on October 24. In the meantime you can learn more and get involved with StarlingX though theses channels: • Website: starlingx.io • Git: git.starlingx.io • Docs: docs.starlingx.io • Freenode IRC: #starlingx • Mailing Lists: lists.starlingx.io Thanks, Claire [1] https://etherpad.openstack.org/p/openstack-community-meeting <https://etherpad.openstack.org/p/openstack-community-meeting>

1 0

Zuul Project Update 2018-10-08
by corvus＠inaugust.com 15 Oct '18

15 Oct '18

Hi, We've had a number of events that were nearly back-to-back, so it's been a while since I've sent one of these. I think it's a good time to catch up. Several of us just returned from AnsibleFest where the Zuul project had a community sponsor booth, and Clint, Paul, and Ricardo gave presentations. Our presence at the event was an enormous success, exceeding everyone's expectations. Ashlee and Claire from the Foundation set up the booth and Monty, Clark, Paul, Clint and I spent as much time with them there as possible. We spent the entire two days of the event talking to people -- our voices were hoarse by the end. When we find the right audience, there's a huge amount of interest in Zuul. Unfortunately, a lot of those people are about to go try out Zuul and follow our setup instructions, which we all know aren't complete. There was a lot of interest (between potential users and the Zuul developers at AnsibleFest) in having a container-based quickstart. This is also something we've discussed in the past. The idea is to bootstrap a complete Zuul system in a sandbox environment with one command, and walk users through the process of setting up initial jobs. I think all of the pieces are in place for that now, so I've begun working on that. That's my highest priority for this week -- I'd like to finish that and land it ASAP so that any potential new users find that and have a good experience. I have a WIP patch at https://review.openstack.org/608344 and would appreciate it if folks would start looking at it and trying it out. Other news from AnsibleFest: Jesse is finishing up work on adding support for the GitHub checks api to github3.py; once that is complete, we should be able to support it in Zuul. We met a large number of folks who are interested in GitLab, one of whom has volunteered to work on implementing support for it. He's still working on getting up to speed on Zuul development, but hopefully will be joining us soon. I'm sure I'm forgetting other very important things from AnsibleFest, but other folks were there and can chime in. :) Monty and I found (via reports from new users at AnsibleFest and my work on the container quickstart) that we were not including the built javascript in either our pypi uploads or our container images. I believe Monty has either fixed those, or has patches in progress to fix them. We're ready to land the patch to convert the dashboard to react; I expect us to do that soon now that more of us are around again. David and Tristan are working on testing for the Nodepool kubernetes and OpenShift patches. The patches themselves look good; once we have testing in place I think we'll be comfortable merging them. David and Clint have been working on the ec2 driver for Nodepool. Once it is a little more complete and has some testing (likely something fake or mock-based) we can merge it. There was, as you might expect, some interest in that at AnsibleFest as well. Matthieu has a spec and code up regarding using JWT for authenticated actions in the web interface. I'm hoping that after we clear the quickstart docs and container drivers from our plate, we'll have time to take that up in earnest. I'm sure I'm missing things, but this is pretty long already. Feel free to reply with more. I plan to send out updates regularly again, so if you have a moment to update https://etherpad.openstack.org/p/zuul-update-email please do so. -Jim

10 13

Toggle gzip in upload-logs
by Paul Belanger 12 Oct '18

12 Oct '18

Greetings, I've proposed a patch to upload-logs[1] that will toggle the compressing of our console logs files. The main change is, by default we no longer compress them. Jobs would need to set zuul_log_compress: true to work as we do today. The main reason for this, is to reduce the amount of extra work a user needs to view our console logs in apache / nginx. Today, if compressed, the browser will try to download the file not render it. I think we could add documentation explaining how to enable compression and which settings would be needed for apache / nginx but in this case, seems like an optimization over necessity. Thoughts? [1] https://review.openstack.org/567696/

3 4

OpenStack Summit Forum Submission Process Extended
by Jimmy McArthur 28 Sep '18

28 Sep '18

Hello Everyone We are extended the Forum Submission process through September 30, 11:59pm Pacific (6:59am GMT). We've already gotten a ton of great submissions, but we want to leave the door open through the weekend in case we have any stragglers. Please submit your topics here: https://www.openstack.org/summit/berlin-2018/call-for-presentations If you'd like to review the submissions to date, you can go to https://www.openstack.org/summit/berlin-2018/vote-for-speakers. There is no voting period, this is just so Forum attendees can review the submissions to date. Thank you! Jimmy

1 0

Spec for a tenant-scoped admin web API
by Matthieu Huin 26 Sep '18

26 Sep '18

Hello, Following Jim's example with the containers spec, I've opened a spec review in the zuul repository: https://review.openstack.org/562321 I've been toying around with adding a few "admin" endpoints to zuul and nodepool for some time now [1][2]. Using Software Factory's authorization engine to limit access to these endpoints to specific users, and consuming germqtt's event feed to react to comments on a review, I demonstrated PoCs for features that I think would be of interest to Zuul users in general: * Ability to re-enqueue builds from zuul's web UI * Ability to trigger a autohold by commenting "autohold <job> on <tenant>" on gerrit review The aforementioned patches were meant as experiments; now I would like to go further and work to have this integrated into Zuul in a way that would be tenant-scoped. Before I rework these extensively I seek your guidance and opinions on how to proceed with this; hence this spec review. There are surely things I have overlooked, and my proposal to use groups to implement policies might need discussing. Please let me know what you think, also if it is the right place for the spec (the review on infra-specs is still open) MHU [1] https://review.openstack.org/536319 [2] https://review.openstack.org/539004

1 1

Feedback on Zuul Overview One Pager
by Claire Massey 26 Sep '18

26 Sep '18

Hi Zuul team, We'll have a Zuul branded booth at AnsibleFest in Austin, October 2-3. More information about our booth presence and how you can participate will be shared in the coming weeks. For now we’re working on creating all of the branded materials needed for the booth, including a one page piece of collateral that gives a nice overview of Zuul. A working draft of the one pager is available here: https://etherpad.openstack.org/p/Zuul_one_pager <https://etherpad.openstack.org/p/Zuul_one_pager>. If you have any feedback or edits on the copy, please provide it in the etherpad by next Monday, August 20. The completed document will end up looking polished like the Kata Containers overview one pager shown at https://katacontainers.io/media/uploads/katacontainers/uploads/katacontaine… <https://katacontainers.io/media/uploads/katacontainers/uploads/katacontaine…>. Thanks! Claire

7 12

Sign Up - Zuul booth at AnsibleFest, Oct 2-3
by Claire Massey 26 Sep '18

26 Sep '18

Hey everyone, If you’re planning to attend AnsibleFest in Austin next week, we’d love to have you volunteer to work a shift or two in the Zuul booth. Members of the Foundation staff including myself, Clark, Chris, Ashlee and Lauren will be onsite to help staff the booth alongside you. *Please sign-up for booth duty here: https://etherpad.openstack.org/p/zuul_ansiblefest2018 <https://etherpad.openstack.org/p/zuul_ansiblefest2018>.* The etherpad also includes helpful information about the event as well as a preview of the new Zuul swag that we’ll be distributing! Let us know if you have any questions. Thanks, Claire

1 0

[Forum] Forum Topic Submission Period - Time Running out!
by Jimmy McArthur 24 Sep '18

24 Sep '18

Just a reminder that there is a little more than 60 hours left to submit your forum topics. Please make haste to the Forum submission tool: https://www.openstack.org/summit/berlin-2018/call-for-presentations Cheers, Jimmy > Jimmy McArthur <mailto:jimmy@openstack.org> > September 17, 2018 at 11:13 AM > Hello Everyone! > > The Forum Topic Submission session started September 12 and will run > through September 26th. Now is the time to wrangle the topics you > gathered during your Brainstorming Phase and start pushing forum > topics through. Don't rely only on a PTL to make the agenda... step on > up and place the items you consider important front and center. > > As you may have noticed on the Forum Wiki > (https://wiki.openstack.org/wiki/Forum) we're reusing the normal CFP > tool this year. We did our best to remove Summit specific language, > but if you notice something, just know that you are submitting to the > Forum. URL is here: > > https://www.openstack.org/summit/berlin-2018/call-for-presentations > > Looking forward to seeing everyone's submissions! > > If you have questions or concerns about the process, please don't > hesitate to reach out. > > Cheers, > Jimmy > > _______________________________________________ > Openstack-track-chairs mailing list > Openstack-track-chairs(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs

1 0

Fixing Github pull-request API limitation
by Simon.Westphahl＠bmw.de 24 Sep '18

24 Sep '18

Hi, I want to discuss a possible solution for a problem we have in our Zuul setup in combination with Github. The problem is a limitation of the Github pull request (PR) API?, which will only returning 300 files max. (see [0] -- btw. what Github calls "response" doesn't mean HTTP response, but a max limit that also applies when using pagination) In case the PR has more than 300 changed files, it might be that Zuul is not considering changes to the config and not using them speculatively due to the incomplete file list. Also jobs using file filters might not be considered in case the relevant files are not in the list of files for a change. I took a stab at fixing this [1] and would like to get some feedback before proceeding. One main problem I see already with my proposal is, that the "fileschanges" gear job is done synchronous. This might block threads in places where this is undesired. Best regards, Simon [0] https://developer.github.com/v3/pulls/#list-pull-requests-files [1] https://review.openstack.org/#/c/603287/ -- BMW Car IT GmbH Simon Westphahl Spezialist Entwicklung Lise-Meitner-Straße 14 89081 Ulm Phone: +49 731 3780-4069 Mobil: +49 1525 4980469 Mail: simon.westphahl(a)bmw-carit.de Mail (BMW): simon.westphahl(a)bmw.de Web: http://www.bmw-carit.de/ BMW Car IT GmbH Geschäftsführer: Kai-Uwe Balszuweit und Christian Salzmann Sitz und Registergericht: München HRB 134810

1 0

Nodepool driver interface
by David Shrewsbury 13 Sep '18

13 Sep '18

Greetings all, We just merged the last piece needed [1] for all of Tristan's various experimental nodepool drivers. This has reminded me of an idea that I've had in the back of my mind for a while now. Currently, there are reviews up for Kubernetes, Amazon EC2, Ansible, OpenContainer, VMWare, and Azure. I'd like for us to at least discuss the possibility of a pluggable driver system so that the drivers can be developed independently from nodepool, rather than in-tree to nodepool. Why? A. Having them in-tree leads to the assumption that they are tested. We do not have access to all of these services, so actively testing them within nodepool is not (currently) possible. B. Nodepool reviewers will be responsible for maintaining the drivers. I, myself, do not have experience with these services, nor access to them, nor the desire or time to learn some of them. I would venture a guess that most of us share some of these deficiencies. This puts us in a spot of just guessing at fixes, or just ignoring them hoping someone else will pick up the ball. C. With plugins, users could get driver updates/fixes without waiting for a nodepool release. I think keeping the OpenStack driver in-tree makes sense because we actively test and use it. I can even get on board with keeping the static driver too. But the others, to me, would be best kept out of tree *unless* we can add access to those services to our Zuul environment and actively test them as well (that at least takes care of the A reason above, and most of B if we have excellent tests setup). Thoughts? [1] https://review.openstack.org/535555 -- David Shrewsbury (Shrews)

7 9

6 days left for the Forum Brainstorming Period...
by Jimmy McArthur 05 Sep '18

05 Sep '18

Hello All! The Forum Brainstorming session ends September 11 and the topic submission phase begins September 12. Thank you to all of the projects that have created a wiki and begun the Brainstorming Phase. I'd like to encourage projects that have not yet created an etherpad to do so at https://wiki.openstack.org/wiki/Forum/Berlin2018 This is an opportunity to get feedback, vet ideas, and garner support from the community on your ideas. Don't rely only on a PTL to make the agenda... step on up and place the items you consider important front and center :) If you have questions or concerns about the process, please don't hesitate to reach out. Cheers, Jimmy

1 0

Zuul Preparations for Berlin Forum
by Claire Massey 31 Aug '18

31 Aug '18

Hi everyone, In preparation to the OpenStack Summit in Berlin it’s time to start thinking about the Forum (https://wiki.openstack.org/wiki/Forum <https://wiki.openstack.org/wiki/Forum>) that consists of a set of 40-minute long collaborative sessions in parallel to the conference.The Forum provides a feedback loop between developers and users and gives an opportunity to do strategic planning for upcoming releases of OSF projects. The participants use etherpad to plan the agenda and capture the notes from the sessions which have a format of open discussion. We're currently in the brainstorming phase for Forum topics. The purpose of this period is to identify areas for feedback or new requirements and collect ideas for cross-project discussions and planning that are also relevant for users. To collect ideas we've created an etherpad: https://etherpad.openstack.org/p/BER-zuul-brainstorming <https://etherpad.openstack.org/p/BER-zuul-brainstorming>. Please start adding topics to the etherpad now. Note that the Forum session submission period starts on September 12 and closes on September 26. CALL TO ACTION: We’re also looking for *one* active member of the Zuul team to volunteer to serve as a representative on the Selection Committee for the Forum. If you're interested in volunteering for this role, please reply to this thread by August 30. The Forum Selection Committee is comprised of members of the OSF staff, OpenStack TC and UC and one representative each from new top level projects StarlingX, Airship, Zuul and Kata Containers. The Selection Committee will collaboratively select the topics and program the Forum agenda between September 26 - October 24. Once the Zuul representative is confirmed, our team, led by Jimmy McArthur, will work closely with them on next steps. Please let me know if you have any questions. Thanks, Claire

2 1

ARA Records Ansible 0.16.0 has been released
by David Moreau Simard 30 Aug '18

30 Aug '18

Hi, Please take note that ARA 0.16 has been released today. It includes a bugfix which prevented customized configuration on the sqlite WSGI middleware which may be deployed for browsing Zuul ARA reports. A new feature can be used to prevent specific host facts from being saved -- by default this includes 'ansible_env' due to the potential sensitivity of the vars that can be in the Ansible shell environment. You can find the full changelog here [1]. Please let me know if you see any issues or have any questions ! [1]: https://github.com/openstack/ara/releases/tag/0.16.0 David Moreau Simard Service Reliability Engineering | OpenShift dmsimard = [irc, github, twitter]

1 0

Re: Invitation: Aug 30 Community Webinar: OSF Project Updates
by Jeremy Stanley 30 Aug '18

30 Aug '18

On 2018-08-27 12:30:54 -0500 (-0500), Claire Massey wrote: > This Thursday, August 30, we’ll host an OSF community webinar at > 8:00am PT (1500 UTC). This meeting is open to all community > members so please join us! > > We’ll cover: [...] > --High level updates from new OSF projects: Airship, Kata > Containers, StarlingX, and Zuul [...] For the sake of transparency, the OSF marketing team was looking on fairly short notice for someone to riff on Zuul for 5 minutes and I volunteered. Anne has a pair of slides one of which is just "Zuul: Stop Merging Broken Code; Get Involved: zuul-ci.org" and the other has "Zuul is Continuous Everything: Merged CI/CD workflow w/ Ansible; Fully dynamic configuration; Gerrit and GitHub support" on it. I'm just going to be dialling in on a phone because I don't trust Zoom's proprietary browser wizardry and am not set up with a webcam anyway. The target audience is mostly marketers and press analysts who have at best heard the name before, so it'll be really low-level. I'll spend a few seconds acknowledging that Zuul originated in Spring of 2012, growing out of the need to scale more traditional CI models to test the ever increasing volume of developer activity in the OpenStack community with a focus on parallel predictive testing of multiple future code states, with the third major release marking six years of continued effort in that regard, but that it also now brings a broader scope with in-repository configuration tested securely on proposal, adoption of a real automation system (Ansible) as its job configuration language and executor implementation, and adds support for integrating dependencies across multiple code review platforms (ability to mix multiple Gerrit and GitHub connections). That's right... rambling. Nevertheless if you decide to dial in and have concerns about what I'm saying, I'll be in #zuul on Freenode the entire time should you wish to course-correct my comments. ;) -- Jeremy Stanley

1 1

Early Bird Pricing Ends Tomorrow - OpenStack Summit Berlin
by Claire Massey 27 Aug '18

27 Aug '18

Hi everyone, Friendly reminder that the early bird ticket price deadline for the OpenStack Summit Berlin <http://www.openstack.org/summit> is tomorrow, August 28 at 11:59pm PT (August 29, 6:59 UTC). Register NOW <https://openstacksummit2018berlin.eventbrite.com/> before the price increases to $999 USD! In Berlin, there will be sessions and workshops around open infrastructure use cases, including CI/CD, container infrastructure, edge computing, HPC / AI / GPUs, private & hybrid cloud, public cloud and NFV. In case you haven’t seen it, the initial agenda with CFP selected content is now live <https://www.openstack.org/summit/berlin-2018/summit-schedule/> and includes sessions and workshops from Ocado Technology, Metronom, Oerlikon, and more! In addition, make sure to check out the Edge Hackathon hosted by Open Telekom Cloud <https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22723/b…> the weekend prior to the Summit. Interested in sponsoring the Summit? Find out more here <https://www.openstack.org/summit/berlin-2018/sponsors/> or email summit(a)openstack.org <mailto:summit@openstack.org>. Thanks, Claire

1 0

Invitation: Aug 30 Community Webinar: OSF Project Updates
by Claire Massey 27 Aug '18

27 Aug '18

Hi Zuul team, This Thursday, August 30, we’ll host an OSF community webinar at 8:00am PT (1500 UTC). This meeting is open to all community members so please join us! We’ll cover: —What’s new in OpenStack’s latest release - Rocky. (This info will still be at a fairly high level, so might not be new information if you’re someone who stays up to date in the OpenStack dev ML or is actively involved in upstream work) —High level updates from new OSF projects: Airship, Kata Containers, StarlingX, and Zuul —What you can expect at the Berlin Summit in November This meeting will be run over Zoom and will be recorded, so if you can’t make the time, don’t panic! Details below. Thanks, Claire When: Aug 30, 2018 8:00 AM Pacific Time (US and Canada) Topic: OSF Community Meeting Please click the link below to join the webinar: https://zoom.us/j/551803657 <https://zoom.us/j/551803657> Or iPhone one-tap : US: +16699006833,,551803657# or +16468769923,,551803657# Or Telephone: Dial(for higher quality, dial a number based on your current location): US: +1 669 900 6833 or +1 646 876 9923 Webinar ID: 551 803 657 International numbers available: https://zoom.us/u/bh2jVweqf <https://zoom.us/u/bh2jVweqf>

1 0

Berlin Community Contributor Awards
by Kendall Nelson 24 Aug '18

24 Aug '18

Hello Everyone! As we approach the Summit (still a ways away thankfully), its time to kick off the Community Contributor Award nominations! For those of you that might already know what they are, here is the form[1]. For those of you that have never heard of the CCA, I'll briefly explain what they are :) We all know people in the community that do the dirty jobs, we all know people that will bend over backwards trying to help someone new, we all know someone that is a savant in some area of the code we could never hope to understand. These people rarely get the thanks they deserve and the Community Contributor Awards are a chance to make sure they know that they are appreciated for the amazing work they do and skills they have. So go forth and nominate these amazing community members[1]! Nominations will close on October 21st at 7:00 UTC and winners will be announced at the OpenStack Summit in Berlin[2]. -Kendall Nelson (diablo_rojo) [1] https://openstackfoundation.formstack.com/forms/berlin_stein_ccas [2]https://www.openstack.org/summit/berlin-2018/

1 1

Zuul project update 2018-08-20
by corvus＠inaugust.com 20 Aug '18

20 Aug '18

Hi, Several folks are on vacation around this time of year, so things are quieter than normal (and may be for the next couple of weeks). Shrews is working on getting images published via pbrx[1]. I plan to continue work on per-project/tenant ssh keys. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim [1] https://review.openstack.org/#/q/topic:push-job

1 0

Proposal to use React for the web dashboard
by Tristan Cacqueray 19 Aug '18

19 Aug '18

Hello, As we are about to add more functionalities to the web dashboard, I would like to suggest we replace Angular by React. In the context of the log-classify project, I experimented with different frameworks to write user web interfaces. I tried Angular, VueJs and React, and the latter works best. React seems less obscure than Angular and it is more widely used than VueJs. The main advantage for me is that React's components are in charge of the rendering and you can follow the code from main() to actual page code. Here is a prototype for the Zuul web interface re-written in React: https://review.openstack.org/#/c/591604/ The highlights are: * The status page jquery code has been rewritten, * It uses patternfly for the new "Masthead" bar, * Tenant name is displayed on the top right, and * Routing uses HTML5 history API (links doesn't have .html extension) The downsides are: * The new routing needs extra rewrite rules, and * Sub-directory deployment needs to rebuild the files with the desired static assets serving location. We could simplify the sub-directory use-case using a custom dynamic discovery mecanism, for example with a /.well-known/zuul-manifest.json file, but this has not been implemented yet. There may be other options, but by default React app are served from the / top location. Please find further details about this specific issue. Are there many zuul web dashboards served from a sub-directory? Regards, -Tristan Details about sub-directory and static files location: I first tried to set the "homepage" setting to "./" so that static files would be loaded relative to the requested path, but that doesn't work for url like "/job/{name}" where static files will be loaded from "/job/static/js/..." The "homepage" default settings is "/" and the index.html loads static assets from "/static/". The current solution is to manually change the "homepage" to the desired location, e.g. "/zuul/" and rebuild the UI so that the index.html loads static assets from "/zuul/static", even when it is accessed from "/zuul/job/{name}" Or without rebuilding, the zuul static assets needs to be served from /, even when client are using a sub-directory. This is not very clean and it may cause conflicts. Another solution is to use the HashRouter instead of the BrowserRouter, but then url will look like this: "zuul.openstack.org/#/job/{name}"

1 0

Berlin Summit Schedule Live!
by Claire Massey 14 Aug '18

14 Aug '18

Hi Zuul team, The schedule for the Berlin Summit <http://www.openstack.org/summit/berlin-2018/summit-schedule> is live with all of the sessions that were accepted via the CFP. It includes a several talks <https://www.openstack.org/summit/berlin-2018/summit-schedule/global-search?…> that feature Zuul. You can see the full CI/CD track here <https://www.openstack.org/summit/berlin-2018/summit-schedule/#track=249>. Check out 100+ sessions, demos, and workshops covering 35+ open source projects in the following Tracks: • CI/CD • Container Infrastructure • Edge Computing • HPC / GPU / AI • Private & Hybrid Cloud • Public Cloud • Telecom & NFV Log in with your OpenStackID <https://openstackid.org/> and start building your schedule now! Register for the Summit <https://openstacksummit2018berlin.eventbrite.com/> - Get your Summit ticket for USD $699 before the price increases on August 21 at 11:59pm PT (August 22 at 6:59 UTC) For speakers with accepted sessions, look for an email from speakersupport(a)openstack.org <mailto:speakersupport@openstack.org> for next steps on registration. Thank you to our Programming Committee! They have once again taken time out of their busy schedules to help create another round of outstanding content for the OpenStack Summit. The OpenStack Foundation relies on the community-nominated Programming Committee, along with your Community Votes to select the content of the summit. If you're curious about this process, you can read more about it here <https://www.openstack.org/summit/berlin-2018/call-for-presentations/selecti…> where we have also listed the Programming Committee members. Interested in sponsoring the Berlin Summit? Learn more here <https://www.openstack.org/summit/berlin-2018/sponsors/#how-to-sponsor>. Thanks, Claire

1 0

Zuul project update 2018-08-13
by corvus＠inaugust.com 13 Aug '18

13 Aug '18

Hi, Tristan is continuing work on the container spec implementation. I'm finishing up support for line comments as well as logs in swift. Later this week, I plan to start working on implementation of an item initially described in the Zuul v3 spec but which was not implemented, the per-project and per-tenant ssh keys described here: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#co… The recent discussions about add_host have reminded us that with that piece in place, add_host would become much more useful for untrusted projects. -Jim

1 0

zuul-jobs : Removal of readthedocs playbook
by Ian Wienand 08 Aug '18

08 Aug '18

Hello, The zuul-jobs:playbooks/python/readthedocs.yaml playbook is scheduled for removal [1] in two weeks on or about 27th August, 2018 (per the deprecation policy [2]). This playbook was used to trigger documentation builds on the readthedocs.org service. The playbook uses an API endpoint that is no longer available [3]. The trigger-readthedocs role is provided for anyone wishing to migrate to the authenticated API endpoint [4]. For more details, see [5]. Thanks, -i [1] https://review.openstack.org/#/c/583836/ [2] https://docs.openstack.org/infra/zuul-jobs/deprecation.html [3] https://github.com/rtfd/readthedocs.org/commit/a66bd56bada66f99c778aa424042… [4] https://zuul-ci.org/docs/zuul-jobs/roles.html#role-trigger-readthedocs [5] http://lists.openstack.org/pipermail/openstack-dev/2018-August/132836.html

1 0

Zuul project update 2018-08-06
by corvus＠inaugust.com 08 Aug '18

08 Aug '18

Hi, Sorry for the delay, I was a bit distracted earlier this week. Tobias has fixed issue with implicit branch matchers[1]. I've mostly completed the effort to store logs in swift. We still need to add the icons if we want them to show up in the generated HTML indexes, but otherwise, it should be ready. For those interested, I expect OpenStack to start using this at scale shortly. The changes to add line comments to Gerrit (and add the framework for GitHub to use later) are in. I still need to see if we can map line numbers across speculative changes in the mergers. We released Zuul 3.2.0. Folks also may be interested to know that in OpenStack, we've seen a significant performance improvement on the executors by upgrading to more recent kernels (4.15.0). Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim [1] https://review.openstack.org/588201

1 0

Software Factory 3.1 released
by Tristan Cacqueray 06 Aug '18

06 Aug '18

We are pleased to announce the release of Software Factory version 3.1: http://www.softwarefactory-project.io/releases/3.1 This new release adds support for tenant deployment and Zuul configuration management from the resources. Here is the list of features we picked in the Software Factory version of Zuul and Nodepool that are still under review upstream. We are confident these features will eventually land upstream, but they might evolve by then. They should therefore be considered a "tech preview", although we will do our best to integrate these in the least disruptive way possible: * Improved dashboard: https://review.openstack.org/#/q/topic:zuul-web-routes https://review.openstack.org/#/q/topic:zuul-ui-pages Nodepool drivers: * RunC: https://review.openstack.org/535556 * Kubernetes: https://review.openstack.org/535557 * Openshift: https://review.openstack.org/570667 Best regards, The Software Factory Team

1 0

Interview with Podcast.__init__
by Monty Taylor 03 Aug '18

03 Aug '18

Hey all, Podcast.__init__, which is a python related podcast, recently interviewed me about Zuul: https://www.podcastinit.com/zuul-with-monty-taylor-episode-172/ If you think you can handle listening to me talk for an hour, check it out. For those of you who think you've heard all of my old grandpa stories ... this podcast contains the story of when Jim and I first met. Monty

1 0

Zuul project update 2018-07-30
by corvus＠inaugust.com 30 Jul '18

30 Jul '18

Hi, This past week has seen: Paul doing some work to fix and better test the ansible synchronize task. Also fixing an issue related to the child-jobs changes from last week. I made progress on both the swift log uploads and file comments efforts. I should be able to test swift uploads this week. Tobias has been working on the ability to pause a job while its children run (this will also be helpful for container work). Tristan has staged several changes implementing the containers spec. I plan on releasing Zuul 3.2.0 shortly. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Zuul project update 2018-07-23
by corvus＠inaugust.com 23 Jul '18

23 Jul '18

Hi, I've published what I hope is the final revision of the container spec. Please take a look and leave feedback and a vote. I spent part of last week working on a new zuul-jobs role to upload logs to swift. I will continue that this week. I'm hoping that with two viable log upload roles, we'll end up with less ambiguity around log uploading. Paul is also performing work to improve roles around log handling. Once that's done, I'll return to my work on line comments. I believe Tristan has fixed the problem with zuul-web which was blocking our release last week. However, we did notice a memory leak in OpenStack-Infra. It's possible that it's already fixed by patches that Tobias happened to already be working on, or it may not. I'd like to observe our instance for a few more days to see how it behaves. But hopefully we can release Zuul this week. We're also planning on burning in the current master version of Nodepool with the idea of making a release there before Shrews goes on vacation. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 1

Rethinking the design of base jobs
by Paul Belanger 19 Jul '18

19 Jul '18

Greetings, I'd like to open a discussion around the idea of changing how base jobs work, today base jobs are stored in a trusted config project. Mostly, this is done because certain aspects of a base jobs need a secret, for example uploading logs to logs.exexmple.org. We also have other tasks in a base job, lets say configure-mirrors, which pre-populates the mirror info on a node from nodepool. As a recap, the main thing to remember about a config project, or trusted job, is we cannot leverage the speculative testing around jobs. We first need to merge a patch, then hope the job continues to work. In openstack-infra, we've create an idea of base / base-test jobs and some instructions[1] on how to interact with them. Now back to configure-mirrors role above, I believe there isn't really a reason for the role to live in a trusted job, nothing it does will affect zuul security (think secrets) or interact with zuul-executor (delagate_localhost). And given that changes to the role need to do the base / base-test dance outlined above, I'd like to make the following suggestion. We keep the idea of base job, but we split it into base (trusted) and base (untrusted). Given that today zuul cannot support 2 jobs with the same name, I think it would look more like. foo-zuul-jobs (untrusted project) - name: base parent: base-minimal config-project (trusted) - name: base-minimal parent: null This means all jobs, now parent to an untrusted project job, with the main benefit that speculative to base are more easily tested. And things that really need to be locked down (eg secrets), still live in the trusted base-minimal job (NOTE: open to renames). In the case of openstack-infra[2], I actually think a lot of the roles in pre.yaml can be moved into untrusted jobs, if we followed this idea. The post-run tasks, are almost all trusted context. Mind you, we don't really need to make this change in zuul, but I'm more thinking we should use it as the default recommendation for new users. I'm looking to start implementing the change in rdoproject.org but wanted to get the idea out before I proposed changes. [1] http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jo… [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks…

2 2

Potential enhancements to logging
by corvus＠inaugust.com 18 Jul '18

18 Jul '18

Hi, Some recent (and not-so-recent) issues with log hosting in OpenStack Infra have caused a renewed interest in reworking our log storing infrastructure. Some of the motivation and much of the initial work is OpenStack-specific, so I have written a post to the openstack-infra list describing that. Technically, none of what is in there affects the wider Zuul community, but folks reading this may want to read that message for background (or, if you follow our patterns closely, it may affect you as well). That message is here: http://lists.openstack.org/pipermail/openstack-infra/2018-July/006020.html Also, like that message, this one is the result of several conversations, especially with Monty; many of these ideas are his. Essentially, that message describes a process for us to begin hosting our logs in swift instead of on a file server. I'll note here that Zuul is (and will continue to be, at least for the foreseeable future) agnostic about where logs are hosted. Log storage is implemented as roles in base jobs, so any site can store logs where and how they wish. We can help each other by sharing the roles which perform these tasks. OpenStack moving its log storage to swift doesn't mean anyone else needs to -- storing logs on a sever (or even the executor) will still be supported. As would storing in S3, or any other system for which someone is motivated to write a role. That message also describes decommissioning our log-hosting server which happens to provide a number of useful services today. Among the these are dynamic generation of directory indexes, and HTMLification of text logs. Our plan is to pre-generate that data, however, that's not ideal, as it expands and duplicates the amount of data stored, and it also creates more work for the executors. We can expand the role of the Zuul dashboard to take on some of those duties, thereby reducing the storage requirements, and offloading the computation to the browser of the user requesting it. In the process, we can make a better experience for users navigating log files, all without compromising Zuul's independence from backend storage mechanisms. All of what follows should apply equally to file or swift-based storage. Let's start by creating a per-build page in the dashboard. Currently, we show a list of builds, and link to the log locations for those builds. In the future, we should link to a new page with details about the build. Not only so that we can show more info (which we already collect) than can be shown on one line, but also so that we have a place for the additional functionality which follows. If we have the uploader also create and upload a file manifest (say, zuul-manifest.json), then the build page can fetch[1] this file from the log storage and display an index of artifacts for the build. This can allow for access to artifacts directly within the Zuul web user interface, without the discontinuity of hitting an artifact server for the index. Since swift (and other object storage systems) may not be capable of dynamically creating indexes, this allows the functionality to work in all cases and removes the need to pre-generate index pages when using swift. We can also extend Zuul to store additional artifact information about a build. For example, in our docs build jobs, we override the success-url to link to the generated doc content. If we return the build location as additional information about the build, we can highlight that in a prominent place on the build page. In the future, we may want to leave the success-url alone so that users visit the build page and then follow a link to the preview. It would be an extra click compared to what we do today, but it may be a better experience in the long run. Either way would still be an option, of course, according to operator preference. Another important feature we currently have in OpenStack is a piece of middleware we call "os-loganalyze" (or OSLA) which HTMLifies text logs. It dynamically parses log files and creates anchors for each line (for easy hyperlink references) and also highlights and allows filtering by severity. We could implement this in javascript, so that when a user browses from the build page to a text file, the javascript component could fetch[1] the file and perform the HTMLification as OSLA does today. If it can't parse the file, it will just pass it through unchanged. And in the virtual file listing on the build page, we can add a "raw" link for direct download of the file without going through the HTMLification. This would eliminate the need for us to pre-render content when we upload to swift, and by implementing this in a generic manner in javascript in Zuul, more non-OpenStack Zuul users would benefit from this functionality. Any unknown files, or existing .html files, should just be direct links which leave the Zuul dashboard interface (we don't want to attempt to proxy or embed everything). Finally, even today Zuul creates a job-output.json, with the idea that we would eventually use it as a substitute for job-output.txt when reviewing logs. The build page would be a natural launching point for a page which read this json file and created a more interactive and detailed version of the streaming job output. In summary, by moving some of the OpenStack-specific log processing into Zuul-dashboard javascript, we can make it more generic and widely applicable, and at the same time provide better support for multiple log storage systems. -Jim [1] Swift does support specifying CORS headers. Users would need to do so to support this, and users of static file servers would need to do something similar.

3 5

Zuul project update 2018-07-16
by corvus＠inaugust.com 17 Jul '18

17 Jul '18

Hi Last week Tristan fixed issues with direct enqueueing of changes with ambiguous project names, Improved timer trigger to skip un-needed projects, and improving Github Event filter debugging. Tobias fixed a potentially serious bug with contamination of configuration. He's working on improving job freeze performance, and parent jobs which pause and continue to run while their children run (for resources and artifact exchange during a buildset). This week I plan on finishing the container spec. I've also written up some ideas for log storage which I may do some work on. We're waiting on a resolution to an issue with multi-tenancy in the new Javascript code. Once that's resolved, or we revert the new JS code, we should be in position to make a new release. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

2 1

Building container images in nodepool
by corvus＠inaugust.com 16 Jul '18

16 Jul '18

Hi, I think the container spec is just about ready, but one part of it recently prompted a discussion that I think could use some wider input. Specifically, this is regarding the "container-as-machine" use-case. So think "run pep8 in a container" when you're thinking about this. There's a whole bunch of stuff in the spec about container-native workloads. This isn't it. The issue is whether or not Nodepool should be able to build container images, much as we do today for machines. Note that regardless, Nodepool will be able to use "upstream" container images from the local Kubernetes registry or Docker Hub. So this would be an optional feature (just as Nodepool can use extant cloud images). There are a few reasons to build local images (of any kind): * To provide a git cache. * To have more up to date packages than those provided externally. * To have a backup of a working image if something upstream changes. * To have a standardized image. * To add local requirements. To be fair, the last two might be the same thing said different ways. In an offline discussion prompted by the spec, Monty has suggested that running a git mirror in Kubernetes may obviate the need for using custom images for a git cache [related: applying this to VM images as well could reduce image sizes there; he's planning on writing up thoughts on this more fully later]. So if we take that out of the equation, the question becomes: Are the remaining reasons for building an image locally sufficient to build this functionality into Nodepool? A quick clarification: we're obviously not going to write a new image builder either way. We're talking about having Nodepool run some existing container image process on a periodic schedule and upload the result to the local registry. The main benefit to Nodepool doing this is using the same images across different registries, having a daemon performing the image builds automatically, and the ability to roll-back to previous versions. If you have thoughts on whether this is useful, or if it should be omitted in favor of relying more heavily on base images or external systems, please let me know. I'll try to collect feedback and update the spec early next week. Then we can merge it. :) -Jim

6 6

Improving new user experience
by corvus＠inaugust.com 11 Jul '18

11 Jul '18

Hi, Recently Artem has rightly pointed out that the experience for new users wanting to run Zuul is not good. We have some work in progress to improve that, and some efforts which are stalled and could use help. Here's what I think we should do: 1) Continue work on the Zuul From Scratch guide. This part of the documentation ends right after the Zuul services start up, but without telling anyone how to actually get Zuul to do anything useful. It needs to continue to explain: * Configuring a pipeline * What a base job needs to do * How to create a base job from scratch * How to use or copy the existing base jobs in zuul-base-jobs * How to use jobs from zuul-jobs * Configuring a project * Actually reporting on a change or pull request I'll note that the original Zuul From Scratch document focused only on GitHub and Fedora. Obviously there's a lot of options, and folks have added sections on Gerrit and other operating systems. The most important aspect of this document, however, is that it walk someone through the entire process of setting up Zuul for at least *one* configuration. If we can handle more, that's fine. Ideall, we can keep divergent steps to a minimum. For example, we may want to describe how to push a configuration change up once (for Github, and once for Gerrit) and then subsequently refer to that process generally when we discuss how to build pipelines, jobs, etc. Otherwise we'll have to write the entire document twice. 2) Separately, we should make the process of bootstrapping a running system easier. While Zuul From Scratch should describe everything needed to build an entire Zuul system, well, "from scratch", we should also have a system that makes it easy to deploy a working Zuul so someone can at least start playing with it, and perhaps even continue to run it in production. I'll call this the "quickstart", and it should replace the current erroneously titled "quickstart" in our docs. Monty is working on jobs which will automatically build and upload container images of Zuul to Dockerhub. Once those are in place, we can write a helm chart, or Ansible playbook, or something similar to deploy those with a very minimal configuration. The goal should be to get something up and running with the minimum amount of work and also the minimum amount of documentation. If you want to learn how a Zuul works, read Zuul From Scratch. If you want to just start breaking it, use the quickstart. I think any progress we make on these two items will drastically improve the experience for new users. -Jim

3 4

zuul-apb to install Zuul on K8/openshift using ansible-service-broker
by Artem Goncharov 10 Jul '18

10 Jul '18

Hi all, as promised in the IRC I am publishing POC work on preparing zuul-apb. It is intended to help (currently only myself) to install Zuul on top of Openshift (Kubernetes should be supported also, but with limited functionality and not tried so far). The project started to be initially used in my company, where after running zuul on VMs I decided to shift it to using Openshift. It is currently located in my personal repo in github with intention to be moved to corporate repo shortly. But before I do this I wanted to offer it to Zuul community for discussion. If there is enough interest in adopting it, I am open and will be glad to shift development under the Openstack hat. So here are some details: - Repo: https://github.com/gtema/zuul-apb - project is built to make use of source-to-image <https://github.com/openshift/source-to-image> to build own images. This is, however, only possible if you use Openshift (not the S2I tool itself, but presence of BuildConfigs). Example repo <https://github.com/gtema/s2i-zuul> with s2i image scripts. For plain Kubernetes this might be later solved by separate pod invoking s2i inside. - regular project-config repo is also mandatory, since it is used to fetch initial components configuration. It should contain zuul/{zuul.conf,main.yaml}, nodepool{nodepool.yaml} files. It is also planned to watch project-config changes and redeploy configMaps. - gearman certificates and all other secrets are passed as APB parameters (unless you use `dev` plan and bootstrap special variables file inside for dev purposes) and stored as Secret - zookeper cluster is created also part of the project (external cluster might be used, but I decided to initially provide own) - project is not yet used in production, therefore not everything is how it should be. I intend to work on this further. There are also some issues with AnsibleServiceBroker itself, which caused some delays and few iterations with upstream. So for example the Dockerfile bases on my upstream fork due to issues with ansible 2.7 (some bugs in `lookup`, causing line breaks to be broken). When changes upstream are published I will rebase back. - work is based on Tobias's shared Openshift templates - thanks Tobias - currently 2 components (nodepool-builder and zuul-executor) are requiring privileged accounts to work. This is kind of problem with APB approach (and generally using containers), since there is currently no way to deliver this nicely and it poses big security risk to the whole cluster. Therefore there is a manual step required for deployment to grant those privileges. On the other hand nodepool-builder is very poor on running inside of container due to lots of required privileges, security concerns, and bugs in `elements` with respect to i.e. SELinux handling ( https://bugs.launchpad.net/diskimage-builder/+bug/1779273) potential problems with cross-platform builds. Tristian had an idea of using nodepool-launcher to start VM with nodepool-builder (I personally like the approach). Otherwise it is anyway funny - VM is running container, which is building VM image ;-). Additionally Tristian volunteered to work also on zuul-executor not to required privileged account. - together with work on containerization of zuul jobs this might become a full solution of running zuul under K8/Openshift and complement Windmill with a VM installation approach. I would be glad, if you share your thoughts. Especially whether community is interested in having this as an official project. Regards and have a nice weekend, Artem

5 12

Canceling a running job
by Matthieu Huin 10 Jul '18

10 Jul '18

Hello, Yesterday we ran into a case where a periodic job had to be stopped in the middle of its execution. We ended up doing the following: * check the console stream to grab the nodes' IP addresses, from the ansible role that sets up firewall rules (this was a multi-nodes build) * SSH into each node and manually reboot the nodes This is obviously suboptimal, and possibly not pullable on single-node builds. Is there any better way to do this, or at least to get the nodes' IP addresses for any given build? Generally speaking, this use case is rather generic, as one might want to be able to stop a running job for a variety of reasons (free up resources, prevent "something bad" from happening accidentally, etc). Is this something we could add into Zuul? Is there a spec for this or should we start one? Cheers, MHU

4 3

Exciting opportunity to bikeshed over a naming question!
by Monty Taylor 09 Jul '18

09 Jul '18

Hey everybody, I've been working on jobs to build zuul container images. The obvious next step, once the build job works - is to publish them to dockerhub. This is where the bikeshed comes in. A while back, I registered dockerhub orgs for both "zuul" and "zuulci". Which org should we publish container images to? A part of me wants to use the 'zuul' one, since we have it. OTOH, the website is zuul-ci - so the other part of me thinks we should just use that. The end result of this will be docker images that look like: zuul/zuul-base zuul/zuul-executor zuul/zuul-scheduler zuul/zuul-web etc or zuulci/zuul-base zuulci/zuul-executor zuulci/zuul-scheduler zuulci/zuul-web etc Thoughts? Monty

5 6

Zuul project update 2018-07-09
by corvus＠inaugust.com 09 Jul '18

09 Jul '18

Hi, Last week Shrews tracked down and fixed a Nodepool bug preventing the launcher from finding unmanaged images by ID[1]. Paul is working on a series of changes to enable jobs to specify that specific child jobs should be skipped[2][3]. I have been "helping" Tobias fix a bug related to dynamic configuration. Discussion is continuing on the container spec, but I think it's getting very close to approval. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim [1] https://review.openstack.org/579664 [2] https://review.openstack.org/578181 [3] https://review.openstack.org/578230 [4] https://review.openstack.org/560136

1 0

Dashboard routing proposals
by Tristan Cacqueray 05 Jul '18

05 Jul '18

Hi, with the recent angular6 refactor, the web interface components routing slightly changed as it is now managed by the navigation javascript component. Unfortunately this broke multi-tenant setup and we need an extra fixes to support it. Here are some options: A) Make the navigation component call a relative 'api/info' endpoint onInit to be able to introspect the deployment mode: the returned json may contains a 'tenant' value to indicate this is a white label setup. B) Always assume white-label and document that multi-tenant needs to redirect the entrypoint to 't/tenants.html' C) Always assume multi-tenant and document that white-label needs a rewrite rule to redirect the entry-point to 'status.html' D) Rewrite the navigation endpoint to rely on a '#!' anchor, this would be a mix of the first option (A) to introspect the setup mode, but using a single 'index.html' file instead. I would prefer D because the current design makes multi-tenant setup looks special with the needed '/t/' path and it feels like a second-class citizen. On the other hand, B or C sounds simple enough and this wouldn't delay much the next release. Regards, -Tristan

2 2

Zuul project update 2018-07-02
by corvus＠inaugust.com 02 Jul '18

02 Jul '18

Hi, This past week has seen: Monty upgrade the zuul dashboard to Angular 6 and typescript. It seems like that's mostly working now, though there may still be an issue in the multi-tenant hosting configuration. Mohammed fixed status page to not show empty headers when filtering for a specific project (a long requested feature)! Tristan found more issues with latest Nodepool master version regarding multiple pool using different labels (fixed in [1] by David), worked on changes to the log-inventory role[2], and contributed to the Angular 6 work. He investigated a couple of exception being raised in the github driver, and started to work on improving logging by promoting important log messages from debug to info[3]. Started to work on selenium tests[4]. I started looking into adding support for leaving in-line comments. I'll continue to work a bit on that this week, but mostly focus on finishing up the container spec and hopefully getting it merged. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim [1] https://review.openstack.org/578831 [2] https://review.openstack.org/#/q/topic:log-inventory-fixes [3] https://review.openstack.org/#/q/topic:improve-logging [4] https://review.openstack.org/579039

1 0

Deprecation: zuul-info/inventory.yaml creation moved from validate-host to log-inventory role (v2)
by Ian Wienand 01 Jul '18

01 Jul '18

Hello (again), This is a follow-up to a prior mail http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-June/000446.html --- The "validate-host" role [1] currently creates the "inventory.yaml" [2] file, which contains the ansible inventory information for the job. For some history, the role was originally called "debug-ansible" but renamed to "validate-host". Since it collects other ansible information, saving the inventory was added to the role shortly after the rename. As copying this inventory information requires access to the executor context, it must be run as a privileged job. However, the role also includes quite a number of other operations that do not have similar requirements. Due to not currently having an automated CI testing framework for privileged jobs, this leaves changes to these unrelated parts (like [3]) with insufficient testing coverage. The creation of the inventory file has been moved into a separate role "log-inventory" [4]. We now wish to remove the inventory creation from the "validate-host" role [5]. This will lower the surface-area of privileged jobs and also enable more straight-forward testing of the rest of the "validate-host" role [6]. For the OpenStack ecosystem, the new role will be added to the base job [7] so this should be entirely transparent. If you build your own base templates you will need to add the "log-inventory" role to continue collecting the inventory.yaml. You can do this *now*, and thus be unaffected when the creation is removed from "validate-host", with one caveat: Note that testing uncovered an issue with the original role -- the "inventory_file" variable is not set by Ansible for localhost only jobs. Thus "log-inventory" will not break any jobs where "validate-host" was working, but you should be careful not to additionally introduce it into localhost (that is to say, executor-only) jobs without considering this. Longer term, the issue has been fixed and "log-inventory" is safe to use in all contexts. A separate variable for the inventory file has been added to the executor context with [8] and is now conditionally used in "log-inventory" with [9] (thus if you are using zuul from master, you are unaffected). Per the deprecation policy for requiring new zuul features [10], this new variable will be used unconditionally to copy the file with [11] around 4 weeks after the next zuul release. Taking into consideration: * reviews posted for several weeks * pending debugging of gate issues with "validate-host" * the informational/debugging nature of the output * the relative ease of adding the new role to maintain the status quo * interaction with current users of the roles I would suggest we merge [7] - addition of "log-inventory" to base jobs in project-config [5] - removal of inventory copy from "validate-host" To complete the initial part of this transition. Thank you -i [1] https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/validat… [2] see any job results zuul-info/inventory.yaml [3] https://review.openstack.org/576351 [4] https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/log-inv… [5] https://review.openstack.org/563790 [6] https://review.openstack.org/563788 [7] https://review.openstack.org/563789 [8] https://review.openstack.org/578235 [9] https://review.openstack.org/578234 [10] https://zuul-ci.org/docs/zuul-jobs/deprecation.html [11] https://review.openstack.org/578238

1 0

Removing the add_host exclusion
by Monty Taylor 30 Jun '18

30 Jun '18

Hey everybody, While working on the OpenStack Infra spec for updating our config management to use Ansible, which involves some pieces of using Zuul for CD, I ran in to an issue related to add_host - which we currently prevent people from using. I'd like to use add_host in jobs in a non-trusted repo - but add_host is on the list of things we block for security reasons. I **think** we're blocking it as a hold-over from the time before we had per-job ssh keys (we were worried about a user using add_host to connect to another job's nodes) Since we do per-build keys now, that vector is gone - and I can't come up with any more reasons why we should be blocking add_host from use. It can't be used to attack anything that we don't already protect from in a different place (like add_host with a connection_type: local would still get blocked on usage by the 'can't execute on localhost' exclusions) So I'd like to relax the exclusion - unless someone has a reason we should keep it? Thanks! Monty

3 2

Matching tags against branches
by corvus＠inaugust.com 29 Jun '18

29 Jun '18

Hi, Recently a discussion came up in OpenStack about moving some job invocations out of our global config-project and into individual project repos. Some of the jobs in question run in our "release" pipeline, which is triggered by tag events. These pipelines have Tag items enqueued into them, rather than Change items. Tags have a ref (the name of the tag), but they don't have a branch. This is because, in git, a tag may point to a commit that is on one branch, many branches, or no branches at all. Moreover, because of this, none of the code review systems attempt to provide "branch" information for tags. The current behavior in Zuul is that explicit branch matchers only match branches or refs. So you can define a job with "branches: stable", and it only run on changes to the "stable" branch. Or you can define a job with "branches: 3\..*" and it will only run on 3.x tags. Or you can say "branches: [stable, 3\..*]" and it will run on both. Implicit branch matchers match branches if the item has one (eg, a Change). But if the item does not (eg, a Tag), they always match. This means that a project can add a job to a tag-based pipeline in-repo, and it will work. The project stanza will get an implicit branch matcher, but that will always match the tag item, so the job will always run. But this means that ever project stanza for the project matches. So if the stable branch says "run foo on release" and the master branch says "run bar on release" both foo and bar will run on release. There is no way to implicitly have different jobs running on tags "from different branches" (I put that in scare quotes because that's an artificial construct that we're placing on git). Of course this can be done with explicit branch matchers, but determining which version numbers correspond to which branches and building those regexes can be high maintenance. If we want to make this a bit more intuitive, there are two approaches we could take. One would be to perform a dynamic reconfiguration for all tag items to determine the actual in-repo content for that tag, and use it to the exclusion of all other in-repo content from that project. That would produce the behavior of running whatever was defined in that commit. This is very intuitive for folks coming from other CI systems, however, it's rather resource intensive for Zuul, and could pose problems if, for example, a stable branch project-pipeline used a job defined on the master branch. The other approach would be to attempt to create the fiction that tags have branches. We could ask the mergers to run "git branch --contains" on the tagged commit and find out which branches it is in. We can then compare that set of branches to any explicit and implicit branch matchers. If any of the branches match, we'll consider the Tag item to have matched. This will probably do the expected thing in the typical case. There are still some edge cases, but they generally degrade to the situation we have today, in that multiple versions of a job will all be run. Of course, in any case (now or with this change) full control can always be achieved in a config-project with explicit branch matchers. I've begun implementing this idea, mostly as an exercise in exploration. I'm tentatively inclined to think that it's an improvement and we should consider proceeding with it. But I'd really like other folks to think about whether it's a good idea, and whether it might produce more problems than it solves. Here's the initial implementation; it's lacking only documentation at this point. Some of the edge cases are described in the commit message, so I refer you to that for further reading: https://review.openstack.org/578557 Thanks, Jim

3 4

Deprecation: zuul-info/inventory.yaml creation moved from validate-host to log-inventory role
by Ian Wienand 26 Jun '18

26 Jun '18

Hello, The "validate-host" role [1] currently creates the "inventory.yaml" [2] file, which contains the ansible inventory information for the job. For some history, the role was originally called "debug-ansible" but renamed to "validate-host". Since it collects other ansible information, saving the inventory was added to the role shortly after the rename. As copying this inventory information requires access to the executor context, it must be run as a privileged job. However, the role also includes quite a number of other operations that do not have similar requirements. Due to not currently having an automated CI testing framework for privileged jobs, this leaves changes to these unrelated parts (like [3]) with insufficient testing coverage. The creation of the inventory file has been moved into a separate role "log-inventory" [4]. We now wish to remove the inventory creation from the "validate-host" role [5]. This will lower the surface-area of privileged jobs and also enable more straight-forward testing of the rest of the "validate-host" role [6]. For the OpenStack ecosystem, the new role will be added to the base job [7] so this should be entirely transparent. However, if you build your own base templates, you will need to add the "log-inventory" role to continue creating the inventory.yaml. Note that you can do this *now*, and thus be unaffected when the creation is removed from "validate-host". Per the deprecation policy [8], but taking into consideration * reviews posted for several weeks * pending debugging of gate issues with "validate-host" * the informational/debugging nature of the output * the relative ease of adding the new role to maintain the status quo I would suggest we merge these changes at the start of next week, around the 25th June Thank you -i [1] https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/validat… [2] see any job results zuul-info/inventory.yaml [3] https://review.openstack.org/576351 [4] https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/log-inv… [5] https://review.openstack.org/563790 [6] https://review.openstack.org/563788 [7] https://review.openstack.org/563789 [8] https://zuul-ci.org/docs/zuul-jobs/deprecation.html

3 4

Zuul project update 2018-06-26
by corvus＠inaugust.com 26 Jun '18

26 Jun '18

Hi, I was away giving a talk on Zuul last week. It looks like it's been a week of bugfixes: Tristan found a bunch of issues after upgrading their deployment: * Zuul refresh_deps doesn't work well with gerrit 2.14 schema change (change/patchset number are now int): https://review.openstack.org/577154 * Zuul configloader doesn't skip project that include: [] https://review.openstack.org/576016 * Nodepool repeatedly creates new deleting node when OpenStack provider fails to remove an instance: https://review.openstack.org/576288 He also split the initial connection loop out of the zk-retry patches: https://review.openstack.org/576047 and https://review.openstack.org/576048 . This make Zuul restart properly if the control plane reboots. Rebased runC, k8s and openshift nodepool drivers in preparation of the upcoming next release of Nodepool. Tobias worked on tenant reconfigure behavior[1] and on possible multi-enqueue/missing-dequeue issue in the github driver. [1] https://review.openstack.org/#/q/project:openstack-infra/zuul+branch:master… Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Re: Unintended variable disclosure on Ansible unreachable tasks despite no_log
by Jeremy Stanley 19 Jun '18

19 Jun '18

On 2018-06-18 13:47:24 +0000 (+0000), Tobias.Henkel(a)bmw.de wrote: > We discovered that if nodes become offline during the build the > no_log attribute of a task got ignored. If the unreachable error > occurred in a task used with a loop variable (ex: with_items), the > contents of the loop items would be printed in the console. This > could lead to accidentally leaking credentials or secrets. [...] MITRE has assigned CVE-2018-12557 to this vulnerability. -- Jeremy Stanley

1 0

Zuul project update 2018-06-18
by corvus＠inaugust.com 18 Jun '18

18 Jun '18

Hi, TristanC has started experimenting with graphQL. The idea is to see how it can be used to simplify the REST apis as well as enable powerful UI using appolo, in particular for pagination. Worked on extra-vars, ansible-tags and ansible-skip-tags job attributes. Found a couple more issues with the Angular6 patch when working on the new project and job page. I worked with Tobias and Clark on fixing a log streaming problem due to the ansible 2.5 upgrade. This week I'm traveling to China to deliver a talk about zuul at OpenInfra Days. I won't be around in channel as much, but should be able to review changes while in transit. Shrews completed static driver rewrite to pre-register nodes. Awaiting review/merge. Identified and fixed an issue where paused request handlers were not removed from active handlers if an exception from a provider operation occured. This potentially caused leaked nodes that remained ready+locked until a launcher restart. Tobiash worked on log streaming problem as mentioned above. Fixed ignored no_log caused by unrechable nodes (a security issue). Fixed caching of empty config. This week: work on improving tenant reconfigure behavior with github. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Zuul participation in OpenCI
by corvus＠inaugust.com 15 Jun '18

15 Jun '18

Hi, A number of us have been involved in the OpenCI effort: https://openci.io/ I think those of us who are, are bringing our experience with Zuul to the group, so it's fair to characterize the Zuul project as participating. It has been suggested that our logo could be added to the OpenCI web page under the "Practitioners from the following Communities" section. Does anyone think this is a good or bad idea? Please feel free to reply to me publicly or privately with any feedback. -Jim

4 3

September PTG attendance
by corvus＠inaugust.com 14 Jun '18

14 Jun '18

Hi, So that folks can start planning for the next PTG, I'd like to find out if we think it's valuable to have a formal Zuul track there, and if folks are planning to attend. I've set up an etherpad to collect information: https://etherpad.openstack.org/p/zuul-denver-ptg-2018 If you could fill it out and indicate whether you: plan to attend, or may attend if there's a standalone Zuul track, or if you won't be able to attend at all. Also, if you have a topic you would like to discuss in person at the PTG, please describe it there. I'll try to collect these within a couple days and we can have a better idea of whether we should make an effort to have a Zuul track, or skip this event in favor of working through our existing backlog. -Jim

1 0

Adding tags, extra-vars files and multiple run to job
by Tristan Cacqueray 14 Jun '18

14 Jun '18

Hi, TripleO folks need some features to be able to re-use deployment playbook natively from a Zuul job. Would it be possible to: * https://storyboard.openstack.org/#!/story/2001353 Adds "tags" and "skip-tags" job attributes that would be used as "--tags" and "--skip-tags" ansible-playbook command line arguments. * https://storyboard.openstack.org/#!/story/2001354 Adds a "extra-vars" job attribute that would be used as "--extra-vars @file.yaml" ansible-playbook command line arguments * https://storyboard.openstack.org/#!/story/2002543 Make the "run" job attribute a list of playbook and make the run phase chain ansible-playbook executions. It seems like those features are simple to implement, so if there are no objections I'd like to work on them. Regards, -Tristan

5 6

Zuul project update 2018-06-12
by corvus＠inaugust.com 12 Jun '18

12 Jun '18

Hi, We landed the changes to upgrade to Ansible 2.5. Things mostly seem to be going well, but we're currently working through a few issues described in story 2002528. Tristan has rebased the extra zuul-web routes with the new cherrypy interface: https://review.openstack.org/#/q/topic:zuul-web-routes and rebased the extra webui pages with the new angular6 interface: https://review.openstack.org/#/q/topic:zuul-ui-pages Monty seems to be updating the angular6 patches, which I guess prompted Tristan's update. Shrews reports that the multi-label changes have merged and he is ready to focus on static driver work now. He also Reviewed and carried Jim's driver API changes through to merge. (Thanks!) Tobias has a number of bug fixes in review. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Multi-tenancy architecture
by Mohammed Naser 05 Jun '18

05 Jun '18

Hi everyone: We're evaluating running Zuul in a multi-tenant environment by leveraging the following feature: https://docs.openstack.org/infra/zuul/admin/components.html#attr-scheduler.… Now, the idea is to make it as self-serve as possible. I've been toying around with a few ideas in terms of implementing tenants and I've come to the following simplistic idea: - Every 'user' has a 'tenant' inside this environment - All projects share the same `config-projects` which contains the base job and publishing artifact and logs roles (config-project is not controlled by tenant) - `untrusted-projects` are all dynamically loaded, provided by the user This would mean that user would supply a project using $method, and as long as they drop in a .zuul.yml file that is appropriate, things will get going automatically (implying that a basic check/gate/post pipeline is in place for all users). The only challenge with this at the moment is that multi-tenancy in Nodepool doesn't seem to exist, which means that everyone will be using the same pool of resources, rather than possibly having every tenant run their jobs inside their own project. Is there any plans to look into addressing the nodepool issue (or maybe ways to workaround it?) and as a more general question, is this an "ok" way of doing things? Your responses are appreciated! Thanks, Mohammed

3 4

Upgrade to Ansible 2.5
by corvus＠inaugust.com 04 Jun '18

04 Jun '18

Hi, We have some patches ready to upgrade Zuul to use Ansible 2.5, but there is some concern about whether there may be incompatibilities with playbooks and roles in zuul-jobs. I know some folks are running with local patches to use later Ansible versions -- if you are one of them, and you are using zuul-jobs, please let me know so we have an idea of what to expect. How do folks feel about merging the upgrade-to-2.5 changes and releasing those as Zuul 3.1.0 -- say, next Thursday, May 10? If we do that, and we find that there are incompatibilities, we will need to begin merging fixes to zuul-jobs immediately. Hopefully, however, those fixes can use the "ansible_version" variable in order to be backwards compatible, so zuul-jobs continues to work for folks who haven't upgraded yet. Finally, we hope to soon have a spec detailing multi-version support for Zuul, so that future upgrades can be handled more gracefully. -Jim

5 8

Zuul project update 2018-06-04
by corvus＠inaugust.com 04 Jun '18

04 Jun '18

Hi, I skipped a few of these due to the Summit and related discombobulation, but it's back! The Summit and OpenDev was really good. We introduced Zuul to a lot of folks. A number of Zuul devs managed to find some time to talk in person about the container spec. I've pushed another revision of that, hopefully it's close to ready. Artem Goncharov (gtema) has started implementation of nodepool security group support. Tristan followed up on summit discussion around container and worked on a prototype implementation of an openshift resource provider[1]. He also started some howtos, recommended practice documentation about PTI[2], renamed the proposed OCI driver to RUNC, and added documentation to nodepool builder about setting diskimage env-vars in secure configuration. I finished the patches to convert zuul-web to cherrypy, those merged at the end of last week. I also pushed up two changes to implement features I've been promising for a while: overridable file matchers, and a new pipeline manager ("supercedent"). I'm hoping we merge the switch to Ansible 2.5 this week. Shrews has been working on refactoring the nodepool API to better support our different drivers, and especially the multi-label support needed by the static driver. Monty has been refreshing his angular v1 -> v5 and related patches. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/topic:openshift-zuul-build-resource [2] https://review.openstack.org/571420

1 0

Behavior change for file matchers
by corvus＠inaugust.com 01 Jun '18

01 Jun '18

Hi, A while back we had a discussion about changing the behavior of file matchers to be more intuitive. The consensus seemed to be that treating them as normal overridable attributes would be the best solution. In particular, that would allow folks to more easily define a file matcher in a job, and then override it at the project-template or project level. See this thread for the full context: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000397.html I've implemented Proposal 2 from that message: https://review.openstack.org/571745 I think we should merge that, and, after we're satisfied with it, release version 3.1 with it, to signify that it's an end-user visible behavior change. It will cause changes in which jobs are run, in ways that would be difficult to enumerate ahead of time. But it should only do so in more complex situations, and hopefully the folks in those situations will be relieved that the solution is easier to comprehend than the status quo. Along with that change I have also proposed a followup which causes files and irrelevant-files to become mutually exclusive options: https://review.openstack.org/571804 If one is used in a parent and the other in a child, the attribute used in the child wins and clears that which was set in the parent. This makes them easier to reason about, though it does so largely by making them less useful. I suspect that the class of things that it lets us express (example in that commit message and in the previously linked thread) can be expressed today with a single regular expression. However, we're looking to move these matchers to use re2 which lacks negative lookaheads. That may mean that if we merge both this change and the re2 switch, we may not be able to express that configuration at all anymore. It may still be worth merging this change to save Zuul admins from pulling their hair out when people ask about the interaction of the two options. Or we may want to keep the ability to use both because, when used well, it does express something unique and succinct. I'd like for folks to consider this change carefully and provide as much feedback as possible before we decide what to do. If we do decide to merge it, it probably makes sense to do so in the same release (3.1) as the previous change. -Jim

1 0

Templates in job vars
by Ian Wienand 29 May '18

29 May '18

Hi, In templating out some diskimage-builder jobs, there is a situation where we pass a list of tests to run, but some platforms can't run various tests. So something like --- - job: name: base abstract: yes ... vars: dib_functests: - gentoo - debian - job: name: test-on-bionic parent: base - job: name: test-on-xenial parent: base vars: dib_functests: - gentoo - debian - suse # also run suse, redefine whole list --- That the base list of tests is repeated to override the dib_functests is a little ugly. I would prefer if the base tests that all platforms can run was defined only once (so if it's updated, you can't miss one). To my mind, it would be better if I could jinja on the variable, so I could add (or remove) a test; above it would become like: # also run opensuse tests on xenial vars: dib_functests: "{{ dib_functests + ['opensuse'] }}" I ended up just appending a second variable [1], but it feels a bit hacky (if you had more levels, you'd need to keep appending more and more variables). Seeing as I'm pretty sure jinja templates are Turing complete, I can also see issues with expanding each vars: entry with it :) Terrible idea? Already better ways I don't know about? Cheers, -i [1] https://review.openstack.org/#/c/566779/60/roles/dib-functests/defaults/mai…

2 1

Zuul project update 2018-05-14
by corvus＠inaugust.com 14 May '18

14 May '18

Hi, Shrews is beginning implementation of multi-label support in nodepool as part of his continuing work on better integrating the static driver. I spent most of last week and will continue to spend most of this week preparing for the upcoming OpenStack Summit and OpenDev events. However, at the end of the week, I needed a break from that, so I started work on replacing our use of aiohttp with cherrypy. Aiohttp (and related libraries) have broken us a few times recently by rapidly dropping support for what they consider "older" python versions -- yet versions that are still under support on the operating systems we use. Also, several of us have found the python async implementation unnecessarily complex and verbose. Cherrypy supports a thread-based model which fits in well with the rest of Zuul. I don't plan on sending this email next week due to the Summit, so expect the next installment on May 28. Please help write that email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0

Mission statement and tagline
by corvus＠inaugust.com 09 May '18

09 May '18

Hi, The Foundation marketing team has indicated that it would be good to have a mission statement and tagline. The mission statement can be used to help convey what we're trying to accomplish as a community, and the tagline will likely feature prominently on the website and other material. I'd like us all to be involved in creating these, but we need to have the process finished by May 10 in order to be able to update marketing material before the Summit/OpenDev. I propose the following process: * We make (and refine) suggestions this week. Add suggestions to https://etherpad.openstack.org/p/zuul-mission and discuss in the etherpad or here on the mailing list. Feedback from the zuul-core team and the Foundation marketing team (some of whom are subscribed to this list) will be important to formulate good candidates. * Next Monday, the zuul-core team and Foundation marketing team veto any suggestions which are inappropriate. I don't think it's fair to have a mission statement the core team doesn't agree with, and we should trust the marketing folks as domain experts to help us avoid pitfalls. Email me with veto requests; I'll produce a final list. * If there isn't a clear favorite, well have a condorcet election with the remaining candidates, to be complete by May 10. I'll run the election. * If we've vetoed everything, we'll have a special IRC meeting to come up with *something*. To start things off, I've populated the etherpad with a mission statement suggestion from Clark, and some of the favored tagline suggestions from earlier brainstorming. -Jim

2 2

Zuul Stickers
by James Cole 08 May '18

08 May '18

Hey everyone! I just wanted to share a pic of the stickers we’re giving away at the Summit in Vancouver. A link is below in case you can’t see the attachment. https://www.dropbox.com/s/dwb8df3wuqey5i6/Zuul_Stickers.jpg?dl=0 Come grab one in Vancouver! James Cole Graphic Designer OpenStack Foundation

1 0

Polling users of Nodepool's static node driver
by David Shrewsbury 08 May '18

08 May '18

Hi, Is anyone currently using the static driver? And if so, are you making us of the multi-label feature it supports for the static nodes? I ask because I've been trying to change the behavior of the driver to pre-populate ZooKeeper with the static nodes (more in line with how the OpenStack driver works), rather than only having the ZK nodes appear when the node is actually in-use. Doing so fundamentally changes how the static driver works, and this deep dive of the code has revealed this multi-label usage which I think many of us (certainly me) have missed. Changing the driver to operate more like how we expect will mean changing nodepool to support multiple labels across *all* drivers. That's not a simple change, and affects other things, such as statsd reporting (which reports per-label stats). (And if you were wondering how the static driver was getting away with supporting multiple labels before, it was the fact that it *didn't* pre-register its nodes with ZK that allowed that.) One one hand, I can see how supporting multiple labels might be useful (e.g., a node labeled with both ubuntu-xenial and ubuntu-latest). But I don't see a great benefit to doing so. If no one is depending on this multi-label behavior, I'd like to remove it from the static driver. -Dave -- David Shrewsbury (Shrews)

4 6

Zuul project update 2018-05-07
by corvus＠inaugust.com 07 May '18

07 May '18

Hi, This past week has seen the following: Shrews began attempting to fix nodepool static driver to pre-register static nodes in ZooKeeper. This led to the discussion in [1] I pushed patches to add our new logo to the website and docs, as well as some other changes to the website suggested by the marketing team. The OpenDev conference and OpenStack summit are fast approaching. I have a lot to prepare before then -- both in terms of preparing for talks as well as polishing the website and documentation as much as possible, so I'm probably going to focus on those for the next two weeks rather than take up any significant new development work. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000197.html

1 0

Re: [openstack-dev] Overriding project-templates in Zuul
by Joshua Hesketh 02 May '18

02 May '18

> > I think in actuality, both operations would end up as intersections: > > ================ ======== ======= ======= > Matcher Template Project Result > ================ ======== ======= ======= > files AB BC B > irrelevant-files AB BC B > ================ ======== ======= ======= > > So with the "combine" method, it's always possible to further restrict > where the job runs, but never to expand it. Ignoring the 'files' above, in the example of 'irrelevant-files' haven't you just combined the results to expand when it runs? ie, A and C are /not/ excluded and therefore the job will run when there are changes to A or C? I would expect the table to be something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC ABC ================ ======== ======= ======= > > So a job with "files: tests/" and "irrelevant-files: docs/" would do > > whatever it is that happens when you specify both. > > In this case, I'm pretty sure that would mean it reduces to just "files: > tests/", but I've never claimed to understand irrelevant-files and I > won't start now. > Yes, I think you are right that this would reduce to that. However, what about the use case of: files: tests/* irrelevant-files: tests/docs/* I could see a use case where both of those would be helpful. Yes you could describe that as one regex but to the end user the above may be expected to work. Unless we make the two options mutually exclusive I feel like this is a feature we should support. (That said, it's likely a separate feature/functionality than what is being described now). Anyway, I feel like Proposal #2 is more how I would expect the system to behave. I can see an argument for combining the results (and feel like you could evaulate that at the end after combining the branch-matching variants) to give something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC ABC ================ ======== ======= ======= However, that gives the user no way to remove a previously listed option. Thus overwriting may be the better solution (ie proposal #2 as written) unless we want to explore the option of allowing a syntax that says "extend" or "overwrite". Yours in hoping that made sense, Josh

2 1

Re: [openstack-dev] Overriding project-templates in Zuul
by corvus＠inaugust.com 01 May '18

01 May '18

Joshua Hesketh <joshua.hesketh(a)gmail.com> writes: > I might be misunderstanding at which point a job is chosen to be ran and > therefore when it's too late to dissuade it. However, if possible, would it > make more sense for the project-local copy of a job to overwrite the > supplied files and irrelevant-files? This would allow a project to run a > job when it otherwise doesn't match. Imagine that a project with branches has a job added via a template. project-config/zuul.yaml@master: - job: name: my-job vars: {jobvar: true} - project-template: name: myjobs check: jobs: - my-job: vars: {templatevar: true} project/zuul.yaml@master: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} project/zuul.yaml@stable: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} The resulting project config is: - project: jobs: - my-job (branches: master; project-local job) - my-job (branches: master; project-template job) - my-job (branches: stable; project-local job) - my-job (branches: stable; project-template job) When Zuul decides what to run, it goes through each of those in order, evaluates their matchers, and pulls in parents and their variants for each that matches. So a change on the master branch would collect the following variants to apply: my-job (branch: master; project-local job) my-job (job) base (job) my-job (branch: master; project-template job) my-job (job) base (job) It would then apply them in this order: base (job) my-job (job) my-job (branch: master; project-template job) my-job (branch: master; project-local job) To further restrict a project-local job with a "files:" matcher would cause the project-local job not to match, but the project-template job will still match, so the job gets run. That's the situation we have today, which is what I meant by "it's too late to dissuade it". Regarding the suggestion to overwrite it, we would need to decide which of the possible variants to overwrite. Keep in mind that there are 3 independent matchers operating on all the variants (branches, files, irrelevant-files). Does a project-local job with a "files:" matcher overwrite all of the variants? Just the ones which match the same branch? That would probably be the most reasonable thing to do. In my mind, that effectively splits the matchers into two categories: branch matchers, and file matchers. And they would behave differently. Zuul could collect the variants as above, considering only the branch matchers. It could then apply all of the variants in the normal manner, treating files and irrelevant-files as normal attributes which can be overwritten. Then, once it has composed the job to run based on all the matching variants, it would only *then* evaluate the files matchers. If they don't match, then it would not run the job after all. I think that's a very reasonable way to solve the issue as well, and I believe it would match people's expectations. Ultimately, the outcome will be very similar to the proposal I made except that rather than being combined, the matchers will be overwritten. That means that if you want to expand the set of irrelevant-files for a job, you would have to copy the set from the parent. There's one other aspect to consider -- it's possible to create a job like this: - job: name: doc-job - jobs: name: doc-job files: docs/index.rst vars: {rebuild_index: true} Which means: there's a normal docs job with no variables, but if docs/index.rst is changed, set the rebuild_index variable to true. Either approach (combine vs overwrite) eliminates the ability to do this within a project or project-template stanza. But the "combine" approach still lets us do this at the job level. We could still support this in the overwrite approach, however, I think it might be simpler to work with if we eliminated it as well and just always treated files and irrelevant-files matchers as overwriteable attributes. It would no longer be possible to implement the above example, but I'm not sure it's that useful anyway? > What happens when something is in both files and irrelevant-files? If the > project-template is trying to say A is in 'files', but the project-local > says A is in 'irrelevant-files', should that overwrite it? I think my statement and table below was erroneous: >> This effectively causes the "files" and "irrelevant-files" attributes on >> all of the project-local job definitions matching a given branch to be >> combined. The combination of multiple files matchers behaves as a >> union, and irrelevant-files matchers as an intersection. >> >> ================ ======== ======= ======= >> Matcher Template Project Result >> ================ ======== ======= ======= >> files AB BC ABC >> irrelevant-files AB BC B >> ================ ======== ======= ======= I think in actuality, both operations would end up as intersections: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC B ================ ======== ======= ======= So with the "combine" method, it's always possible to further restrict where the job runs, but never to expand it. As to your question about what if something appears in both -- in my "combine" proposal, I would continue to treat them independently, so if a project-local variant has an "irrelevant-files:" matcher, and a project-template variant has a "files:" matcher, then the usual nonsense would happen: they'd both have to match. So a job with "files: tests/" and "irrelevant-files: docs/" would never run because it's impossible to satisfy both. That actually matches some current behavior we have -- a job must match a "real" job definition as well as at least one project-template or project-local job definition, so a "job" with "files: tests/" and a project-template with "irrelevant-files: docs/" will behave the same way today. However, if we go with the "overwrite" proposal, since we're altering the behavior of the non-branch matchers anyway, that might be a fine time to pair them up and treat them as a unit. Then, whichever is the latest would win. So a job with "files" and a project-template with "irrelevant-files" would end up only having an irrelevant-files attribute. Then a further project-local stanza with only "files" would eliminate the irrelevant-files attribute leaving only the files value. Okay, let's summarize: Proposal 1: All project-template and project-local job variants matching the item's branch must also match the item. * Files and irrelevant-files on project-template and project stanzas are essentially combined in a set intersection. * It's possible to further reduce the scope of jobs, but not expand. * Files and irrelevant-files are still independent matchers, and if both are present, both must match. * It's not possible to alter a job attribute by adding a project-local variant with only a files matcher (it would cause the whole job to run or not run). But it's still possible to do that in the main job definition itself. Proposal 2: Files and irrelevant-files are treated as overwriteable attributes and evaluated after branch-matching variants are combined. * Files and irrelevant-files are overwritten, so the last value encountered when combining all the matching variants (looking only at branches) wins. * Files and irrelevant-files will be treated as a pair, so that if "irrelevant-files" appears, it will erase a previous "files" attribute. * It's possible to both reduce and expand the scope of jobs, but the user may need to manually copy values from a parent or other variant in order to do so. * It will no longer be possible to alter a job attribute by adding a variant with only a files matcher -- in all cases files and irrelevant-files are used solely to determine whether the job is run, not to determine whether to apply a variant. I think both would be good solutions to the problem. The key points for me are whether we want to keep the "alter a job attribute with variant with a files matcher" functionality (the "rebuild_index" example from above), and whether the additional control of overwriting the matchers (at the cost of redundancy in configuration) is preferable to combining the matchers. -Jim

1 1

Zuul Logo
by James Cole 30 Apr '18

30 Apr '18

Hello Zuul Enthusiasts, I’m James, a graphic designer with the OpenStack Foundation, and I’ve been working with Jim and the Zuul-core team to design an official Zuul logo. As you probably know, the goal has been to create something unique that can be used across a variety of mediums, such as an avatar on GitHub, on the website, as badge, and on other types of Zuul communications. After a few iterations, the team is leaning towards the concept featured in the attached PDF. The symbol in the logo was inspired by the geometry of the gate in Ghostbusters, since Zuul focuses on project gating and the name “Zuul” originated from the Zuul the Gatekeeper in the movie. More generally, the logo consists of a triangular, linework shape with the Zuul word mark placed underneath. The logo can be used a number of different ways and still maintain legibility at smaller sizes. The mockups in the attached PDF show a few different use cases, and offers some supporting typography and color information at the end. In case the email attachment did not make it to you, here is a link to view the document on Dropbox <https://www.dropbox.com/s/vv4yszyev1q9pbs/Zuul_Logo.pdf?dl=0>. We need to finalize something soon, since Zuul will be featured at the OpenStack Summit in May. So, we would love to hear what you think about this idea, if there is a detail we missed, or if you have any other feedback. Thank you! James Cole Graphic Designer OpenStack Foundation

10 22

Overriding project-templates
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, We tried to make the Zuul v3 configuration language as intuitive as possible, and incorporated a lot that we learned from our years running Zuul v2. One thing that we didn't anticipate was how folks would end up wanting to use a job in both project-templates *and* local project stanzas. Essentially, we had assumed that if you wanted to control how a job was run, you would add it to a project stanza directly rather that use a project-template. It's easy to do so if you use one or the other. However, it turns out there are lots of good reasons to use both. For example, in a project-template we may want to establish a recommended way to run a job, or that a job should always be run with a set of related jobs. Yet a project may still want to indicate that job should only run on certain changes in that specific repo. To be very specific -- a very commonly expressed frustration is that a project can't specify a "files" or "irrelevant-files" matcher to override a job that appears in a project-template. Reconciling those is difficult, largely because once Zuul decides to run a job (for example, by a declaration in a project-template) it is impossible to dissuade it from running that job by adding any extra configuration to a project. We need to tread carefully when fixing this, because quite a number of related concepts could be affected. For instance, we need to preserve branch independence (a change to stop running a job in one branch shouldn't affect others). And we need to preserve the ability for job variants to layer on to each other (a project-local variant should still be able to alter a variant in a project-template). I propose that we remedy this by making a small change to how Zuul determines that a job should run: When a job appears multiple times on a project (for instance if it appears in a project-template and also on the project itself), all of the project-local variants which match the item's branch must also match the item in order for the job to run. In other words, if a job appears in a project-template used by a project and on the project, then both must match. This effectively causes the "files" and "irrelevant-files" attributes on all of the project-local job definitions matching a given branch to be combined. The combination of multiple files matchers behaves as a union, and irrelevant-files matchers as an intersection. ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC B ================ ======== ======= ======= I believe this will address the shortcoming identified above, but before we get too far in implementing it, I'd like to ask folks to take a moment and evaluate whether it will address the issues you've seen, or if you foresee any problems which I haven't anticipated. Thanks, Jim

1 0

Zuul project update 2018-04-30
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, This past week has seen the following: We released Zuul 3.0.2. Note that we added a dependency on fb-re2, a python wrapper for the re2 library, which is now a binary dependency. The long-term plan is to start moving all of our regular expression parsing to re2, mostly because it is resistant to denial-of-service attacks. Clint has some patches in progress to do that. But in the mean time, any new uses of regular expressions should use re2 so that we don't create additional debt. We plan on issuing a Nodepool release this week with several accumulated bugfixes and improvements. Tobiash improved the merger so that it is more able to recover from an unclean shutdown. He also has a change [1] in review to add regular expression support to the project stanza so that you can, for example, specify that a job should run on all projects. Last week (and over the weekend) we merged my work to reduce memory consumption for configuration changes. It's now reasonable to consider configuration changes as not any more expensive than normal changes. The CPU time needed to calculate configuration changes is also reduced. This week I plan on returning to reviewing our outstanding specs. Tristan has been working on adding MQTT schema documentation. He also has been working on adding OpenAPI documentation but couldn't find a sphinx doc generator or how to embed swagger-ui in zuul-web. It sounds like we could use some investigation in to how to best handle documentation for the REST API. He also has a patch [2] to support secure diskimage env-vars in nodepool. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/535713 [2] https://review.openstack.org/564687

1 0

Proposed updates to website to empower our non-coder friends
by Monty Taylor 26 Apr '18

26 Apr '18

Hey everybody! As part of the work with the Foundation staff on getting Zuul set up, we've chatted a bit about how to manage the website content. Obviously from the dev side, we find it important to be able to edit content like source code, propose those changes to gerrit and to review each other's changes. There are, however, folks out there who would like to help us do things like marketing who are not crazy about the concept of editing raw html. SO - we went and did a little bit of exploration to see what we could find. Actually, what really happened is that we agreed we'd just go with the html thing we have currently for now and circle back later to figure out a CMS + code review system that would work for everyone. But then I had a weekend and a personal project that had pretty much the same set of requirements. End result - what I'd like to propose is that we adopt a combination of using gatsby [1] as a static site generator and netlify-cms [2] as a web-based CMS interface for editing content and submitting changes to gerrit. Both gatsby and netlify-cms are completely client-side - there are no server side components to run. I found them by looking at https://headlesscms.org/ and https://www.staticgen.com/ gatsby ------ gatsby is a Javascript/React-based static site generator that allows managing content in markdown files. It has netlify-cms plugin support, and the build toolchain works very similarly to the new javascript toolchain for zuul-dashboard. I've got a first rough-draft of this part done that I'll push up for review today or tomorrow. netlify-cms ----------- netlify-cms is a Javascript/React-based CMS for static site generators. It has support for either pushing changes directly in to git or for submitting them as proposed changes. It currently basically only supports github - but the github interface is broken out into a driver layer. In fact, they have two different options- one that talks to github directly and one that talks to a netlify git-gateway service, so the plugin layer isn't just theoretical. We'll need to write a gerrit driver, but that shouldn't be terribly hard. Proposed Plan ------------- We can keep this pretty simple. Step one- migrate existing site from static html to gatsby + zuul jobs. Step two - write gerrit driver for netlify-cms. Step three - add netlify-cms to website. Why not use ... --------------- There are many options out there for static site generation. I like gatsby because it's pure-javascript rather than having a go, ruby or python component. I've personally come to really like keeping web things in javascript ... especially since our zuul-dashboard is doing javascript build tooling now too. There is also tight integration support with netlify-cms (which is also a good choice for the same reason) - and it was super easy for me to get up and going with one of their examples. [1] https://www.gatsbyjs.org/ [2] https://www.netlifycms.org/

3 3

Zuul project update 2018-04-23
by corvus＠inaugust.com 25 Apr '18

25 Apr '18

Hi, This past week has seen the following: Shrews started testing Ansible 2.5 and has related changes up for review[1]. I made a video[2] to upload to the website which explains the gating algorithm. I expect it to be merged soon. I've also resumed work on the configuration loading refactor which will allow us to use less memory for dynamic configuration changes. I plan on continuing that this week. Tobiash has been improving scalability with parent-change-enqueued event and many github app installations[3]. Part or that stack improves github testing. Next week, he plans on working on weak job dependencies (jobs that still run even if their parents are skipped -- so that Zuul can decide that an image build job does or doesn't need to run), along with finishing up GitHub branch status protection handling. Clarkb has (with Shrews) made zuul-web more asynchronous. He points out that we've merged a number of bugfixes, including updates to zuul-changes and encrypt_secret. We've alo removed the zuul-cloner command. He also asks whether it's time to make another release. TristanC is working on `allowed-triggers` and `allowed-reporters` tenant settings to better support a multi-tenant environment with a multitude of connections. The nodepool driver work mentioned last week has merged. Another change[4] is proposed to make the OpenStack NodeLauncher generic. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/topic:zuul-ansible-2.5 [2] https://review.openstack.org/562005 [3] https://review.openstack.org/#/q/topic:parent-change-enqueued+(status:open+… [4] https://review.openstack.org/535555

2 1

Re: Zuul + OpenStack Foundation update
by Lauren Sell 24 Apr '18

24 Apr '18

Hi everyone, Thanks very much Jim for providing the update on our plans heading into the Vancouver Summit. I want to sincerely apologize for the miscommunication around the board meeting. We presented an update on making Zuul an independent project at the Gold Member meeting yesterday and again at the Board Meeting today, but unfortunately the entire presentation today was scheduled during the executive session and not able to be heard by the public. It was a definitely oversight and miscommunication. I want to share the deck we presented here and am happy to do a call for those who want to run through it, or simply chat about it in IRC. We welcome any feedback on the plans: https://docs.google.com/presentation/d/1iR_Q0ntV5BAs1TNjN-b1JBnsfahb2hI4IxM… The primary feedback we received during the gold member meeting was trying to understand where Zuul would fit under the OpenStack Foundation (ie. is it still part of OpenStack, or an adjacent project with independent governance?) and where Zuul fits in the CI/CD landscape (ie. what’s competitive and complementary). For people who are newer to the space, there has been a lot more activity recently around XCI and the OpenCI initiative and of course existing tools like Jenkins and new ones like Spinnaker, and they need help parsing what’s an effort by a group of people versus what’s actually software and when you would use which software :) Hope that is useful, and again, apologies for the miscommunication. I’m now subscribed to the mailing list and am happy to discuss here :) Thanks, Lauren PS If you can’t view the deck in google docs, I’m happy to send over a PDF. I just didn’t want to send an attachment to the mailing list... >> Begin forwarded message: >> >> From: corvus(a)inaugust.com <mailto:corvus@inaugust.com> (James E. Blair) >> Subject: Zuul + OpenStack Foundation update >> Date: April 11, 2018 at 5:28:08 PM CDT >> To: Zuul Discussion List <zuul-discuss(a)lists.zuul-ci.org <mailto:zuul-discuss@lists.zuul-ci.org>> >> >> Hi, >> >> Several of us on the zuul-core team have been working with the OpenStack >> Foundation staff on establishing and launching Zuul as a top-level >> project under the Foundation's umbrella. >> >> As a free/open-source project, and one that follows OpenStack's "four >> opens" we generally strive to use public channels for collaboration. >> However, when making a change like this, that is not always practical, >> especially in the early stages. I think we'll get there soon, but in >> the mean time, I ask you to accept Monty, Clark, Jeremy and I as >> delegates of the project to work with the Foundation on this. >> >> I'll try to keep everyone updated as best as possible with what we're >> working on, though sometimes those updates may need to be vague. Please >> feel free to contact me or the others privately if you have specific >> concerns you want to ensure we address. Here are updates from our most >> recent discussions: >> >> * There will be discussion about Zuul at the OpenStack board of >> directors meeting on April 24. That will be a phone call to which >> anyone is welcome to listen. >> >> * The Foundation will promote Zuul at the upcoming OpenStack Summit in >> Vancouver. It will be officially announced as a Foundation project on >> the keynote stage and via press release. >> >> * We are working with the Foundation's design team on a logo. We want >> to have something that can be adapted to work as an avatar (i.e., on >> GitHub), on the website, as a badge, and on slide decks. >> >> * The Foundation marketing team has begun a review of the website >> content and will be assisting with changes. Included in this is an >> effort to collect user stories for Zuul. >> >> * We're looking into implementing a system to manage the website content >> which is both Gerrit-friendly and has a workflow suitable for those >> unaccustomed to using Git. Monty has volunteered to research and >> report back with some options to explore. >> >> -Jim >> >> _______________________________________________ >> Zuul-discuss mailing list >> Zuul-discuss(a)lists.zuul-ci.org <mailto:Zuul-discuss@lists.zuul-ci.org> >> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss >

1 0

Simplified DSL
by Tristan Cacqueray 21 Apr '18

21 Apr '18

Hello folks, Some users may find job written in Ansible hard to reproduce, and I was wondering if we could add a new "exec" job attribute that would simply be shell task to be executed on a single node. This would let us write zuul.yaml file more similar to other in-repo ci system. For example a zuul.yaml could be: - job: name: my-job exec: - mvn build - mvn test - project: check: jobs: - my-job gate: jobs: - my-job Moreover we could go a step further and streamline that full description in a single stanza: - ci-job: - mvn build - mvn test Which could be translated into a unique job name attached to the check and gate pipeline by default. What do you think? -Tristan

8 9

Question regarding using etcd instead of Zookeeper
by Tristan Cacqueray 21 Apr '18

21 Apr '18

Hello folks, From a Kubernetes operator point of view, the Zookeeper component sounds redundant with the already running etcd service. Moreover etcd seems more active and it shows better overall performance. So my question is why Zookeeper was picked in the first place, and would it be possible to support/migrate to etcd in the future? Thanks in advance, -Tristan

9 11

Reverse of required-projects.override-checkout?
by Mike Pagel 20 Apr '18

20 Apr '18

Hi all, I am using zuul now for a while at work and I am struggling to properly configure the following use-case. Hope I don't miss the obvious. I have two repositories A and B. A is an internal, B is an external project from another department. We define our jobs in A's zuul.yaml. B is using different branch names (e.g., because they deliver to other teams as well). A/zuul.yaml (master branch): # build the "master-a" branch of B for A's "master": job: required-projects: name: B override-checkout: master-a Similarly, when we have a maintenance branch "v1.0" in A, B might call it "v1.0-a": A/zuul.yaml (v1.0 branch): # build the "v1.0-a" branch of B for A's "v1.0": job: required-projects: name: B override-checkout: v1.0-a Now, without duplication of these job definitions, how would I run the correct job variant when checking a change in B, let's say in the "v1.0-a" branch, where I cannot fall back to "master" as default branch? AnsibleJob.resolveBranch() has some logic, but I think that's still the forward reference. Could I use job.branches for this, as in: A/zuul.yaml (v1.0 branch): # build this "v1.0" branch in A if a change occurred in (any) "v1.0-a" branch: job: branches: v1.0-a required-projects: name: B override-checkout: v1.0-a I see code in and calls to model.Job.changeMatches() that could suggest this behavior. Or would it try to als check out A's "v1.0-a" branch, which is non-existent? Any hints are greatly appreciated. Mike

2 3

GitHub OAuth application
by Tristan Cacqueray 18 Apr '18

18 Apr '18

Hello folks, Would it be possible to adapt the GitHub driver to use user's access instead of the Zuul unique app key/api token? Ideally, users would register on the scheduler and get redirected to grant access to their GitHub identity through an OAuth app. Then zuul could manage the project's hook. That would be the easiest workflow from an user point of view. Though this would require some sort of user management. Another solution would be to create the access_token out of bound and extend the driver configuration to support using multiple tokens to be used per projects or oganizations. If this can be implemented without a major rewrite of the driver, that sounds like something worthy as it would enable new use cases and it may mitigate rate limiting issue for large deployment. Regards, -Tristan

2 2

Zuul project update 2018-04-16
by corvus＠inaugust.com 16 Apr '18

16 Apr '18

Hi, Last week, we agreed to suspend weekly IRC meetings and instead try to conduct more of our business over email, or in ad-hoc IRC meetings as needed. In an attempt to keep the cadence provided by the meetings, we're going to send out a weekly email update. Hopefully this will help us all keep up with what's going on in the project. Please feel free to reply to this email with further discussion. Last week, we released Zuul 3.0.1. Shrews fixed a common bug seen in long streaming involving multi-byte characters across read boundaries, and is continuing work on making zuul-web more async. Also in progress: improvements to the zuul-from-scratch document. [1] I posted a spec about container support [2]. It's more of a starting point for discussion than a complete plan at this point. It's collected a number of comments so far, and will certainly be revised several times before it is finalized. This week, I plan to work on resuming the work I started before the 3.0.0 release to reduce the memory used by dynamic configuration changes. Mordred has switched us to using nodeenv for javascript dependencies which makes it easier to run the tests locally and means we can use the standard tox-py35 job. He also upgraded webpack to version 4. This week has us moving the dashboard from angularjs v2 to angular.io v5. [3] Tristan has updated the MQTT driver which should be ready now and updated several jobs in zuul-jobs (ansible-lint, review, spec and upload-to-galaxy). In nodepool, the run_handler refactor (which is intended to make adding more drivers easier) is ready for review. He also updated zuul-changes.py to support the new zuul-web api routes. Tobiash fixed a number of bugs, and has patches up to refactor the executor's governor into a sensor API for easier extension. He added support for postgres (we run tests on PG and mysql now), and regex matching of github status. This week he plans on trying out Ansible 2.5, improving scalability of parent-change-enqueued event across many github app installations, and adding tests to status protection handling. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+b… [2] https://review.openstack.org/560136 [3] https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+b…

1 0

Next Zuul release (3.0.1?)
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, We have some mildly-urgent bugfixes merged or ready to merge in Zuul, so I'd like to make a release sometime in the next few days. By mildly-urgent, I mean that the sooner we release the fixes, the less problems people will have in the future (either by creating jobs relying on wrong behavior, or system configurations which are problematic). The items I have on the list are: * Roles checkout fix (merged) * Github3.py version bump (ready to merge) * Postgress fixes/support (merging as I write this) We've also merged some other bugfixes, and may slip some others in during the remaining time. Are there any other items of similar urgency to the three above that we need to release ASAP? Once we're satisfied (probably tomorrow?) I'll release 3.0.1. I think that since we're only focusing on bugfixes, the micro version bump is appropriate. I don't think Nodepool needs a corresponding release. -Jim

4 3

Zuul + OpenStack Foundation update
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, Several of us on the zuul-core team have been working with the OpenStack Foundation staff on establishing and launching Zuul as a top-level project under the Foundation's umbrella. As a free/open-source project, and one that follows OpenStack's "four opens" we generally strive to use public channels for collaboration. However, when making a change like this, that is not always practical, especially in the early stages. I think we'll get there soon, but in the mean time, I ask you to accept Monty, Clark, Jeremy and I as delegates of the project to work with the Foundation on this. I'll try to keep everyone updated as best as possible with what we're working on, though sometimes those updates may need to be vague. Please feel free to contact me or the others privately if you have specific concerns you want to ensure we address. Here are updates from our most recent discussions: * There will be discussion about Zuul at the OpenStack board of directors meeting on April 24. That will be a phone call to which anyone is welcome to listen. * The Foundation will promote Zuul at the upcoming OpenStack Summit in Vancouver. It will be officially announced as a Foundation project on the keynote stage and via press release. * We are working with the Foundation's design team on a logo. We want to have something that can be adapted to work as an avatar (i.e., on GitHub), on the website, as a badge, and on slide decks. * The Foundation marketing team has begun a review of the website content and will be assisting with changes. Included in this is an effort to collect user stories for Zuul. * We're looking into implementing a system to manage the website content which is both Gerrit-friendly and has a workflow suitable for those unaccustomed to using Git. Monty has volunteered to research and report back with some options to explore. -Jim

1 0

Deprecation notice: Remove support for extensions as lists in zuul-jobs/roles/stage-output - change https://review.openstack.org/540890
by Andreas Jaeger 11 Apr '18

11 Apr '18

Andrea Frittoli has updated roles/stage-output in zuul-jobs to handle dicts instead of lists for extensions - and updated all usage in OpenStack itself. His review https://review.openstack.org/540890 removes now handling of lists. Please check that none of your playbooks and jobs uses extensions with stage-output and update them. I put a WIP on the change and will remove it in two weeks so that we can merge, please update your files as needed and feel free to review the change, Andreas -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

1 0

Spec for using containers as Zuul build resources
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, We've all discussed using containers in Zuul for some time, but it's been difficult to get on the same page. Partly because of differing use cases and assumptions, and partly because Zuul brings some unique things to the table which need to be integrated into the container mindset. I've uploaded the first draft of a spec which I hope will address this and set a path for us to successfully integrate containers: https://review.openstack.org/560136 It's long on exposition and short on work items at this point. I think first it's useful for us to look at it from a distance and see if it handles the use cases we think we need to cover, and if the general approaches to workflow will work. Then, we should focus on the details, and expand it to ensure that we describe the actual processes we're going to implement. And finally, it should grow a list of work items so we can get things done. If you haven't reviewed a spec like this before, know this: it's a very iterative process. Please leave detailed inline review comments in Gerrit, where we may engage in a back-and-forth discussion for a while. If it gets awkward to have that discussion in Gerrit, we can move it to the mailing list, but in general, it's preferable to keep the discussion in Gerrit where the context and history are all together. After some discussion, I will update the spec with suggested changes and we'll go through the process again. The goal is to achieve consensus during the review process and merge a polished result, not to merge the change ASAP. This also establishes a specs section of the Zuul developer documentation. We had a brief discussion in an earlier IRC meeting about how to handle specs now that openstack-infra/infra-specs is no longer appropriate. I think putting them directly in the Zuul repo is worth trying. The main danger there is accumulating too much documentation which does not reflect the current state of the project, but this includes big red warnings, so maybe that will help. It also looks like a significant amount of this work may end up happening in the nodepool repo as opposed to zuul, but as long as we don't have a dedicated place for cross-project specs, I think the zuul repo is a reasonable place to start. -Jim

2 1

Zuul meeting time
by Tobias.Henkel＠bmw.de 09 Apr '18

09 Apr '18

Hi zuul community, Currently the zuul meeting is at 22UTC which is currently 23:00 localtime or soon 0:00 within summer time at my location. That makes it impossible for me to participate regularly. However I’d really like to attend the zuul meeting. So I’d like to ask zuul folks about your opinion about shifting it around 20ish UTC. I wouldn’t have any strong opinion about the day. Thanks and kind regards Tobias

9 15

Fedora packages
by Tristan Cacqueray 02 Apr '18

02 Apr '18

Greeting folks, I have created a Nodepool package[0] in Fedora and it is proposed for f27 and f28 updates here[1] and there[2]. Please leave a +1 if the package works for you. The package contains service files, default configurations and man pages. The drivers are splitted in sub packages so that the openstacksdk is only pulled by the nodepool-driver-openstack package. A quick start install doc using dnf could then be reduced to: dnf install nodepool-launcher nodepool-driver-static systemctl start nodepool-launcher Regarding a Zuul package, it doesn't seems feasible at the moment since Fedora ships Ansible version 2.4 and 2.5. Though perhaps with modularity we can have a Zuul module that depends on Ansible 2.3. Regards, -Tristan [0]: https://apps.fedoraproject.org/packages/nodepool [2]: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d2da815b3e [3]: https://bodhi.fedoraproject.org/updates/FEDORA-2018-75021b046f

2 1

Software Factory 3.0 released
by Tristan Cacqueray 30 Mar '18

30 Mar '18

Greeting folks, We are pleased to announce the release of Software Factory version 3.0: http://www.softwarefactory-project.io/releases/3.0 It is a software development forge that integrates the last version of Zuul and Nodepool along with a few patches that couldn't make it into the upstream release: * Ansible-2.4 support: https://review.openstack.org/535839 * Dynamic tenant configuration: https://review.openstack.org/535878 * MQTT reporter: https://review.openstack.org/535543 * Job and Projects dashboard: https://review.openstack.org/550979 * Broken config protection: https://review.openstack.org/535511 * Zookeeper auto-retry: https://review.openstack.org/535537 * OpenContainer Nodepool provider: https://review.openstack.org/535556 * Kubernetes Nodepool provider: https://review.openstack.org/535557 Best regards, The Software Factory Team

3 3

Next steps after 3.0.0 release
by corvus＠inaugust.com 29 Mar '18

29 Mar '18

Hi, Now that 3.0.0 is released[1] (yay!), it's a good time to go over what we need to do next. The first order of business is to land the changes to add reno: https://review.openstack.org/536861 https://review.openstack.org/262597 You should read the docs that those changes introduce (they'll be in the developer docs for future reference), but the key bit is: When adding a noteworthy feature, fixing a noteworthy bug or introducing a behavior change that a user or operator should know about, it is a good idea to add a release note to the same patch. Every release from this point out will have release notes included in the documentation. We relaxed our review standards significantly during the feature branch phase, but we've been returning to normal for the past few months. At this point, the master branches of Nodepool and Zuul should always be kept working and ready for release. Gating usually handles the technical side of that, but don't lose track of the intent when it comes to other aspects. Please make sure that changes have documentation if they affect a user-visible aspect of the system, have a release note if the change is noteworthy, and have tests if they alter behavior. If a change is missing these things, please ask about it. These rules don't need to be too rigid, but if we flex them, we should be in the habit of discussing it and establishing that there's a good reason. We need to be careful about making changes to the zuul-jobs and zuul-base-jobs repos now. We'll write up a policy and include it in the docs for those repos, but the important thing is that we can't use new Zuul features in those repos until we've given folks a chance to upgrade (so probably some number of weeks after the feature appears in a release). And we can't remove features from Zuul until we've removed their use from zuul-jobs, and given folks who might be running a fork of zuul-jobs time to update. I expect us to release Zuul and Nodepool much more often now. Any time we make a significant change and establish that the software is working as intended, we should release. We've assigned some very specific meaning to version numbers recently, but don't think that will continue. The versions we release will be what they are depending on what has landed when we release. Thanks everyone for your work on the release, and I look forward to reviewing all the patches that are waiting in the wings! -Jim [1] http://lists.zuul-ci.org/pipermail/zuul-announce/2018-March/000006.html

1 0

Docs and git repos on zuul-ci.org
by corvus＠inaugust.com 28 Mar '18

28 Mar '18

Hi, We now have all of the Zuul documentation hosted on the Zuul website[1], so folks may want to start referencing those URLs, rather than docs.openstack.org. We also have the Zuul git repos mirrored under the Zuul domain at [2]. That page shows only the Zuul-related git repos. Additionally, if you're referencing the zuul-jobs or zuul-base-jobs repos, you may want to switch your connection to use git.zuul-ci.org instead. Note that if you do that, the project name will be "git.zuul-ci.org/zuul-jobs" -- with no "openstack-infra" prefixed to the name, as there are no organization prefixes on repositories mirrored at git.zuul-ci.org. [1] https://zuul-ci.org/docs/ [2] http://git.zuul-ci.org/ -Jim

2 2