- Zuul-discuss - lists.opendev.org

Zuul project update 2018-05-07
by corvus＠inaugust.com 08 May '18

08 May '18

Hi, This past week has seen the following: Shrews began attempting to fix nodepool static driver to pre-register static nodes in ZooKeeper. This led to the discussion in [1] I pushed patches to add our new logo to the website and docs, as well as some other changes to the website suggested by the marketing team. The OpenDev conference and OpenStack summit are fast approaching. I have a lot to prepare before then -- both in terms of preparing for talks as well as polishing the website and … [View More]

1 0

Re: [openstack-dev] Overriding project-templates in Zuul
by Joshua Hesketh 02 May '18

02 May '18

> > I think in actuality, both operations would end up as intersections: > > ================ ======== ======= ======= > Matcher Template Project Result > ================ ======== ======= ======= > files AB BC B > irrelevant-files AB BC B > ================ ======== ======= ======= > > So with the "combine" method, it's always possible to further restrict > where the job runs, but never to expand it.… [View More] Ignoring the 'files' above, in the example of 'irrelevant-files' haven't you just combined the results to expand when it runs? ie, A and C are /not/ excluded and therefore the job will run when there are changes to A or C? I would expect the table to be something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC ABC ================ ======== ======= ======= > > So a job with "files: tests/" and "irrelevant-files: docs/" would do > > whatever it is that happens when you specify both. > > In this case, I'm pretty sure that would mean it reduces to just "files: > tests/", but I've never claimed to understand irrelevant-files and I > won't start now. > Yes, I think you are right that this would reduce to that. However, what about the use case of: files: tests/* irrelevant-files: tests/docs/* I could see a use case where both of those would be helpful. Yes you could describe that as one regex but to the end user the above may be expected to work. Unless we make the two options mutually exclusive I feel like this is a feature we should support. (That said, it's likely a separate feature/functionality than what is being described now). Anyway, I feel like Proposal #2 is more how I would expect the system to behave. I can see an argument for combining the results (and feel like you could evaulate that at the end after combining the branch-matching variants) to give something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC ABC ================ ======== ======= ======= However, that gives the user no way to remove a previously listed option. Thus overwriting may be the better solution (ie proposal #2 as written) unless we want to explore the option of allowing a syntax that says "extend" or "overwrite". Yours in hoping that made sense, Josh [View Less]

2 1

Re: [openstack-dev] Overriding project-templates in Zuul
by corvus＠inaugust.com 02 May '18

02 May '18

Joshua Hesketh <joshua.hesketh(a)gmail.com> writes: > I might be misunderstanding at which point a job is chosen to be ran and > therefore when it's too late to dissuade it. However, if possible, would it > make more sense for the project-local copy of a job to overwrite the > supplied files and irrelevant-files? This would allow a project to run a > job when it otherwise doesn't match. Imagine that a project with branches has a job added via a template. project-config/… [View More]zuul.yaml@master: - job: name: my-job vars: {jobvar: true} - project-template: name: myjobs check: jobs: - my-job: vars: {templatevar: true} project/zuul.yaml@master: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} project/zuul.yaml@stable: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} The resulting project config is: - project: jobs: - my-job (branches: master; project-local job) - my-job (branches: master; project-template job) - my-job (branches: stable; project-local job) - my-job (branches: stable; project-template job) When Zuul decides what to run, it goes through each of those in order, evaluates their matchers, and pulls in parents and their variants for each that matches. So a change on the master branch would collect the following variants to apply: my-job (branch: master; project-local job) my-job (job) base (job) my-job (branch: master; project-template job) my-job (job) base (job) It would then apply them in this order: base (job) my-job (job) my-job (branch: master; project-template job) my-job (branch: master; project-local job) To further restrict a project-local job with a "files:" matcher would cause the project-local job not to match, but the project-template job will still match, so the job gets run. That's the situation we have today, which is what I meant by "it's too late to dissuade it". Regarding the suggestion to overwrite it, we would need to decide which of the possible variants to overwrite. Keep in mind that there are 3 independent matchers operating on all the variants (branches, files, irrelevant-files). Does a project-local job with a "files:" matcher overwrite all of the variants? Just the ones which match the same branch? That would probably be the most reasonable thing to do. In my mind, that effectively splits the matchers into two categories: branch matchers, and file matchers. And they would behave differently. Zuul could collect the variants as above, considering only the branch matchers. It could then apply all of the variants in the normal manner, treating files and irrelevant-files as normal attributes which can be overwritten. Then, once it has composed the job to run based on all the matching variants, it would only *then* evaluate the files matchers. If they don't match, then it would not run the job after all. I think that's a very reasonable way to solve the issue as well, and I believe it would match people's expectations. Ultimately, the outcome will be very similar to the proposal I made except that rather than being combined, the matchers will be overwritten. That means that if you want to expand the set of irrelevant-files for a job, you would have to copy the set from the parent. There's one other aspect to consider -- it's possible to create a job like this: - job: name: doc-job - jobs: name: doc-job files: docs/index.rst vars: {rebuild_index: true} Which means: there's a normal docs job with no variables, but if docs/index.rst is changed, set the rebuild_index variable to true. Either approach (combine vs overwrite) eliminates the ability to do this within a project or project-template stanza. But the "combine" approach still lets us do this at the job level. We could still support this in the overwrite approach, however, I think it might be simpler to work with if we eliminated it as well and just always treated files and irrelevant-files matchers as overwriteable attributes. It would no longer be possible to implement the above example, but I'm not sure it's that useful anyway? > What happens when something is in both files and irrelevant-files? If the > project-template is trying to say A is in 'files', but the project-local > says A is in 'irrelevant-files', should that overwrite it? I think my statement and table below was erroneous: >> This effectively causes the "files" and "irrelevant-files" attributes on >> all of the project-local job definitions matching a given branch to be >> combined. The combination of multiple files matchers behaves as a >> union, and irrelevant-files matchers as an intersection. >> >> ================ ======== ======= ======= >> Matcher Template Project Result >> ================ ======== ======= ======= >> files AB BC ABC >> irrelevant-files AB BC B >> ================ ======== ======= ======= I think in actuality, both operations would end up as intersections: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC B ================ ======== ======= ======= So with the "combine" method, it's always possible to further restrict where the job runs, but never to expand it. As to your question about what if something appears in both -- in my "combine" proposal, I would continue to treat them independently, so if a project-local variant has an "irrelevant-files:" matcher, and a project-template variant has a "files:" matcher, then the usual nonsense would happen: they'd both have to match. So a job with "files: tests/" and "irrelevant-files: docs/" would never run because it's impossible to satisfy both. That actually matches some current behavior we have -- a job must match a "real" job definition as well as at least one project-template or project-local job definition, so a "job" with "files: tests/" and a project-template with "irrelevant-files: docs/" will behave the same way today. However, if we go with the "overwrite" proposal, since we're altering the behavior of the non-branch matchers anyway, that might be a fine time to pair them up and treat them as a unit. Then, whichever is the latest would win. So a job with "files" and a project-template with "irrelevant-files" would end up only having an irrelevant-files attribute. Then a further project-local stanza with only "files" would eliminate the irrelevant-files attribute leaving only the files value. Okay, let's summarize: Proposal 1: All project-template and project-local job variants matching the item's branch must also match the item. * Files and irrelevant-files on project-template and project stanzas are essentially combined in a set intersection. * It's possible to further reduce the scope of jobs, but not expand. * Files and irrelevant-files are still independent matchers, and if both are present, both must match. * It's not possible to alter a job attribute by adding a project-local variant with only a files matcher (it would cause the whole job to run or not run). But it's still possible to do that in the main job definition itself. Proposal 2: Files and irrelevant-files are treated as overwriteable attributes and evaluated after branch-matching variants are combined. * Files and irrelevant-files are overwritten, so the last value encountered when combining all the matching variants (looking only at branches) wins. * Files and irrelevant-files will be treated as a pair, so that if "irrelevant-files" appears, it will erase a previous "files" attribute. * It's possible to both reduce and expand the scope of jobs, but the user may need to manually copy values from a parent or other variant in order to do so. * It will no longer be possible to alter a job attribute by adding a variant with only a files matcher -- in all cases files and irrelevant-files are used solely to determine whether the job is run, not to determine whether to apply a variant. I think both would be good solutions to the problem. The key points for me are whether we want to keep the "alter a job attribute with variant with a files matcher" functionality (the "rebuild_index" example from above), and whether the additional control of overwriting the matchers (at the cost of redundancy in configuration) is preferable to combining the matchers. -Jim [View Less]

1 1

Zuul Logo
by James Cole 01 May '18

01 May '18

Hello Zuul Enthusiasts, I’m James, a graphic designer with the OpenStack Foundation, and I’ve been working with Jim and the Zuul-core team to design an official Zuul logo. As you probably know, the goal has been to create something unique that can be used across a variety of mediums, such as an avatar on GitHub, on the website, as badge, and on other types of Zuul communications. After a few iterations, the team is leaning towards the concept featured in the attached PDF. The symbol in the … [View More]

10 22

Overriding project-templates
by corvus＠inaugust.com 01 May '18

01 May '18

Hi, We tried to make the Zuul v3 configuration language as intuitive as possible, and incorporated a lot that we learned from our years running Zuul v2. One thing that we didn't anticipate was how folks would end up wanting to use a job in both project-templates *and* local project stanzas. Essentially, we had assumed that if you wanted to control how a job was run, you would add it to a project stanza directly rather that use a project-template. It's easy to do so if you use one or the … [View More]other. However, it turns out there are lots of good reasons to use both. For example, in a project-template we may want to establish a recommended way to run a job, or that a job should always be run with a set of related jobs. Yet a project may still want to indicate that job should only run on certain changes in that specific repo. To be very specific -- a very commonly expressed frustration is that a project can't specify a "files" or "irrelevant-files" matcher to override a job that appears in a project-template. Reconciling those is difficult, largely because once Zuul decides to run a job (for example, by a declaration in a project-template) it is impossible to dissuade it from running that job by adding any extra configuration to a project. We need to tread carefully when fixing this, because quite a number of related concepts could be affected. For instance, we need to preserve branch independence (a change to stop running a job in one branch shouldn't affect others). And we need to preserve the ability for job variants to layer on to each other (a project-local variant should still be able to alter a variant in a project-template). I propose that we remedy this by making a small change to how Zuul determines that a job should run: When a job appears multiple times on a project (for instance if it appears in a project-template and also on the project itself), all of the project-local variants which match the item's branch must also match the item in order for the job to run. In other words, if a job appears in a project-template used by a project and on the project, then both must match. This effectively causes the "files" and "irrelevant-files" attributes on all of the project-local job definitions matching a given branch to be combined. The combination of multiple files matchers behaves as a union, and irrelevant-files matchers as an intersection. ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC B ================ ======== ======= ======= I believe this will address the shortcoming identified above, but before we get too far in implementing it, I'd like to ask folks to take a moment and evaluate whether it will address the issues you've seen, or if you foresee any problems which I haven't anticipated. Thanks, Jim [View Less]

1 0

Zuul project update 2018-04-30
by corvus＠inaugust.com 01 May '18

01 May '18

Hi, This past week has seen the following: We released Zuul 3.0.2. Note that we added a dependency on fb-re2, a python wrapper for the re2 library, which is now a binary dependency. The long-term plan is to start moving all of our regular expression parsing to re2, mostly because it is resistant to denial-of-service attacks. Clint has some patches in progress to do that. But in the mean time, any new uses of regular expressions should use re2 so that we don't create additional debt. We … [View More]

1 0

Proposed updates to website to empower our non-coder friends
by Monty Taylor 27 Apr '18

27 Apr '18

Hey everybody! As part of the work with the Foundation staff on getting Zuul set up, we've chatted a bit about how to manage the website content. Obviously from the dev side, we find it important to be able to edit content like source code, propose those changes to gerrit and to review each other's changes. There are, however, folks out there who would like to help us do things like marketing who are not crazy about the concept of editing raw html. SO - we went and did a little bit of … [View More]exploration to see what we could find. Actually, what really happened is that we agreed we'd just go with the html thing we have currently for now and circle back later to figure out a CMS + code review system that would work for everyone. But then I had a weekend and a personal project that had pretty much the same set of requirements. End result - what I'd like to propose is that we adopt a combination of using gatsby [1] as a static site generator and netlify-cms [2] as a web-based CMS interface for editing content and submitting changes to gerrit. Both gatsby and netlify-cms are completely client-side - there are no server side components to run. I found them by looking at https://headlesscms.org/ and https://www.staticgen.com/ gatsby ------ gatsby is a Javascript/React-based static site generator that allows managing content in markdown files. It has netlify-cms plugin support, and the build toolchain works very similarly to the new javascript toolchain for zuul-dashboard. I've got a first rough-draft of this part done that I'll push up for review today or tomorrow. netlify-cms ----------- netlify-cms is a Javascript/React-based CMS for static site generators. It has support for either pushing changes directly in to git or for submitting them as proposed changes. It currently basically only supports github - but the github interface is broken out into a driver layer. In fact, they have two different options- one that talks to github directly and one that talks to a netlify git-gateway service, so the plugin layer isn't just theoretical. We'll need to write a gerrit driver, but that shouldn't be terribly hard. Proposed Plan ------------- We can keep this pretty simple. Step one- migrate existing site from static html to gatsby + zuul jobs. Step two - write gerrit driver for netlify-cms. Step three - add netlify-cms to website. Why not use ... --------------- There are many options out there for static site generation. I like gatsby because it's pure-javascript rather than having a go, ruby or python component. I've personally come to really like keeping web things in javascript ... especially since our zuul-dashboard is doing javascript build tooling now too. There is also tight integration support with netlify-cms (which is also a good choice for the same reason) - and it was super easy for me to get up and going with one of their examples. [1] https://www.gatsbyjs.org/ [2] https://www.netlifycms.org/ [View Less]

3 3

Zuul project update 2018-04-23
by corvus＠inaugust.com 26 Apr '18

26 Apr '18

Hi, This past week has seen the following: Shrews started testing Ansible 2.5 and has related changes up for review[1]. I made a video[2] to upload to the website which explains the gating algorithm. I expect it to be merged soon. I've also resumed work on the configuration loading refactor which will allow us to use less memory for dynamic configuration changes. I plan on continuing that this week. Tobiash has been improving scalability with parent-change-enqueued event and many github … [View More]

2 1

Re: Zuul + OpenStack Foundation update
by Lauren Sell 25 Apr '18

25 Apr '18

Hi everyone, Thanks very much Jim for providing the update on our plans heading into the Vancouver Summit. I want to sincerely apologize for the miscommunication around the board meeting. We presented an update on making Zuul an independent project at the Gold Member meeting yesterday and again at the Board Meeting today, but unfortunately the entire presentation today was scheduled during the executive session and not able to be heard by the public. It was a definitely oversight and … [View More]miscommunication. I want to share the deck we presented here and am happy to do a call for those who want to run through it, or simply chat about it in IRC. We welcome any feedback on the plans: https://docs.google.com/presentation/d/1iR_Q0ntV5BAs1TNjN-b1JBnsfahb2hI4IxM… The primary feedback we received during the gold member meeting was trying to understand where Zuul would fit under the OpenStack Foundation (ie. is it still part of OpenStack, or an adjacent project with independent governance?) and where Zuul fits in the CI/CD landscape (ie. what’s competitive and complementary). For people who are newer to the space, there has been a lot more activity recently around XCI and the OpenCI initiative and of course existing tools like Jenkins and new ones like Spinnaker, and they need help parsing what’s an effort by a group of people versus what’s actually software and when you would use which software :) Hope that is useful, and again, apologies for the miscommunication. I’m now subscribed to the mailing list and am happy to discuss here :) Thanks, Lauren PS If you can’t view the deck in google docs, I’m happy to send over a PDF. I just didn’t want to send an attachment to the mailing list... >> Begin forwarded message: >> >> From: corvus(a)inaugust.com <mailto:corvus@inaugust.com> (James E. Blair) >> Subject: Zuul + OpenStack Foundation update >> Date: April 11, 2018 at 5:28:08 PM CDT >> To: Zuul Discussion List <zuul-discuss(a)lists.zuul-ci.org <mailto:zuul-discuss@lists.zuul-ci.org>> >> >> Hi, >> >> Several of us on the zuul-core team have been working with the OpenStack >> Foundation staff on establishing and launching Zuul as a top-level >> project under the Foundation's umbrella. >> >> As a free/open-source project, and one that follows OpenStack's "four >> opens" we generally strive to use public channels for collaboration. >> However, when making a change like this, that is not always practical, >> especially in the early stages. I think we'll get there soon, but in >> the mean time, I ask you to accept Monty, Clark, Jeremy and I as >> delegates of the project to work with the Foundation on this. >> >> I'll try to keep everyone updated as best as possible with what we're >> working on, though sometimes those updates may need to be vague. Please >> feel free to contact me or the others privately if you have specific >> concerns you want to ensure we address. Here are updates from our most >> recent discussions: >> >> * There will be discussion about Zuul at the OpenStack board of >> directors meeting on April 24. That will be a phone call to which >> anyone is welcome to listen. >> >> * The Foundation will promote Zuul at the upcoming OpenStack Summit in >> Vancouver. It will be officially announced as a Foundation project on >> the keynote stage and via press release. >> >> * We are working with the Foundation's design team on a logo. We want >> to have something that can be adapted to work as an avatar (i.e., on >> GitHub), on the website, as a badge, and on slide decks. >> >> * The Foundation marketing team has begun a review of the website >> content and will be assisting with changes. Included in this is an >> effort to collect user stories for Zuul. >> >> * We're looking into implementing a system to manage the website content >> which is both Gerrit-friendly and has a workflow suitable for those >> unaccustomed to using Git. Monty has volunteered to research and >> report back with some options to explore. >> >> -Jim >> >> _______________________________________________ >> Zuul-discuss mailing list >> Zuul-discuss(a)lists.zuul-ci.org <mailto:Zuul-discuss@lists.zuul-ci.org> >> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss > [View Less]

1 0

Simplified DSL
by Tristan Cacqueray 22 Apr '18

22 Apr '18

Hello folks, Some users may find job written in Ansible hard to reproduce, and I was wondering if we could add a new "exec" job attribute that would simply be shell task to be executed on a single node. This would let us write zuul.yaml file more similar to other in-repo ci system. For example a zuul.yaml could be: - job: name: my-job exec: - mvn build - mvn test - project: check: jobs: - my-job gate: jobs: - my-job Moreover we could … [View More]

8 9