- Zuul-discuss - lists.opendev.org

Zuul project update 2018-06-04
by corvus＠inaugust.com 04 Jun '18

04 Jun '18

Hi, I skipped a few of these due to the Summit and related discombobulation, but it's back! The Summit and OpenDev was really good. We introduced Zuul to a lot of folks. A number of Zuul devs managed to find some time to talk in person about the container spec. I've pushed another revision of that, hopefully it's close to ready. Artem Goncharov (gtema) has started implementation of nodepool security group support. Tristan followed up on summit discussion around container and worked on a … [View More]

1 0

Behavior change for file matchers
by corvus＠inaugust.com 02 Jun '18

02 Jun '18

Hi, A while back we had a discussion about changing the behavior of file matchers to be more intuitive. The consensus seemed to be that treating them as normal overridable attributes would be the best solution. In particular, that would allow folks to more easily define a file matcher in a job, and then override it at the project-template or project level. See this thread for the full context: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000397.html I've implemented … [View More]

1 0

Templates in job vars
by Ian Wienand 29 May '18

29 May '18

Hi, In templating out some diskimage-builder jobs, there is a situation where we pass a list of tests to run, but some platforms can't run various tests. So something like --- - job: name: base abstract: yes ... vars: dib_functests: - gentoo - debian - job: name: test-on-bionic parent: base - job: name: test-on-xenial parent: base vars: dib_functests: - gentoo - debian - suse # also … [View More]

2 1

Zuul project update 2018-05-14
by corvus＠inaugust.com 15 May '18

15 May '18

Hi, Shrews is beginning implementation of multi-label support in nodepool as part of his continuing work on better integrating the static driver. I spent most of last week and will continue to spend most of this week preparing for the upcoming OpenStack Summit and OpenDev events. However, at the end of the week, I needed a break from that, so I started work on replacing our use of aiohttp with cherrypy. Aiohttp (and related libraries) have broken us a few times recently by rapidly dropping … [View More]

1 0

Mission statement and tagline
by corvus＠inaugust.com 10 May '18

10 May '18

Hi, The Foundation marketing team has indicated that it would be good to have a mission statement and tagline. The mission statement can be used to help convey what we're trying to accomplish as a community, and the tagline will likely feature prominently on the website and other material. I'd like us all to be involved in creating these, but we need to have the process finished by May 10 in order to be able to update marketing material before the Summit/OpenDev. I propose the following … [View More]

2 2

Zuul Stickers
by James Cole 09 May '18

09 May '18

Hey everyone! I just wanted to share a pic of the stickers we’re giving away at the Summit in Vancouver. A link is below in case you can’t see the attachment. https://www.dropbox.com/s/dwb8df3wuqey5i6/Zuul_Stickers.jpg?dl=0 Come grab one in Vancouver! James Cole Graphic Designer OpenStack Foundation

1 0

Polling users of Nodepool's static node driver
by David Shrewsbury 08 May '18

08 May '18

Hi, Is anyone currently using the static driver? And if so, are you making us of the multi-label feature it supports for the static nodes? I ask because I've been trying to change the behavior of the driver to pre-populate ZooKeeper with the static nodes (more in line with how the OpenStack driver works), rather than only having the ZK nodes appear when the node is actually in-use. Doing so fundamentally changes how the static driver works, and this deep dive of the code has revealed this … [View More]

4 6

Zuul project update 2018-05-07
by corvus＠inaugust.com 08 May '18

08 May '18

Hi, This past week has seen the following: Shrews began attempting to fix nodepool static driver to pre-register static nodes in ZooKeeper. This led to the discussion in [1] I pushed patches to add our new logo to the website and docs, as well as some other changes to the website suggested by the marketing team. The OpenDev conference and OpenStack summit are fast approaching. I have a lot to prepare before then -- both in terms of preparing for talks as well as polishing the website and … [View More]

1 0

Re: [openstack-dev] Overriding project-templates in Zuul
by Joshua Hesketh 02 May '18

02 May '18

> > I think in actuality, both operations would end up as intersections: > > ================ ======== ======= ======= > Matcher Template Project Result > ================ ======== ======= ======= > files AB BC B > irrelevant-files AB BC B > ================ ======== ======= ======= > > So with the "combine" method, it's always possible to further restrict > where the job runs, but never to expand it.… [View More] Ignoring the 'files' above, in the example of 'irrelevant-files' haven't you just combined the results to expand when it runs? ie, A and C are /not/ excluded and therefore the job will run when there are changes to A or C? I would expect the table to be something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC ABC ================ ======== ======= ======= > > So a job with "files: tests/" and "irrelevant-files: docs/" would do > > whatever it is that happens when you specify both. > > In this case, I'm pretty sure that would mean it reduces to just "files: > tests/", but I've never claimed to understand irrelevant-files and I > won't start now. > Yes, I think you are right that this would reduce to that. However, what about the use case of: files: tests/* irrelevant-files: tests/docs/* I could see a use case where both of those would be helpful. Yes you could describe that as one regex but to the end user the above may be expected to work. Unless we make the two options mutually exclusive I feel like this is a feature we should support. (That said, it's likely a separate feature/functionality than what is being described now). Anyway, I feel like Proposal #2 is more how I would expect the system to behave. I can see an argument for combining the results (and feel like you could evaulate that at the end after combining the branch-matching variants) to give something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC ABC ================ ======== ======= ======= However, that gives the user no way to remove a previously listed option. Thus overwriting may be the better solution (ie proposal #2 as written) unless we want to explore the option of allowing a syntax that says "extend" or "overwrite". Yours in hoping that made sense, Josh [View Less]

2 1

Re: [openstack-dev] Overriding project-templates in Zuul
by corvus＠inaugust.com 02 May '18

02 May '18

Joshua Hesketh <joshua.hesketh(a)gmail.com> writes: > I might be misunderstanding at which point a job is chosen to be ran and > therefore when it's too late to dissuade it. However, if possible, would it > make more sense for the project-local copy of a job to overwrite the > supplied files and irrelevant-files? This would allow a project to run a > job when it otherwise doesn't match. Imagine that a project with branches has a job added via a template. project-config/… [View More]zuul.yaml@master: - job: name: my-job vars: {jobvar: true} - project-template: name: myjobs check: jobs: - my-job: vars: {templatevar: true} project/zuul.yaml@master: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} project/zuul.yaml@stable: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} The resulting project config is: - project: jobs: - my-job (branches: master; project-local job) - my-job (branches: master; project-template job) - my-job (branches: stable; project-local job) - my-job (branches: stable; project-template job) When Zuul decides what to run, it goes through each of those in order, evaluates their matchers, and pulls in parents and their variants for each that matches. So a change on the master branch would collect the following variants to apply: my-job (branch: master; project-local job) my-job (job) base (job) my-job (branch: master; project-template job) my-job (job) base (job) It would then apply them in this order: base (job) my-job (job) my-job (branch: master; project-template job) my-job (branch: master; project-local job) To further restrict a project-local job with a "files:" matcher would cause the project-local job not to match, but the project-template job will still match, so the job gets run. That's the situation we have today, which is what I meant by "it's too late to dissuade it". Regarding the suggestion to overwrite it, we would need to decide which of the possible variants to overwrite. Keep in mind that there are 3 independent matchers operating on all the variants (branches, files, irrelevant-files). Does a project-local job with a "files:" matcher overwrite all of the variants? Just the ones which match the same branch? That would probably be the most reasonable thing to do. In my mind, that effectively splits the matchers into two categories: branch matchers, and file matchers. And they would behave differently. Zuul could collect the variants as above, considering only the branch matchers. It could then apply all of the variants in the normal manner, treating files and irrelevant-files as normal attributes which can be overwritten. Then, once it has composed the job to run based on all the matching variants, it would only *then* evaluate the files matchers. If they don't match, then it would not run the job after all. I think that's a very reasonable way to solve the issue as well, and I believe it would match people's expectations. Ultimately, the outcome will be very similar to the proposal I made except that rather than being combined, the matchers will be overwritten. That means that if you want to expand the set of irrelevant-files for a job, you would have to copy the set from the parent. There's one other aspect to consider -- it's possible to create a job like this: - job: name: doc-job - jobs: name: doc-job files: docs/index.rst vars: {rebuild_index: true} Which means: there's a normal docs job with no variables, but if docs/index.rst is changed, set the rebuild_index variable to true. Either approach (combine vs overwrite) eliminates the ability to do this within a project or project-template stanza. But the "combine" approach still lets us do this at the job level. We could still support this in the overwrite approach, however, I think it might be simpler to work with if we eliminated it as well and just always treated files and irrelevant-files matchers as overwriteable attributes. It would no longer be possible to implement the above example, but I'm not sure it's that useful anyway? > What happens when something is in both files and irrelevant-files? If the > project-template is trying to say A is in 'files', but the project-local > says A is in 'irrelevant-files', should that overwrite it? I think my statement and table below was erroneous: >> This effectively causes the "files" and "irrelevant-files" attributes on >> all of the project-local job definitions matching a given branch to be >> combined. The combination of multiple files matchers behaves as a >> union, and irrelevant-files matchers as an intersection. >> >> ================ ======== ======= ======= >> Matcher Template Project Result >> ================ ======== ======= ======= >> files AB BC ABC >> irrelevant-files AB BC B >> ================ ======== ======= ======= I think in actuality, both operations would end up as intersections: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC B ================ ======== ======= ======= So with the "combine" method, it's always possible to further restrict where the job runs, but never to expand it. As to your question about what if something appears in both -- in my "combine" proposal, I would continue to treat them independently, so if a project-local variant has an "irrelevant-files:" matcher, and a project-template variant has a "files:" matcher, then the usual nonsense would happen: they'd both have to match. So a job with "files: tests/" and "irrelevant-files: docs/" would never run because it's impossible to satisfy both. That actually matches some current behavior we have -- a job must match a "real" job definition as well as at least one project-template or project-local job definition, so a "job" with "files: tests/" and a project-template with "irrelevant-files: docs/" will behave the same way today. However, if we go with the "overwrite" proposal, since we're altering the behavior of the non-branch matchers anyway, that might be a fine time to pair them up and treat them as a unit. Then, whichever is the latest would win. So a job with "files" and a project-template with "irrelevant-files" would end up only having an irrelevant-files attribute. Then a further project-local stanza with only "files" would eliminate the irrelevant-files attribute leaving only the files value. Okay, let's summarize: Proposal 1: All project-template and project-local job variants matching the item's branch must also match the item. * Files and irrelevant-files on project-template and project stanzas are essentially combined in a set intersection. * It's possible to further reduce the scope of jobs, but not expand. * Files and irrelevant-files are still independent matchers, and if both are present, both must match. * It's not possible to alter a job attribute by adding a project-local variant with only a files matcher (it would cause the whole job to run or not run). But it's still possible to do that in the main job definition itself. Proposal 2: Files and irrelevant-files are treated as overwriteable attributes and evaluated after branch-matching variants are combined. * Files and irrelevant-files are overwritten, so the last value encountered when combining all the matching variants (looking only at branches) wins. * Files and irrelevant-files will be treated as a pair, so that if "irrelevant-files" appears, it will erase a previous "files" attribute. * It's possible to both reduce and expand the scope of jobs, but the user may need to manually copy values from a parent or other variant in order to do so. * It will no longer be possible to alter a job attribute by adding a variant with only a files matcher -- in all cases files and irrelevant-files are used solely to determine whether the job is run, not to determine whether to apply a variant. I think both would be good solutions to the problem. The key points for me are whether we want to keep the "alter a job attribute with variant with a files matcher" functionality (the "rebuild_index" example from above), and whether the additional control of overwriting the matchers (at the cost of redundancy in configuration) is preferable to combining the matchers. -Jim [View Less]

1 1