May 2018 - Zuul-discuss - lists.opendev.org

Re: [openstack-dev] Overriding project-templates in Zuul
by Joshua Hesketh 02 May '18

02 May '18

> > I think in actuality, both operations would end up as intersections: > > ================ ======== ======= ======= > Matcher Template Project Result > ================ ======== ======= ======= > files AB BC B > irrelevant-files AB BC B > ================ ======== ======= ======= > > So with the "combine" method, it's always possible to further restrict > where the job runs, but never to expand it.… [View More] Ignoring the 'files' above, in the example of 'irrelevant-files' haven't you just combined the results to expand when it runs? ie, A and C are /not/ excluded and therefore the job will run when there are changes to A or C? I would expect the table to be something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC ABC ================ ======== ======= ======= > > So a job with "files: tests/" and "irrelevant-files: docs/" would do > > whatever it is that happens when you specify both. > > In this case, I'm pretty sure that would mean it reduces to just "files: > tests/", but I've never claimed to understand irrelevant-files and I > won't start now. > Yes, I think you are right that this would reduce to that. However, what about the use case of: files: tests/* irrelevant-files: tests/docs/* I could see a use case where both of those would be helpful. Yes you could describe that as one regex but to the end user the above may be expected to work. Unless we make the two options mutually exclusive I feel like this is a feature we should support. (That said, it's likely a separate feature/functionality than what is being described now). Anyway, I feel like Proposal #2 is more how I would expect the system to behave. I can see an argument for combining the results (and feel like you could evaulate that at the end after combining the branch-matching variants) to give something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC ABC ================ ======== ======= ======= However, that gives the user no way to remove a previously listed option. Thus overwriting may be the better solution (ie proposal #2 as written) unless we want to explore the option of allowing a syntax that says "extend" or "overwrite". Yours in hoping that made sense, Josh [View Less]

2 1

Re: [openstack-dev] Overriding project-templates in Zuul
by corvus＠inaugust.com 01 May '18

01 May '18

Joshua Hesketh <joshua.hesketh(a)gmail.com> writes: > I might be misunderstanding at which point a job is chosen to be ran and > therefore when it's too late to dissuade it. However, if possible, would it > make more sense for the project-local copy of a job to overwrite the > supplied files and irrelevant-files? This would allow a project to run a > job when it otherwise doesn't match. Imagine that a project with branches has a job added via a template. project-config/… [View More]zuul.yaml@master: - job: name: my-job vars: {jobvar: true} - project-template: name: myjobs check: jobs: - my-job: vars: {templatevar: true} project/zuul.yaml@master: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} project/zuul.yaml@stable: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} The resulting project config is: - project: jobs: - my-job (branches: master; project-local job) - my-job (branches: master; project-template job) - my-job (branches: stable; project-local job) - my-job (branches: stable; project-template job) When Zuul decides what to run, it goes through each of those in order, evaluates their matchers, and pulls in parents and their variants for each that matches. So a change on the master branch would collect the following variants to apply: my-job (branch: master; project-local job) my-job (job) base (job) my-job (branch: master; project-template job) my-job (job) base (job) It would then apply them in this order: base (job) my-job (job) my-job (branch: master; project-template job) my-job (branch: master; project-local job) To further restrict a project-local job with a "files:" matcher would cause the project-local job not to match, but the project-template job will still match, so the job gets run. That's the situation we have today, which is what I meant by "it's too late to dissuade it". Regarding the suggestion to overwrite it, we would need to decide which of the possible variants to overwrite. Keep in mind that there are 3 independent matchers operating on all the variants (branches, files, irrelevant-files). Does a project-local job with a "files:" matcher overwrite all of the variants? Just the ones which match the same branch? That would probably be the most reasonable thing to do. In my mind, that effectively splits the matchers into two categories: branch matchers, and file matchers. And they would behave differently. Zuul could collect the variants as above, considering only the branch matchers. It could then apply all of the variants in the normal manner, treating files and irrelevant-files as normal attributes which can be overwritten. Then, once it has composed the job to run based on all the matching variants, it would only *then* evaluate the files matchers. If they don't match, then it would not run the job after all. I think that's a very reasonable way to solve the issue as well, and I believe it would match people's expectations. Ultimately, the outcome will be very similar to the proposal I made except that rather than being combined, the matchers will be overwritten. That means that if you want to expand the set of irrelevant-files for a job, you would have to copy the set from the parent. There's one other aspect to consider -- it's possible to create a job like this: - job: name: doc-job - jobs: name: doc-job files: docs/index.rst vars: {rebuild_index: true} Which means: there's a normal docs job with no variables, but if docs/index.rst is changed, set the rebuild_index variable to true. Either approach (combine vs overwrite) eliminates the ability to do this within a project or project-template stanza. But the "combine" approach still lets us do this at the job level. We could still support this in the overwrite approach, however, I think it might be simpler to work with if we eliminated it as well and just always treated files and irrelevant-files matchers as overwriteable attributes. It would no longer be possible to implement the above example, but I'm not sure it's that useful anyway? > What happens when something is in both files and irrelevant-files? If the > project-template is trying to say A is in 'files', but the project-local > says A is in 'irrelevant-files', should that overwrite it? I think my statement and table below was erroneous: >> This effectively causes the "files" and "irrelevant-files" attributes on >> all of the project-local job definitions matching a given branch to be >> combined. The combination of multiple files matchers behaves as a >> union, and irrelevant-files matchers as an intersection. >> >> ================ ======== ======= ======= >> Matcher Template Project Result >> ================ ======== ======= ======= >> files AB BC ABC >> irrelevant-files AB BC B >> ================ ======== ======= ======= I think in actuality, both operations would end up as intersections: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC B ================ ======== ======= ======= So with the "combine" method, it's always possible to further restrict where the job runs, but never to expand it. As to your question about what if something appears in both -- in my "combine" proposal, I would continue to treat them independently, so if a project-local variant has an "irrelevant-files:" matcher, and a project-template variant has a "files:" matcher, then the usual nonsense would happen: they'd both have to match. So a job with "files: tests/" and "irrelevant-files: docs/" would never run because it's impossible to satisfy both. That actually matches some current behavior we have -- a job must match a "real" job definition as well as at least one project-template or project-local job definition, so a "job" with "files: tests/" and a project-template with "irrelevant-files: docs/" will behave the same way today. However, if we go with the "overwrite" proposal, since we're altering the behavior of the non-branch matchers anyway, that might be a fine time to pair them up and treat them as a unit. Then, whichever is the latest would win. So a job with "files" and a project-template with "irrelevant-files" would end up only having an irrelevant-files attribute. Then a further project-local stanza with only "files" would eliminate the irrelevant-files attribute leaving only the files value. Okay, let's summarize: Proposal 1: All project-template and project-local job variants matching the item's branch must also match the item. * Files and irrelevant-files on project-template and project stanzas are essentially combined in a set intersection. * It's possible to further reduce the scope of jobs, but not expand. * Files and irrelevant-files are still independent matchers, and if both are present, both must match. * It's not possible to alter a job attribute by adding a project-local variant with only a files matcher (it would cause the whole job to run or not run). But it's still possible to do that in the main job definition itself. Proposal 2: Files and irrelevant-files are treated as overwriteable attributes and evaluated after branch-matching variants are combined. * Files and irrelevant-files are overwritten, so the last value encountered when combining all the matching variants (looking only at branches) wins. * Files and irrelevant-files will be treated as a pair, so that if "irrelevant-files" appears, it will erase a previous "files" attribute. * It's possible to both reduce and expand the scope of jobs, but the user may need to manually copy values from a parent or other variant in order to do so. * It will no longer be possible to alter a job attribute by adding a variant with only a files matcher -- in all cases files and irrelevant-files are used solely to determine whether the job is run, not to determine whether to apply a variant. I think both would be good solutions to the problem. The key points for me are whether we want to keep the "alter a job attribute with variant with a files matcher" functionality (the "rebuild_index" example from above), and whether the additional control of overwriting the matchers (at the cost of redundancy in configuration) is preferable to combining the matchers. -Jim [View Less]

1 1

Zuul Logo
by James Cole 30 Apr '18

30 Apr '18

Hello Zuul Enthusiasts, I’m James, a graphic designer with the OpenStack Foundation, and I’ve been working with Jim and the Zuul-core team to design an official Zuul logo. As you probably know, the goal has been to create something unique that can be used across a variety of mediums, such as an avatar on GitHub, on the website, as badge, and on other types of Zuul communications. After a few iterations, the team is leaning towards the concept featured in the attached PDF. The symbol in the … [View More]

10 22