Hi,
After merging the feature/zuulv3 branches of Nodepool and Zuul back into
master, a number of open changes on the master branch remained from
before the merge (and, in many cases, much older than that).
Some of us have gone through to try to triage those changes. Some of
them were implemented on the feature/zuulv3 branch or otherwise just
don't apply anymore (a component was removed, or functions
differently). These changes were abandoned with a short explanation of
why, and an …
[View More]invitation to restore them if the, admittedly cursory,
analysis was wrong.
Some of them may still apply, but in those cases, almost certainly need
to be rebased. These changes we marked with Workflow -1: Work in
Progress, with the hope that the author will update them when time
permits.
At this point, all of the open changes on Zuul and Nodepool which are
not marked WIP should be relevant to the current state of the tree, and
should be reviewed. So if you've only been looking at a subset of open
changes to avoid the older changes on master which may not be relevant,
you should be able to remove that filter and consider all open, non-WIP
changes.
Thanks,
Jim
[View Less]
Hi,
We've run into an issue a few times where a complex job defined in a
project repo doesn't run because it can't find a parent job that matches
the branch.
In all the cases I've seen so far, it's children of OpenStack's devstack
job which are at issue. That's because the devstack job is defined in
the devstack repo (an untrusted project repo) with branches. That means
that it has implied branch matchers attached to it.
We could, of course, turn those off, however, the intention is for …
[View More]the
devstack job to be branched. Running the Queens release is not the same
as running Ocata, so the different branch variants should be used under
different circumstances.
The problem occurs when someone wants to inherit from the devstack job
in a different repo, and there is not a devstack branch variant which
matches the branch of the child job.
The way Zuul pieces together the final job that is run is to take the
name of the job specified in the project-pipeline config, and then
collect all the variants of that job which match the change. Each time
it finds a match, it looks at the parent of that variant, and
recursively collects all the variants of the parent which match the
change (eliminating duplicates). If it can't find at least one variant
of the parent which matches the change, the job is not run.
The first time this came to my attention was for the tempest job.
During development, the devstack job has only been defined on the master
branch of devstack. The tempest job is in a repo with no branches, so
it doesn't have an implied branch matcher. When any project invokes it,
it will match the branch, and then Zuul will look for its parent job,
'devstack', and as long as the change is on the master branch, it will
find it and run. However, if the change is on stable/ocata, and there
is no devstack job defined in the stable/ocata branch of devstack, then
it won't run.
I think the solution to this case is to backport the devstack job to
devstack's stable branches. That's the situation we will ultimately be
in after our natural branching progression anyway, and we should treat
this as a one-time cost of the transition to Zuul v3.
The other two instances where this has come up are for projects which
should run a child of the devstack job on a branch which does not, and
never will, exist in devstack.
When nodepool had a feature/zuulv3 branch, we were unable to run the
devstack integration test job on that branch because of this. Ansible's
default branch is 'devel', which means in order for us to run the shade
integration test job on changes to Ansible, Zuul looks for a devstack
variant which matches the 'devel' branch.
In all cases, these issues could technically be solved by creating
devstack job variants which matched these other branches. However, it
should not be incumbent on the devstack maintainers to anticipate all of
the branch names of other projects which might want to use the devstack
job. We could create a devstack job variant with an explicit branch
matcher which matches all branches, and treat the stable branch variants
as deltas from that, however, that is more difficult to manage and
doesn't fit our branch creation model.
I can think of two solutions to this problem:
1) Have Zuul use a default branch when searching for parents with no
matching branch variants.
Every project in Zuul has a default branch (which can be specified in
the 'project' config stanza, but the default default is 'master'). If,
when searching for the parents of a job, Zuul finds no matching
variants, it could perform the search a second time with the default
branch of the parent job's project (all variants of a job are defined in
the same repo, so we know which project the job is in, so there's only
one default branch to search for each parent).
2) Allow someone creating a child job to specify a fallback branch for
Zuul to use when searching for parent jobs.
Essentially, we would modify the algorithm in the same way as the
previous suggestion, except instead of determining the fallback branch
based on the default branch of the parent job, we would let the child
job supply the fallback branch. At each level of recursion as we walk
up the inheritance path, a new fallback branch could be provided.
I'm inclined to implement option (1) at least. That is very similar to
what already happens today with git repo checkouts. I.e., if you did
manage to run one of the jobs in the examples I described, you would get
the master branch of devstack checked out in the job because of the
existing fallback behavior. It makes sense that the job matching
algorithm would be similar. And I think in many of the cases where this
issue comes up, it would do the right thing automatically (certainly it
would have in all the examples thus far).
I'm on the fence about option (2). I see no reason not to add it other
than perhaps it's unnecessary complexity. Maybe we should avoid
implementing it until we find a compelling case for it? Or do folks
anticipate a need for that now?
Thanks,
Jim
[View Less]
Hi,
Today we merged the feature/zuulv3 branches of Nodepool and Zuul into
master.
In order to delete the feature branch, now that it's no longer
necessary, we need to close all of the changes on that branch. In order
to do that, I have marked all of them as abandoned.
I'd like to stress that this is a technicality and of course the changes
themselves are still very much welcome.
If you have a pending change to that branch, please re-propose the
change to the master branch. If, when doing …
[View More]so, you could include a
link to the previous version of the change, especially if previous
reviews had been completed, that would be very helpful to reviewers.
I'm including a summary of all the changes which were open on
feature/zuulv3 shortly before I abandoned them.
All 113 changes in status:open branch:feature/zuulv3
Number Subject Project Branch Topic Owner Updated C V W
383437 Increase build worker main loop delay nodepool feature/zuulv3 nodepool-zk James E. Blair 2017-12-18 1 -1
408951 Add command list into server.py zuul feature/zuulv3 launcher Joshua Hesketh 2017-12-18 -1 -1
426861 WIP: Add reporter for Federated Message Bus (fedmsg) zuul feature/zuulv3 426861 Paul Belanger 2017-12-18 -1 -1
429850 WIP: Re-enable test_time_database test zuul feature/zuulv3 enable-tests Paul Belanger 2017-12-18 -1 -1
446172 Add reconfigure_handler for executor logging zuul feature/zuulv3 Paul Belanger 2017-12-18 1 -1
450897 Re-enable test_timer_sshkey zuul feature/zuulv3 test-conflict Clint 'SpamapS' B 2017-12-18 -1
454826 [WIP] Integration: Get static nodes from nodepool zuul feature/zuulv3 Cullen Taylor 2017-12-18 -1
456090 Set the git sshkey for the connection zuul feature/zuulv3 zuulv3 Joshua Hesketh 2017-12-18 -1 -1
465852 Add Dockerfile nodepool feature/zuulv3 Tobias Henkel 2017-05-23 -1 1 -1
465912 Add Dockerfile zuul feature/zuulv3 Tobias Henkel 2017-12-28 2 1 -1
466071 Switch from testrepository to stestr zuul feature/zuulv3 stestr Clint 'SpamapS' B 2017-09-21 1 -1
468670 Re-enable test_merge_conflict_reports zuul feature/zuulv3 re-enable-tests Clint 'SpamapS' B 2017-09-21 2 -1
471175 WIP: Status branch protection checking for github zuul feature/zuulv3 branch-protection Jamie Lennox 2017-12-22 1
473811 Add build.started state flag zuul feature/zuulv3 streamer_testing Monty Taylor 2017-12-18 1 -1 -1
473985 Only prepend hostname on multi-node plays zuul feature/zuulv3 streamer_testing Monty Taylor 2017-12-18 1 -1
478675 Add ability to auto-generate simple one-line shell playbooks zuul feature/zuulv3 Monty Taylor 2017-12-18 -1 -1
480843 Ensure build.start_time is defined zuul feature/zuulv3 Tristan Cacqueray 2017-09-21 -1 1
481134 Add v3 update slides zuul feature/zuulv3 480759 Monty Taylor 2017-12-18 1 -1
487538 Use yarn and webpack to manage zuul-web javascript zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1 -1
528295 ├─Add babel transpiling enabling use of ES6 features zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1
528296 │ └─Add StandardJS linting and analysis zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1
528373 │ └─Fix source_url handling for jobs view zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1
528297 │ └─Fix StandardJS warnings and turn them to errors zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1
528298 │ └─Add bundle analysis to the lint target zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-22 2 1
529193 │ └─Inject url endpoint information zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 1
528374 │ └─Make bundle of build web content zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 -1
528437 │ └─Remove use strict zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 2 -1
529686 ├─Add nodeenv and plumb it in to tox zuul feature/zuulv3 zuul-javascript-t Monty Taylor 2017-12-21 -1
529690 └─Add web server URL tests zuul feature/zuulv3 James E. Blair 2017-12-21 -1
494655 WIP Replace paste/webob with aiohttp zuul feature/zuulv3 aiohttp Monty Taylor 2017-09-21 -1
497962 Add zookeeper-server to bindep for Red Hat nodepool feature/zuulv3 Monty Taylor 2017-08-25 -1 -1
499224 Keep a cache of branches for projects and update on events zuul feature/zuulv3 github-ratelimit Monty Taylor 2017-09-21 -1 -1
499969 Add /node-list to the webapp nodepool feature/zuulv3 Tristan Cacqueray 2018-01-09 2 1
500254 └─Add /label-list to the webapp nodepool feature/zuulv3 Tristan Cacqueray 2018-01-09 2 1
531390 └─Refactor status functions, add web endpoints, allow params nodepool feature/zuulv3 500254 Matthieu Huin 2018-01-09 1
531703 └─Add a separate module for node management commands nodepool feature/zuulv3 500254 Matthieu Huin 2018-01-09 1
531718 └─webapp: add optional admin endpoint nodepool feature/zuulv3 500254 Matthieu Huin 2018-01-10 1
500159 Always generate the ARA report, even on failure zuul feature/zuulv3 zuulv3 David Moreau Sima 2017-09-21 1 -1
502468 Emit a message to the job log if ansible crashes zuul feature/zuulv3 Monty Taylor 2017-09-21 -1
503166 Don't gather host keys for non ssh connections nodepool feature/zuulv3 windows-support Tobias Henkel 05:42 AM -1
504112 └─Add connection-port to provider diskimage nodepool feature/zuulv3 windows-support Tobias Henkel 05:45 AM -1
504238 Delete IncludeRole object from result object for include_role tas zuul feature/zuulv3 David Moreau Sima 2018-01-09 1 -1
504267 Move github webhook from webapp to zuul-web zuul feature/zuulv3 zuul-web Jesse Keating 2017-12-11 1 1
504464 Consume server.id from shade create exception nodepool feature/zuulv3 Monty Taylor 2017-12-18 2 -1
504526 Add SELinux type enforcement zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-09-21 1
504809 WIP: add abstract job attribute zuul feature/zuulv3 James E. Blair 2017-09-21 -1
505419 Disable action and lookup plugins from 2.4 zuul feature/zuulv3 zuulv3 Monty Taylor 2017-09-21 2 -1
505430 └─Port in changes from ansible 2.4 command module zuul feature/zuulv3 zuulv3 Monty Taylor 2017-09-21 -1
506871 Add restricted-node-labels global limits zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-11-07 1
508774 Add noop job description zuul feature/zuulv3 zuul-web Tristan Cacqueray 05:11 PM 2 -2 1
508793 WIP add repl zuul feature/zuulv3 James E. Blair 2017-10-02
508960 Add memory awareness to system load governor zuul feature/zuulv3 Clint 'SpamapS' B 05:03 AM 2 1
509294 Add jinja filters for manipulating zuul.projects zuul feature/zuulv3 Monty Taylor 2017-10-03 -1
509484 Log exceptions from MODULE FAILURE more consistently zuul feature/zuulv3 Monty Taylor 2017-10-05 2 -1
509531 Provide resource_name in logging as structured data nodepool feature/zuulv3 zuulv3 Monty Taylor 2017-12-18 -1 -1
509903 Remove references to pipelines, queues, and layouts on dequeue zuul feature/zuulv3 zuulv3-fixes James E. Blair 2017-10-06 -1 -1
510301 Add a CRD multi-branch test zuul feature/zuulv3 James E. Blair 2017-10-07 -1
512130 Enabled ssh retries for ansible zuul feature/zuulv3 zuulv3-fixes Paul Belanger 2017-10-17 2 1 -1
513003 WIP: Add script for deterministic key generation zuul feature/zuulv3 Tobias Henkel 2017-10-18 -1
513368 Add regex support to project stanzas zuul feature/zuulv3 Tobias Henkel 2018-01-05 1
513387 Add node list to webapp nodepool feature/zuulv3 Tobias Henkel 2017-10-19 1
513885 Refactor Zuul job to use lightweight Vue components zuul feature/zuulv3 cleanup-status Mohammed Naser 2017-12-12 1 1
513886 └─Switch change list to use Vue.js zuul feature/zuulv3 cleanup-status Mohammed Naser 2017-10-21 1
515169 Support autoholding nodes for specific changes/refs zuul feature/zuulv3 autohold-changese Krzysztof Klimond 2017-12-14 1
516920 builder: do not cleanup image for driver not managing image nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 2 1
468624 └─Implement a static driver for Nodepool nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 -1
526325 └─Refactor run_handler to be generic nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 1
532450 └─Refactor NodeLauncher to be generic nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 1
468753 └─Implement an OpenContainer driver nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 -2 -1
521356 └─Implement a Kubernetes driver nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 1
528982 └─Implement an Amazon EC2 driver nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 1
517067 Add BaseSource.getProjectReadonly and refactor zuul feature/zuulv3 517078 Clint 'SpamapS' B 2017-11-03 -1 -1
517078 └─Do not add invalid projets via the /keys API zuul feature/zuulv3 517078 Clint 'SpamapS' B 2017-11-16 2 -1
518279 mqtt: add basic reporter zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-11-29 2 1
519222 Add client unit testing zuul feature/zuulv3 enqueue-ref Ian Wienand 2017-12-06 2 1
519596 Allow run to be list of playbooks zuul feature/zuulv3 Paul Belanger 2017-11-20 -1 1
519654 Override HOME environment variable in bubblewrap zuul feature/zuulv3 bwrap-set-home Clint 'SpamapS' B 2017-11-16 -1 -1
520657 Run image object autocleanup after uploading images nodepool feature/zuulv3 Monty Taylor 2017-11-29 -1
520664 WIP: Convert from legacy to native devstack job nodepool feature/zuulv3 devstack_native David Shrewsbury 2018-01-02 1
520855 Ensure the merger prunes on update zuul feature/zuulv3 fetch-prune Ian Wienand 2017-11-20 -1 1
521625 Add angular to fetch-dependencies.sh zuul feature/zuulv3 Monty Taylor 2017-11-21 -1 1
523640 zk: use kazoo retry facilities nodepool feature/zuulv3 zuulv3 Tristan Cacqueray 05:28 AM -2 1
523697 executor: add log_stream_port and log_stream_file settings zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-11-30 -1 1
524463 Remove zuul._projects zuul feature/zuulv3 project-transitio Ian Wienand 2017-12-01 -1 -1
524773 handler: fix support for handler without launch_manager nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 2018-01-17 2 1
524620 └─Add a plugin interface for drivers nodepool feature/zuulv3 nodepool-drivers Tristan Cacqueray 02:53 AM 1
525851 zk: automatically retry command when connection is lost zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-12-27 -2 1
527579 web: add /{tenant}/jobs/{job_name} route zuul feature/zuulv3 zuul-web Tristan Cacqueray 2017-12-13 -1 1
528588 Use bandit security scanner zuul feature/zuulv3 bandit Clint 'SpamapS' B 2017-12-17 1
528729 requirements: remove paramiko <2.0 cap zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-05 2 1
528969 config: add statsd-server config parameter nodepool feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-15 2 -1
529013 web: add OpenAPI documentation zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2017-12-27 2
529060 Make Zuul able to start with a broken config zuul feature/zuulv3 529060 Fabien Boucher 2018-01-08 1
529293 Set remote url on every getRepo in merger zuul feature/zuulv3 remote-url Tobias Henkel 2018-01-08 1
529821 Add winrm certificate handling zuul feature/zuulv3 windows-support Tobias Henkel 2018-01-17 1
530521 Slack driver zuul feature/zuulv3 slack-reporter Clint 'SpamapS' B 2018-01-10 -2 1
531009 Allow Ansible 2.4 zuul feature/zuulv3 ansible24 David Shrewsbury 2018-01-11 -2 1
531057 license: remove dos line break nodepool feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-04 -1
531171 WIP Rework log streaming to use logging zuul feature/zuulv3 zuul-stream-rewor Monty Taylor 2018-01-04 -1
531510 Delete stale jobdirs at startup zuul feature/zuulv3 James E. Blair 2018-01-12 -1 -1
531742 Also prefix the indexes if needed zuul feature/zuulv3 prefix-indexes Tobias Henkel 2018-01-08 1
531934 Add job for testing against devel of ansible zuul feature/zuulv3 turn-on-ansible Monty Taylor 2018-01-16 1
532023 Fix sql reporting with postgres zuul feature/zuulv3 fix-postgres Tobias Henkel 2018-01-09 1
532310 Switch to python3-dev / python3-devel in bindep.txt zuul feature/zuulv3 Paul Belanger 2018-01-10 2 1
532718 scheduler: better handle format status error zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-11 1
533256 Add DIB faq to docs nodepool feature/zuulv3 dib-faq Clark Boylan 2018-01-12 1
533509 ansible: honor command no_log module attribute zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-15 1
534190 Use re2 for change_matcher zuul feature/zuulv3 fix-zuul-re Clint 'SpamapS' B 2018-01-16 -1
534588 pep8: ignore E124 rule zuul feature/zuulv3 zuulv3 Tristan Cacqueray 2018-01-17 1
534956 Handle secrets in branches zuul feature/zuulv3 James E. Blair 2018-01-17 2 1
534974 └─Handle nodesets in branches zuul feature/zuulv3 James E. Blair 05:23 PM 2 1
534988 zk: check for client in properties nodepool feature/zuulv3 zuulv3 Tristan Cacqueray 02:20 AM 1
535423 WIP: Add provider info command nodepool feature/zuulv3 info_cmd David Shrewsbury 05:44 PM -1
Thanks for your understanding and your help,
Jim
[View Less]
Hi,
Currently our handling of secrets in branches is less than ideal. There
are two problems:
1) A secret may not be used by a job in another branch of the same
project.
2) A secret may not be defined in more than one branch of the same
project.
The first issue is inconvenient for users which want to, for example,
upload an image to dockerhub for all of the branches of their project.
The second prohibits the obvious fix for the first issue, but it also
means that as soon as a project with …
[View More]a secret definition branches, Zuul
will have a configuration error.
I believe the original intent was not to restrict secrets to a single
branch. If we accept that a secret should be available for use by any
branch of a project, the fix for that is simple.
The fix for the second issue is also not too difficult -- we can simply
permit duplicate definitions, but only on different branches of the same
project and with the same content. That allows us to create a new
branch from an existing one without a configuration error.
I've implemented these fixes in this change:
https://review.openstack.org/534956
In an IRC conversation, we briefly considered supporting distinct
secrets on multiple branches -- that is, a secret with the same name on
multiple branches of the same project but with potentially different
values on each branch, but that's a bit more difficult to reason about,
and ultimately doesn't make this more secure.
An important thing to note about the change is that by permitting the
use of secrets across branches, if a project has different group
maintaining the different branches, it could allow a secret to leak from
one group to the other. In other words, if a project uses secrets, you
need to trust the maintainers of all the branches of that project
equally.
It's also worth noting that even in the current situation, where we
(perhaps accidentally) only allowed the secret to be used within a
single branch, it is equally trivial for a maintainer of another branch
of the same project to simply copy the secret to their branch under a
different name and expose it, since all branches of a project share a
single private key. Preventing this would require different keys for
different branches, which would make branching impossible.
I think the proposed solution is the best way of handling this, given
the constraints. I know some folks would like to use secrets in a
production-promotion system, where distinct groups have differing levels
of access control. I think in those cases, those secrets and jobs will
have to be located in distinct projects, not branches.
Please let me know your thoughts and if this affects your use of
secrets.
-Jim
[View Less]
Hey everyone,
I wanted to start a discussion here of something I would like to setup in
at least two environments and also consolidate into a tutorial/blog for
others to build on/integrate.
Please see image of what I have setup as one of the environments.
Unfortunately one of the things documentation (could be I just have not
come across it) does not explain is the flow/dependencies upon each other
of the different daemons. I know some folks tried to explain what could be
done but I was …
[View More]still a little loss so hoping this thread can flush out the
details.
--
Kind regards,
Melvin Hillsman
mrhillsman(a)gmail.com
mobile: (832) 264-2646
[View Less]