- Zuul-discuss - lists.opendev.org

Re: [OpenStack-Infra] Zuul: Question about Github webhook
by Melvin Hillsman 22 Mar '18

22 Mar '18

I also have went through the "ZFS" at least once and ran into quite a few issues I think and started documenting[0] them. Bernd not sure if you have run into additional issues but feel free to add to the etherpad; it is not organized at all right now but the plan was/is to report back a list of things and patch(es). [0] https://etherpad.openstack.org/p/zuul-quickstart-and-zfs(zuul-from-scratch) On Wed, Mar 21, 2018 at 3:26 AM, <Tobias.Henkel(a)bmw.de> wrote: > Hi Bernd, > > … [View More]On 21.03.18, 08:54, "Bernd Bausch" <berndbausch(a)gmail.com> wrote: > > Sorry for this newbie intrusion on the distribution list. I am trying > to > follow the Zuul from Scratch instructions[1] to set up Zuul at home. > The > purpose is twofold: Learn about Zuul, and check whether Zuul from > Scratch > instructions are accurate and make sense. > > Currently I am at a roadblock. > > The problem is the Github webhook, through which Github sends data to > Zuul. > It should be > http://<zuul-hostname>/connection/github/payload [2], but on the Zuul > server > that > I have deployed so far, nothing listens at port 80. Thus, Github gets > a "connection refused" when trying to contact my server. I used > tcpdump to > confirm that my server does indeed receive an http POST from Github. > > Either something is wrong with my setup, which prevents Zuul from > listening > at > 80. Or the webhook isn't documented correctly. > > How can I confirm what could be the problem? > > Bernd > > [1] https://docs.openstack.org/infra/zuul/admin/zuul-from-scratch.html > [2] https://docs.openstack.org/infra/zuul/admin/drivers/github.html, > https://docs.openstack.org/infra/zuul/admin/zuul-from- > scratch.html#configure > -github > > > It looks like you discovered a bug in the zuul-from-scratch docs. Zuul-web > by default > listens on port 9000. While the documented webhook config in github > assumes port 80. > So you need to either change the webhook to use 9000 or change the zuul > config to use > port 80 [1]. > > Further I pushed up a change to fix the doc [2]. > > Sidenote: we now have zuul specific mailing lists [3] so you might want to > also subscribe > to them. > > [1] https://docs.openstack.org/infra/zuul/admin/components. > html#attr-web.port > [2] https://review.openstack.org/554829 > [3] http://lists.zuul-ci.org > > Kind regards > Tobias > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -- Kind regards, Melvin Hillsman mrhillsman(a)gmail.com mobile: (832) 264-2646 [View Less]

3 4

Services missing tenant scope (nodepool-*, zuul-executor)
by Tristan Cacqueray 19 Mar '18

19 Mar '18

Hello folks, Is there a plan to add a tenant scope to all the services? For example, we could add the tenant name to NodeRequest and a 'tenant' setting in nodepool.yaml so that a provider and its labels could be restricted to a particular tenant. Similarly for the executors, we could be able to isolate tenant job execution from each other. Has this been discussed at the PTG? Regards, -Tristan

2 2

Zookeeper ConnectionLoss/SessionExpired issues
by Tristan Cacqueray 16 Mar '18

16 Mar '18

Hello folks, Here is a recap of the issues happening when the connection to Zookeeper is unstable: * Nodepool marks dib build as failed, * ConnectionLoss/SessionExpired exceptions happen when the connection drop after the "self._zk.suspended or self._zk.lost" checks, * Zuul lost NodeRequest which may lock change in queued state, and * Zookeeper needs to be running before starting Zuul or Nodepool, otherwise the daemon stay stuck in limbo. This used to happen frequently in our deployment (… [View More]

5 16

Disabling nullwrap
by corvus＠inaugust.com 14 Mar '18

14 Mar '18

Hi, The recent security fixes have underscored that bubblewrap is currently key to Zuul executor security. While Zuul does have an option to run without bubblewrap (called "nullwrap"), it is not recommended, and it doesn't support secrets, which is a fairly important feature. Nullwrap was written with the idea of aiding those use-cases where it is not possible to run with bubblewrap. However, I think that if such a case still exists, we should work to address that -- nullwrap really shouldn'… [View More]

4 3

Landing changes to zuul-web javascript
by Monty Taylor 09 Mar '18

09 Mar '18

Hey everybody! Something just occurred to me while reviewing the next set of patches from Tristan to zuul-web that I think is important to share more broadly. In OpenStack's deployment of Zuul, HTML/JS changes to zuul-web go live basically immediately (give or take 15 minutes) No services need to be restarted or anything. However, while the python code will also be _installed_ at the same time, it will not go live immediately. It's also completely possible for someone to deploy web … [View More]

3 4

working on zuul trigger plugin for jenkins
by Darragh Bailey 08 Mar '18

08 Mar '18

Hi, Spend some time last week hacking away at a zuul trigger plugin for jenkins, really just about getting some better understanding of how this should hang together and what questions need to be asked. Lots of time spent reading details on gRPC as it was suggested at https://etherpad.openstack.org/p/zuulv3-jenkins-integration So far I've got a (very) rudimentary gRPC server that responds to a single status request with OK working as a Jenkins plugin that starts up automatically. I'll try to … [View More]get it released somewhere soon so it's possible to discuss code, though if it is of interest to the OpenStack community maybe it should be hosted on review.openstack.org? Some general things I've noted: - authenticating requests doesn't appear to be straight forward with gRPC - unclear whether this can share the default jenkins http port even though gRPC and REST (currently using dedicated port) can technically do this - given the number of messages exchanged for a single job, should this use bidirectional streaming - started with a proto layout that is mostly unary client calls and server streaming responses - but should the trigger really be a single connection with multiple messages around the hand-off coordination? I've also had a bunch of questions to bounce off Monty around how to deal with Zuulv3 and Jenkins node coordination, the upshot of those is to leave that to nodepool and probably help out with the Jenkins nodepool plugin and make that a dependency of the zuul trigger plugin. For more detail, originally I was worrying about how to ensure zuul trigger plugin would work without using the nodepool plugin, and it would end up needing to query Jenkins as to whether it had the resources to execute the on before triggering, and this would require a deeper integration into Zuul. It was suggested that this would complicate it deeply and would be better to have nodepool manage the resources, so that zuul would only start the ansible job that would call the zuul trigger API endpoints if nodepool had sufficient resources for Jenkins to use to execute the job. It appears clear any migration path to Zuulv3 involving Jenkins would want to use Zuulv2 -> Gearman -> Jenkins -> Nodepoolv3 first, then migrate Zuulv3. So I'm looking to help on the Jenkins Nodepool plugin first, and that will probably make it much more straight forward to understand what needs to be done in the trigger plugin. A couple of weeks ago someone was in the Zuul room chatting about working on the Nodepool plugin for Jenkins, wondering if there has been any progress there, and what I can help out with in the future. -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool" [View Less]

1 0

[nodepool] Webapp label_list
by David Shrewsbury 01 Mar '18

01 Mar '18

Greetings! We have recently merged a change to add the label_list command to nodepool's webapp [1]. For that change, we briefly discussed in IRC [2] the meaning of the "count" parameter that was originally within the output of that command. The tl;dr from me in that discussion was that count without considering node state has no meaning. The final PS of that review left count out of the results. There is now another change proposing to add that count back in [3]. I think this functionality … [View More]

4 8

[nodepool] Restricting images to specific nodepool builders
by Ian Wienand 20 Feb '18

20 Feb '18

Hi, How should we go about restricting certain image builds to specific nodepool builder instances? My immediate issue is with ARM64 image builds, which I only want to happen on a builder hosted in an ARM64 cloud. Currently, the builders go through the image list and check "is the existing image missing or too old, if so, build" [1]. Additionally, all builders share a configuration file [2]; so builders don't know "who they are". I'd propose we add an arbitrary tag/match system so that … [View More]

6 10

How to determine what has changed in a test
by corvus＠inaugust.com 08 Feb '18

08 Feb '18

Hi, The problem of how to determine what has changed in the change under test has come to my attention twice today, and it's still early, so I thought it'd be a good topic for a ML discussion. The two issues that have come to my attention represent two similar kinds of tests: a) Inspect the specific commit(s) under test (e.g., inspect the commit message). b) Inspect the delta of applying the patch (e.g., run coverage twice and calculate the difference). Simon Westphahl has proposed a … [View More]patch focused on the first case, which adds a zuul.commits variable with the sha(s) of the patch(es) under test: https://review.openstack.org/539975 Meanwhile, here is an example of a test which relies on the existence of an origin remote: http://logs.openstack.org/58/473658/19/check/openstack-tox-cover/0f6849c/jo… We also have the minor example of pip emitting warnings if there is no origin remote; if we can fix that here, that's fine, but I don't think it's significant enough to drive our design. The patch to add zuul.commits makes case (a) fairly simple -- you can have ansible iterate over the values in the variable. I think that case (b) might be a bit harder with zuul.commits -- specialized tooling would be needed to create the alternative repo state (ie, a new branch) in order to perform the git checkout or diff operations. Because in (b) we are focused more on the previous state than the commit, we can't just checkout zuul.commit, we need to checkout the parent of zuul.commit. However, if we were to create an origin remote, and populate it with the "previous" git repo state, (that is, either the current state of the tree, or that with the changes ahead of this one applied), it seems we can accommodate both (a) and (b) in an intuitive manner. You could get the values that would be put in zuul.commits by performing 'git diff origin/master..master', which addresses (a), and you could address (b) with 'git checkout origin/master'. Among the reasons we don't have any remotes on the git checkouts are: * What would the URL be? * What would the content be? * Concern about jobs altering the git repo under test by performing naïve and incorrect git operations. We've addressed the content question, that leaves the other two. Ideally, if we did this, I'd like to have a remote with no URL. However, while that may be possible, some investigation suggests that we may want to avoid that as it looks like it may be a bit of an edge case that not all tools or libraries would support correctly. However, if we set it to a bogus URL, that may be sufficient to satisfy the tooling, as well as prevent git repo manipulation errors. I'm inclined to favor this approach, as I think it addresses both cases and is fairly intuitive. Do folks think it would work? Are there things we would be missing as compared to the zuul.commits approach? Thanks, Jim [View Less]

3 3

Zuul website
by corvus＠inaugust.com 01 Feb '18

01 Feb '18

Hi, We now have a website at https://zuul-ci.org/ The content is in git in the openstack-infra/zuul-website repo: https://git.openstack.org/cgit/openstack-infra/zuul-website/tree/ I put together the site using a CC-BY 3.0 template, and I expect that the content of that repo will be licensed CC-BY as well (as most of our docs-only repos are). I enthusiastically invite other folks to contribute to the website by proposing changes to that repo. Currently the zuul-core team has approval … [View More]

2 1