- Zuul-discuss - lists.opendev.org

Potential enhancements to logging
by corvus＠inaugust.com 19 Jul '18

19 Jul '18

Hi, Some recent (and not-so-recent) issues with log hosting in OpenStack Infra have caused a renewed interest in reworking our log storing infrastructure. Some of the motivation and much of the initial work is OpenStack-specific, so I have written a post to the openstack-infra list describing that. Technically, none of what is in there affects the wider Zuul community, but folks reading this may want to read that message for background (or, if you follow our patterns closely, it may affect … [View More]you as well). That message is here: http://lists.openstack.org/pipermail/openstack-infra/2018-July/006020.html Also, like that message, this one is the result of several conversations, especially with Monty; many of these ideas are his. Essentially, that message describes a process for us to begin hosting our logs in swift instead of on a file server. I'll note here that Zuul is (and will continue to be, at least for the foreseeable future) agnostic about where logs are hosted. Log storage is implemented as roles in base jobs, so any site can store logs where and how they wish. We can help each other by sharing the roles which perform these tasks. OpenStack moving its log storage to swift doesn't mean anyone else needs to -- storing logs on a sever (or even the executor) will still be supported. As would storing in S3, or any other system for which someone is motivated to write a role. That message also describes decommissioning our log-hosting server which happens to provide a number of useful services today. Among the these are dynamic generation of directory indexes, and HTMLification of text logs. Our plan is to pre-generate that data, however, that's not ideal, as it expands and duplicates the amount of data stored, and it also creates more work for the executors. We can expand the role of the Zuul dashboard to take on some of those duties, thereby reducing the storage requirements, and offloading the computation to the browser of the user requesting it. In the process, we can make a better experience for users navigating log files, all without compromising Zuul's independence from backend storage mechanisms. All of what follows should apply equally to file or swift-based storage. Let's start by creating a per-build page in the dashboard. Currently, we show a list of builds, and link to the log locations for those builds. In the future, we should link to a new page with details about the build. Not only so that we can show more info (which we already collect) than can be shown on one line, but also so that we have a place for the additional functionality which follows. If we have the uploader also create and upload a file manifest (say, zuul-manifest.json), then the build page can fetch[1] this file from the log storage and display an index of artifacts for the build. This can allow for access to artifacts directly within the Zuul web user interface, without the discontinuity of hitting an artifact server for the index. Since swift (and other object storage systems) may not be capable of dynamically creating indexes, this allows the functionality to work in all cases and removes the need to pre-generate index pages when using swift. We can also extend Zuul to store additional artifact information about a build. For example, in our docs build jobs, we override the success-url to link to the generated doc content. If we return the build location as additional information about the build, we can highlight that in a prominent place on the build page. In the future, we may want to leave the success-url alone so that users visit the build page and then follow a link to the preview. It would be an extra click compared to what we do today, but it may be a better experience in the long run. Either way would still be an option, of course, according to operator preference. Another important feature we currently have in OpenStack is a piece of middleware we call "os-loganalyze" (or OSLA) which HTMLifies text logs. It dynamically parses log files and creates anchors for each line (for easy hyperlink references) and also highlights and allows filtering by severity. We could implement this in javascript, so that when a user browses from the build page to a text file, the javascript component could fetch[1] the file and perform the HTMLification as OSLA does today. If it can't parse the file, it will just pass it through unchanged. And in the virtual file listing on the build page, we can add a "raw" link for direct download of the file without going through the HTMLification. This would eliminate the need for us to pre-render content when we upload to swift, and by implementing this in a generic manner in javascript in Zuul, more non-OpenStack Zuul users would benefit from this functionality. Any unknown files, or existing .html files, should just be direct links which leave the Zuul dashboard interface (we don't want to attempt to proxy or embed everything). Finally, even today Zuul creates a job-output.json, with the idea that we would eventually use it as a substitute for job-output.txt when reviewing logs. The build page would be a natural launching point for a page which read this json file and created a more interactive and detailed version of the streaming job output. In summary, by moving some of the OpenStack-specific log processing into Zuul-dashboard javascript, we can make it more generic and widely applicable, and at the same time provide better support for multiple log storage systems. -Jim [1] Swift does support specifying CORS headers. Users would need to do so to support this, and users of static file servers would need to do something similar. [View Less]

3 5

Zuul project update 2018-07-16
by corvus＠inaugust.com 17 Jul '18

17 Jul '18

Hi Last week Tristan fixed issues with direct enqueueing of changes with ambiguous project names, Improved timer trigger to skip un-needed projects, and improving Github Event filter debugging. Tobias fixed a potentially serious bug with contamination of configuration. He's working on improving job freeze performance, and parent jobs which pause and continue to run while their children run (for resources and artifact exchange during a buildset). This week I plan on finishing the container … [View More]

2 1

Building container images in nodepool
by corvus＠inaugust.com 17 Jul '18

17 Jul '18

Hi, I think the container spec is just about ready, but one part of it recently prompted a discussion that I think could use some wider input. Specifically, this is regarding the "container-as-machine" use-case. So think "run pep8 in a container" when you're thinking about this. There's a whole bunch of stuff in the spec about container-native workloads. This isn't it. The issue is whether or not Nodepool should be able to build container images, much as we do today for machines. Note … [View More]

6 6

Improving new user experience
by corvus＠inaugust.com 11 Jul '18

11 Jul '18

Hi, Recently Artem has rightly pointed out that the experience for new users wanting to run Zuul is not good. We have some work in progress to improve that, and some efforts which are stalled and could use help. Here's what I think we should do: 1) Continue work on the Zuul From Scratch guide. This part of the documentation ends right after the Zuul services start up, but without telling anyone how to actually get Zuul to do anything useful. It needs to continue to explain: * Configuring … [View More]a pipeline * What a base job needs to do * How to create a base job from scratch * How to use or copy the existing base jobs in zuul-base-jobs * How to use jobs from zuul-jobs * Configuring a project * Actually reporting on a change or pull request I'll note that the original Zuul From Scratch document focused only on GitHub and Fedora. Obviously there's a lot of options, and folks have added sections on Gerrit and other operating systems. The most important aspect of this document, however, is that it walk someone through the entire process of setting up Zuul for at least *one* configuration. If we can handle more, that's fine. Ideall, we can keep divergent steps to a minimum. For example, we may want to describe how to push a configuration change up once (for Github, and once for Gerrit) and then subsequently refer to that process generally when we discuss how to build pipelines, jobs, etc. Otherwise we'll have to write the entire document twice. 2) Separately, we should make the process of bootstrapping a running system easier. While Zuul From Scratch should describe everything needed to build an entire Zuul system, well, "from scratch", we should also have a system that makes it easy to deploy a working Zuul so someone can at least start playing with it, and perhaps even continue to run it in production. I'll call this the "quickstart", and it should replace the current erroneously titled "quickstart" in our docs. Monty is working on jobs which will automatically build and upload container images of Zuul to Dockerhub. Once those are in place, we can write a helm chart, or Ansible playbook, or something similar to deploy those with a very minimal configuration. The goal should be to get something up and running with the minimum amount of work and also the minimum amount of documentation. If you want to learn how a Zuul works, read Zuul From Scratch. If you want to just start breaking it, use the quickstart. I think any progress we make on these two items will drastically improve the experience for new users. -Jim [View Less]

3 4

zuul-apb to install Zuul on K8/openshift using ansible-service-broker
by Artem Goncharov 11 Jul '18

11 Jul '18

Hi all, as promised in the IRC I am publishing POC work on preparing zuul-apb. It is intended to help (currently only myself) to install Zuul on top of Openshift (Kubernetes should be supported also, but with limited functionality and not tried so far). The project started to be initially used in my company, where after running zuul on VMs I decided to shift it to using Openshift. It is currently located in my personal repo in github with intention to be moved to corporate repo shortly. But … [View More]before I do this I wanted to offer it to Zuul community for discussion. If there is enough interest in adopting it, I am open and will be glad to shift development under the Openstack hat. So here are some details: - Repo: https://github.com/gtema/zuul-apb - project is built to make use of source-to-image <https://github.com/openshift/source-to-image> to build own images. This is, however, only possible if you use Openshift (not the S2I tool itself, but presence of BuildConfigs). Example repo <https://github.com/gtema/s2i-zuul> with s2i image scripts. For plain Kubernetes this might be later solved by separate pod invoking s2i inside. - regular project-config repo is also mandatory, since it is used to fetch initial components configuration. It should contain zuul/{zuul.conf,main.yaml}, nodepool{nodepool.yaml} files. It is also planned to watch project-config changes and redeploy configMaps. - gearman certificates and all other secrets are passed as APB parameters (unless you use `dev` plan and bootstrap special variables file inside for dev purposes) and stored as Secret - zookeper cluster is created also part of the project (external cluster might be used, but I decided to initially provide own) - project is not yet used in production, therefore not everything is how it should be. I intend to work on this further. There are also some issues with AnsibleServiceBroker itself, which caused some delays and few iterations with upstream. So for example the Dockerfile bases on my upstream fork due to issues with ansible 2.7 (some bugs in `lookup`, causing line breaks to be broken). When changes upstream are published I will rebase back. - work is based on Tobias's shared Openshift templates - thanks Tobias - currently 2 components (nodepool-builder and zuul-executor) are requiring privileged accounts to work. This is kind of problem with APB approach (and generally using containers), since there is currently no way to deliver this nicely and it poses big security risk to the whole cluster. Therefore there is a manual step required for deployment to grant those privileges. On the other hand nodepool-builder is very poor on running inside of container due to lots of required privileges, security concerns, and bugs in `elements` with respect to i.e. SELinux handling ( https://bugs.launchpad.net/diskimage-builder/+bug/1779273) potential problems with cross-platform builds. Tristian had an idea of using nodepool-launcher to start VM with nodepool-builder (I personally like the approach). Otherwise it is anyway funny - VM is running container, which is building VM image ;-). Additionally Tristian volunteered to work also on zuul-executor not to required privileged account. - together with work on containerization of zuul jobs this might become a full solution of running zuul under K8/Openshift and complement Windmill with a VM installation approach. I would be glad, if you share your thoughts. Especially whether community is interested in having this as an official project. Regards and have a nice weekend, Artem [View Less]

5 12

Canceling a running job
by Matthieu Huin 10 Jul '18

10 Jul '18

Hello, Yesterday we ran into a case where a periodic job had to be stopped in the middle of its execution. We ended up doing the following: * check the console stream to grab the nodes' IP addresses, from the ansible role that sets up firewall rules (this was a multi-nodes build) * SSH into each node and manually reboot the nodes This is obviously suboptimal, and possibly not pullable on single-node builds. Is there any better way to do this, or at least to get the nodes' IP addresses for … [View More]

4 3

Exciting opportunity to bikeshed over a naming question!
by Monty Taylor 10 Jul '18

10 Jul '18

Hey everybody, I've been working on jobs to build zuul container images. The obvious next step, once the build job works - is to publish them to dockerhub. This is where the bikeshed comes in. A while back, I registered dockerhub orgs for both "zuul" and "zuulci". Which org should we publish container images to? A part of me wants to use the 'zuul' one, since we have it. OTOH, the website is zuul-ci - so the other part of me thinks we should just use that. The end result of this will … [View More]

5 6

Zuul project update 2018-07-09
by corvus＠inaugust.com 10 Jul '18

10 Jul '18

Hi, Last week Shrews tracked down and fixed a Nodepool bug preventing the launcher from finding unmanaged images by ID[1]. Paul is working on a series of changes to enable jobs to specify that specific child jobs should be skipped[2][3]. I have been "helping" Tobias fix a bug related to dynamic configuration. Discussion is continuing on the container spec, but I think it's getting very close to approval. Please help write next week's email by updating this etherpad: https://etherpad.… [View More]

1 0

Dashboard routing proposals
by Tristan Cacqueray 06 Jul '18

06 Jul '18

Hi, with the recent angular6 refactor, the web interface components routing slightly changed as it is now managed by the navigation javascript component. Unfortunately this broke multi-tenant setup and we need an extra fixes to support it. Here are some options: A) Make the navigation component call a relative 'api/info' endpoint onInit to be able to introspect the deployment mode: the returned json may contains a 'tenant' value to indicate this is a white label setup. B) Always assume … [View More]

2 2

Zuul project update 2018-07-02
by corvus＠inaugust.com 03 Jul '18

03 Jul '18

Hi, This past week has seen: Monty upgrade the zuul dashboard to Angular 6 and typescript. It seems like that's mostly working now, though there may still be an issue in the multi-tenant hosting configuration. Mohammed fixed status page to not show empty headers when filtering for a specific project (a long requested feature)! Tristan found more issues with latest Nodepool master version regarding multiple pool using different labels (fixed in [1] by David), worked on changes to the log-… [View More]

1 0