January 2023 - Zuul-discuss - lists.opendev.org

Re: Environment variables passed to a tox job from encrypted secrets leak into the build log, how do I suppress this?
by Florian Haas 01 Feb '23

01 Feb '23

On 26/01/2023 19:26, Clark Boylan wrote: >> I have been unable to pinpoint exactly where and how that specific >> log is generated by the job. If somebody could point me to it, and >> let me know what overrides I need to add to the job to suppress >> this log altogether, or to remove the env variables from it, or >> better still configure it to replace the values of env variables >> matching a glob or regex with "<redacted>" or "*********" or some >> such, I'd be most grateful. > > Tox itself produces these log files and writes them to > .tox/$envname/log then the fetch-tox-output role [0] collects them > for you. Looking at this role there doesn't appear to be a flag to > disable it, just flags to modify its behavior. It also doesn't look > like there is a way to stop Tox from logging these files [1]. You > might just want to drop the fetch-tox-output role from your jobs and > not copy the files. Or maybe you can get away with a job task that > truncates the files before they are copied. Thanks for the pointer; that helped a lot. It turns out that {work_dir}/{env_name}/log did already exist in tox 3. However, the logs it contained did *not* dump all the environment variables, and so those never ended up in the Zuul web interface. As of tox 4, they do.[1] This might have security implications for anyone that runs tox from Zuul, hasn't locked down their Zuul web interface, uses sensitive information that is passed into a tox environment via environment variables, and just upgraded from tox 3 to tox 4. They might not be aware that their secrets are now accessible via the web interface, when with tox 3 they were not. Since the fetch-tox-output role looks explicitly in {work_dir}/{env_name}/log to find those logs, an applicable workaround to make those logs not show up in the Zuul build log (though, of course, they still get generated) would be to put envlogdir = {toxinidir}/dummylog into the testenv definition in tox.ini (for any testenvs that might pass sensitive information to tox via environment variables). Then, Zuul's fetch-tox-output role can't find anything in the directory it looks into, and it can only produce the job-output.txt log which — as far as I've been able to tell — does not contain a dump of environment variables. tox will generate envlogdir if it does not exist. It it NOT advisable to use some random existing directory; tox 4 will remove all files contained therein at the start of the run (tox 3 didn't do *that,* either). Again, thanks for the pointer! Cheers, Florian [1] More precisely, I see the new log format when testing with tox 4.4.2; I see the old one with 3.28.0. So the format would have changed at some point between those releases, though I haven't looked into exactly when.

2 1

Environment variables passed to a tox job from encrypted secrets leak into the build log, how do I suppress this?
by Florian Haas 26 Jan '23

26 Jan '23

Hi everyone, first post to list, though I know a few folks who are subscribed personally — to those of you, no long time no see! Here's my issue: I want to keep Zuul from leaking secrets into the build logs, and in this specific instance I don't know where to look for the fix. In more detail: I am running a tox job from Zuul that uploads some generated files to a Ceph radosgw service via its S3 API. This is my "upload" testenv as defined in my tox.ini: [testenv:upload] deps = awscli passenv = AWS_* setenv = file|build.env commands = aws --endpoint-url={env:S3_ENDPOINT} \ s3 sync \ {toxinidir}/build/ \ s3://{env:S3_BUCKET}/ I set S3_ENDPOINT and S3_BUCKET in my build.env file, and I want to pass AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment through to the aws s3 sync command, hence the passenv entry. Since I would prefer to not have to maintain a separate job definition in a config project for this, I have the following secret and job definition in my repo's .zuul.yaml: - secret: name: tox_environment_secret data: AWS_ACCESS_KEY_ID: !encrypted/pkcs1-oaep - # encrypted secret redacted AWS_SECRET_ACCESS_KEY: !encrypted/pkcs1-oaep - # encrypted secret redacted - job: name: tox-upload parent: tox secrets: - name: tox_environment secret: tox_environment_secret pass-to-parent: true vars: tox_envlist: "upload" As described in https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.secrets, this results in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY being passed into the tox run managed by the tox-upload job (or rather by its parent, the upstream tox job), as environment variables, with their values set to the decrypted secrets. Now, when I check the build log at log/job-output.txt (relative to my build URL), I see no trace of the secrets. So far, so good. But there is another job log at log/tox/upload/1-commands[0].log, which contains *all* the environment variables passed into the tox job, including the decrypted values for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY: name: upload run_id: commands[0] env LANG: C.UTF-8 env AWS_ACCESS_KEY_ID: <redacted> env AWS_SECRET_ACCESS_KEY: <redacted> [etc.] I'd obviously prefer for those to not leak into the build logs. I have been unable to pinpoint exactly where and how that specific log is generated by the job. If somebody could point me to it, and let me know what overrides I need to add to the job to suppress this log altogether, or to remove the env variables from it, or better still configure it to replace the values of env variables matching a glob or regex with "<redacted>" or "*********" or some such, I'd be most grateful. Thanks! Cheers, Florian

2 1

Feedback on Changes to Supported Python Version in Zuul
by Clark Boylan 24 Jan '23

24 Jan '23

Hello everyone, Eventually (hopefully soon), Zuul will add Ansible 7 (Ansible Core 2.14 support). According to the Ansible docs [0] this new version of Ansible drops support for Python 3.8. Currently Zuul is tested on Python 3.8 and 3.11 with the intent of supporting Zuul across Python 3.8, 3.9, 3.10, and 3.11. Unfortunately, Ansible's dropping of Python 3.8 support means that Zuul will also need to drop support for Python 3.8 when Ansible 7 support is added to Zuul. On top of that, Zuul test suite runs in CI are approximately 30 minutes quicker running on python 3.10+ than on older Pythons. Unfortunately, I don't have the best data to illustrate this since we currently only run jobs for 3.8 and 3.11, but the major performance improvements appear to have shown up with 3.10. You can get a sense for this difference comparing the 3.8 runtimes [1] to the 3.11 runtimes [2]. As a developer I selfishly think that this improvement is worth considering, and we might also take this opportunity to drop Python 3.9 support (and testing). For anyone running on Python 3.8 today now is probably a good time to start looking at updated platforms. Is there anyone out there running Zuul on Python 3.9 today? If you are running Zuul on Python < 3.10 we'd love to hear from you to ensure we understand the potential impact of changes like this. [0] https://docs.ansible.com/ansible/latest/installation_guide/intro_installati… [1] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py38&pipeline=gate… [2] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py311&pipeline=gat… Clark

5 10

The OpenInfra Summit 2023 CFP is closing soon!
by Helena Spease 10 Jan '23

10 Jan '23

Hi Everyone! The CFP for the 2023 OpenInfra Summit (June 13-15, 2023) is closing in just a few days[1]! Check out the full list of tracks and submit a talk on your topic of expertise [2]. The CFP closes January 10, 2023, at 11:59 p.m. PT. See what that is in your timezone [3] We are also now accepting submissions for Forum sessions [4]! Looking for other resources? Find information on registration, sponsorships, travel support and visa requests at https://openinfra.dev/summit/ <https://openinfra.dev/summit/vancouver-2023> If you have any questions feel free to reach out :) Cheers, Helena [1] https://cfp.openinfra.dev/app/vancouver-2023/19/presentations <https://cfp.openinfra.dev/app/vancouver-2023/19/presentations> [2] https://openinfra.dev/summit/vancouver-2023/summit-tracks/ <https://openinfra.dev/summit/vancouver-2023/summit-tracks/> [3] https://www.timeanddate.com/worldclock/fixedtime.html?msg=2023+OpenInfra+Su… <https://www.timeanddate.com/worldclock/fixedtime.html?msg=2023+OpenInfra+Su…> [4] https://cfp.openinfra.dev/app/vancouver-2023/20/ <https://cfp.openinfra.dev/app/vancouver-2023/20/>

1 1

Future of the ensure-kubernetes role
by Ian Wienand 09 Jan '23

09 Jan '23

Hello, The ensure-kubernetes role in zuul-jobs sets up a k8s environment on the test node so we can do some API testing against, and run, small containers. I think it's fair to say this isn't aimed at being a production installation, but rather the minimal parts of k8s that basic functional testing can validate against. Currently the ensure-kubernetes role deploys k8s via minikube bionic/focal/jammy with a choice of two backends; cri-o and docker. I worry about this whole toolchain. It took some effort to get it working on Jammy, and it's not very elegant. Some of the problems - kubectl installs from a repo called "https://apt.kubernetes.io/ kubernetes-xenial main" [1]. It seems to get updates; and maybe it refers to the lowest version. I don't know, but it's not reassuring. - minikube isn't packaged and we download a blob directly. This is probably neither here-nor-there; but it is less appealing than using something from a distro. - everything is written to use the minikube "none" driver, which avoids having to worry about nesting VM's. - minikube with the "none" driver only supports a CRI compatible backend and has a lot of caveats [2] - cri-o is CRI compatible, but is not upstream packaged for Ubuntu. We rely on packaging of the tools from the SUSE kubic program, which has now been retired and is in a sort of best-effort community maintenance [3]. What prompted me looking at this was an expired GPG key in this project which broke testing. This was fixed, of course bugs happen to everyone. But we are 100% dependent on this repo (I could not get the docker back-end to work without some of the runtime tools from it, so it is not completely independent). - docker is packaged externally, but widely used (by OpenDev in production even). But to use it with minikube, we need to run a CRI api -> dockerd shim that we download as a binary blob [4] - With all these disparate parts, we get conflicts. For example, bits of podman are packaged on Jammy, so now "containernetworking-plugins" comes from the distro. Which of course installs things in a different location to the kubic versions. Which means we have to monkey-patch over the top of patching that minikube does [5]. - I don't think we actually got any of this working for Focal; we just had jobs on bionic and skipped straight to jammy [6]. So we're not keeping up. I think that's probably evidence enough that setting up a test k8s environment with minikube + <cri-o|docker> on Ubuntu is not very compelling. I've personally spent much more time debugging all this than anything to do with actual k8s support in tools. minikube seems to have a docker driver now [7], which I'm sure did not exist when any of this was written and might suit us as it says it does not need nested virt. However, I would propose that we are not making good use of our limited resources by continuing along this path. Ubuntu's preferred method for deploying k8s is via microk8s which they package as snap images [8]. I am not a huge fan of snap images for distribution, but I am a fan of not trying to maintain a bunch of parts that don't really want to work together. I have written changes in https://review.opendev.org/q/topic:ensure-kubernetes-microk8s to setup microk8s in ensure-kubernetes, and to make it so when using use-buildset-repository with microk8s, k8s will pull through the buildset repository. I have converted the nodepool job to use this. What I would propose is that we deprecate the other paths in ensure-kubernetes and focus our efforts on microk8s on jammy (and future distros going forward). This role is not heavily used, I don't see our time well spent supporting multiple installation methods across multiple releases (and it's never been cross-distro -- which I also think is fine). I see this having a greater role in the future as part of the work such as the nodepool->zuul conversion, where it may get a lot more exposure. Thoughts are welcome :) -i [1] https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/insta… [2] https://minikube.sigs.k8s.io/docs/drivers/none/ [3] https://kubic.opensuse.org/blog/2022-06-10-kubic-retired/ [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-kubernete… [5] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-kubernete… [6] https://review.opendev.org/c/zuul/zuul-operator/+/863948/3/.zuul.yaml [7] https://minikube.sigs.k8s.io/docs/drivers/docker/ [8] https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s#1-ove…

2 1

Project lead election
by James E. Blair 09 Jan '23

09 Jan '23

Hi, The position of Zuul Project Lead is an elected position described in our governance document[1]. I currently hold the position, and would be happy to continue in that position for another year. If elected, I plan to continue my current hands-off approach. I believe that all of the Zuul Maintainers are empowered to collectively decide on and execute our vision for the project. My purpose is to help articulate that vision when necessary, drive the process of attaining consensus, and be a point of contact and representative for the project in circumstances where a single person is needed (but in those cases, I promise to share as much of that work as possible with the rest of the team). I have uploaded a change[2] to update the expiration date of my term. If any other Zuul Maintainers would like to nominate themselves for the position, please upload a similar change and send a reply to this message to this list. The nomination period will close on January 16, 2023. If there are other candidates, we will hold a Condorcet election at that time. -Jim [1] https://zuul-ci.org/docs/zuul/governance.html#zuul-project-lead [2] https://review.opendev.org/869609

1 0

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 09 Jan '23

09 Jan '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we continue working on planning for the OpenInfra Summit as well as Foundation-wide surveys. Thanks, Amy (spotz) 0 - https://etherpad.opendev.org/p/diversity-wg-agenda

1 0

Fwd: 2023 PTGs + Virtual Team Signup
by Kendall Nelson 03 Jan '23

03 Jan '23

Hello Everyone! The next PTG on the calendar will be our usual virtual format and it will take place March 27 -31. The project signup survey is already ready to go! If your team is interested in participating, sign them up here[1]! Registration is also open now[2] We have also heard many requests from the contributor community to bring back in person events and also heard that with global travel reduction, it would be better travel wise for the community at large to combine it with the OpenInfra Summit[3]. So, an in-person PTG will take place Wednesday and Thursday! June 14 -15th. OpenInfra Summit registration will be required to participate, and contributor pricing and travel support[4] will be available. We will also be planning another virtual PTG in the latter half of the year. The exact date is still being finalized. This will give you three opportunities to get your team together in 2023: two virtual and one in-person. - Virtual: March 27-31 - In- Person: June 14-15 collocated with the OpenInfra Summit - Virtual: TBD, but second half of the year As any project in our community, you are free to take advantage of all or some of the PTGs this coming year. I, for one, hope to see you and your project at as many as make sense for your team! To make sure we are as inclusive as possible I will continue to encourage team moderators to write summaries of their discussions to gather and promote after the events as well. Stay tuned for more info as we get closer, but the team signup survey[1] has opened for the first virtual PTG and is ready for you! -Kendall (diablo_rojo) [1] https://openinfrafoundation.formstack.com/forms/march2023_vptg_survey [2] https://openinfra-ptg.eventbrite.com [3]https://openinfra.dev/summit/vancouver-2023 [4] https://openinfrafoundation.formstack.com/forms/openinfra_tsp

1 1