- Zuul-announce - lists.opendev.org

Zuul 13.0.1 released [security]
by James E. Blair 23 Oct '25

23 Oct '25

The Zuul team announces the release of Zuul 13.0.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci This release addresses the following security issue: Jan Gutter observed that Zuul's streaming log handling for the win_command and win_shell modules did not correctly observe the no_log directive. Previously Zuul would suppress output to the job-output.json file, but not output to job-output.txt or live log streams. This has been corrected and no_log is now honored under both circumstances. The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/13.0.1/releasenotes.html Security Issues =============== * Streaming log handling for the win_command and win_shell modules did not correctly observe the no_log directive. Previously Zuul would suppress output to the job-output.json file, but not output to job-output.txt or live log streams. Bug Fixes ========= * The Zuul module for ansible.builtin.command and ansible.builtin.shell now correctly handles the case when no_log: True is set. Previously the task would deadlock if the process output filled the buffer.

1 0

Zuul 13.0.0 released
by James E. Blair 11 Sep '25

11 Sep '25

The Zuul team is pleased to announce the release of Zuul 13.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/13.0.0/releasenotes.html New Features ============ * Ansible version 11 is now available. The default Ansible version is still 9, but version 11 may be selected by using job.ansible-version. * The authorization-rule.conditions attribute can now include conditions for boolean claims. * Implemented support for the reject attribute in the GitLab driver. It allows specifying Merge Requests which should not be enqueued into a pipeline. The supported conditions cover the same Merge Requests parameters as available in the existing require attribute – filtering whether the Merge Request must (not) be open, merged, approved or contain some labels. * If a change alters a file in an job.include-vars, job.pre-run, job.run, or job.post-run attribute of a job, then that job will run regardless of any file matchers configured on the job (unless job.match-on-config-updates is disabled). This means that jobs with file matchers should no longer need to explicitly list their playbooks or include-vars files in order to run on changes to those files. * The nodeset.alternatives feature will now be used if Ansible reports an unreachable host while running a job. This can be used to fall back to a more reliable nodeset if connectivity to the nodeset is interrupted while running. Upgrade Notes ============= * Support for Ansible version 8 has been removed. Migrate any existing jobs which rely on this version to Ansible version 9 before upgrading. * The previously deprecated status-url attribute of the Github and Pagure reporter has been removed.

1 0

Security update to upload-logs-s3 and upload-artifactory roles
by James E. Blair 04 Sep '25

04 Sep '25

|James Blair recently discovered that two roles in the zuul-jobs repo could be subverted to expose the credentials used for uploads. The roles "upload-logs-s3" and "upload-artifactory" used undocumented variables to enable pre-merge testing of these roles in test environments. The variables could be used to cause the upload tasks to run on a host other than the executor, and if that host was compromised, the credentials could be exposed. This technique predates more recent changes to Zuul which permit the testing of roles of this nature without such overrides. Therefore, these variables have now been removed from the roles. The zuul-jobs repo has been updated to correct this problem, so if you include it in your tenant configuration directly, it will automatically be updated. If you use a local mirror of zuul-jobs, you should update it immediately. The update to zuul-jobs is this commit: https://opendev.org/zuul/zuul-jobs/commit/15828c7a8595fe615f05308e7f53b8e4a… |

1 0

Changing the default for configure_mirrors_extra_repos to False in zuul/zuul-jobs configure-mirrors
by Clark Boylan 27 Aug '25

27 Aug '25

This is your warning that in two weeks, September 9, we will be changing the default for configure_mirrors_extra_repos from True to False within zuul-jobs' configure-mirrors role. When set to True (the current default) this configuration option enables additional package repositories like Debian Backports and CentOS Stream PowerTools. Debian will delete backport repos for older releases (as they have done recently for Debian Bullseye) which means that the default configuration for this role produces a Debian config that is broken on Bullseye. We intend to change the default from True to False to better mimic a default package repo configuration for these platforms and to avoid problems as the platform ages and packages repos are removed. If your jobs rely on packages in these extra repositories you can either set configure_mirrors_extra_repos back to True in your jobs or add the extra repos as an additional step. This short term inconvenience should help avoid future issues and ensure that our test environments more closely mimic an "off the shelf" installation. If you have any questions or concerns feel free to reply to this message or followup on the proposed change in Gerrit [0]. [0] https://review.opendev.org/c/zuul/zuul-jobs/+/958605 Clark

1 0

(zuul-jobs) Removing deprecated js jobs and defaults
by Damian Fajfer 14 Aug '25

14 Aug '25

Hello, Since both of these changes got +2 on Code-Review 921067: Remove deprecated javascript jobs | https://review.opendev.org/c/zuul/zuul-jobs/+/921067 957219: Remove version defaults for nodejs jobs | https://review.opendev.org/c/zuul/zuul-jobs/+/957219 I would like to announce a 2 week notice before merging the changes. - For deprecated jobs the alternatives are listed in description. - Regarding defaults, you need to provide them as a variable instead of relying on (old and ancient) defaults. For example in Zuul it's done on project.var level[1], but currently was not working due to job.var having a higher precendence. [1] https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L383 Thank you, Damian

1 0

Ansible 11 and the multi-node-bridge role
by Clark Boylan 14 Aug '25

14 Aug '25

Zuul is growing support for Ansible 11. One of the issues we've found with Ansible 11 is that it no longer includes the openvswitch.openvswitch collection which provides the openvswitch_bridge Ansible module. The zuul/zuul-jobs multi-node-bridge role currently relies on openvswitch_bridge to configure its overlay networking and is broken under Ansible 11 as a result. I have proposed a change [0] to vendor the openvswitch_bridge module into the multi-node-bridge role in zuul/zuul-jobs. Testing within OpenDev indicates that this unmodified vendored code continues to work under Ansible 11; however, updates to Ansible or openvswitch may break it in the future. For this reason we're considering this role effectively deprecated at this point in time. Ideally this role would get rewritten to use Linux bridges instead of openvswitch as this set of tools is more common. This might also be a good time to consider using Geneve instead of VXLAN, but there is a good chance that VXLAN's age means it is more readily available on the various platforms Zuul jobs run on. If you are using this role today and have an interest in updating the role to replace openvswitch with Linux bridges we would appreciate the help. Feel free to reach out in response to this email or follow up on Matrix. [0] https://review.opendev.org/c/zuul/zuul-jobs/+/957188 Clark

1 0

Changing defaults in limit-log-files role in zuul-jobs
by James E. Blair 10 Jul '25

10 Jul '25

Hi, We are planning on merging https://review.opendev.org/953854 on Thursday, July 24, 2025 to update a default value for the limit-log-files role. Currently, the role has a default limit of 100 files. The change above updates the role to allow an unlimited number of files (so that the role might be used only to limit the overall size), and changes the default to unlimited. If you are using this role and relying on the default, please update your use of the role to set an explicit value for `limit_log_files_file_limit` before July 24 in order to avoid disruption. Thanks, Jim

1 0

Zuul 12.1.0 released
by James E. Blair 18 Jun '25

18 Jun '25

The Zuul team is pleased to announce the release of Zuul 12.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/12.1.0/releasenotes.html New Features ============ * Added support for OpenID Connect (OIDC) compatible workload identity federation. When configured using secret.oidc, Zuul will generate an OIDC ID token and supply it to the job, which can be used to authenticate to external services that trust Zuul as an OIDC Identity Provider. See Secret for more information. * An option has been added to allow multiple timer-triggered queue items for the same project-branch may now be enqueued if the branch head has advanced. Previously timer-triggered items only included information about the project and branch, and therefore if a job on such an item ran longer than the timer interval, Zuul would not enqueue a second copy of the item because it would have been seen as identical to the first (even if the underlying repo had changed. If the new dereference attribute of the timer trigger is set, Zuul will now include information about the branch head SHA of the queue item, so that now if a timer trigger fires while a queue item for the same project-branch is already in the queue, the second item will be enqueued if the branch has changed since the first item was enqueued. If the SHA of the branch head is the same for both items, the second item will not be enqueued. When set, the newrev is now available in the job's zuul variables for such queue items. The previous behavior remains the default, where derefence is not set, and newrev is not available. * Most pipeline triggers now accept a debug attribute which may be used to add a trigger to allow users to request extra debug information about why a change may or may not have been enqueued into a pipeline and why certain jobs ran or did not run. For example, a site may add a trigger that matches the comment "check debug" to request an expanded report on a change. * Tenant administrators may now use the web UI or zuul-client commands to pause and unpause event processing for a tenant.

1 0

Zuul 12.0.0 released
by James E. Blair 22 Apr '25

22 Apr '25

The Zuul team is pleased to announce the release of Zuul 12.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/12.0.0/releasenotes.html New Features ============ * Jobs now have the ability to set a job.pre-timeout value. This timeout limits the amount of time that pre-run playbooks may consume. This timeout counts against the normal job.timeout value and if job.pre-timeout is unset then job.timeout is used. * Add replication_timeout to Gerrit connection configuration. Setting this to a positive value makes Zuul’s Gerrit connection replication aware. Zuul will wait up to this many seconds for replication to complete for events like patchset-created, change-merged, and ref-updated before continuing to process those primary events. This is useful if you have configured Zuul to look at a Gerrit replica for git data. * The Zuul Executor has a new governor sensor that detects when the Executor is nearing the limit on new processes. Running Zuul jobs in the Executor requires a number of process including but not limited to: Bubblewrap, Ansible, SSH, and SSH Agent. This new governor sensor helps to avoid jobs failing due to an inability to fork one of these many processes. Instead the Executor will pause its efforts until it backs away from the process limit. * Live console log streaming is now available for win_shell and win_command tasks on Windows hosts. Upgrade Notes ============= * When using Zuul with a MariaDB database, configure the database.dburi setting to use the mariadb dialect with mariadb+pymysql://. * The web.status_url configuration option is no longer supported and using the status_url, change and changes string substitutions in pipeline reporter messages will result in an error. * Due to the addition of live console log streaming for Windows hosts, one of the following is necessary: * Add the zuul_win_console: task to the first pre-run playbook in a base job for Windows hosts, and allow connections to port 19886 on the hosts. This will allow for automatic live log streaming in the same manner as under Posix systems. * If the above is not practical on long-lived Windows hosts, set zuul_console_disabled: True for those hosts. This will disable writing console output to spool files on the remote hosts which would otherwise not be deleted. Note that the Posix and Windows log streaming servers operate on different ports (19885 and 19886 respectively) in order to allow both to coexist on the same host. Bug Fixes ========= * Job json API responses now include the job.post-timeout configuration value for the job.

1 0

Zuul 11.3.0 released
by James E. Blair 05 Mar '25

05 Mar '25

The Zuul team is pleased to announce the release of Zuul 11.3.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/11.3.0/releasenotes.html New Features ============ * Individual attributes of jobs may now be marked as final so that any attempt to override that attribute will cause an error. See job.attribute-control. * Zuul will now run any job that is a hard dependency of a job that is set to run. Previously if job B had a hard dependency on job A, and job A did not run due to a file matcher, Zuul would report an error. Now it will instead ignore the file matcher on job A as long as job B is to be run. This will allow for simpler file matchers, more intuitive behavior, and enable a new pattern for job graphs where job A might have a file matcher that never matches, but job B will cause it to run regardless. * A new options, job.workspace-checkout is available to disable checkouts of repositories on the Zuul executor. This may be useful to save time or space when preparing large repositories which will be unused on the executor. Upgrade Notes ============= * The Gerrit driver now has an additional option, pipeline.reporter.<gerrit reporter>.notify which configures the notification handling of reviews as noted in the Gerrit changes API Docs. used to suppress reporting job results as review comments. Due to the configuration syntax for Gerrit reporters, the word “notify” may no longer be used as a review label. * The maximum combined stdout and stderr output from a single Ansible task has been limited to 1GiB. In the unlikely event that an existing job legitimately exceeds this limit, a new executor configuration option, executor.output_max_bytes, has been provided to temporarily increase the limit. This option is likely to be removed in a future version of Zuul. To avoid this issue, Ansible tasks with large volumes of output should be adjusted to redirect that output to a file which is separately collected and processed. * Schedulers will no longer trigger a smart-reconfig after startup and with that also not pick up modifications to the local tenant config and system attributes in the zuul config file with a restart of a scheduler. This means that changes to the tenant config or the zuul config always require an explicit (smart) reconfiguration.

1 0

[zuul-jobs] Update remove-registry-tag role to use explicit registry type
by Vladimir Kozhukalov 15 Jan '25

15 Jan '25

Hello, [1] is going to be merged in two weeks from the moment on Jan/28/2025 or around this date. It removes the field `remove_registry_tag_api_type` and replaces it with the `type` field in the `container_registry_credentials` (see example below) dictionary that allows one to explicitly define the type of the registry used. It also makes it aligned with other roles like `ensure-quay-repo` regarding how registry type is defined. ```yaml container_registry_credentials: quay.io: type: 'quay' api_token: 'abcd1234' docker.io: type: 'docker' username: 'username' password: 'password' ``` By default if the registry type is not given it is guessed from the registry url. This also improves the usage experience of the role. After we merge [1] it is going to be possible to use this role within a loop in case one would like to publish images to multiple registries. At the moment the condition [2] prevents the variable `remove_registry_tag_api_type` to be redefined on the second and later iterations in a loop. [1] https://review.opendev.org/c/zuul/zuul-jobs/+/881524 [2] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/remove-registry-… -- Best regards, Kozhukalov Vladimir

1 0

Zuul 11.2.0 released
by James E. Blair 02 Dec '24

02 Dec '24

The Zuul team is pleased to announce the release of Zuul 11.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/11.2.0/releasenotes.html New Features ============ * The project.<pipeline>.fail-fast attribute now also applies to node failures of voting jobs. * Jobs may now read variables from an in-repo file using the job.include-vars job attribute. This allows jobs with many (or frequently changing) variables to load their variables from a file in order to reduce the complexity of Zuul configuration. Because these files are read when the job is executed, changes to their values will not cause Zuul tenant reconfigurations. Upgrade Notes ============= * Ansible version 8 is now deprecated in Zuul since it is unmaintaned, and it will be removed from a future version of Zuul. Ansible 9 is now the default version in Zuul. * Github Enterprise versions older than 2.21 are not supported. * Include-vars behavior has changed when running on tags. Previously, the use of job.include-vars in a job that ran on a ref (i.e., tag or branch rather than a change) would use either the override-checkout or the project default branch to source the variables file. In the case that the job is running on a ref of the include-vars project, it will now source the file from the ref checkout itself; this is expected to be more useful and intuitive for users. To restore the previous behavior, the job.include-vars.use-ref may be used.

1 0

Nodepool 11.0.0 released
by James E. Blair 04 Nov '24

04 Nov '24

The Zuul team is pleased to announce the release of Nodepool 11.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/11.0.0/releasenotes.html New Features ============ * Limited support for AWS dedicated hosts is now available using the providers.[aws].pools.labels.dedicated-host option. This allows for a dedicated host to be allocated to serve a single instance (the dedicated host can not be used for multiple instances). This enables Nodepool to launch certain instance types which require dedicated hosts. * The AWS driver now support EC2 Fleet API to launch instances whose types are selected from among a specified set based on certain optimization criteria. * The following new statsd keys are available for builders: nodepool.image.<diskimage name>.image_build_requests nodepool.builder.<hostname>.current_builds nodepool.builder.<hostname>.current_uploads nodepool.builder.<hostname>.build_workers nodepool.builder.<hostname>.upload_workers nodepool.builder.<hostname>.image.<image name>.build.state nodepool.builder.<hostname>.image.<image name>.provider.<provider name>.upload.state * Nodepool-builder may now be configured to delete disk image files after uploads are complete, but while the corresponding images are still in service. The diskimages.delete-after-upload and diskimages.keep-formats configuration options may be used to configure this behavior. * The AWS driver now supports optionally uploading diskimages using the EBS direct APIs. This may be faster and more efficient since it bypasses S3, but it may incur additional costs. * The metastatic driver now supports a providers.[metastatic].pools.labels.max-age attribute to set the maximum age for a backing node. After this age is reached, the backing node will be gracefully removed from service. * The metastatic driver now supports a providers.[metastatic].pools.labels.min-retention-time attribute to set the minimum retention time for a backing node (useful for cloud resources with minimum billing intervals). * Nodepool now exports a new metric called nodepool.provider.<provider>.pool.<pool>.addressable_requests. This gauge shows provider pools, and the number of open node requests which they can address.

1 0

zuul-jobs: Deleting legacy nodejs jobs
by Monty Taylor 20 Sep '24

20 Sep '24

As part of the recent work to fix the broken default version in the nodejs jobs, we'd like to clean up the legacy nodejs jobs which have been deprecated for at least 4 years now. The jobs: build-javascript-tarball build-javascript-content build-javascript-content-tarball nodejs-npm nodejs-npm-run-test nodejs-npm-run-lint nodejs-npm-run-docs While we do not expect anyone is using them, as we don't believe they're functional, out of an abundance of caution we will wait 2 weeks until Thursday, Oct 3 to delete them. The change has already been proposed [0]. You can provide feedback there or in response to this email if necessary. The correct non-deprecated and much much better jobs are: build-javascript-source-tarball build-javascript-deployment js-build nodejs-run-test nodejs-run-lint nodejs-run-docs Thanks! Monty [0] https://review.opendev.org/c/zuul/zuul-jobs/+/921067

1 0

Zuul 11.1.0 released
by James E. Blair 11 Sep '24

11 Sep '24

The Zuul team is pleased to announce the release of Zuul 11.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/11.1.0/releasenotes.html New Features ============ * Untrusted projects may now be allowed (via explicit configuration in the tenant config file) to configure jobs that are run by certain specified other untrusted projects. This allows a “super-project” to configure the jobs run by its “sub-projects”. See tenant.untrusted-projects.<project>.configure-projects for details. * Project stanzas may now include an explicit branch configuration via the project.branches attribute. This enables projects that configure other projects (whether the configuring projects are trusted or untrusted) to better control what jobs run on certain branches. * The tenant configuration may now specify the maximum number of changes that should be enqueued in a pipeline in order to protect Zuul from resource exhaustion. This is not necessary in most circumstances, so the default remains no limit. Deprecation Notes ================= * The status-url attribute of the Github and Pagure reporter is deprecated. Reporters will currently ignore this setting and automatically use the best URL for information about the item instead. Bug Fixes ========= * The zuul_return Ansible plugin will now validate the schema of the data supplied to it; particularly the data for warnings and artifacts. Previously the behavior on incorrectly structured data was undefined and ranged from being silently ignorred to causing exceptions in pipeline processing. Data format errors will now be detected while the job is running and will cause Ansible errors and (if not ignored) job failures.

1 1

Zuul 11.0.1 released
by James E. Blair 19 Jul '24

19 Jul '24

The Zuul team is pleased to announce the release of Zuul 11.0.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/11.0.1/releasenotes.html Bug Fixes ========= * The recent security fix to run cleanup-run playbooks interleaved with post-run playbooks incorrectly caused jobs to retry if a cleanup playbook encountered an unreachable host. This has been corrected and a cleanup playbook will never cause a job to retry if it encounters an unreachable host.

1 0

Zuul 11.0.0 released [security]
by James E. Blair 11 Jul '24

11 Jul '24

The Zuul team announces the release of Zuul 11.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci This release addresses the following security issue: Albin Vass discovered that an attacker could access the nodepool ssh key or potentially access Zuul secrets from trusted playbooks by crafting a cleanup-run playbook designed to access these secrets and triggering its run by causing Zuul to abort the job in a carefully timed manner. The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/11.0.0/releasenotes.html Security Issues =============== * A vulnerability was discovered which could potentially allow jobs with a malicious cleanup-run playbook to access the credentials of their parents or the nodepool ssh key. * To protect against this, cleanup-run playbooks are now executed during the post-run phase of the job, and Zuul will only execute the post-run playbooks corresponding to the inheritance level of previously-executed pre-run playbooks. Upgrade Notes ============= * Existing cleanup-run playbooks are now executed during the post-run phase of the job. This is a change in behavior that will run playbooks in a different order than before. Previously all cleanup-run playbooks would run after all post-run playbooks. Zuul will now run post-run, then cleanup-run playbooks at each level of the inheritance hierarchy (interleaving post-run and cleanup-run playbooks from different levels of the hierarchy). Note that this may have an impact on some jobs. In particular, be on the lookout for jobs with post-run playbooks that assume that other, more-nested, post-run playbooks always run (for example, log collection playbooks). Ensure that log collection happens no more nested than at the job level where logs are created. Also be aware of cleanup-run playbooks that remove artifacts required by post-run playbooks. Since more-nested cleanup-run playbooks can now run before less-nested post-run playbooks, it may be necessary to move cleanup actions to less-nested levels. To facilitate upgrades, Zuul will ignore the result of cleanup-run playbooks when specified using the old syntax, but once migrated to the new syntax using post-run, these playbooks may cause a POST_FAILURE just like any other post-run playbook. Deprecation Notes ================= * The job.cleanup-run attribute is deprecated. Instead, list cleanup playbooks under job.post-run and set the job.post-run.cleanup flag.

1 0

Zuul 10.2.0 released
by James E. Blair 27 Jun '24

27 Jun '24

The Zuul team is pleased to announce the release of Zuul 10.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/10.2.0/releasenotes.html New Features ============ * A new job variable, zuul.build_refs, is available to indicate for which particular refs in a circular dependency queue item a deduplicated job is being run. * The scheduler now accepts the argument --disable-pipelines which will instruct it to discard all pipeline events. This facilitaties the creation of a system with a running Zuul config that does not start any jobs or make any reports. * The gerrit driver now supports a new synthetic trigger, pipeline.trigger.<gerrit source>.approval-change. This is similar to approval, but only triggers when the value of the approval changes instead of simply being present. * gitlab driver now handles the merge request 'close' action. * The GitHub driver now supports excluding locked (read-only) branches with the exclude-locked-branches tenant configuration setting. Deprecation Notes ================= * The replacement fields change, changes, and status_url available in some pipeline reporter configurations are deprecated. Use item_url, which is automatically the best URL for information about the item instead. Bug Fixes ========= * The pseudo-file "/COMMIT_MSG" is not considered at all when determining whether a change matches files or irrelevant-files matchers. It was previously not considered in most cases, but in the case of a negated regular expression in an irrelevant-files stanza, it could still have an effect. This has been corrected and any regexes that explicitly mention "/COMMIT_MSG" may remove it. * The job.files and job.irrelevant-files attributes now fully support the new regular expression syntax, including negation. Other Notes =========== * Zuul now checks the submitWholeTopic setting in Gerrit when it starts. If this setting is changed, or Gerrit is upgraded, all running Zuul schedulers should be restarted in order to see the change.

1 0

Zuul 10.1.0 released
by James E. Blair 24 Apr '24

24 Apr '24

The Zuul team is pleased to announce the release of Zuul 10.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/10.1.0/releasenotes.html New Features ============ * The executor now monitors filesystem inode usage as well as storage usage. The threshold for accepting jobs can be configured independenty with executor.min_avail_inodes. Inode usage is also reported separately with the zuul.executor.<executor>.pct_used_inodes metric. * The topic of a change (if any) is now reported in the zuul.topic job variable. * Zuul now supports substring searches on the Builds and Buildsets pages on its web UI. This is supported by placing asterisk (*) placeholders anywhere in the search term, representing any characters. This is enabled for job name, branch, project, and pipeline filters. * Information about all of the changes (refs) associated with a circular dependency queue item is now available in the zuul.buildset_refs variable. * When a remote host in a job is found to be unreachable, Zuul will automatically add it to a new group named zuul_unreachable for all subsequent playbooks. This can be used to avoid running certain post-run steps on hosts already known to be unreachable. Deprecation Notes ================= * Specifying pipeline.trigger.<github source>.event as a list is deprecated. Update any instances to specify multiple triggers each with a single associated event instead. * The use of requested has been deprecated for some time already. Its use will now produce a configuration syntax warning, and in a future version of Zuul, an error. * The use of pipeline.trigger.<github source>.require-status has been deprecated for some already. Its use will now produce a configuration syntax warning, and in a future version of Zuul, an error. Bug Fixes ========= * Fixes a bug where Zuul would incorrectly enqueue a Gerrit change that could not be submitted due to missing labels in a dependent pipeline. Specifically, if change had a score for any label that would provided by Zuul, all other missing labels would be incorrectly ignored. * Monitoring stats for per-branch queues are now distinct from shared-branch queues. Shared branch queue stats are at: zuul.tenant.<tenant>.pipeline.<pipeline>.queue. Per-branch queue stats are now at: zuul.tenant.<tenant>.pipeline.<pipeline>.queue.<queue>.branch. Prior to this change, per-branch queue stats for one branch queue may have overwritten the stats from another queue resulting in incomplete or incorrect data. * In 10.0.0, supercedent pipelines could raise exceptions when enqueing changes that were part of a dependency cycle (regardless of whether the changes merged or not). This has been corrected, and the documentation has been updated to clarify that circular dependencies are ignored in supercedent pipelines. * The value of zuul.items.project.src_dir did not take into account the job’s workspace scheme and was always constructed using the golang scheme. That has been corrected so that it now reflects the job’s scheme.

1 0

Nodepool 10.0.0 released
by James E. Blair 29 Feb '24

29 Feb '24

The Zuul team is pleased to announce the release of Nodepool 10.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/10.0.0/releasenotes.html New Features ============ * Support for requiring IMDSv2 in AWS is now available using these options: providers.[aws].pools.labels.imdsv2 and providers.[aws].diskimages.imds-support * The metastatic driver now includes the providers.[metastatic].pools.host-key-checking and providers.[metastatic].pools.labels.host-key-checking options. This can be used to verify that a backing node is still functioning correctly before allocating new metastatic nodes to it. * It is now possible to provide an custom pod definition for the Kubernetes and OpenShift drivers using the providers.[kubernetes].pools.labels.spec, providers.[openshift].pools.labels.spec and providers.[openshiftpods].pools.labels.spec attributes. These can be used to supply parameters not otherwise supported by Nodepool, or to create complex pods with multiple containers. Upgrade Notes ============= * The default providers.[aws].diskimages.volume-type for AWS diskimages has been changed from gp2 to gp3. * The hostname-format option, previously available for the OpenStack and AWS drivers, but unused for some time, has been removed. Remove any uses of it from nodepool config files before upgrading; otherwise they will fail config validation.

1 0

Zuul-Client 10.0.0 released
by James E. Blair 23 Feb '24

23 Feb '24

The Zuul team is pleased to announce the release of Zuul-Client 10.0.0. Download it from PyPI: https://pypi.org/project/zuul-client Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul-client/10.0.0/releasenotes.html New Features ============ * Add freeze-job subcommand to display information about jobs as they would be run in a project’s pipeline. * Support for freezing and displaying the job graph has been added via the zuul-client job-graph subcommand. * Zuul-client is now compatible with the data format used by Zuul version 10, as well as older versions.

1 0

Zuul 10.0.0 released
by James E. Blair 22 Feb '24

22 Feb '24

The Zuul team is pleased to announce the release of Zuul 10.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/10.0.0/releasenotes.html Prelude ======= This release includes a significant refactoring of the internal handling of circular dependencies. This requires some changes for consumers of Zuul output (via some reporters or the REST API) and requires special care during upgrades. In the case of a dependency cycle between changes, Zuul pipeline queue items will now represent multiple changes rather than a single change. This allows for more intuitive behavior and information display as well as better handling of job deduplication. New Features ============ * Ansible version 9 is now available. The default Ansible version is still 8, but version 9 may be selected by using job.ansible-version. Upgrade Notes ============= * Support for Ansible version 6 has been removed. Migrate any existing jobs which rely on this version to Ansible version 8 before upgrading. * Zuul can not be upgraded to this version while running. To upgrade: * Stop all Zuul components running the previous version (stopping Nodepool is optional). * On a scheduler machine or image (with the scheduler stopped) and the new version of Zuul, run the command: zuul-admin delete-state –keep-config-cache This will delete all of the pipeline state from ZooKeeper, but it will retain the configuration cache (which contains all of the project configuration from zuul.yaml files). This will speed up the startup process. * Start all Zuul components on the new version. * The MQTT reporter now includes a job_uuid field to correlate retry builds with final builds. Deprecation Notes ================= * The syntax of string substitution in pipeline reporter messages has changed. Since queue items may now represent more than one change, the {change} substitution in messages is deprecated and will be removed in a future version. To maintain backwards compatability, it currently refers to the arbitrary first change in the list of changes for a queue item. Please upgrade your usage to use the new {changes} substitution which is a list. * The syntax of string substitution in SMTP reporter messages has changed. Since queue items may now represent more than one change, the {change} substitution in messages is deprecated and will be removed in a future version. To maintain backwards compatability, it currently refers to the arbitrary first change in the list of changes for a queue item. Please upgrade your usage to use the new {changes} substitution which is a list. * The MQTT and Elasticsearch reporters now include a changes field which is a list of dictionaries representing the changes included in an item. The correspending scalar fields describing what was previously the only change associated with an item remain for backwards compatability and refer to the arbitrary first change is the list of changes for a queue item. These scalar values will be removed in a future version of Zuul. Please upgrade yur usage to use the new changes entries. * The zuul.bundle_id variable is deprecated and will be removed in a future version. For backwards compatability, it currently duplicates the item uuid.

1 0

Zuul 9.5.0 released
by James E. Blair 13 Feb '24

13 Feb '24

The Zuul team is announces the release of Zuul 9.5.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.5.0/releasenotes.html Upgrade Notes ============= * Ansible versions 6 is now deprecated in Zuul since it is unmaintaned, and it will be removed from a future version of Zuul. Ansible 8 is now the default version in Zuul.

1 0

Zuul 9.4.0 released
by James E. Blair 08 Feb '24

08 Feb '24

The Zuul team is announces the release of Zuul 9.4.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.4.0/releasenotes.html New Features ============ * The Gerrit driver now supports the hashtags-changed event as a trigger as well as using hashtags as trigger or pipeline requirements. Bug Fixes ========= * The MQTT reporter now sets the log_url for retried builds correctly. The link to the build result page is reported in a dedicated web_url field.

1 0

Nodepool 9.1.0 released
by James E. Blair 14 Dec '23

14 Dec '23

The Zuul team is pleased to announce the release of Zuul 9.1.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/9.1.0/releasenotes.html New Features ============ * The AWS driver now supports an providers.[aws].image-import-timeout option to control automatic retries and timeouts when AWS import task resource limits are reached. * The AWS driver now supports volume quota. It will automatically register the limits from the cloud and ensure that labels that specify EBS volume attributes stay under the limit. * The Azure driver now supports specifying the size of the OS disk. * The Kubernetes and OpenShift drivers now support adding dynamic metadata, i.e. Pod and Namespace labels, with information about the corresponding node request. This is analogous to the existing dynamic tags of the OpenStack, AWS, and Azure drivers. See providers.[kubernetes].pools.labels.dynamic-labels, providers.[openshift].pools.labels.dynamic-labels, and providers.[openshiftpods].pools.labels.dynamic-labels for details. * The Azure driver now support specifying community and shared gallery images. * The metastatic driver will now automatically use the node-attributes from backing nodes as default values for node-attributes of metastatic nodes. Any node-attribute values specified in the metastatic pool config will override those from the backing node. * An upload timeout can be configured on OpenStack providers for use in the case that image uploads to a provider take longer than the default of one hour. Upgrade Notes ============= * The Azure driver now uses the "Standard" SKU for all public IP addresses. Previously it would chose either "Standard" or "Basic" depending on the selection of IPv4 or IPv6 addresses. Pricing for public IP addresses may differ between the SKU levels. Standard IP addresses block all incoming traffic by default, therefore the use of a Network Security Group is required in order to allow incoming traffic. If you are not currently using a Network Security Group, then before upgrading Nodepool it is recommended to create one, add any required rules, and attach it to the subnet that Nodepool uses. * The Azure driver no longer creates and deletes Disks, Network Interfaces, or Public IP Addresses as separate steps. Nodepool and user-supplied tags will no longer be applied to Network Interfaces, or Public IP Addresses. This also limits Nodepool's ability to detect leaks of these resources (however this is unlikely since Azure is now responsible for deleting them).

1 0

Zuul 9.3.0 released
by James E. Blair 07 Dec '23

07 Dec '23

The Zuul team is announces the release of Zuul 9.3.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.3.0/releasenotes.html New Features ============ * Monitoring stats for builds are now emitted for more results. Previously they were emitted for some build results such as SUCCESS, FAILURE, and ERROR but now include NODE_FAILURE and other result values. * Support for using Google Cloud Pub/Sub as an event source has been added to the Gerrit driver. Upgrade Notes ============= * A significant SQL schema migration is included which requires extra care when upgrading. It is recommended to make a database backup before upgrading in case of problems. It is also recommended to test the migration with a copy of the database in a development environment first in order to ascertain expected runtime and detect any data integrity problems early. Zuul is unable to operate as normal during the schema upgrade. The following procedure is recommended: * Perform a database backup * Stop all Zuul components * Start one scheduler and wait for it to complete the migration and initialization * Start the rest of Zuul If the migration fails and the backing database is PostgreSQL, the migration will be rolled back and Zuul may be restarted on the previous version. If the backing database is MySQL, if the error happens early enough the migration may be rolled back (look for "Early error in schema migration, rolling back" in scheduler logs). If an error happens late in the migration, manual intervention may be required. * Zuul's GitHub driver has updated the default merge mode to better align with GitHub's default behavior. This change requires Git 2.33.0 or newer unless you explicitly override Zuul's default merge mode for a project. Additionally, if you aren't overriding the merge mode then the merge algorithm chosen may change depending on your Git and GitHub versions. This is expected to provide more deterministic results. * Input data from a parent job (return data or artifacts) no longer influences whether a child job can be deduplicated or not. Bug Fixes ========= * Correct the behavior of the timer driver to not trigger for branches configured as always-dynamic.

1 0

Zuul 9.2.0 released
by James E. Blair 26 Sep '23

26 Sep '23

The Zuul team is announces the release of Zuul 9.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.2.0/releasenotes.html New Features ============ * Gerrit and GitHub repositories will now automatically use the default branch as specified in their respective systems as the project’s default branch. This may still be overridden in zuul using the project.default-branch setting. * A new job configuration option, job.failure-output, has been added to allow users to specify regular expressions which, when seen in the streaming job output, will trigger the early failure detection in Zuul so that dependent pipeline gate resets may begin before the completion of a failed job. * Two new settings are available to protect Zuul from resource exhaustion from processing too many dependencies among changes. The Gerrit driver supports setting <gerrit connection>.max_dependencies to limit internal dependency processing during event processing, and a new tenant setting of tenant.max-dependencies is available to limit tenant processing while enqueing changes in pipelines. * Additional pipeline stats for individual project queues are available at zuul.tenant.<tenant>.pipeline.<pipeline>.queue. * Configuration options which accept regular expressions have been updated to accept a new syntax which allows specifying that the regular expression should be treated as a negative match. * An upper limit on the dependent pipeline actionable window can now be specified using the pipeline.window-ceiling setting. The default of no upper limit remains. Upgrade Notes ============= * If using the GitHub driver in app mode, you should consider also generating and adding an api_token. The option is not strictly required but will increase rate limits and add functionality for projects the app is not installed in. Deprecation Notes ================= * Regular expressions using Python syntax are deprecated in favor of RE2 syntax instead, for additional speed and safety. Negative lookahead assertions may be replaced using the negate keyword. See Regular Expressions for more information. If Zuul detects a regular expression using Python syntax that is not supported by RE2, it will register a configuration syntax warning. These may be viewed on the configuration errors page in the web interface. A future version of Zuul will remove Python regex support completely and these warnings will become errors.

1 0

[zuul-jobs] Dropping support for ensure-cabal and ensure-haskell-stack
by Tristan Cacqueray 23 Aug '23

23 Aug '23

Dear Zuulers, Since ghcup is the new recommended installer for the Haskell toolchain, I propose to remove the legacy roles in this change: https://review.opendev.org/c/zuul/zuul-jobs/+/847111 This only affects external playbooks using the ensure-cabal and ensure-haskell-stack roles directly. These tasks are now performed by the ensure-ghc role. The change is scheduled to land in two weeks from now, on 2023-09-05. With regards, -Tristan

1 0

Deleting the zuul-jobs nox-py27 job on September 5, 2023
by Clark Boylan 23 Aug '23

23 Aug '23

We discovered recently that the zuul-jobs nox-py27 job stopped working because nox depends on virtualenv and new virtualenv releases don't support python2.7 virtualenvs. A workaround for this was merged, but it brought up the question of whether or not we need the job in the first place. Python 2.7 is quickly on its way out now. Debian Bookworm does not include any Python 2.7 packages. We could only find one project using this job in OpenDev's Zuul instance. This led to the realization that we probably don't need this job if it isn't well used and has been broken for some time. As a result we will delete the nox-py27 job entirely on September 5, 2023. This gives everyone two weeks to migrate away from it. The change has already been proposed [0]. You can provide feedback there or in response to this email if necessary. [0] https://review.opendev.org/c/zuul/zuul-jobs/+/892408 Clark

1 0

Zuul 9.1.0 released
by James E. Blair 22 Aug '23

22 Aug '23

The Zuul team is announces the release of Zuul 9.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.1.0/releasenotes.html New Features ============ * A new job attribute, ansible-split-streams has been added to instruct Zuul to keep stdout and stderr separate when running Ansible command tasks. Deprecation Notes ================= * Zuul is changing the behavior of its customized Ansible command module to match the standard Ansible behavior. The previous behavior and current default is to combine stdout and stderr streams. In a later Zuul release, the ansible-split-streams default value will change from false to true in order to match the standard Ansible behavior and keep the streams separate. A further Zuul release will remove the option altogether. Bug Fixes ========= * Ansible task failures in block/rescue stanzas could cause Zuul to erroneously trigger early-failure behavior for the build, which could result in inconsistent behavior in a dependent pipeline. Task failures in Ansible blocks are no longer considered for early failure detection, and if a build encounters an early failure, it will cause the build result to be reported as a failure in all cases.

1 0

Nodepool 9.0.0 released
by James E. Blair 17 Aug '23

17 Aug '23

The Zuul team is pleased to announce the release of Zuul 9.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/9.0.0/releasenotes.html New Features ============ * The AWS driver now supports importing images using either the "image" or "snapshot" import methods. The "snapshot" method is the current behavior and remains the default and is the fastest and most efficient in most circumstances. The "image" method is available for images which require certain AWS licensing metadata that can only be added via that method. * The AWS driver now supports launching Amazon EC2 Spot instances (https://aws.amazon.com/ec2/spot/) when specifying providers.[aws].pools.labels.use-spot. * Many more drivers now report metrics for leaked resources. * The OpenStack driver now supports volume quota. It will automatically register the limits from the cloud and ensure that labels that utilize boot-from-volume stay under the limit. Limits can also be specified at the pool and tenant level in Nodepool’s configuration. * Add support for requesting gpu resources in kubernetes and openshift drivers. * Added support for specifying Kubernetes and OpenShift pod resource limits separately from requests. * Added support for specifying the scheduler name, additional metadata, and volume mounts in Kubernetes and OpenShift drivers. Upgrade Notes ============= * New diskimage build ids are now in UUID format instead of integer sequences with leading zeroes. This facilitates faster restoration of images from backup data on systems with large numbers of image builds. Existing builds will continue to use the integer format and can coexist with the new format. * Python 3.11 is now the only version of Python with which Nodepool is tested. * Nodepool now requires ZooKeeper version 3.6.0 or later (note that at the time of this writing, the oldest supported ZooKeeper version is later than that).

1 0

Zuul 9.0.0 released
by James E. Blair 17 Aug '23

17 Aug '23

The Zuul team is pleased to announce the release of Zuul 9.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/organization/zuul-ci The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/9.0.0/releasenotes.html New Features ============ * Ansible version 8 is now available. The default Ansible version is still 6, but version 8 may be selected by using job.ansible-version. * Support for using AWS Kinesis as an event source has been added to the Gerrit driver. * A new job variable, zuul.commit_id has been added. This provides a stable identifier for all types of pipeline queue items (changes, refs, tags, or branches) from any driver to identify the commit which triggered the queue item. This can be used to interface with external systems which identify changes by commit. * Zuul will now begin the process of aborting following jobs and re-ordering pipelines immediately after the first Ansible task fails. * Support for using Kafka as an event source has been added to the Gerrit driver. * A new tenant project configuration option, implied-branch-matchers has been added. This is useful in conjunction with the branch exclusion options which may cause Zuul to change behavior as the set of load branches switches between one and more than one. The operator can fix the appropriate behavior in the tenant config file rather than relying on the heuristic in these cases. * A new Zuul job variable zuul_will_retry available in post and cleanup playbooks, which indicates whether the current job will be retried. Upgrade Notes ============= * Python 3.11 is now the only version of Python with which Zuul is tested. Bug Fixes ========= * Duplicate queue configuration items are now permitted on multiple branches of the same project as long as their configuration is identical. Similar to secrets and semaphores, this can help avoid spurious configuration errors when branching a project with in-repo configuration.

1 0

[zuul-jobs] Dropping support for Helm v2 and fixing v3
by Radosław Piliszek 19 Jul '23

19 Jul '23

Dear Zuulers, Since Helm v2 is long gone in practice, I have proposed to drop its support in [1]. This affects the helm-template role and fixes its issue with Helm v3 not setting the release name properly. This change is scheduled to land in two weeks from now, on August 2nd. [1] https://review.opendev.org/c/zuul/zuul-jobs/+/885987 Kind regards, Radosław Piliszek -yoctozepto

1 0

Zuul 8.3.1 released [security]
by James E. Blair 18 May '23

18 May '23

The Zuul team is pleased to announce the release of Zuul 8.3.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/repository/zuul-ci/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.3.1/releasenotes.html Security Issues =============== * Zuul will execute bwrap with –disable-userns set if two conditions hold. 1) The version of bwrap is 0.8.0 or newer and 2) User namespaces are enabled in the zuul-executor runtime context. Doing so will prevent the zuul-executor bwrap runtimes from creating additional user namespaces which fortifies Zuul's security position.

1 0

Zuul 8.3.0 released
by James E. Blair 17 May '23

17 May '23

The Zuul team is pleased to announce the release of Zuul 8.3.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://quay.io/repository/zuul-ci/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.3.0/releasenotes.html New Features ============ * Added a new dark theme for the Zuul web interface. * Added theme selection for the Zuul web interface. The default theme is set to Auto which means your system/browsers preference determines if the Light or Dark theme should be used. Either can be explicitly set in the settings for the web interface by clicking the cogs in the top right. * Gerrit pipeline triggers now support embedded require and reject filters in order to match. Any conditions set for the pipeline in require or reject filters may also be set for event trigger filters. This can be used to construct pipelines which trigger based on certain events but only if certain other conditions are met. It is distinct from pipeline requirements in that it only affects items that are directly enqueued whereas pipeline requirements affect dependencies as well. * All Gerrit "requires" filters are now available as "reject" filters as well. * GitHub pipeline triggers now support embedded require and reject filters in order to match. Any conditions set for the pipeline in require or reject filters may also be set for event trigger filters. This can be used to construct pipelines which trigger based on certain events but only if certain other conditions are met. It is distinct from pipeline requirements in that it only affects items that are directly enqueued whereas pipeline requirements affect dependencies as well. * The nodepool.slot variable has been added to host vars. This is supplied by the nodepool static and metastatic drivers starting with version 8.0.0. It may be used to avoid build directory collisions on nodes that run more than one job. Deprecation Notes ================= * The Elasticsearch reporter now filters zuul data from the job returned vars. The job returned vars under the zuul key are meant for Zuul and may include large amount of data such as file comments. * The require-approval and reject-approval Gerrit trigger attributes are deprecated. Use pipeline.trigger.<gerrit source>.require and pipeline.trigger.<gerrit source>.reject instead. * The require-status GitHub trigger attribute is deprecated. Use pipeline.trigger.<github source>.require instead. Bug Fixes ========= * The zuul-admin prune-database command did not completely delete expected data from the database. It may not have deleted all of the buildsets older than the specified cutoff time, and it may have left orphaned data in ancillary tables. This has been corrected and it should now work as expected. Additionally, a –batch-size argument has been added so that it may delete data in multiple transactions which can facilitate smoother operation when run while Zuul is operational. Users who have previously run the command may need to manually delete rows from the zuul_build, zuul_build_event, zuul_artifact, and zuul_provides tables which do not have corresponding entries in the zuul_buildset table. * The cherry-pick merge mode will now silently skip commits that have already been applied to the tree when cherry-picking, instead of failing with an error. The exception to this is if the source of the cherry-pick is an empty commit, in which case it is always kept. Skipping commits that have already been applied is important in a pipeline triggered by the Gerrit change-merged event (like the deploy pipeline), since the scheduler would previously try to cherry-pick the change on top of the commit that just merged and fail. * Zuul will now attempt to honor Gerrit "submit requirements" when determining whether to enqueue a change into a dependent (i.e., "gate") pipeline. Zuul previously honored only Gerrit's older "submit records" feature. The new checks will avoid enqueing changes in "gate" pipelines in the cases where Zuul can unambiguously determine that there is no possibility of merging, but some non-mergable changes may still be enqueued if Zuul can not be certain whether a rule should apply or be disregarded (in these cases, Gerrit will fail to merge the change and Zuul will report the buildset as a MERGE_FAILURE). This requires Gerrit version 3.5.0 or later, and Zuul to be configured with HTTP access for Gerrit.

1 0

New container image location
by James E. Blair 27 Apr '23

27 Apr '23

Hi, The Zuul project has changed publishing locations for the container images it builds. The new location is "quay.io/zuul-ci". So if you were previously specifying, for example, the zuul-scheduler image as: zuul/zuul-scheduler:latest or docker.io/zuul/zuul-scheduler:latest You should instead use: quay.io/zuul-ci/zuul-scheduler:latest Do note that both the host and the organization part of the reference has changed. It is unfortunate that we were unable to use our preferred organization name, but it was not available. We are not changing the name of the project; it's still Zuul. All of the existing release tags have been copied over, so if you are running a release, you may switch that now as well. We will no longer be publishing any new images to Docker Hub. Existing images will remain in place until May 25, at which point they will be deleted. Please update your references as soon as possible, and certainly before that date. Thanks, James Blair on behalf of the Zuul team

1 0

Zuul 8.2.0 released
by James E. Blair 22 Feb '23

22 Feb '23

The Zuul team is pleased to announce the release of Zuul 8.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.2.0/releasenotes.html New Features ============ * The change message (commit message, or pull request message depending on the driver) is now available in plain text form annotated with the Ansible !unsafe tag as zuul.change_message. Upgrade Notes ============= * Fixes tenant.untrusted-projects.<project>.include-branches being lower priority than tenant.untrusted-projects.<project>.exclude-branches to match the documentation and expected user behavior. This only affects projects that are using both include-branches and exclude-branches at the same time. Now, include-branches has priority over exclude-branches for any branches that match both. Practically speaking, this means that exclude-branches is ignored if include-branches is set. Deprecation Notes ================= * The base64 encoded version of the change message available as zuul.message is deprecated and will be removed in a future version of Zuul. Use zuul.change_message instead.

1 0

Nodepool 8.2.0 released
by James E. Blair 22 Feb '23

22 Feb '23

The Zuul team is pleased to announce the release of Nodepool 8.2.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/8.2.0/releasenotes.html New Features ============ * Added support for privileged pods in Kubernetes and OpenShift drivers via providers.[kubernetes].pools.labels.privileged, providers.[openshift].pools.labels.privileged, and providers.[openshiftpods].pools.labels.privileged.

1 0

Change to default value for enable_fips parameter in enable-fips playbook in zuul-jobs
by Ade Lee 23 Jan '23

23 Jan '23

Hi all, This is just an announcement that the enable_fips parameter in https://opendev.org/zuul/zuul-jobs/src/branch/master/playbooks/enable-fips/… is being changed from being True by default to being False by default. Any jobs that use this playbook (and wish to enable fips) will need to explicitly set enable_fips to True. The reason for this change is as follows: Turing FIPS on requires a reboot. For most consistent results, we want this to occur early in the node setup so that subsequent setup is not affected by a reboot. To do this, we need to run the playbook as part of a base job. Defaulting enable_fips to false allows us to add this playbook to most jobs. Descendant jobs can then simply enable FIPS mode by simply setting enable_fips mode to True. More details can be found in the change: https://review.opendev.org/c/zuul/zuul-jobs/+/866881 Thanks, Ade Lee

1 0

Nodepool 8.1.0 released
by James E. Blair 19 Jan '23

19 Jan '23

The Zuul team is pleased to announce the release of Nodepool 8.1.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/8.1.0/releasenotes.html While not addressed directly by Nodepool, note that container images starting with this release include a version of git patched to address CVE-2022-23521. Upgrade Notes ============= * In the OpenStack driver, when using `min-ram` in combination with a `flavor-name`, the first flavor to be found that satisfied the `min-ram` requirements and contained the substring `flavor-name` would be used. The order of the flavors to be searched was dependent on what the cloud returned. From this release, the available flavors are now alphabetically sorted before matching `flavor-name`.

1 0

Zuul 8.1.0 released [security]
by James E. Blair 19 Jan '23

19 Jan '23

The Zuul team is pleased to announce the release of Zuul 8.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.1.0/releasenotes.html Benjamin Schanzel discovered that pipeline requirements were not checked for dependent changes in a post-review independent pipeline. If an independent pipeline was configured so that it only runs jobs for reviewed code, dependent changes could be used to bypass that requirement and cause unreviewed code to be included in jobs. This vulnerability has been present since Zuul version 3.0.0. Zuul will now check pipeline requirements for all changes (including dependent changes that are only enqueued for the purpose of providing speculative git repo states). While not addressed directly by Zuul, note that container images starting with this release include a version of git patched to address CVE-2022-23521. Security Issues =============== * Non-live items are now subject to pipeline requirements for independent pipelines. Previously, an optimization for independent pipelines skipped checking that a change met the pipeline requirements. If an independent pipeline is intended only to run reviewed code, this could allow running unreviewed code by updating dependent changes. Now both non-live and live items are subject to pipeline requirements in all pipeline managers. The new allow-other-connections pipeline configuration option may now be used to ensure that only changes from connections which are mentioned in the pipeline configuration (such as triggers, reporters, or pipeline requirements) are enqueued. This allows the construction of a pipeline where, for example, code review requirements are strictly enforced, even for dependencies which are not normally directly enqueued. New Features ============ * A new pipeline attribute, pipeline.allow-other-connections, has been added to ensure that only changes from connections which are mentioned in the pipeline configuration (such as triggers, reporters, or pipeline requirements) are enqueued. * Zuul now stores the pause and resume events for a build together with their timestamp and reports them to the SQL database and via MQTT. * Individual playbooks may now be wrapped by a semaphore. Zuul will start the job and proceed up to the point of a playbook which requires a semaphore and then wait until it is able to aquire the semaphore before proceeding. It releases the semaphore after the end of that individual playbook. The same semaphore may be used for both jobs and playbooks, but an individual job may not use the same semaphore for both purposes. Upgrade Notes ============= * Additional syntax checking is performed on job configuration changes so that Zuul will report an error if a job inherits from a non-permitted parent (such as a final job, intermediate job when the child is not abstract, or a protected job in another project). Previously, these situations might only be discovered when freezing a job graph. If any jobs currently in this situation exist, they will be reported as configuration errors.

1 0

Nodepool 8.0.1 released
by James E. Blair 12 Jan '23

12 Jan '23

The Zuul team is pleased to announce the release of Nodepool 8.0.1. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/8.0.1/releasenotes.html New Features ============ * The nodepool command now includes a “hold” subcommand. Bug Fixes ========= * The Kubernetes driver now supports version 1.24+

1 0

ensure-tox and tox v4
by James E. Blair 08 Dec '22

08 Dec '22

The tox project has released a major version upgrade (v4) with some backwards incompatible changes[0]. This has caused some problems for the ensure-tox role (and the many tox-related jobs that use it) in zuul-jobs. Clark Boylan was able to address several of the compatibility issues that affected how tox is used internally in the role[1] (e.g., siblings installation and log copying). But it became apparent that some of the breaking changes are related to the configuration of projects that use tox. For example, the "whitelist_externals" option has been renamed and using the old option produces an error. The ensure-tox role has, to date, not had a method for specifying what version of tox to install, instead, always installing the latest. To enable projects to upgrade their support to tox v4 at their own pace, we are adding a new role variable, "ensure_tox_version" which can be used to specify what tox version the ensure-tox role should install. This variable has been added, and it currently defaults to "<4" so that the latest tox 3.x version is installed. On December 22, we will update the default to be "" (the empty string) which will once again mean that ensure-tox will install the latest version of tox[2]. We recommend that you use this time to update your tox configuration to work with tox v4. If you are unable to do so before December 21, you can set the "ensure_tox_version" variable to "<4" on your jobs. [0] https://discuss.python.org/t/tool-tox-4-on-the-horizon/6537 [1] https://review.opendev.org/866935 [2] https://review.opendev.org/866943

1 0

Potential log upload credential exposure in zuul-jobs
by James E. Blair 17 Nov '22

17 Nov '22

Clark Boylan recently discovered that some of the log upload roles in the zuul-jobs repo were missing `no_log: true` settings and can write credentials to the job-output.json file. These are written to the file after the logs have been uploaded, and so are not included in the uploaded logs. This means that accidental exposure in typical usage is unlikely. A malicious user could write a cleanup-run playbook to read in the contents of the file and send the credentials off-host. The zuul-jobs repo has been updated to correct this problem, so if you include it in your tenant configuration directly, it will automatically be updated. If you use a local mirror of zuul-jobs, you should update it immediately. The update to zuul-jobs is this commit: https://opendev.org/zuul/zuul-jobs/commit/622baa65bfa02a592024a867c5bd94a8a…

1 0

Zuul 8.0.1 released
by James E. Blair 04 Nov '22

04 Nov '22

The Zuul team is pleased to announce the release of Zuul 8.0.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.0.1/releasenotes.html Bug Fixes ========= * An error which prevented results from appearing in the web UI on the "Buildsets" tab has been corrected.

1 0

Zuul 8.0.0 released
by James E. Blair 02 Nov '22

02 Nov '22

The Zuul team is pleased to announce the release of Zuul 8.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/8.0.0/releasenotes.html New Features ============ * Read-level access to tenants or the tenant list may now be restricted to authorized users using the tenant.access-rules and api-root.access-rules attributes. * The rebase merge-mode is now supported for GitHub. Upgrade Notes ============= * Support for Ansible version 5 has been removed. Migrate any existing jobs which rely on this version to Ansible version 6 before upgrading. * In preparation for expanded access control features in the web interface, and REST API, the admin-rule tenant configuration object has been renamed to authorization-rule. When applied to a tenant, the tenant attribute is still admin-rules since it determines admin access to that tenant. This change will allow similar rules to be applied to non-admin level access in the future. Tenant configs should now follow this example: - authorization-rule: name: example-rule - tenant: name: example-tenant admin-rules: - example-rule The old form is still permitted for backwards compatability, but will be removed in a later version of Zuul. * The default merge mode used by Zuul for preparing git repos was previously merge-resolve in all cases, but is now merge for all drivers except Gerrit, where merge-resolve is still the default. This makes the merge operations performed by Zuul more closely match the operations that will be performed by the code review systems. Bug Fixes ========= * The use of implied branch matchers in jobs where override-checkout was specified could cause some jobs to include unintended variants. Specifically: job variants with implied branch matchers on branches which are substring matches of a branch specified in the override-checkout job attribute may have been used when not intended. * This has been corrected so that the same job variant matching process happens whether the change’s branch or the branch specified by override-checkout is used.

1 0

Nodepool 8.0.0 released
by James E. Blair 02 Nov '22

02 Nov '22

The Zuul team is pleased to announce the release of Nodepool 8.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/8.0.0/releasenotes.html New Features ============ * The AWS driver now support specifying volume IOPS and throughput; see: providers.[aws].pools.labels.iops, providers.[aws].pools.labels.throughput, providers.[aws].diskimages.iops, and providers.[aws].diskimages.throughput. * The OpenStack, AWS, and Azure drivers now support adding tags (AKA metadata, AKA properties) to instances with dynamic data about the corresponding node request using string formatting. See providers.[openstack].pools.labels.instance-properties, providers.[aws].pools.labels.dynamic-tags, and providers.[azure].pools.labels.dynamic-tags for details. * The AWS and Azure drivers now support adding tags to images via the providers.[aws].diskimages.tags, and providers.[azure].diskimages.tags attributes. The OpenStack driver already supported similar behavior with its meta attribute. The diskimages.metadata attribute has been added and will supply default values to the provider image tag attributes mentioned above. * A new configuration option for K8s Pod type labels was added to limit the amount of ephemeral storage allocatable in a container (cf. K8s Local ephemeral storage resource documentation <https://kubernetes.io/docs/concepts/configuration/manage-resources-containe…>) This limit can be set via the integer value of providers.[kubernetes].pools.labels.storage and is treated as Megabytes. Also, a pool-scoped default value can be specified via providers.[kubernetes].pools.default-label-storage

1 0

Zuul and OpenSSL 3.x Security Vulnerabilities
by Clark Boylan 01 Nov '22

01 Nov '22

OpenSSL disclosed two security vulnerabilities [0] affecting OpenSSL 3.0 and newer. Prior to disclosure these issues were rated critical but on further analysis were degraded to high. We believe the impact to Zuul is minimal, but are sending this email as this situation has received a lot of attention. Zuul users should be informed about the inner workings of the software they run. Zuul does use OpenSSL for a number of functions. In particular, cert based client auth is performed on the Zookeeper connection (and potentially with OpenTelemetry if enabled). This means that Zuul installations connecting to a malicious Zookeeper server (or OpenTelemetry endpoint) could potentially be exploited. The official Zuul Docker images are based on Debian Bullseye which does not include OpenSSL 3.x and should not be affected. Users who have installed on Ubuntu Jammy or RHEL 9 (or any of its derivatives) should update their OpenSSL packages to avoid any problems. The Ubuntu page for this CVE [1] notes that stack protections should limit the impact of this bug on Ubuntu to denials of service further reducing the potential impact to Zuul. [0] https://www.openssl.org/news/secadv/20221101.txt [1] https://ubuntu.com/security/CVE-2022-3602

1 0

Zuul 7.1.0 released
by James E. Blair 19 Oct '22

19 Oct '22

The Zuul team is pleased to announce the release of Zuul 7.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/7.1.0/releasenotes.html Upgrade Notes ============= * Ansible versions 5 is now deprecated in Zuul since it is unmaintaned, and it will be removed from a future version of Zuul. Ansible 6 is now the default version in Zuul. New Features ============ * The GitHub driver now supports specifying the draft status of a PR as a pipeline requirement. See pipeline.require.<github source>.draft. * Details about the configuration and current usage of semaphores are now available in the web UI under the "Semaphores" tab.

1 0

Zuul 7.0.0 released
by James E. Blair 05 Oct '22

05 Oct '22

The Zuul team is pleased to announce the release of Zuul 7.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/7.0.0/releasenotes.html New Features ============ * The MQTT driver now supports including data returned from a job in its reports. See pipeline.<reporter>.<mqtt>.include-returned-data. * Nodesets may now express an ordered list of alternatives so that if Nodepool is unable to fulfill a request for certain labels, one or more alternative Nodesets may be attempted instead. See nodeset.alternatives for details. Upgrade Notes ============= * Support for Ansible versions 2.8 and 2.9 has been removed. Migrate any existing jobs which rely on these versions to Ansible version 5 or 6 before upgrading. * The previously deprecated merge-failure and merge-failure-message pipeline configuration options have been removed. Use merge-conflict and merge-conflict-message respectively instead. * The deprecated syntax of specifying project change queues on pipeline configurations has been removed. Specify queues using the project stanza now. See queue for more information.

1 0

Zuul 6.4.0 released
by James E. Blair 16 Sep '22

16 Sep '22

The Zuul team is pleased to announce the release of Zuul 6.4.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/6.4.0/releasenotes.html New Features ============ * Ansible version 6 is now available. The default Ansible version is still 5, but version 6 may be selected by using job.ansible-version. Upgrade Notes ============= * Ansible versions 2.8 and 2.9 are now deprecated in Zuul since they are both unmaintaned. Ansible 5 is now the default version in Zuul.

1 0

Zuul 6.3.0 released
by James E. Blair 06 Sep '22

06 Sep '22

The Zuul team is pleased to announce the release of Zuul 6.3.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/6.3.0/releasenotes.html New Features ============ * A new pipeline.config-error pipeline reporter is available for customizing reporter actions related to Zuul configuration errors.

1 0

Nodepool 6.2.0 released
by James E. Blair 05 Aug '22

05 Aug '22

The Zuul team is pleased to announce the release of Nodepool 6.2.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/6.2.0/releasenotes.html New Features ============ * The AWS driver now supports multiple quotas for specific instance types. This support is automatic, but also includes corresponding enhancements to provider, pool, and tenant limits configured in Nodepool. * A new statsd metric, nodepool.image_build_requests is available. It reports the number of outstanding manual image build requests. * A new image-status command and accompanying web endpoint are available to easily see what images have been paused via the image-pause command and have pending manual build requests via the build-image command. Bug Fixes ========= * The AWS, Azure, and IBMVPC drivers now check provider quota before accepting requests. This allows them to decline requests which can not possibly be satisfied given provider quota constraints.

1 0

Zuul 6.2.0 released
by James E. Blair 05 Aug '22

05 Aug '22

The Zuul team is pleased to announce the release of Zuul 6.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/6.2.0/releasenotes.html New Features ============ * Added build set url to Zuul comment. This provide a quick way for user to reach the build set page from Gerrit. * Added a new tenant.untrusted-projects.<project>.always-dynamic-branches tenant project configuration option. This may be used to specify branches from which Zuul should never load static configuration and instead treat every change as if it newly proposed dynamic configuration. This is potentially useful for large numbers of rarely-used feature branches. * The new ssh_server option for gerrit connections may be used to specify a hostname to use for SSH connections while the normal server option specifies the hostname to use for HTTP connections. * Support for global (cross-tenant) semaphores has been added. See Global Semaphore. * Pipeline timer triggers with jitter now apply the jitter to each project-branch individually (instead of to the pipeline as a whole). This can reduce the thundering herd effect on external systems for periodic pipelines with many similar jobs. * The scheduler now accepts an argument --wait-for-init which will cause it to wait until all tenants have been initialized before it begins processing pipelines. This may help large systems with excess scheduler capacity perform a rolling restart of schedulers more quickly. Bug Fixes ========= * Zuul now treats job dependencies that reference undefined jobs as a configuration error. Previously a job which depended on another job which did not exist would pass initial syntax validation and only cause a failure in freezing the job graph when Zuul attempted to run the job. Now incorrect or missing job dependencies are detected during configuration. This means that new config errors may be prevented from merging. It also means that existing erroneous job or project configurations will be regarded as configuration errors at startup.

1 0

[zuul-jobs] ensure-pip: removing venv package install via ensure_pip_from_upstream_interpreters
by Ian Wienand 22 Jun '22

22 Jun '22

Hello, We plan to merge [1] to the ensure-pip role on or about 6th July, 2022. Keying venv package installs from ensure_pip_from_upstream_interpreters is not idiomatic to the role and we wish to remove this behaviour to prevent further confusion. Its use has been limited and this is unlikley to affect any current users, but if you have concerns please reach out so we can find a better way to ensure this role is working for you. Thank you, -i [1] https://review.opendev.org/c/zuul/zuul-jobs/+/846248

1 0

Zuul 6.1.0 released
by James E. Blair 22 Jun '22

22 Jun '22

The Zuul team is pleased to announce the release of Zuul 6.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/6.1.0/releasenotes.html New Features ============ * The cherry-pick merger mode can now handle merges between branches by performing a git merge instead of git cherry-pick if the change has multiple parents. Previously, this would fail because git doesn’t allow a merge to be cherry-picked. * If identical jobs are run for multiple changes in a dependency cycle, Zuul may now deduplicate them under certain circumstances. See job.deduplicate for details. * Added new tenant project configuration options tenant.untrusted-projects.<project>.include-branches and tenant.untrusted-projects.<project>.exclude-branches. Similar to exclude-unprotected-branches, these may be used to reduce the set of branches from which Zuul will load configuration. * A new command, prune-database has been added to zuul-admin in order to remove database entries older than a certain age. * The GitHub driver now has support for using a GitHub Enterprise repository cache. See <github connection>.repo_cache for information on how to configure it. Upgrade Notes ============= * A SQL schema migration is added in order to add an updated column to the buildset table which will be used by the prune-database command to determine which buildsets to delete. The migration will attempt to use the most recent timestamp associated with the buildset when initializing this column, however, if no timestamps are available, it will initialize it to 1970-01-01. Since this is considerably before Zuul’s birthdate, this means that any buildsets without timestamp information will be pruned the first time prune-database is run. It is expected that buildsets with no timestamps, even very recent ones, are typically uninteresting and therefore this should not cause a hardship. If this is not the case for your installation, you may want to inspect the database and change the updated column to a more recent value in these cases. Deprecation Notes ================= * The zuul CLI is renamed zuul-admin. The zuul command will remain usable until a future version, then will be phased out. Likewise, tenant-scoped, workflow affecting commands such as autohold, enqueue, dequeue, promote are deprecated from the zuul-admin CLI and will be phased out in a future version. They can still be performed via the zuul-client CLI.

1 0

Nodepool 6.1.0 released
by James E. Blair 22 Jun '22

22 Jun '22

The Zuul team is pleased to announce the release of Nodepool 6.1.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/6.1.0/releasenotes.html New Features ============ * Config options for kubernetes providers were added to define default limits for cpu and memory for pod-type labels. * providers.[kubernetes].pools.default-label-cpu * providers.[kubernetes].pools.default-label-memory These values will apply to all pod-type labels within the same pool that do not override these limits. This allows to enforce resource limits on pod labels. It thereby enables to account for pool and tenant quotas in terms of cpu and memory consumption. New config options for kubernetes pools therefore also include * providers.[kubernetes].pools.max-cores * providers.[kubernetes].pools.max-ram The exsisting tenant quota settings apply accordingly. Note that cpu and memory quotas can still not be considered for labels that do not specify any limits, i.e. neither a pool default, nor label specific limit is set. * Priority may now be set at the provider or provider-pool level. Providers or pools with the highest priority will fulfill requests first, until they are at quota, at which point lower-priority providers will begin to be used. See providers.priority for details. Upgrade Notes ============= * Due to an internal change in how Nodepool launchers communicate with each other, all launchers should be upgraded to the same version within a short period of time. They will generally continue to work at different versions, but the mechanism that allows them to yield to specific providers when requested is being changed and so that will not function correctly unless they are upgraded near-simultaneously. Bug Fixes ========= * Fixes an exception that was raised by the OpenStack driver when attempting to reset the quota timestamp and ignore-provider-quota is true. This exception prevented nodepool from seeing quota errors from OpenStack, causing it to fail the node request instead of retrying as it does for other providers. * Removes diskimage.meta checks from the OpenStack driver. The limit of only 5 entries is anachronistic and now removed. Rather than trying to pre-guess what OpenStack wants the metadata is now passed as-is and OpenStack will reject it at upload time. * Previously, metadata was checked by nodepool and invalid values would cause all metadata to be silently ignored. Now, metadata will be passed directly to glance, and an API error will occur. This may mean that images that previously uploaded (with no metadata) will now cause an API error when uploading.

1 0

Zuul-Client 0.1.0 released
by James E. Blair 22 Jun '22

22 Jun '22

The Zuul team is pleased to announce the release of Zuul-Client 0.1.0. Download it from PyPI: https://pypi.org/project/zuul-client Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul-client/0.1.0/releasenotes.html New Features ============ * Add the build-info subcommand, allowing a user to fetch the details of a given build by its UUID. * Add the builds subcommand to zuul-client, allowing users to search through builds using filters. * Add a "tenant" field in the configuration file, so that a user doesn't have to specify this argument in the CLI when using a predefined configuration.

1 0

Default java version in ensure-java role changing
by James E. Blair 21 Jun '22

21 Jun '22

Hi, On July 5 we will merge https://review.opendev.org/846909 which will change the default java version from 8 to 11 in the ensure-java role in zuul-jobs. Please take this time to adjust your usage of that role if necessary. -Jim

1 0

Nodepool 6.0.0 released
by James E. Blair 03 May '22

03 May '22

The Zuul team is pleased to announce the release of Nodepool 6.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/6.0.0/releasenotes.html Prelude ======= The AWS driver has been updated to achieve parity with other Nodepool drivers. New Features ============ * The AWS driver now supports rate limiting. It utilizes a two-tier rate limiting system to match AWS’s request token buckets. The rate specified in the config file is used as the rate for mutating requests. Non-mutating requests will have their rate limited to 10 times that amount. * The AWS driver now supports quota. AWS only provides a quota value for the number of cores. * The AWS driver now support diskimage uploads. * The AWS driver uses a new state machine framework within Nodepool with significant caching in order to improve performance at scale. * The AWS driver now supports IPv6 addresses. * openshift and openshiftpods drivers now supports pods using images from private registries by configuring image-pull-secrets. * It is now possible to configure the zookeeper-timeout. which can help to avoid zookeeper session losses on busy systems. Upgrade Notes ============= * The AWS driver will now ignore the "Name" tag if specified. Instead, it matches the behavior of other Nodepool drivers and sets the instance name to the Nodepool hostname (which is derived from the node name; e.g, "np0000000001") Deprecation Notes ================= * In AWS providers, the public-ip-address setting is deprecated. Use public-ipv4 or public-ipv6 instead. * In AWS providers, specifying image filter values as non-string values is deprecated. The current behavior is that Nodepool coerces non-string values (such as true or integers) into strings, but a later version of Nodepool will produce an error. Please update config files to use literal (quoted if necessary) YAML strings. Other Notes =========== * Kubernetes 1.8 or newer is required by the Kubernetes driver. This was necessary to support Kubernetes 1.22.0 and newer which requires using APIs that are not supported before version 1.8.

1 0

Zuul 6.0.0 released
by James E. Blair 03 May '22

03 May '22

The Zuul team is pleased to announce the release of Zuul 6.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/6.0.0/releasenotes.html New Features ============ * Ansible version 5 is now available. The default Ansible version is still 2.9, but version 5 may be selected by using job.ansible-version. * Pipelines can now trigger on wip-state-changed and filter events on the wip state of a change with pipeline.require.<gerrit source>.wip. Upgrade Notes ============= * This is the first version of Ansible added to Zuul since the Ansible project began releasing the Ansible community package. Zuul includes the Ansible community package since it includes a wide selection of useful modules, many of which were included by default in previous versions of Ansible. Only the major version of Ansible community is specified (e.g. ansible-version: 5). This corresponds to a single minor release of Ansible core (e.g., Ansible community 5 corresponds to Ansible core 2.12). Ansible releases minor versions of the community package which may contain updates to the included Ansible collections as well as micro version updates of Ansible core (e.g. Ansible community 5.6 includes ansible-core 2.12.4). Zuul does not specify the minor version of Ansible community, therefore the latest available micro-version will be installed at build-time. If you need more control over the version of Ansible used, see the help text for zuul-manage-ansible. * The built-in support for the ARA Ansible callback plugin has been removed. * The restricted Ansible environment used for untrusted playbooks has been relaxed. Zuul previously attempted to restrict the actions of playbooks running in the untrusted execution context on the executor so that users would not be able to load custom Ansible plugins, execute code on the executor, or use certain functions of built-in Ansible modules. This was done in an attempt to improve the security of the Zuul executor. However, the approach has proved laborious, prone to error, and increasingly incompatible with newer versions of Ansible. Therefore it has been removed, and now playbooks within both the trusted and untrusted execution contexts have access to the full suite of Ansible modules. See the Security Considerations section for information on caveats relating to executor security. Bug Fixes ========= * Nodepool host variables are now available even for unreachable hosts. This is useful for network appliances which start the job in an unreachable state.

1 0

Changing deb-src behavior on Debian in configure-mirrors role
by Clark Boylan 27 Apr '22

27 Apr '22

Hello, We recently noticed that configure-mirrors enables deb-src repositories on Debian but not Ubuntu. Typically deb-src repositories are not required by CI systems as jobs don't need those packages. These packages do consume significant disk space on your mirrors and the current configure-mirrors behavior means that you must replicate and include these package repos. We would like this to no longer be the case so we've pushed a change to enable deb-src behind a configure-mirrors option [0]. Currently this flag defaults to True to preserve the preexisting behavior. In two weeks we will flip the default to False [1] improving the situation for people running the CI system and its package mirrors. If jobs need the source packages they can toggle the flag in their jobs or manually configure deb-src repo configuration themselves. [0] https://review.opendev.org/c/zuul/zuul-jobs/+/839592/ [1] https://review.opendev.org/c/zuul/zuul-jobs/+/839593/ Clark

1 0

Zuul 5.2.5 released
by James E. Blair 27 Apr '22

27 Apr '22

The Zuul team is pleased to announce the release of Zuul 5.2.5. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes for 5.2.3 through 5.2.5, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/5.2.5/releasenotes.html Bug Fixes ========= * Projects and jobs on branches whose names have special characters in regular expressions could fail to match changes as intended. Implied branch matchers automatically generated from branch names are now treated as requiring exact matches. Any user-specified branch matcher (including in job.branches and pragma.implied-branches) are still treated as regular expressions. * Project name regex handling has been updated to return all possible matches. Previously if there were collisions with short names it was an error. The point of the regex system is to simplify configuration and apply configs to all projects that match. Collisions don’t impact this behavior so we don’t need to raise an error in these cases. * A change in 5.2.3 began reporting NO_JOBS results to the database which was unintentional. This has been corrected and they are no longer recorded in the database. * Further instances of reporting NO_JOBS results to the database when not necessary have been corrected.

1 0

Zuul 5.2.2 released
by James E. Blair 31 Mar '22

31 Mar '22

The Zuul team is pleased to announce the release of Zuul 5.2.2. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes for 5.2.2 and 5.2.1, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/5.2.2/releasenotes.html Bug Fixes ========= * An issue introduced in Zuul version 5.2.0 which could cause jobs running on Windows nodes to fail with the error Could not find imported module support code for ‘Ansible.ModuleUtils.Legacy’” has been corrected. * Fixed an issue in the Gerrit driver introduced in 5.2.0 which could cause infinite loops while querying changes. New Features ============ * The queue.dependencies-by-topic setting is added for Zuul users who wish to emulate the Gerrit submitWholeTopic behavior for some projects without enabling it site-wide in Gerrit. * The promote administrative action now functions with all pipeline managers. Previously it would only have an impact on dependent pipelines, but it will now re-order change queues as well as changes within any type of pipeline. Upgrade Notes ============= * The promote administrative action will no longer restart jobs for changes which have not been re-ordered within their change queue.

1 0

Zuul 5.2.0 released (security)
by James E. Blair 25 Mar '22

25 Mar '22

The Zuul team announces the release of Zuul 5.2.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/5.2.0/releasenotes.html Security ======== * A vulnerability which allowed the execution of untrusted code on the executor was fixed. Zuul restricts the Ansible modules and plugins which can be used in the `untrusted` security context (i.e., untrusted projects). It also prohibits running programs on the Zuul executor in the untrusted security context. Ansible 2.8 and later versions support referencing builtin modules using the `ansible.builtin.<name>` alias. Playbooks which use this mechanism can bypass Zuul's security restrictions and run arbitrary local code or otherwise restricted modules. Zuul's use of bubblewrap means that any commands executed via this vulnerability would still be contained within the restricted environment, meaning that they can not access files outside of the build directory or continue running longer than the playbook. But they may have been able to access files within the build directory but outside of the `work/` directory, as well as potentially exploit any kernel or hypervisor privilege escalation vulnerabilities. The Zuul team now considers the restricted Ansible environment to be ineffective as a security mechanism and is developing plans to remove the restrictions and rely entirely on bubblewrap in the future. These changes will occur in a future release of Zuul (likely 6.0.0) and will be preceded by more details about the change. New Features ============ * Changes in dependent queues are now automatically dequeued if they no longer meet the mergability requirements set by the upstream code review system. * Zuul now matches Gerrit’s behavior when change.submitWholeTopic is set in Gerrit. When this setting is enabled in Gerrit and a change is submitted (merged), all changes with the same topic are submitted simultaneously. Zuul will now query for changes which are set to be submitted together by Gerrit when enqueing them and treat them as if they are a set of circular dependencies. If the projects are not part of pipeline queues which are configured to allow circular dependencies, then Zuul will report failure on enqueue. Be sure that the submitWholeTopic setting in Gerrit and the allow-circular-dependencies setting in Zuul match. This functionality requires an HTTP connection to Gerrit. If only an SSH connection is available, then changes submitted together will be ignored. * The ability to configure Microsoft Login as an OpenID Connect authentication provider has been added. * Zuul now reports total resource usage statistics. The following statistic is emitted: zuul.nodepool.resources.total.tenant.{tenant}.{resource} Gauge with the currently used resources by tenant in total, i.e., all nodes belonging to a tenant, regardles of their state. * It is now possible for jobs to skip retries caused by failures in ‘pre-run’ by returning retry: false: - zuul_return: data: zuul: retry: false Upgrade Notes ============= * Zuul now reports total and in-use resource usage statics separately. To distinquish those, the following statistics have been renamed: zuul.nodepool.resources.tenant.{tenant}.{resource} -> zuul.nodepool.resources.in_use.tenant.{tenant}.{resource} zuul.nodepool.resources.project.{project}.{resource}: -> zuul.nodepool.resources.in_use.project.{tenant}.{resource}

1 0

Zuul 5.1.0 released
by James E. Blair 21 Mar '22

21 Mar '22

The Zuul team is pleased to announce the release of Zuul 5.1.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/5.1.0/releasenotes.html New Features ============ * Added support for adding and removing merge request labels in the GitLab driver, as well as triggering pipelines on label removal (label addition was already supported). * A new buildset result MERGE_FAILURE is stored in the build database in the case where a buildset is unable to be merged by the upstream code review system. * The following new statsd metrics are available in order to monitor Zuul system performance: * zuul.tenant.<tenant>.pipeline.<pipeline>.event_enqueue_time * zuul.tenant.<tenant>.pipeline.<pipeline>.merge_request_time * zuul.tenant.<tenant>.pipeline.<pipeline>.layout_generation_time * zuul.tenant.<tenant>.pipeline.<pipeline>.job_freeze_time * zuul.tenant.<tenant>.pipeline.<pipeline>.repo_state_time * zuul.tenant.<tenant>.pipeline.<pipeline>.node_request_time * zuul.tenant.<tenant>.pipeline.<pipeline>.job_wait_time * zuul.tenant.<tenant>.pipeline.<pipeline>.event_job_time * zuul.tenant.<tenant>.pipeline.<pipeline>.read_time * zuul.tenant.<tenant>.pipeline.<pipeline>.write_time * zuul.tenant.<tenant>.pipeline.<pipeline>.read_objects * zuul.tenant.<tenant>.pipeline.<pipeline>.write_objects * zuul.tenant.<tenant>.pipeline.<pipeline>.read_znodes * zuul.tenant.<tenant>.pipeline.<pipeline>.write_znodes * zuul.tenant.<tenant>.pipeline.<pipeline>.read_bytes * zuul.tenant.<tenant>.pipeline.<pipeline>.write_bytes * zuul.tenant.<tenant>.pipeline.<pipeline>.event_process * zuul.tenant.<tenant>.pipeline.<pipeline>.handling * zuul.tenant.<tenant>.reconfiguration_time * A new command zuul-scheduler tenant-reconfigure has been added. It allows an operator to perform a reconfiguration of a single tenant. This may be helpful in clearing up issues after connection problems with the code hosting system. * New monitoring metrics specific to zuul-web are available. Upgrade Notes ============= * The buildset result MERGER_FAILURE has been renamed to MERGE_CONFLICT, and the pipeline reporter configuration merge-failure has been renamed to merge-conflict. These are more descriptive of the most common errors actually reported, and so are expected to be less confusing to users. This is also in service of a future change to support a new buildset result MERGE_FAILURE which will indicate that the change was unable to be merged in the upstream repository. When upgrading, it is recommended to stop all schedulers briefly (i.e, when the first scheduler of the new version starts, there should be no schedulers running the old version). The new scheduler will perform a database migration when it starts and update all existing MERGER_FAILURE buildset results to MERGE_CONFLICT. If old schedulers are running, they may continue to add MERGER_FAILURE entries which will need to be manually updated in order to be visible in the web UI or rest API. Deprecation Notes ================= * The merge-failure and merge-failure-message pipeline configuration options have been renamed to merge-conflict and merge-conflict-message respectively. The old settings are retained for backwards compatibility, but will be removed in a later version. Please update your usage of them as soon as possible. Bug Fixes ========= * The labels pipeline requirement in the gitlab driver erroneously treated the labels as a boolean or but should have treated them as a boolean and (i.e., all listed labels are required). The behavior has been updated to match the documentation and other drivers.

1 0

Nodepol 5.0.0 released
by James E. Blair 01 Feb '22

01 Feb '22

The Zuul team is pleased to announce the release of Nodepool 5.0.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/5.0.0/releasenotes.html This release is nearly identical to 4.4.0. The major version increment to 5.0 is to re-align with Zuul.

1 0

Zuul 5.0.0 released
by James E. Blair 01 Feb '22

01 Feb '22

The Zuul team is pleased to announce the release of Zuul 5.0.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/5.0.0/releasenotes.html Upgrade Notes ============= * Zuul no longer uses gearman. It is no longer required to run a Gearman server, and the Zuul scheduler no longer does so. All gearman-related settings in zuul.conf files are now ignored and may be removed. Commands such as zuul enqueue which were previously executed via Gearman may still be used by creating an authentication token and adding a web section to zuul.conf. * Support for configuring the database as a connection in zuul.conf was deprecated in Zuul version 4.0 and has now been removed.

1 0

Zuul 4.12.0 released
by James E. Blair 25 Jan '22

25 Jan '22

The Zuul team is pleased to announce the release of Zuul 4.12.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.12.0/releasenotes.html Upgrade Notes ============= * Due to a change in the ZooKeeper data format, the following upgrade procedure is required: * Stop all Zuul components * Run zuul delete-state * Start all Zuul components * Zuul-web now requires access to the Zookeeper keystore. Ensure that the keystore.password option is included in zuul.conf on at least the scheduler, executor, and web servers (it may be included on all components if desired). * Zuul-web requires information for all defined connections. Previously, zuul-web may have started without all of the connections fully defined in its config file, or with some requirements (such as keys used for connecting to remote services) present. They are now required in order for zuul-web to start.

1 0

Nodepol 4.4.0 released
by James E. Blair 12 Jan '22

12 Jan '22

The Zuul team is pleased to announce the release of Nodepool 4.4.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.4.0/releasenotes.html New Features ============ * Added support for filtering Azure images. Use the providers.[azure].cloud-images.image-filter setting to specify a private image using filters (tags, for example). * The node-attributes setting has been added to the Azure driver. * The Azure driver now supports setting an admin password, which is required in order to launch Windows images on Azure. * The shell-type setting has been added to the Azure driver. * The Azure driver now supports user-data and custom-data. * Two new nodepool commands, nodepool export-image-data and nodepool import-image-data have been added to back up the image data in ZooKeeper to a file in case the ZooKeeper cluster is lost.

1 0

Zuul 4.11.0 released
by James E. Blair 17 Dec '21

17 Dec '21

The Zuul team is pleased to announce the release of Zuul 4.11.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.11.0/reference/releasenotes.html New Features ============ * The new variable zuul.playbook_context as well as new variables under zuul.projects have been added to help debug or audit the playbooks and roles used by Ansible when running jobs. * These variables describe the repo configuration used for the playbooks and roles of each Ansible execution. These repos may have a different state than the workspace repos. * It is now possible to disable connection pooling in the Gitlab driver with <gitlab connection>.disable_connection_pool. This option may be useful under adverse network conditions. * The optional prometheus service now also includes endpoints for readiness and liveness checks. See Liveness Probes for details. * Allow an authorized user to dequeue changes from the Web UI's status page. * Add authentication in the web UI. Zuul's web UI can be configured to authenticate users against an Identity Provider supporting the OpenID Connect protocol. Upgrade Notes ============= * Users upgrading from previous versions of zuul will need to run zuul delete-state to delete Zuul's ephemeral state from Zookeeper. The ephemeral state produced by older versions of zuul is not compatible with this release. This command should be run while all Zuul services are stopped to prevent writes to the Zookeeper database. More information on this command can be found in the documentation: https://zuul-ci.org/docs/zuul/reference/client.html#delete-state * The prometheus endpoint would previously serve metrics at any URI. It now only returns metrics on the / and /metrics URI. The latter is recommended.

1 0

Zuul 4.10.4 released
by James E. Blair 29 Oct '21

29 Oct '21

The Zuul team is pleased to announce the release of Zuul 4.10.4. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.10.4/reference/releasenotes.html Bug Fixes ========= * A fix was made to mitigate an issue where a ZooKeeper disconnect could cause too many threads to be spawned resulting in errors.

1 0

Zuul 4.10.3 released
by James E. Blair 22 Oct '21

22 Oct '21

The Zuul team is pleased to announce the release of Zuul 4.10.3. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.10.3/reference/releasenotes.html Bug Fixes ========= * A bug with the change cache cleanup routine which could have caused items to be stuck in pipelines without running jobs has been corrected. * Gerrit driver: Zuul error responses could overflow the default message length with large configurations leading to Gerrit rejecting the comment (and hence no notification from Zuul on the change). Such messages are now truncated to a safe length.

1 0

Zuul 4.10.2 released
by James E. Blair 16 Oct '21

16 Oct '21

The Zuul team is pleased to announce the release of Zuul 4.10.2. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.10.2/reference/releasenotes.html Bug Fixes ========= * A bug which prevented change cache cleanup (and therefore caused ZooKeeper usage to grow without limits) was fixed.

1 0

Zuul 4.10.1 released
by James E. Blair 13 Oct '21

13 Oct '21

The Zuul team is pleased to announce the release of Zuul 4.10.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.10.1/reference/releasenotes.html New Features ============ * Add support for squashing MR with gitlab. * On the description page of a buildset in the zuul web dashboard, a user can display a timeline of the builds making up the buildset. Bug Fixes ========= * Fix a bug in the cleanup routine for the new Zookeeper based change cache. Previously this caused the change cache to leak entries. Note the fixed routine will clean up the leaked entries automatically once Zuul is restarted on this new version. * Fix detection of the failed merge in gitlab driver.

1 0

Zuul 4.10.0 released
by James E. Blair 06 Oct '21

06 Oct '21

The Zuul team is pleased to announce the release of Zuul 4.10.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.10.0/reference/releasenotes.html Upgrade Notes ============= * Nodepool 4.3.0 is now required. Zuul stores additional information in node records in preparation for supporting multiple schedulers. * The scheduler time database has been removed. This was stored in the scheduler state directory, typically /var/lib/zuul/times. The entire state directory on the scheduler is no longer used and may now be removed. Zuul now derives its estimated build duration times from the SQL database.

1 0

Nodepol 4.3.0 released
by James E. Blair 10 Sep '21

10 Sep '21

The Zuul team is pleased to announce the release of Nodepool 4.3.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.3.0/releasenotes.html New Features ============ * Added the option to set quota on resources on a per-tenant basis (i.e. Zuul tenants). A new top-level config structure tenant-resource-limits has been added under which one can specify a number of tenants, each with max-servers, max-cores, and max-ram limits. These limits are valid globally, i.e., for all providers. This differs from currently existing provider and pool quotas, which only are considered for nodes of the same provider. This feature is optional and tenant quotas are ignored for any NodeRequests that do not deliver tenant information with them. Also no quota is evaluated for tenants that have no limits configured for them.

1 0

Zuul 4.9.0 released
by James E. Blair 09 Sep '21

09 Sep '21

The Zuul team is pleased to announce the release of Zuul 4.9.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.9.0/reference/releasenotes.html New Features ============ * The zuul delete-state command may be used to delete all of the ephemeral state stored by Zuul in ZooKeeper. Normally Zuul is able to detect and correct errors on its own, but in case it is unable to, this may prove a useful utility for manual recovery. * The new commands zuul copy-keys and zuul delete-keys may be useful when renaming projects in order to move the keys from the old to the new project names without service interruption. * Add new statsd gauge metrics of current open node requests exported as zuul.nodepool.tenant.<tenant>.current_requests. This metric tracks the currently open node requests per tenant. It drills down the overall zuul.nodepool.current_requests metric. Upgrade Notes ============= * Zuul no longer reads or writes project private key files from the scheduler’s filesystem. In order to load existing keys into ZooKeeper, run version 4.6.0 of the scheduler at least once, if you haven’t already. * A new command zuul export-keys has been added to export the encrypted keys from ZooKeeper onto the filesystem for backup. Likewise, zuul import-keys will load a previously-exported backup into ZooKeeper. It is recommended that you use these commands in system backup scripts.

1 0

zuul-jobs: potentially breaking change to tox base job
by Sorin Sbarnea 08 Sep '21

08 Sep '21

There is a change at https://review.opendev.org/c/zuul/zuul-jobs/+/807702 which is changing the default tox arguments to add "--skip-missing-interpreters false". This overrides the value from tox.ini file and disallow tox from using another python version than the requested one. The goal of this change is to prevent accidents where some project endup running tox in CI and believe that these jobs are testing one version of python but in fact another version is used instead. We think that this has the potential to affect only users of jobs with `parent: tox`. Users that are using `tox-py3*` zuul-jobs are not affected because we force tox to run using that specific python version. We will wait 2 weeks before merging this change, so people would have time to adapt their configurations. Potentially affected projects from opendev: https://codesearch.opendev.org/?q=skip_missing_interpreters%20%3D%20%5BtT%5… If you find your project on the list above it would be wise to just apply the tox.ini changes mentioned below before we merge that change If you are affected you have two options: - update your tox.ini file and use generic "py" environment on envlist and remove the skip_missing_interpreters entry. - override tox_extra_args in your tox job in order to keep using the old behavior Cheers Sorin Sbarnea

1 0

Zuul 4.8.1 released
by James E. Blair 24 Aug '21

24 Aug '21

The Zuul team is pleased to announce the release of Zuul 4.8.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.8.1/reference/releasenotes.html Upgrade Notes ============= * An error was found in a change related to Zuul’s internal configuration cache which could cause Zuul to use cached in-repo configuration files which no longer exist. If a zuul.yaml (or zuul.d/* or any related variant) file was deleted or renamed, Zuul would honor that change immediately, but would attempt to load both the old and new contents from its cache upon the next restart. This error was introduced in version 4.8.0. If upgrading from 4.8.0, run zuul-scheduler full-reconfigure in order to correctly update the cache. Bug Fixes ========= * Restored the job.success-message and job.failure-message functionality which was inadvertently removed.

1 0

Zuul has moved from IRC to Matrix
by James E. Blair 21 Aug '21

21 Aug '21

Hi, The Zuul project has completed its move from IRC to Matrix. If you would like to chat with us, please join our room, #zuul:opendev.org -- you can follow this link for convenience: https://matrix.to/#/#zuul:opendev.org You can read detailed instructions on how to get started using Matrix (it's easy!) here: https://zuul-ci.org/docs/zuul/howtos/matrix.html Those instructions also include information on how to use Matrix to participate in bridged IRC channels, so you can use the same client to participate in communities that use IRC. If you are still in the IRC channel, please go ahead and /part in order to aid us in shutting it down. -Jim

1 0

Moving from IRC to Matrix
by James E. Blair 19 Aug '21

19 Aug '21

Hi, The Zuul community will be moving our official chat platform to Matrix on Saturday, August 21. The room has not been created yet, but when it is, the name will be #zuul:opendev.org We will update the website and instructions in the documentation accordingly over the next few days. Starting next week, we will no longer use the #zuul IRC channel on OFTC, and will encourage anyone remaining in that channel to /part. You don't need to do anything right now -- because Matrix preserves history, you can join the new room at any point in the future and have access to messages sent before you joined, so you won't miss anything by joining after the move. -Jim

1 0

Zuul 4.8.0 released
by James E. Blair 12 Aug '21

12 Aug '21

The Zuul team is pleased to announce the release of Zuul 4.8.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.8.0/reference/releasenotes.html New Features ============ * Zuul now can route live log streams via finger gateways to make it possible to distribute executors over multiple datacenters without the possibility to directly contact every executor from within zuul-web. This is typically the case in a Kubernetes based deployment. * The finger gateway and executor log streaming system now supports TLS connections. Normally zuul-web makes a direct connection to an executor in order to stream logs. With this new option, that connection can be encrypted if it crosses an untrusted network. The ability to route log streaming connections through finger gateway servers was recently added; these will also use TLS if required. The finger gateway server can also be used by end-users; in that case it may need a TLS certificate to use if it is required to connect to an encrypted executor or finger gateway to stream logs. An option to disable using TLS when acting as a server is provided for this case, since there are no TLS-enable finger clients. See fingergw.tls_cert and related options to enable encrypted connections for all three components.

1 0

Zuul 4.7.0 released
by James E. Blair 30 Jul '21

30 Jul '21

The Zuul team is pleased to announce the release of Zuul 4.7.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.7.0/reference/releasenotes.html New Features ============ * The Github driver can now use "skipped" and "neutral" status for pipeline.<reporter>.<github source>.check attribute on the Github reporter. Upgrade Notes ============= * The zuul.scheduler.eventqueues.result gauge was removed * Zuul now requires at least Nodepool version of 4.2.0 due to an internal API change. Deprecation Notes ================= * The following attributes are now ignored: * The report-build-page tenant configuration setting. * The success-url job attribute. * The failure-url job attribute. The URL of the build page is now always reported for every build. This is now also true for in-progress builds, which provides for a more consistent user experience. Since the build page is always reported as the URL, the success and failure URL job attributes are no longer useful, so this functionality has also been removed. Zuul's configuration syntax checker will continue to allow these settings for now (they are simply ignored) but this will be removed in version 5.0 and using them will be considered an error. To achieve a similar result, consider returning the URL as an artifact from the job via zuul_return. This will cause a link to appear in the "Artifacts" section of the build page.

1 0

Nodepool 4.2.2 released
by James E. Blair 22 Jul '21

22 Jul '21

The Zuul team is pleased to announce the release of Nodepool 4.2.2. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.2.2/releasenotes.html Bug Fixes ========= * Fixes a regression in gathering public SSH host keys on slower nodes. We now wait until SSHd on the remote node is started before gathering host keys.

1 0

Nodepool 4.2.1 released
by James E. Blair 09 Jul '21

09 Jul '21

The Zuul team is pleased to announce the release of Nodepool 4.2.1. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.2.1/releasenotes.html Bug Fixes ========= * Fix Kubernetes in-cluster configuration loading if no local config is present. The previous release missed a fallback case which has been corrected.

1 0

Nodepool 4.2.0 released
by James E. Blair 08 Jul '21

08 Jul '21

The Zuul team is pleased to announce the release of Nodepool 4.2.0. Download it from PyPI: https://pypi.org/project/nodepool Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.2.0/releasenotes.html Upgrade Notes ============= * Nodepool 0.3.6 introduced an unintended behaviour change with openstacksdk statsd reporting. Due to a change in the way nodepool manages OpenStack API calls, all API related statistics created during interaction with clouds are now generated by openstacksdk and prefixed with openstack.api instead of being created by nodepool and prefixed with nodepool.provider.<cloud> as in prior versions. If you wish to revert to the prior behaviour, changes have been provided to openstacksdk to allow setting custom prefixes via the cloud configuration file; see statsd documentation

1 0

Zuul 4.6.0 released [security]
by James E. Blair 24 Jun '21

24 Jun '21

The Zuul team announces the release of Zuul 4.6.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.6.0/reference/releasenotes.html This release addresses several security issues: * Paul Belanger and Ian Wienand discovered that container building jobs in the zuul-jobs repo which used the buildset registry could be vulnerable to a situation where if multiple jobs were using the buildset registry and at least one job finished before the others and included buildset registry credentials in its logs, an attacker could obtain those credentials and externally alter the contents of the buildset registry while the remaining jobs were in progress. * Albin Vass discovered that in some circumstances Zuul secrets could be exposed by supplying jobs with a variable that used an Ansible Jinja template referencing the secret. If a playbook with access to a secret referenced a variable that could be overridden by a child job, then the playbook could be tricked into exposing the contents of the secret. * Albin Vass discovered that Ansible "extra vars" can be used to alter the behavior of playbooks that use loop variables in included tasks. The precedence of extra vars is so high that they override even what playbook authors would consider internally defined variables. This could lead to secret exposure or potentially arbitrary code execution. * Albin Vass discovered that the ability to set the ansible_shell_executable and ansible_connection variables on remote nodes could be used to execute arbitrary code in a trusted context. * Albin Vass discovered that it is possible to request nodes or groups with the name "localhost" which could be used to trick Zuul into running on remote nodes trusted playbooks designed to be run only on the executor. This could lead to the exposure of secrets. To address these, we have taken the following steps: Added a facility for jobs to return secret data to Zuul to pass to dependent jobs without those data appearing in normal inventory or logs, and we have updated the buildset registry roles in zuul-jobs to use it. Zuul now freezes the values of variables (without access to secrets) at the start of jobs, and use the frozen values in any trusted playbook or any playbook with a secret. Removed the ability to set Ansible extra vars in Zuul. Restricted the ability to set some ansible_* variables. Removed the ability to request nodes or groups with the name "localhost". Please see the release notes below for details and information on adjustments you may need to make because of these changes. We believe (but have not verified) that these vulnerabilities have been present in Zuul since version 3.0.0. We thank the reporters for finding and responsibly reporting these issues. Security Issues =============== * The following connection-related variables are no longer allowed to be set in job definitions, as they may be used to circumvent security measures: ansible_connection ansible_host ansible_python_interpreter ansible_shell_executable ansible_user They may still be set using the corresponding settings in Nodepool. * The ability to use Ansible Jinja templates in Zuul job variables is partially restricted. It was found that the ability to use Jinja templates in Zuul job variables could be used to expose the contents of secrets. To remedy this, the values of Zuul job variables are frozen at the start of the job and these values are used for trusted playbooks and playbooks with secrets. The freezing action is taken without access to any secrets so they can not be exposed. This means that Zuul job variables which reference non-secret values that are known at the start of the job (including any zuul.* variable) will continue to work as expected. Job variables which reference secrets will not work (they will be undefined). In untrusted playbooks, job variables are still dynamically evaluated and can make use of values that are set after the start of the job. Additionally, job.extra-vars are no longer passed to Ansible using the "-e" command line options. They could be used to expose secrets because they take precedence over some internal playbook variables in some circumstances. Zuul's extra-vars are now passed as normal inventory variables, however, they retain precedence over all other Zuul job variables (vars, host-vars, and group-vars) except secrets. Secrets are also now passed as inventory variables as well for the same reason. They have the highest precedence of all Zuul job variables. Their values are tagged with !unsafe so that Ansible will not evaluate them as Jinja expressions. If you are certain that a value contained within a secret is safe to evaluate as a Jinja expression, you may work around this limitation using the following construct in a playbook: - set_fact: unsafe_var_eval: "{{ hostvars['localhost'].secret.var }}" This will force an explicit evaluation of the variable. This is generally safe to do in a situation where a playbook is accessing a single secret by name, with no other secrets in scope. Do not use this capability with more than one secret that is not under the control of the project where the playbook is defined. Similarly, versions of all the original job variables tagged with !unsafe are available under the unsafe_vars variable hierarchy. For example, the job variable myvar would be available under unsafe_vars.myvar. It is not recommended to evaluate unsafe_vars expressions except in the most controlled of circumstances. They are almost impossible to render safely. New Features ============ * If sensitive data must be returned from a job in order to be provided to dependent jobs, the secret_data attribute of zuul_return attribute may now be used instead of the normal data attribute. The data will be provided via the same mechanism as job secrets, where the data are not written to disk in the work directory. Care must still be taken to avoid displaying or storing sensitive data within the job. For example: tasks: - zuul_return: secret_data: password: foobar data: this_is: not secret

1 0

Zuul 4.5.1 released
by James E. Blair 15 Jun '21

15 Jun '21

The Zuul team is pleased to announce the release of Zuul 4.5.1. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.5.1/reference/releasenotes.html This is a minor bugfix release that corrects an issue with the "zuul tenant-conf-check" command. If you use this command, you may want to upgrade, otherwise, this release is nearly identical to 4.5.0 and an upgrade is not necessary. As a reminder, the Zuul Maintainers are planning on making a release of Zuul with important security updates on June 24, 2021. For the purposes of preparing for that release, running either 4.5.0 or 4.5.1 is satisfactory. Bug Fixes ========= * The zuul tenant-conf-check command no longer needs a ZooKeeper connection to validate the tenants configuration file is valid.

1 0

Upcoming security release
by James E. Blair 10 Jun '21

10 Jun '21

Multiple security issues have been identified in Zuul which could lead to an exposure of secret data. The Zuul Maintainers have been preparing fixes to these issues and plan to make a release of Zuul with them on Thursday, June 24, 2021, around 14:00 UTC. Unfortunately, there is no known mitigation for these vulnerabilities, and so the best advice we can offer for operators is to stop Zuul shortly before the release, and restart it after updating. Additionally, there will be some modification to the functionality offered by Zuul, which may cause some jobs to fail or behave differently. More details will be provided along with the release. We encourage you to use the time between now and June 24th to ensure that you are running the latest version of Zuul (4.5.0) in order to prepare for the update. An announcement of the release and accompanying information will be sent to zuul-announce(a)lists.zuul-ci.org when the release is available. Feel free to follow up to zuul-discuss(a)lists.zuul-ci.org or chat with us in #zuul on OFTC if you have any questions.

1 0

Zuul 4.5.0 released
by James E. Blair 10 Jun '21

10 Jun '21

The Zuul team is pleased to announce the release of Zuul 4.5.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at: https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.5.0/reference/releasenotes.html This is a checkpoint release. This release contains some bugfixes and other user-invisible changes, but its main purpose is to serve as a checkpoint as we move more of Zuul's functionality into ZooKeeper. This version has seen production use in OpenDev and we believe it to be stable and ready for wider use. Please do upgrade to it and report any issues to zuul-discuss. You may downgrade to the previous release if you do encounter any issues. Likewise, operators running the latest development builds may downgrade to this release in the case of issues.

1 0

Zuul 4.4.0 released
by James E. Blair 02 Jun '21

02 Jun '21

The Zuul team is pleased to announce the release of Zuul 4.4.0. Download it from PyPI: https://pypi.org/project/zuul Find container images at https://hub.docker.com/u/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.4.0/reference/releasenotes.html New Features ============ * A new prometheus_port option for the services can be used to start the prometheus python client and exposes metrics. * Jobs may now request multiple semaphores and they will not start until all semaphores are acquired. Use the new job.semaphores (plural) attribute to specify them. Note that the new attribute is additive when considering inheritance and job variants. That is to say that a job definition with a semaphores attribute will extend the list of semaphores supplied by its parent rather than overriding it (which is the behavior for the deprecated attribute). Deprecation Notes ================= * The job attribute job.semaphore (note the singular rather than plural form) job attribute is now deprecated. Use the plural form job.semaphores instead. As with most list items in Zuul configuration, it also accepts a single item without the wrapping list, so to convert existing jobs, simply change the spelling of the attribute, no change to the value is required. The singular form will be removed in Zuul 5.0.

1 0

Zuul 4.3.0 released
by James E. Blair 17 May '21

17 May '21

The Zuul team is pleased to announce the release of Zuul 4.3.0. Download it from PyPI: https://pypi.org/project/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.3.0/reference/releasenotes.html New Features ============ * The MQTT reporter now includes artifact information along with build results. * Jobs may now specify an alternate scheme to use when preparing repositories in the workspace. The default remains the same golang-style, but an alternate scheme called flat is now available. See job.workspace-scheme for more details. * Project secrets keys and SSH keys are now stored in Zookeeper. All private data will be encrypted at rest, which requires a new mandatory setting keystore.password in zuul.conf. * For backup purposes the secrets keys and SSH keys will still exist on the local filesystem of the scheduler as before. Upgrade Notes ============= * The internal git repo caches maintained by the mergers and executors now use a new naming scheme in order to avoid collisions. When existing executors and mergers are restarted, they will remove their git repo caches and re-clone repos using the new scheme. Jobs may be slow to start until the caches are warmed again. * As project secrets keys and SSH keys are stored encrypted in Zookeeper the new keystore.password option in zuul.conf is required. Please add it to your configuration. Deprecation Notes ================= * Zuul now correctly handles the rerequested action on check run trigger definitions (check_run) literally. This has been requested which doesn’t match the GitHub api. The value requested is now deprecated and will be removed in a later release. Bug Fixes ========= * Fixed a bug where multiple connections of the same type would not filter trigger events coming from the wrong connection.

1 0

Upcoming breaking deployment change in Zuul
by James E. Blair 13 Apr '21

13 Apr '21

Hi, We need to add a new required configuration setting in zuul.conf, and once we do, Zuul will fail to start if it is omitted. Please add this setting as soon as possible to prepare for future upgrades. Because it is a new setting, you can add it right now and Zuul will ignore it until it is used. The setting is a shared secret that all Zuul components can use to decrypt data stored in ZooKeeper. This enables us to avoid storing decrypted sensitive data at rest in ZooKeeper -- an important security consideration for some deployments. To add this setting, create a new strong password, and set it as follows in zuul.conf: [keystore] password=CHANGEME That's it. We plan to merge the change which requires this on April 20, so if you're running from the latest source code or container images, make sure you implement this before then or pin to the 4.2.0 release. If you run the latest releases, be aware that 4.3.0 will likely include this change, so prepare for it before upgrading. -Jim

1 0

Zuul 4.2.0 released
by James E. Blair 12 Apr '21

12 Apr '21

The Zuul team is pleased to announce the release of Zuul 4.2.0. Download it from PyPI: https://pypi.org/project/zuul This is a checkpoint release. This release contains some bugfixes and other user-invisible changes, but its main purpose is to serve as a checkpoint as we move more of Zuul's functionality into ZooKeeper. This version has seen production use in OpenDev and we believe it to be stable and ready for wider use. Please do upgrade to it and report any issues to zuul-discuss. You may downgrade to the previous release if you do encounter any issues. Likewise, operators running the latest development builds may downgrade to this release in case of issues.

1 0

Zuul Operator 0.1.0 released
by James E. Blair 12 Apr '21

12 Apr '21

Hi, The Zuul team is pleased to announce the release of Zuul Operator 0.1.0. This tag is now available on Docker Hub. Note that we are currently reviewing changes to the operator which may render the next release (and subsequent :latest tags) incompatible with the current version. If you are using the operator, you may wish to switch to using the 0.1.0 tag for stability during this period.

1 0

Zuul 4.1.0 released
by James E. Blair 22 Mar '21

22 Mar '21

The Zuul team is pleased to announce the release of Zuul 4.1.0. Download it from PyPI: https://pypi.org/project/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.1.0/reference/releasenotes.html New Features ============ * Executors configured with executor.zone now can also be configured to also process jobs without a zone by enabling executor.allow_unzoned. * Zuul now honors the shell-type configuration from nodepool and uses it to set ansible_shell_type, a setting that is required when connecting to Windows workers over ssh. * For Linux workers, there is a long standing ansible issue with using non-default ansible_shell_type and become, so this feature is primarily targeting Windows workers. * Projects can now configure change queues to queue per branch. See queue for more information. * Circular dependencies are now optionally supported (but disabled by default). To permit dependency cycles to be enqueued, see the queue.allow-circular-dependencies option. * A job now can leave warning messages on the change using zuul_return. See Return Values for examples. * Zuul now supports the --validate-tenants switch which can be used to validate the complete configuration of the listed tenants. Upgrade Notes ============= * Two sets of statsd metrics are now reported for executors: zoned and unzoned. The existing statsd keys are now deprecated; new statsd keys are available for both zoned and unzoned executors. See zuul.executors for details. Deprecation Notes ================= * Shared queues should be configured per project now instead per pipeline. Specifying project.<pipeline>.queue is deprecated and will be removed in a future release. Bug Fixes ========= * If zoned executors were used with prior releases of Zuul, the reported executor statistics would only represent a single, unspecified zone. This has now been corrected.

1 0

Nodepool 4.1.0 released
by James E. Blair 22 Mar '21

22 Mar '21

The Zuul team is pleased to announce the release of Nodepool 4.1.0. Download it from PyPI: https://pypi.org/project/nodepool The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/nodepool/4.1.0/releasenotes.html New Features ============ * Multi-line log messages (such as tracebacks from image builds) are now prefixed in the same manner as single-line messages. * AWS EC2, GCE, Kubernetes, Openshift, Openshift Pods, OpenStack and Static drivers now support a shell-type config. Shell-type config is intended to enable setting of cmd or powershell as shell type for Windows workers with connection-type ssh. * For Linux workers, there is a long standing ansible issue with using non-default shell type and become, so care should be taken if using it for such workers. Upgrade Notes ============= * The zuul-public-key configuration attribute in the providers Azure driver has been moved and renamed. Please move this setting to its new location at providers.[azure].cloud-images.key

1 0

Behavior change for upload-logs roles
by Alex 12 Mar '21

12 Mar '21

We'd like to update the upload-logs-* roles in order to allow users can use proxies for Zuul storage endpoints. Updated roles would have a new output, named zuul_upload_logs_results and it will include log file addresses which users can get logs via proxies. This means if you are using upload_results.url to get the log download link in playbooks, need to replace that with zuul_upload_logs_results.url. For us, We also need to change the post-logs playbook in zuul/base-job repo to get quick-download link of Opendev Zuul server. [1] https://review.opendev.org/c/zuul/zuul-jobs/+/776677 [2] https://review.opendev.org/c/opendev/base-jobs/+/777087 We plan to merge this change on March 29, 2021. Please let us know if this would cause a hardship for you. Thank you.

1 0

Zuul 4.0.0 released
by jim＠acmegating.com 23 Feb '21

23 Feb '21

The Zuul team is pleased to announce the release of Zuul 4.0.0. Download it from PyPI: https://pypi.org/project/zuul The release notes, included below, are also available in the documentation with hyperlink references: https://zuul-ci.org/docs/zuul/4.0.0/reference/releasenotes.html Prelude ======= This is the first 4.x release of Zuul. It requires some deployment changes by operators which have been carefully planned in order to facilitate work on Zuul version 5, which will be the first version of Zuul where every component is fault tolerant and able to scale. If you read the release notes for the last 3.x release, you may have already made all of the required changes. If not, please do so before upgrading to version 4. Every required change in version 4 is optionally supported in 3.19, so it is safe to make these changes and then upgrade. Please read all of the notes below, especially in the “Upgrading” section for details. The primary additional requirements are: TLS ZooKeeper connections Network connectivity from all components to ZooKeeper An SQL database With these changes in place, it is anticipated that further upgrades to Zuul made in support of the scale-out-scheduler work will be done with minimal disruption in the course of normal releases between version 4 and 5. New Features ============ * The action “promote” is now available via the tenant-scoped REST API using the zuul cli. * Builds in the SQL reporter have a “held” attribute set to True if the build triggered a autohold request. Builds can be filtered by held status. * Subdirectories of zuul.d or .zuul.d can be ignored when reading configuration by placing a .zuul.ignore file inside them. * The find module is now allowed to run on the executor. * Changes can now be dequeued via the Github checks API. If a github reporter is configured to use the checks API, all running checks will provide a custom “Abort” action. * Zuul now respects GitHub review requirements when enqueuing into gate pipelines. This works for github.com and GitHub Enterprise starting with version 2.21.0. * Jobs may specify the new intermediate flag to note they may only be inherited by abstract jobs. This can be useful if building a job hierarchy where wish to limit where a base job is instantiated. * The builds/ and buildset/ API endpoints now include information about retried builds. They are called non-final as those are all builds that were retried at least once and thus weren’t visible to the user so far. * The builds/ API can filter those retried builds and you might exclude them from API result by setting final=false in the API request. * An option to use the URL of the Zuul build page when reporting has been added. This feature requires that all the pipelines in the tenant have a SQL reporter configured, and at least one of tenant.web-root or web.root must be defined. See tenant.report-build-page. * REST API: authorizations: add a tenant-scoped endpoint at '/api/tenant/{tenant}/authorizations'. Calling the endpoint will return a list of admin tenants limited to the scoped tenant, if the user has admin privileges on it. * Host key checking is now disabled for a host in the generated Ansible inventory if host-key-checking is set to False in the corresponding nodepool config. Upgrade Notes ============= * Support for Ansible 2.7 has been removed. * If using a SQL reporter, the zuul_builds table will be updated with a new 'held' column. The zuul-scheduler and zuul-web services need to be restarted together for the change to take effect. * Previously the SqlReporter would record the job name in the database in place of the url if the url was empty. This has now been updated to store a null in the database for that case. Zuul will automatically run a database migration to correct old values. * The -d switch doesn’t enforce foreground anymore. It only enables debug logging. To start zuul in foreground please add the -f switch instead. * Support for running Zuul under Python 3.5 has been dropped. * As further integration with the web interface is planned, the web.root setting in zuul.conf is marked required and future releases may error if it is missing. Please add it to your configuration now. See tenant.web-root for additional information about whitelabel tenants. * The components zuul-scheduler and zuul-web now require database configuration. There is now only one supported database connection. If multiple sql connections have been configured only the first one will be used. * Signal based triggering of full-reconfiguration in the scheduler (deprecated since 3.3.0) has been removed. Use zuul-scheduler full-reconfigure instead. * The zookeeper section in zuul.conf is required for all components, and all components must now be able to connect to ZooKeeper. Additionally, TLS is now required for all ZooKeeper connections. See Encrypted Connections for more details. Deprecation Notes ================= * Defining database connections using <sql connection> is now deprecated. Refer to Database Configuration how to configure the database now. * REST API: authorizations: the /api/user/authorizations endpoint is deprecated in favor of the tenant-scoped endpoint. It will be removed next release. Bug Fixes ========= * Although the documentation states that the MQTT reporter reports the build's log_url, this was only true as long as tenant.report-build-page was disabled. As soon as the setting was enabled, the MQTT reporter reported the url to the build's result page in Zuul. As MQTT is meant to be consumed by machines, this broke use cases like log post processing. * This was fixed so that the <mqtt schema>.buildset.builds.log_url now always contains the log url, while an additional field <mqtt schema>.buildset.builds.web_url contains the url to the build's result page if tenant.report-build-page is enabled.

1 0