Roadmap for running jobs on OpenShift
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Kube config integration in zuul-executor ---------------------------------------- As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
The Openshift drivers --------------------- https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver.
Inventory update by job ----------------------- We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092
Roles and parent jobs --------------------- The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community.
Define the security context --------------------------- It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
How many users are interested in these topics and what are your review bandwidths? Can we get an estimated release date for OpenShift users?
Regards, -Tristan
Hi Tristan,
I totally agree with the security considerations you mentioned.
As already mentioned this couple of times I am interested in Openshift&Zuul love from both sides and will try to spend some time reviewing mentioned changes next week.
Thanks for your work. Artem (gtema)
On Sun, 18 Nov 2018, 10:34 Tristan Cacqueray, tdecacqu@redhat.com wrote:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
The Openshift drivers
https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver.
Inventory update by job
We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092
Roles and parent jobs
The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community.
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
How many users are interested in these topics and what are your review bandwidths? Can we get an estimated release date for OpenShift users?
Regards, -Tristan _______________________________________________ Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
On 2018-11-18 09:34:24 +0000 (+0000), Tristan Cacqueray wrote: [...]
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
[...]
Would using Kata for the container runtime solve this dilemma by still allowing jobs to do things like install additional packages under execution without exposing the host's kernel to the workload? We've discovered over the years that with a diverse job set, preinstalling everything a job might need in images doesn't scale (and neither does trying to maintain job-specific images).
On Sun, 18 Nov 2018, 12:41 Jeremy Stanley, fungi@yuggoth.org wrote:
On 2018-11-18 09:34:24 +0000 (+0000), Tristan Cacqueray wrote: [...]
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the
same.
I strongly disagree with this approach because containers that run as
root
have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift
cluster
shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set
the
expectation immediately and keep users safe.
[...]
Would using Kata for the container runtime solve this dilemma by still allowing jobs to do things like install additional packages under execution without exposing the host's kernel to the workload? We've discovered over the years that with a diverse job set, preinstalling everything a job might need in images doesn't scale (and neither does trying to maintain job-specific images).
While it might be generally an option there is one certain limitation to remember about Kata - it requires nested virtualization, which might not be available in the Openshift/K8 cluster. This is at least something I have faced now and was discussing with Kata people. Therefore I am personally not sure it would be a right way to put such a limitation/requirement.
Artem
On November 18, 2018 12:16 pm, Artem Goncharov wrote:
On Sun, 18 Nov 2018, 12:41 Jeremy Stanley, fungi@yuggoth.org wrote:
On 2018-11-18 09:34:24 +0000 (+0000), Tristan Cacqueray wrote: [...]
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the
same.
I strongly disagree with this approach because containers that run as
root
have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift
cluster
shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set
the
expectation immediately and keep users safe.
[...]
Would using Kata for the container runtime solve this dilemma by still allowing jobs to do things like install additional packages under execution without exposing the host's kernel to the workload? We've discovered over the years that with a diverse job set, preinstalling everything a job might need in images doesn't scale (and neither does trying to maintain job-specific images).
Perhaps it is possible to install job requirements without root access, e.g. make bindep unpack the header and tool in ~/.local ?
While it might be generally an option there is one certain limitation to remember about Kata - it requires nested virtualization, which might not be available in the Openshift/K8 cluster. This is at least something I have faced now and was discussing with Kata people. Therefore I am personally not sure it would be a right way to put such a limitation/requirement.
Kata or KubeVirt might be a solution but it would be quite a limitation for some users...
-Tristan
On 18/11/2018 10.34, Tristan Cacqueray wrote:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Can we make this work on any distribution of Kubernetes? Or do you need specific things that will work only on OpenShift?
Andreas
[...]
On November 18, 2018 2:31 pm, Andreas Jaeger wrote:
On 18/11/2018 10.34, Tristan Cacqueray wrote:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Can we make this work on any distribution of Kubernetes? Or do you need specific things that will work only on OpenShift?
The "Kube config integration in zuul-executor", "Inventory update by job" and "Define the security context" topics are relevant for any distribution of Kubernetes.
OpenShift comes with extended security and image building processes that should be taken into account for a more complete integration. Thus the "Roles and parent jobs" topic is more challenging to support any distribution.
-Tristan
Tristan Cacqueray tdecacqu@redhat.com writes:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
The Openshift drivers
https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver.
Inventory update by job
We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092
Roles and parent jobs
The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community.
Thanks for the road map -- those all seem like good next steps. I'll leave detailed feedback to the reviews, but I very much agree with the sentiment in the last item.
I also believe that we should be able to get a k8s up and running to attach to OpenStack's Zuul soon -- apparently one of our clouds has a Magnum we can use. That's not necessary for any of this, but it might make some collaboration a little easier.
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
I suspect this is going to take a bit of discussion to get through.
To start with, I think we're proceeding from different base assumptions. You say that "containers that run as root have far more permissions than their workload requires" -- that's not certain to be true. That may be true for some workloads, but certainly not all.
I don't think a job running as root in a container is substantially different than a job running in a VM as root. And being able to run jobs as root is one of Zuul's strong points. We get a lot of users because of that ability.
And it's far from unusual to run containerized processes as the root user. I just spun up a pod with an ssh daemon running as root inside of GKE (perhaps the "reference" kubernetes environment?) using only the defaults, and it works fine.
I agree that we should set expectations early -- and one of those is that Zuul will run whatever workloads your underlying infrastructure will support.
Now, of course, some folks are likely going to want to run jobs on containers (and VMs) without root access. And like VMs, we should support those in containers too. But if those jobs require special images, then those folks are going to have to do more work to make those images.
To get concrete for a minute, consider the tox-py35 job. If it runs on a VM without necessary dependencies installed, it will install them, as root. So it doesn't assume root privileges, unless it is missing dependencies. The same approach should work on containers. In either case, if you want to run on a VM or a container without root, you must supply an image with the necessary dependencies.
The main thing is that support for root-privileged workloads is a first-class feature of Zuul, and that's consistent for containers and VMs. With that out of the way, I'm happy for us to work on supporting both use-cases. The easiest way to proceed will be for us to start by assuming we have root access. Then I think we can work on recommended processes for folks who want a more restricted environment.
-Jim
On 19.11.18, 18:38, "James E. Blair" corvus@inaugust.com wrote:
Tristan Cacqueray tdecacqu@redhat.com writes: ... > Define the security context > --------------------------- > It seems like opinions diverge on the security requirements for running > untrusted jobs in pods. At the summit, I was told that we should keep on > using the root user so that bindep and other service operations stay the same. > I strongly disagree with this approach because containers that run as root > have far more permissions than their workload requires and this could be > abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster > shall be restricted to non-root users to mitigate privilege escallation. > Even though this can be discussed later, I think it's important to set the > expectation immediately and keep users safe.
I suspect this is going to take a bit of discussion to get through.
To start with, I think we're proceeding from different base assumptions. You say that "containers that run as root have far more permissions than their workload requires" -- that's not certain to be true. That may be true for some workloads, but certainly not all.
I don't think a job running as root in a container is substantially different than a job running in a VM as root. And being able to run jobs as root is one of Zuul's strong points. We get a lot of users because of that ability.
And it's far from unusual to run containerized processes as the root user. I just spun up a pod with an ssh daemon running as root inside of GKE (perhaps the "reference" kubernetes environment?) using only the defaults, and it works fine.
I agree that we should set expectations early -- and one of those is that Zuul will run whatever workloads your underlying infrastructure will support.
Now, of course, some folks are likely going to want to run jobs on containers (and VMs) without root access. And like VMs, we should support those in containers too. But if those jobs require special images, then those folks are going to have to do more work to make those images.
To get concrete for a minute, consider the tox-py35 job. If it runs on a VM without necessary dependencies installed, it will install them, as root. So it doesn't assume root privileges, unless it is missing dependencies. The same approach should work on containers. In either case, if you want to run on a VM or a container without root, you must supply an image with the necessary dependencies.
The main thing is that support for root-privileged workloads is a first-class feature of Zuul, and that's consistent for containers and VMs. With that out of the way, I'm happy for us to work on supporting both use-cases. The easiest way to proceed will be for us to start by assuming we have root access. Then I think we can work on recommended processes for folks who want a more restricted environment.
I agree with this. However I have to note that the default on OpenShift is to not allow root. I don't see real problems with that at first glance as this only covers what jobs can do. In order to cope with this I can think of three things we probably should do.
The first is that we should document the fact that a specific job or role in zuul-jobs needs root access. This is an issue with static nodes too as they don't necessarily have root access. Maybe we could enhance zuul-sphinx with a tag for this.
The second is that we could add support in nodepool to specify the scc to use. This makes it easier for openshift admins to allow root for nodepool pods without having to modify the default scc and makes it much easier to justify root (!= privileges, openshift is a bit paranoid about this) because changing the default scc in a multi tenant environment is a no-go.
The third is to make a big note about the openshift behavior in the docs so users know that they either have no root or need to modify the default scc or need to configure nodepool with option 2 if we implement that.
- Tobias
On 2018-11-19 18:32:29 +0000 (+0000), Tobias.Henkel@bmw.de wrote: [...]
The first is that we should document the fact that a specific job or role in zuul-jobs needs root access. This is an issue with static nodes too as they don't necessarily have root access. Maybe we could enhance zuul-sphinx with a tag for this.
[...]
I like this. If we had some standardized means of marking jobs/playbooks/roles as requiring root privs and a means of testing for those which don't but attempt to use it anyway, then we might be able to also reliably early-fail a job with a more helpful error directly from Zuul if it or one of its components needs root but is targeting a backend which doesn't permit that.
On November 19, 2018 7:18 pm, Jeremy Stanley wrote:
On 2018-11-19 18:32:29 +0000 (+0000), Tobias.Henkel@bmw.de wrote: [...]
The first is that we should document the fact that a specific job or role in zuul-jobs needs root access. This is an issue with static nodes too as they don't necessarily have root access. Maybe we could enhance zuul-sphinx with a tag for this.
[...]
I like this. If we had some standardized means of marking jobs/playbooks/roles as requiring root privs and a means of testing for those which don't but attempt to use it anyway, then we might be able to also reliably early-fail a job with a more helpful error directly from Zuul if it or one of its components needs root but is targeting a backend which doesn't permit that.
This may be difficult if the information is in the roles since Zuul doesn't know what roles a job use. How difficult would it be for the scheduler to discover what are the roles associated with any given job?
-Tristan
On 2018-11-20 04:04:17 +0000 (+0000), Tristan Cacqueray wrote:
On November 19, 2018 7:18 pm, Jeremy Stanley wrote:
On 2018-11-19 18:32:29 +0000 (+0000), Tobias.Henkel@bmw.de wrote: [...]
The first is that we should document the fact that a specific job or role in zuul-jobs needs root access. This is an issue with static nodes too as they don't necessarily have root access. Maybe we could enhance zuul-sphinx with a tag for this.
[...]
I like this. If we had some standardized means of marking jobs/playbooks/roles as requiring root privs and a means of testing for those which don't but attempt to use it anyway, then we might be able to also reliably early-fail a job with a more helpful error directly from Zuul if it or one of its components needs root but is targeting a backend which doesn't permit that.
This may be difficult if the information is in the roles since Zuul doesn't know what roles a job use. How difficult would it be for the scheduler to discover what are the roles associated with any given job?
Oh, right. If we want to avoid Zuul abusing Ansible internals to identify jobs using those roles then we would need to stick to having any needs-root-access flag as a job-level construct.
On November 19, 2018 6:32 pm, Tobias.Henkel@bmw.de wrote:
The second is that we could add support in nodepool to specify the scc to use. This makes it easier for openshift admins to allow root for nodepool pods without having to modify the default scc and makes it much easier to justify root (!= privileges, openshift is a bit paranoid about this) because changing the default scc in a multi tenant environment is a no-go.
Users also suggested we could describe the container spec body and the role binding information in the nodepool configuration. We could extend that mechanism to generic object to be applied to the cluster...
However, changing scc needs admin cluster access which is currently not required. The nodepool openshift driver is currently usable by regular developper account and this is the most frequent use-case.
-Tristan
On Mon, Nov 19, 2018, at 9:37 AM, James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
The Openshift drivers
https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver.
Inventory update by job
We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092
Roles and parent jobs
The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community.
Thanks for the road map -- those all seem like good next steps. I'll leave detailed feedback to the reviews, but I very much agree with the sentiment in the last item.
I also believe that we should be able to get a k8s up and running to attach to OpenStack's Zuul soon -- apparently one of our clouds has a Magnum we can use. That's not necessary for any of this, but it might make some collaboration a little easier.
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
I suspect this is going to take a bit of discussion to get through.
To start with, I think we're proceeding from different base assumptions. You say that "containers that run as root have far more permissions than their workload requires" -- that's not certain to be true. That may be true for some workloads, but certainly not all.
I don't think a job running as root in a container is substantially different than a job running in a VM as root. And being able to run jobs as root is one of Zuul's strong points. We get a lot of users because of that ability.
And it's far from unusual to run containerized processes as the root user. I just spun up a pod with an ssh daemon running as root inside of GKE (perhaps the "reference" kubernetes environment?) using only the defaults, and it works fine.
Worth noting that on older GKE clusters the pods had root access to the entire cloud account (maybe not entire account, but to lots of fun things). Root or not in the container itself this would've been bad to expose to CI jobs. They have since clamped this down and it should be fine if you create a new cluster today.
I do think this illustrates that we will likely need to express the security concerns that must be accounted for when running Zuul jobs in a way that is consumable by users such that they can accommodate Zuul in a way that makes sense for their use cases. Simply disallowing root is likely not enough and may not be appropriate in some cases.
I agree that we should set expectations early -- and one of those is that Zuul will run whatever workloads your underlying infrastructure will support.
Now, of course, some folks are likely going to want to run jobs on containers (and VMs) without root access. And like VMs, we should support those in containers too. But if those jobs require special images, then those folks are going to have to do more work to make those images.
To get concrete for a minute, consider the tox-py35 job. If it runs on a VM without necessary dependencies installed, it will install them, as root. So it doesn't assume root privileges, unless it is missing dependencies. The same approach should work on containers. In either case, if you want to run on a VM or a container without root, you must supply an image with the necessary dependencies.
The main thing is that support for root-privileged workloads is a first-class feature of Zuul, and that's consistent for containers and VMs. With that out of the way, I'm happy for us to work on supporting both use-cases. The easiest way to proceed will be for us to start by assuming we have root access. Then I think we can work on recommended processes for folks who want a more restricted environment.
++
More and more CI tools are constraining the problem set they can accommodate. While this may help them avoid dealing with these difficult problems, one of the things that makes Zuul useful to the OpenStack project is that it attempts to address these issues instead. I see this as an opportunity for Zuul if the alternatives push towards "run $command in $prebuildimage" and force that on users.
Clark
On November 19, 2018 6:32 pm, Clark Boylan wrote:
On Mon, Nov 19, 2018, at 9:37 AM, James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
Hello everyone,
Please find below a list of topics I think we should address in order to land the OpenShift driver.
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
The Openshift drivers
https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver.
Inventory update by job
We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092
Roles and parent jobs
The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community.
Thanks for the road map -- those all seem like good next steps. I'll leave detailed feedback to the reviews, but I very much agree with the sentiment in the last item.
I also believe that we should be able to get a k8s up and running to attach to OpenStack's Zuul soon -- apparently one of our clouds has a Magnum we can use. That's not necessary for any of this, but it might make some collaboration a little easier.
That would help curate the roles and jobs upstream if they can be tested directly. Would it be possible to setup an Openshift too?
Define the security context
It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe.
I suspect this is going to take a bit of discussion to get through.
To start with, I think we're proceeding from different base assumptions. You say that "containers that run as root have far more permissions than their workload requires" -- that's not certain to be true. That may be true for some workloads, but certainly not all.
Root privileges are quite extensive and we should refine what it means for Zuul jobs. It seems like we mostly want filesystem capabilities, but root privilege also covers: * kernel auditing * ptrace any process * create special files (mknod) * network interface and firewall administration * raw packets * change process uid * reboot and use kexec * create chroot * load kernel module * change system clock * change process priority
I don't think container workload should require any of the above privileges.
I don't think a job running as root in a container is substantially different than a job running in a VM as root. And being able to run jobs as root is one of Zuul's strong points. We get a lot of users because of that ability.
And it's far from unusual to run containerized processes as the root user. I just spun up a pod with an ssh daemon running as root inside of GKE (perhaps the "reference" kubernetes environment?) using only the defaults, and it works fine.
Worth noting that on older GKE clusters the pods had root access to the entire cloud account (maybe not entire account, but to lots of fun things). Root or not in the container itself this would've been bad to expose to CI jobs. They have since clamped this down and it should be fine if you create a new cluster today.
I do think this illustrates that we will likely need to express the security concerns that must be accounted for when running Zuul jobs in a way that is consumable by users such that they can accommodate Zuul in a way that makes sense for their use cases. Simply disallowing root is likely not enough and may not be appropriate in some cases.
K8S may authorize such privileges by default, but using default settings doesn't seem secure for running multi-tenant workloads. I am not familiar with K8S but we should at least suggest Zuul operators follow: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/
I agree that we should set expectations early -- and one of those is that Zuul will run whatever workloads your underlying infrastructure will support.
Now, of course, some folks are likely going to want to run jobs on containers (and VMs) without root access. And like VMs, we should support those in containers too. But if those jobs require special images, then those folks are going to have to do more work to make those images.
To get concrete for a minute, consider the tox-py35 job. If it runs on a VM without necessary dependencies installed, it will install them, as root. So it doesn't assume root privileges, unless it is missing dependencies. The same approach should work on containers. In either case, if you want to run on a VM or a container without root, you must supply an image with the necessary dependencies.
The main thing is that support for root-privileged workloads is a first-class feature of Zuul, and that's consistent for containers and VMs. With that out of the way, I'm happy for us to work on supporting both use-cases. The easiest way to proceed will be for us to start by assuming we have root access. Then I think we can work on recommended processes for folks who want a more restricted environment.
++
More and more CI tools are constraining the problem set they can accommodate. While this may help them avoid dealing with these difficult problems, one of the things that makes Zuul useful to the OpenStack project is that it attempts to address these issues instead. I see this as an opportunity for Zuul if the alternatives push towards "run $command in $prebuildimage" and force that on users.
Can we please proceed with assuming the least privileges first? Openshift does not allow root-privileges by default, and I would like to ensure that Zuul can be used in the default configuration.
The dependencies management you mentioned can be addressed using BuildConfig. Alternatively, we can update bindep to be usable without system packages, for example, it could extract headers and applications in ~/.local.
Assuming root-privileges in containers does not sound like the right approach, especially when Zuul can execute malicious tasks (e.g. if untrusted-projects with "include: job" are hosted publicly). Supporting that feature warrants a big warning regarding the risk of later exploitation, and we should strongly recommend Openstack or Kata containers be used instead.
Regards, -Tristan
On November 18, 2018 9:34 am, Tristan Cacqueray wrote:
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
[...]
Can we get an estimated release date for OpenShift users?
Hi,
It has been 20 days since the last reply and only one review has been made. Can we please make a tentative schedule for Openshift support?
Drone.io[0] recently announced official support for k8s and there is argoproj[1] which is also building a ci/cd setup for k8s. I think we are only missing the zuul-executor patch to be able to implement similar jobs in Zuul.
Regards, -Tristan
[0]: https://blog.drone.io/drone-goes-kubernetes-native/ [1]: https://argoproj.github.io/
Tristan Cacqueray tdecacqu@redhat.com writes:
On November 18, 2018 9:34 am, Tristan Cacqueray wrote:
Kube config integration in zuul-executor
As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668
[...]
Can we get an estimated release date for OpenShift users?
Hi,
It has been 20 days since the last reply and only one review has been made. Can we please make a tentative schedule for Openshift support?
Drone.io[0] recently announced official support for k8s and there is argoproj[1] which is also building a ci/cd setup for k8s. I think we are only missing the zuul-executor patch to be able to implement similar jobs in Zuul.
I believe the next step on your roadmap is https://review.openstack.org/570668 -- I've reviewed that with only minor suggestions; I think we're pretty close.
Thanks,
Jim
participants (7)
-
Andreas Jaeger
-
Artem Goncharov
-
Clark Boylan
-
corvus＠inaugust.com
-
Jeremy Stanley
-
Tobias.Henkel＠bmw.de
-
Tristan Cacqueray