Should we drop our custom Ansible plugins?
Hi,
When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor).
To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous.
Broadly speaking, there are two classes of actions we might want to block:
1) Actions which may expose private information stored on the executor (including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class.
Here's more background on the bubblewrap addition:
https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-executo...
Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here.
Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use.
Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure.
That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility.
Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter.
Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth.
What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins?
-Jim
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
Hi,
When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor).
To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous.
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class.
Here's more background on the bubblewrap addition:
https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-executo...
Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here.
Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use.
Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure.
That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility.
Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth.
What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins?
I do like the additional layer of protection we added to ansible, but not really surprised we are wanting to remove it. As we get better at doing things with zuul, we've needed to relax the restrictions on some modules to use (eg: add_host / known_hosts recently). What would it look like, if a zuul operator wanted to manage this additional security themselves? Or is that something we'd even consider. For example, in zuul 3.3.1[1] we released a point release with additional filtering for add_host and SSH, we'd now be saying this was actually okay, but still could be an attack vector right?
Are connection plugins restrictions still in place or will they also be removed?
[1] https://zuul-ci.org/docs/zuul/releasenotes.html#relnotes-3-3-1-security-issu...
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Paul Belanger pabelanger@redhat.com writes:
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
...
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
With respect to the first class, what you describe would not be significantly different. We already run Ansible in a container; running it in a different container would not change much.
With respect to the second, it would mean that an operator would be able to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Now is a good time to clarify that there are *significant* technical problems and potential loss of functionality with that approach, and there are no current plans to support it.
I do like the additional layer of protection we added to ansible, but not really surprised we are wanting to remove it. As we get better at doing things with zuul, we've needed to relax the restrictions on some modules to use (eg: add_host / known_hosts recently). What would it look like, if a zuul operator wanted to manage this additional security themselves? Or is that something we'd even consider. For example, in zuul 3.3.1[1] we released a point release with additional filtering for add_host and SSH, we'd now be saying this was actually okay, but still could be an attack vector right?
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
Given that executing arbitrary code on a machine is frequently a path privilege escalation, it might be worth calling that out explicitly. Perhaps that should be a third class of action we're currently trying to prevent.
It's certainly something we paid attention to when we added bubblewrap:
https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-executo...
Are connection plugins restrictions still in place or will they also be removed?
I imagine we would remove them too.
-Jim
On Mon, Mar 11, 2019 at 16:05 James E. Blair wrote:
Paul Belanger pabelanger@redhat.com writes:
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
...
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
With respect to the first class, what you describe would not be significantly different. We already run Ansible in a container; running it in a different container would not change much.
Though Zuul executor job containers are missing cgroup, network namespace and seccomp filter. And we assume the executor process can create containers, which may not be possible in some restricted environment.
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
With respect to the second, it would mean that an operator would be able to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
Now is a good time to clarify that there are *significant* technical problems and potential loss of functionality with that approach, and there are no current plans to support it.
I do like the additional layer of protection we added to ansible, but not really surprised we are wanting to remove it. As we get better at doing things with zuul, we've needed to relax the restrictions on some modules to use (eg: add_host / known_hosts recently). What would it look like, if a zuul operator wanted to manage this additional security themselves? Or is that something we'd even consider. For example, in zuul 3.3.1[1] we released a point release with additional filtering for add_host and SSH, we'd now be saying this was actually okay, but still could be an attack vector right?
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
Regards, -Tristan
Given that executing arbitrary code on a machine is frequently a path privilege escalation, it might be worth calling that out explicitly. Perhaps that should be a third class of action we're currently trying to prevent.
It's certainly something we paid attention to when we added bubblewrap:
https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-executo...
Are connection plugins restrictions still in place or will they also be removed?
I imagine we would remove them too.
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Tristan Cacqueray tdecacqu@redhat.com writes:
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
It adds the requirement of a COE, and one of Zuul's strengths is that it can operate in any environment, bare metal, virtual machine, or container.
There's also the problem of the git repo cache that Paul mentioned. That is far from trivial.
Regarding the container privilege requirement, the best way to remove that would be for someone to identify what the problem is on a technical level. Right now, we know that we have to set "privileged: true" in some situations. Why? What specific technical action performed by bwrap requires that? What capabilities? What seccomp rules?
Basically, what are all of the changes required to get bwrap to work in openshift.
If we knew that, we could look at potential changes to bwrap or future features of the kernel or openshift which may ameliorate the problem.
If you're really interested in this, then please help with that. Understanding the problem is an important first step.
With respect to the second, it would mean that an operator would be able to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
Indeed, we should clarify this. I don't think we can drop our plugins or proceed with a scale-out scheduler if we can't prevent anonymous zk reads. The Jira report involves a custom authentication plugin. Have you tested this with the system we'd be using?
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
That's what bubblewrap does for us today.
-Jim
On Tue, Mar 12, 2019 at 11:57 AM James E. Blair corvus@inaugust.com wrote:
[snip]
With respect to the second, it would mean that an operator would be able
to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
Indeed, we should clarify this. I don't think we can drop our plugins or proceed with a scale-out scheduler if we can't prevent anonymous zk reads. The Jira report involves a custom authentication plugin. Have you tested this with the system we'd be using?
FYI, I did some quick local testing and I don't believe the standard digest/sasl mechanisms are affected. An unauthenticated client can connect and do CRUD ops, but cannot affect any znodes that have appropriate ACLs set on them. So as long as we set the ACLs correctly, I don't see any issue. This will mean we'll need to consider an upgrade path from non-ACL nodes to ACL nodes, but it looks like kazoo provides us a set_acls() API call we use to create a tool to help with that.
-Dave
Hi all,
On Tue, 12 Mar 2019, 18:48 David Shrewsbury, shrews@redhat.com wrote:
On Tue, Mar 12, 2019 at 11:57 AM James E. Blair corvus@inaugust.com wrote:
[snip]
With respect to the second, it would mean that an operator would be able
to worry less about what the Zuul executor has access to, but would
need
to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
Indeed, we should clarify this. I don't think we can drop our plugins or proceed with a scale-out scheduler if we can't prevent anonymous zk reads. The Jira report involves a custom authentication plugin. Have you tested this with the system we'd be using?
FYI, I did some quick local testing and I don't believe the standard digest/sasl mechanisms are affected. An unauthenticated client can connect and do CRUD ops, but cannot affect any znodes that have appropriate ACLs set on them. So as long as we set the ACLs correctly, I don't see any issue. This will mean we'll need to consider an upgrade path from non-ACL nodes to ACL nodes, but it looks like kazoo provides us a set_acls() API call we use to create a tool to help with that.
Doesn't this still leave a DDos or overfill attacks possible then?
On the other hand I would really like to have possibility of keeping and using in-project custom plugins (i.e. project executes some playbooks and use in-project plugin to report statistics to some defined place). This is something I wanted to play with. I haven't tried it yet, but this mail raised concerns in me.
Artem
On Tue, Mar 12, 2019 at 19:09 Artem Goncharov wrote:
Hi all,
On Tue, 12 Mar 2019, 18:48 David Shrewsbury, shrews@redhat.com wrote:
On Tue, Mar 12, 2019 at 11:57 AM James E. Blair corvus@inaugust.com wrote:
[snip]
With respect to the second, it would mean that an operator would be able
to worry less about what the Zuul executor has access to, but would
need
to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
Indeed, we should clarify this. I don't think we can drop our plugins or proceed with a scale-out scheduler if we can't prevent anonymous zk reads. The Jira report involves a custom authentication plugin. Have you tested this with the system we'd be using?
FYI, I did some quick local testing and I don't believe the standard digest/sasl mechanisms are affected. An unauthenticated client can connect and do CRUD ops, but cannot affect any znodes that have appropriate ACLs set on them. So as long as we set the ACLs correctly, I don't see any issue. This will mean we'll need to consider an upgrade path from non-ACL nodes to ACL nodes, but it looks like kazoo provides us a set_acls() API call we use to create a tool to help with that.
Doesn't this still leave a DDos or overfill attacks possible then?
On the other hand I would really like to have possibility of keeping and using in-project custom plugins (i.e. project executes some playbooks and use in-project plugin to report statistics to some defined place). This is something I wanted to play with. I haven't tried it yet, but this mail raised concerns in me.
Another concern is the /zookeeper nodes that would need extra protection. Moreover, note that ZooKeeper ACL are not like regular filesystem permission, they don't implement hierarchy. It may be possible that an anonymous user can create nodes in /nodepool/.
-Tristan
On Tue, Mar 12, 2019 at 08:57 James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
It adds the requirement of a COE, and one of Zuul's strengths is that it can operate in any environment, bare metal, virtual machine, or container.
This could be an optional requirement implemented as an execution context driver.
There's also the problem of the git repo cache that Paul mentioned. That is far from trivial.
Regarding the container privilege requirement, the best way to remove that would be for someone to identify what the problem is on a technical level. Right now, we know that we have to set "privileged: true" in some situations. Why?
As explained in the previous thread about this (install Zuul on K8/openshift) the core issue is that recursive/nested containers are considered dangerous and container runtime like docker disable it by default.
There are progress being made to at least be able to create new namespace inside a container and this is well documented in these slides: https://www.slideshare.net/AkihiroSuda/rootless-containers
However, considering the bad track record of this feature, it may takes time before we can created nested namespace without explicitely using some sort of "privileged: true" or "seccomp: unconfined" options, which may be disabled in multi tenant COE like openshift.
What specific technical action performed by bwrap requires that? What capabilities? What seccomp rules?
The first issue is clone(CLONE_NEWNS). You need the SYS_ADMIN capability. For example a regular user is not able to run "unshare --mount".
Though you can create a user namespace (CLONE_NEWUSER) which grants the SYS_ADMIN capability, thus a regular user can run "unshare --user --mount". Unfortunately this has a lot of security implication.
Then assuming you can create namespaces, you still need access to a proper /proc fs to complete the new mount namespace creation.
Finally you have to overcome the default seccomp/apparmor/SELinux policy which usualy prevent above actions.
More details about this are available in these issues: https://github.com/projectatomic/bubblewrap/issues/269 https://github.com/opencontainers/runc/issues/1658
And finally there seems to be a blocker for user namespace in K8/openshift: https://github.com/kubernetes/enhancements/issues/127
Basically, what are all of the changes required to get bwrap to work in openshift.
Basically, you need these permission today: - https://github.com/gtema/zuul-apb/blob/master/zuul_scc.yaml#L21-L23 If user namespace are available, then you could replace privileged by - enable (through seccomp) the creation of NEWNS|NEWUSER clone. - enable a proper /proc (e.g. unmasked /proc/sys /proc/kcore). - enable setuid/setgid cap for newuidmap/newgidmap needed by user ns creation.
Then when K8S supports user namespace creation, you may be able to drop the setuid/setgit cap.
At least, that is my understanding of the technical issues :-)
Also please note that these only cover namespaces, which is a subset of what "container" usualy means, and bwrap doesn't support cgroup or network isolation.
If we knew that, we could look at potential changes to bwrap or future features of the kernel or openshift which may ameliorate the problem.
If you're really interested in this, then please help with that. Understanding the problem is an important first step.
The core problem with the zuul executor process is that we are running different execution context inside a single pod, which goes against the design of COE environment: With containers you are supposed to split the application into multiple containers each doing one task as opposed to multiple different processes in the same container. This gives you the isolation.
Thus I think nested/recursive containers is great for unprivileged image building, but it's not suitable for the zuul executor.
[snip]
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
That's what bubblewrap does for us today.
I meant, if we are going to enable trusted and untrusted playbook to run with the same workspace, then we may be in a position to use an external pod for the job. IIUC, the issue is that after an untrusted play we have to ensure the trusted content is not tainted, which is the main limitation with regard to implementing a k8s execution context driver.
Regards, -Tristan
Tristan Cacqueray tdecacqu@redhat.com writes:
On Tue, Mar 12, 2019 at 08:57 James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
It adds the requirement of a COE, and one of Zuul's strengths is that it can operate in any environment, bare metal, virtual machine, or container.
This could be an optional requirement implemented as an execution context driver.
That's certainly something to consider, but if it's optional, then it doesn't provide an answer to the question I raised in this thread: Should we drop our custom Ansible plugins?
So, setting aside the fact that you would like a COE driver as a new feature for other reasons, let's bring this back to topic:
Do you think the extra protections of a full container system (i.e., more than bubblewrap) are necessary if we are to drop the custom Ansible plugins?
There's also the problem of the git repo cache that Paul mentioned. That is far from trivial.
Regarding the container privilege requirement, the best way to remove that would be for someone to identify what the problem is on a technical level. Right now, we know that we have to set "privileged: true" in some situations. Why?
As explained in the previous thread about this (install Zuul on K8/openshift) the core issue is that recursive/nested containers are considered dangerous and container runtime like docker disable it by default.
There are progress being made to at least be able to create new namespace inside a container and this is well documented in these slides: https://www.slideshare.net/AkihiroSuda/rootless-containers
However, considering the bad track record of this feature, it may takes time before we can created nested namespace without explicitely using some sort of "privileged: true" or "seccomp: unconfined" options, which may be disabled in multi tenant COE like openshift.
What specific technical action performed by bwrap requires that? What capabilities? What seccomp rules?
The first issue is clone(CLONE_NEWNS). You need the SYS_ADMIN capability. For example a regular user is not able to run "unshare --mount".
Though you can create a user namespace (CLONE_NEWUSER) which grants the SYS_ADMIN capability, thus a regular user can run "unshare --user --mount". Unfortunately this has a lot of security implication.
Then assuming you can create namespaces, you still need access to a proper /proc fs to complete the new mount namespace creation.
Finally you have to overcome the default seccomp/apparmor/SELinux policy which usualy prevent above actions.
More details about this are available in these issues: https://github.com/projectatomic/bubblewrap/issues/269 https://github.com/opencontainers/runc/issues/1658
And finally there seems to be a blocker for user namespace in K8/openshift: https://github.com/kubernetes/enhancements/issues/127
Basically, what are all of the changes required to get bwrap to work in openshift.
Basically, you need these permission today:
If user namespace are available, then you could replace privileged by
- enable (through seccomp) the creation of NEWNS|NEWUSER clone.
- enable a proper /proc (e.g. unmasked /proc/sys /proc/kcore).
- enable setuid/setgid cap for newuidmap/newgidmap needed by user ns creation.
Then when K8S supports user namespace creation, you may be able to drop the setuid/setgit cap.
At least, that is my understanding of the technical issues :-)
Also please note that these only cover namespaces, which is a subset of what "container" usualy means, and bwrap doesn't support cgroup or network isolation.
That is all very helpful information, thanks!
If we knew that, we could look at potential changes to bwrap or future features of the kernel or openshift which may ameliorate the problem.
If you're really interested in this, then please help with that. Understanding the problem is an important first step.
The core problem with the zuul executor process is that we are running different execution context inside a single pod, which goes against the design of COE environment: With containers you are supposed to split the application into multiple containers each doing one task as opposed to multiple different processes in the same container. This gives you the isolation.
Thus I think nested/recursive containers is great for unprivileged image building, but it's not suitable for the zuul executor.
I disagree with your interpretations, but let's not discuss this as a point of philosophy. I don't think that's going to bring us to a resolution.
We can look at this strictly from a technical perspective:
* We will continue to support non-COE Zuul deployments as first-class. * Zuul works as designed today in any COE which allows privileged containers (including dedicated OpenShift so configured). * Executor support for COE could provide some additional advantages in scalability, security, and simplified management. * Executor support for COE could allow Zuul to be used in a shared OpenShift.
Those are reasons enough to consider it. But again, maybe we could do that separately instead of diverting this thread off course?
[snip]
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
That's what bubblewrap does for us today.
I meant, if we are going to enable trusted and untrusted playbook to run with the same workspace, then we may be in a position to use an external pod for the job. IIUC, the issue is that after an untrusted play we have to ensure the trusted content is not tainted, which is the main limitation with regard to implementing a k8s execution context driver.
I am not proposing that we change anything about how trusted and untrusted playbooks are kept separate. They run with the same workspace, but bubblewrap ensures that secrets are only accessible to the individual playbook they're associated with.
The start of the text you quoted was in regards to dropping our custom Ansible plugins, but it seems to me that in your responses you have interpreted it in the context of executor COE support. That's why the conversation is so confused.
The original question I asked is very important, and it's worth us focusing on it. It's fine that, as part of that, we ask "would executor COE support help address the issue?" the answer is that since we will not *require* it, it can not help. So let's set that aside now and explore it in a new thread. Meanwhile, let's get back to analyzing the issues involved in dropping the custom Ansible plugins.
-Jim
On Wed, Mar 13, 2019 at 08:42 James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
On Tue, Mar 12, 2019 at 08:57 James E. Blair wrote:
Tristan Cacqueray tdecacqu@redhat.com writes:
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
It adds the requirement of a COE, and one of Zuul's strengths is that it can operate in any environment, bare metal, virtual machine, or container.
This could be an optional requirement implemented as an execution context driver.
That's certainly something to consider, but if it's optional, then it doesn't provide an answer to the question I raised in this thread: Should we drop our custom Ansible plugins?
So, setting aside the fact that you would like a COE driver as a new feature for other reasons, let's bring this back to topic:
Do you think the extra protections of a full container system (i.e., more than bubblewrap) are necessary if we are to drop the custom Ansible plugins?
I meant if we enable untrusted job to run on localhost, then this should enable such COE driver implementation. However droping the custom Ansible plugins and authorizing arbritary code execution in the context of the executor service sounds like a dangerous feature.
There's also the problem of the git repo cache that Paul mentioned. That is far from trivial.
Regarding the container privilege requirement, the best way to remove that would be for someone to identify what the problem is on a technical level. Right now, we know that we have to set "privileged: true" in some situations. Why?
As explained in the previous thread about this (install Zuul on K8/openshift) the core issue is that recursive/nested containers are considered dangerous and container runtime like docker disable it by default.
There are progress being made to at least be able to create new namespace inside a container and this is well documented in these slides: https://www.slideshare.net/AkihiroSuda/rootless-containers
However, considering the bad track record of this feature, it may takes time before we can created nested namespace without explicitely using some sort of "privileged: true" or "seccomp: unconfined" options, which may be disabled in multi tenant COE like openshift.
What specific technical action performed by bwrap requires that? What capabilities? What seccomp rules?
The first issue is clone(CLONE_NEWNS). You need the SYS_ADMIN capability. For example a regular user is not able to run "unshare --mount".
Though you can create a user namespace (CLONE_NEWUSER) which grants the SYS_ADMIN capability, thus a regular user can run "unshare --user --mount". Unfortunately this has a lot of security implication.
Then assuming you can create namespaces, you still need access to a proper /proc fs to complete the new mount namespace creation.
Finally you have to overcome the default seccomp/apparmor/SELinux policy which usualy prevent above actions.
More details about this are available in these issues: https://github.com/projectatomic/bubblewrap/issues/269 https://github.com/opencontainers/runc/issues/1658
And finally there seems to be a blocker for user namespace in K8/openshift: https://github.com/kubernetes/enhancements/issues/127
Basically, what are all of the changes required to get bwrap to work in openshift.
Basically, you need these permission today:
If user namespace are available, then you could replace privileged by
- enable (through seccomp) the creation of NEWNS|NEWUSER clone.
- enable a proper /proc (e.g. unmasked /proc/sys /proc/kcore).
- enable setuid/setgid cap for newuidmap/newgidmap needed by user ns creation.
Then when K8S supports user namespace creation, you may be able to drop the setuid/setgit cap.
At least, that is my understanding of the technical issues :-)
Also please note that these only cover namespaces, which is a subset of what "container" usualy means, and bwrap doesn't support cgroup or network isolation.
That is all very helpful information, thanks!
I forgot to mention that while these restrictions are enforced by openshift, they are also relevant for multi-tenant k8s. And since zuul executors (and each job they run) are effectively multi-tenant, we should ensure Zuul can work with these restrictions activated.
If we knew that, we could look at potential changes to bwrap or future features of the kernel or openshift which may ameliorate the problem.
If you're really interested in this, then please help with that. Understanding the problem is an important first step.
The core problem with the zuul executor process is that we are running different execution context inside a single pod, which goes against the design of COE environment: With containers you are supposed to split the application into multiple containers each doing one task as opposed to multiple different processes in the same container. This gives you the isolation.
Thus I think nested/recursive containers is great for unprivileged image building, but it's not suitable for the zuul executor.
I disagree with your interpretations, but let's not discuss this as a point of philosophy. I don't think that's going to bring us to a resolution.
We can look at this strictly from a technical perspective:
- We will continue to support non-COE Zuul deployments as first-class.
- Zuul works as designed today in any COE which allows privileged containers (including dedicated OpenShift so configured).
- Executor support for COE could provide some additional advantages in scalability, security, and simplified management.
- Executor support for COE could allow Zuul to be used in a shared OpenShift.
Those are reasons enough to consider it. But again, maybe we could do that separately instead of diverting this thread off course?
It's great to hear you are willing to consider such feature. Then I can propose a plan in another thread.
[snip]
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
That's what bubblewrap does for us today.
I meant, if we are going to enable trusted and untrusted playbook to run with the same workspace, then we may be in a position to use an external pod for the job. IIUC, the issue is that after an untrusted play we have to ensure the trusted content is not tainted, which is the main limitation with regard to implementing a k8s execution context driver.
I am not proposing that we change anything about how trusted and untrusted playbooks are kept separate. They run with the same workspace, but bubblewrap ensures that secrets are only accessible to the individual playbook they're associated with.
The start of the text you quoted was in regards to dropping our custom Ansible plugins, but it seems to me that in your responses you have interpreted it in the context of executor COE support. That's why the conversation is so confused.
Mea culpa I extrapolated the proposal for executor COE support. Please find my reply to the original question in the new thread.
Regards, -Tristan
The original question I asked is very important, and it's worth us focusing on it. It's fine that, as part of that, we ask "would executor COE support help address the issue?" the answer is that since we will not *require* it, it can not help. So let's set that aside now and explore it in a new thread. Meanwhile, let's get back to analyzing the issues involved in dropping the custom Ansible plugins.
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Quoting Tristan Cacqueray (2019-03-11 20:09:24)
On Mon, Mar 11, 2019 at 16:05 James E. Blair wrote:
Paul Belanger pabelanger@redhat.com writes:
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
...
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
With respect to the first class, what you describe would not be significantly different. We already run Ansible in a container; running it in a different container would not change much.
Though Zuul executor job containers are missing cgroup, network namespace and seccomp filter. And we assume the executor process can create containers, which may not be possible in some restricted environment.
A seccomp filter would need to be created for every potential job. Any attempt to make a generic one will have limited benefit as it won't really be blocking much.
What problem that ansible restrictions prevent now, would be prevented by cgroups and/or network namespaces?
Also USER_NS has been in mainline kernels for *years* now. I'm comfortable with that assumption. We also require Python 3.5+, which isn't in a lot of old OS's, so I'd like to think we can continue the aggressive stance on dependencies.
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
Not sure anybody has suggested this isn't possible, but I think what's on the table now is removing the Ansible restrictions without doing external pod execution. Are you suggesting pod execution should be added first?
Also there is no executor privilege requirement if your kernel supports USER_NS.
With respect to the second, it would mean that an operator would be able to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
What a bummer, I didn't know about that.
I believe the workaround for this is to put a TLS terminator in front of Zookeeper that enforces client cert auth. Seems like it would be worth our time to dust off that PR and get it merged and released though.
Now is a good time to clarify that there are *significant* technical problems and potential loss of functionality with that approach, and there are no current plans to support it.
I do like the additional layer of protection we added to ansible, but not really surprised we are wanting to remove it. As we get better at doing things with zuul, we've needed to relax the restrictions on some modules to use (eg: add_host / known_hosts recently). What would it look like, if a zuul operator wanted to manage this additional security themselves? Or is that something we'd even consider. For example, in zuul 3.3.1[1] we released a point release with additional filtering for add_host and SSH, we'd now be saying this was actually okay, but still could be an attack vector right?
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
The current restrictions aren't sufficient anyway. You can write your own Ansible module and do whatever you want that isn't an Ansible action. IMO doing things at this granular level is a neverending time sink, and that's what Jim is suggesting we end: Trust bubblewrap, lock it down further if necessary, and look at your executors for what they are: dangerous remote code executors that need to be watched and controlled.
On Tue, Mar 12, 2019 at 09:57 Clint Byrum wrote:
Quoting Tristan Cacqueray (2019-03-11 20:09:24)
On Mon, Mar 11, 2019 at 16:05 James E. Blair wrote:
Paul Belanger pabelanger@redhat.com writes:
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
...
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
With respect to the first class, what you describe would not be significantly different. We already run Ansible in a container; running it in a different container would not change much.
Though Zuul executor job containers are missing cgroup, network namespace and seccomp filter. And we assume the executor process can create containers, which may not be possible in some restricted environment.
A seccomp filter would need to be created for every potential job. Any attempt to make a generic one will have limited benefit as it won't really be blocking much.
Well the common seccomp filters usualy provides a good defensive layer that shouldn't be available to any given job (such as using SCTP protocol). Here is the one used by docker: https://docs.docker.com/v18.03/engine/security/seccomp/#significant-syscalls...
What problem that ansible restrictions prevent now, would be prevented by cgroups and/or network namespaces?
Cgroup would let us limit the memory and cpu usage. Network namespaces would enable 2 jobs to bind the same port, and we could setup packet filter to prevent job network to access zuul control plane.
Also USER_NS has been in mainline kernels for *years* now. I'm comfortable with that assumption. We also require Python 3.5+, which isn't in a lot of old OS's, so I'd like to think we can continue the aggressive stance on dependencies.
As explained in another mail, USER_NS is not enough to enable using bwrap in k8s.
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
Not sure anybody has suggested this isn't possible, but I think what's on the table now is removing the Ansible restrictions without doing external pod execution. Are you suggesting pod execution should be added first?
I'm suggesting we could have a pod execution context driver that could be used in place of the bubblewrap one.
Also there is no executor privilege requirement if your kernel supports USER_NS.
IIUC, you still need SETUID/SETGID cap to be able to complete the user namespace creation.
Regards, -Tristan
With respect to the second, it would mean that an operator would be able to worry less about what the Zuul executor has access to, but would need to ensure that whatever off-host non-Zuul executor doesn't have access to anything it shouldn't. Basically, it moves the problem. As long as the things that Zuul requires access to (e.g., gearman, zookeeper) can be protected, I don't think it's a significant difference from a security POV.
Note that ZooKeeper access can not be protected without a fix for https://issues.apache.org/jira/browse/ZOOKEEPER-2937 (at least without network isolation between the executor service and the job tasks).
What a bummer, I didn't know about that.
I believe the workaround for this is to put a TLS terminator in front of Zookeeper that enforces client cert auth. Seems like it would be worth our time to dust off that PR and get it merged and released though.
Now is a good time to clarify that there are *significant* technical problems and potential loss of functionality with that approach, and there are no current plans to support it.
I do like the additional layer of protection we added to ansible, but not really surprised we are wanting to remove it. As we get better at doing things with zuul, we've needed to relax the restrictions on some modules to use (eg: add_host / known_hosts recently). What would it look like, if a zuul operator wanted to manage this additional security themselves? Or is that something we'd even consider. For example, in zuul 3.3.1[1] we released a point release with additional filtering for add_host and SSH, we'd now be saying this was actually okay, but still could be an attack vector right?
The vulnerability there is running arbitrary code on the executor, and, if we decide to proceed with this, we would be making the decision that is no longer a vulnerability but is a feature.
That sounds like rather risky, especially since executor are handling job's secrets. How are we going to prevent one play to not taint the one that are protected?
The current restrictions aren't sufficient anyway. You can write your own Ansible module and do whatever you want that isn't an Ansible action. IMO doing things at this granular level is a neverending time sink, and that's what Jim is suggesting we end: Trust bubblewrap, lock it down further if necessary, and look at your executors for what they are: dangerous remote code executors that need to be watched and controlled.
Quoting Tristan Cacqueray (2019-03-12 20:35:48)
On Tue, Mar 12, 2019 at 09:57 Clint Byrum wrote:
Quoting Tristan Cacqueray (2019-03-11 20:09:24)
On Mon, Mar 11, 2019 at 16:05 James E. Blair wrote:
Paul Belanger pabelanger@redhat.com writes:
On Mon, Mar 11, 2019 at 01:47:26PM -0700, James E. Blair wrote:
...
Broadly speaking, there are two classes of actions we might want to block:
- Actions which may expose private information stored on the executor
(including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted).
- Actions which may allow a user to take advantage of privileges
afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services.
One topic I keep hearing from time to time, is moving where the zuul-executor runs ansible-playbook from. Most commonly, I've heard people asking for that to be run inside a pod (off the static zuul-executor host). I understand in our case it is a little more difficult then that, with how we deal with git repo state. Could you speak to if that would provide a better / safer / other place for ansible-playbook to run?
With respect to the first class, what you describe would not be significantly different. We already run Ansible in a container; running it in a different container would not change much.
Though Zuul executor job containers are missing cgroup, network namespace and seccomp filter. And we assume the executor process can create containers, which may not be possible in some restricted environment.
A seccomp filter would need to be created for every potential job. Any attempt to make a generic one will have limited benefit as it won't really be blocking much.
Well the common seccomp filters usualy provides a good defensive layer that shouldn't be available to any given job (such as using SCTP protocol). Here is the one used by docker: https://docs.docker.com/v18.03/engine/security/seccomp/#significant-syscalls...
Indeed. Sometimes it does feel like we're just re-implementing kubernetes and docker with the zuul executor and bubblewrap.
What problem that ansible restrictions prevent now, would be prevented by cgroups and/or network namespaces?
Cgroup would let us limit the memory and cpu usage.
+1, but again, zuul-executor feels like maybe the wrong place to do it if container runtimes already have this.
Network namespaces would enable 2 jobs to bind the same port, and we could setup packet filter to prevent job network to access zuul control plane.
Sounds complicated to get right.
Also USER_NS has been in mainline kernels for *years* now. I'm comfortable with that assumption. We also require Python 3.5+, which isn't in a lot of old OS's, so I'd like to think we can continue the aggressive stance on dependencies.
As explained in another mail, USER_NS is not enough to enable using bwrap in k8s.
I missed that one. How unfortunate. :-/
I wish we would consider using external pods to run jobs, this would removes the executor privilege requirement and simplifies resources scaling.
Not sure anybody has suggested this isn't possible, but I think what's on the table now is removing the Ansible restrictions without doing external pod execution. Are you suggesting pod execution should be added first?
I'm suggesting we could have a pod execution context driver that could be used in place of the bubblewrap one.
I like this train of thought. When I made the interface for bubblewrap, some may remember I added an uncontained execution context because I thought maybe people would prefer to run without bubblewrap in some contexts (it was removed because it was deemed too dangerous). But the abstraction holds, and I think this would be a worthwhile usage of that abstraction. V
Also there is no executor privilege requirement if your kernel supports USER_NS.
IIUC, you still need SETUID/SETGID cap to be able to complete the user namespace creation.
ACK
I'd like to reset this thread because I think we were getting off-topic.
To summarize, there has been some trepidation, some specific concerns, and some support.
The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway.
The specific concerns are:
1) We should ensure that Zookeeper is sufficiently secure that a job can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient.
2) We would be allowing arbitrary code execution which may allow an attacker access to exploit a vulnerability (e.g., in the kernel) on the host.
I've tried to digest the thread so far, but I may have missed something. Are there any other concerns about regressions which this may introduce?
-Jim
On Wed, Mar 13, 2019 at 09:23:10AM -0700, James E. Blair wrote:
I'd like to reset this thread because I think we were getting off-topic.
To summarize, there has been some trepidation, some specific concerns, and some support.
I'm in the cautiously support camp, and see if possible to remove.
The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway.
The specific concerns are:
- We should ensure that Zookeeper is sufficiently secure that a job
can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient.
- We would be allowing arbitrary code execution which may allow an
attacker access to exploit a vulnerability (e.g., in the kernel) on the host.
I admit, this sounds scary, but if we 99.9% believe bubblewrap is going to provide the best protection, this should be fine. I do think our plugins provide another layer of protection, which helps limit the amount of things we can even run on the executor. So far, the majority of jobs haven't needed access to localhost that I have seen.
I've tried to digest the thread so far, but I may have missed something. Are there any other concerns about regressions which this may introduce?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Quoting Paul Belanger (2019-03-13 09:42:46)
On Wed, Mar 13, 2019 at 09:23:10AM -0700, James E. Blair wrote:
I'd like to reset this thread because I think we were getting off-topic.
To summarize, there has been some trepidation, some specific concerns, and some support.
I'm in the cautiously support camp, and see if possible to remove.
The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway.
The specific concerns are:
- We should ensure that Zookeeper is sufficiently secure that a job
can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient.
- We would be allowing arbitrary code execution which may allow an
attacker access to exploit a vulnerability (e.g., in the kernel) on the host.
I admit, this sounds scary, but if we 99.9% believe bubblewrap is going to provide the best protection, this should be fine. I do think our plugins provide another layer of protection, which helps limit the amount of things we can even run on the executor. So far, the majority of jobs haven't needed access to localhost that I have seen.
I've always held the position that the main benefit of the plugins was the friendly error messages. Instead of an EACCESS and whatever code you're executing's terrible error handling, you got a friendly "Hey you can't do that."
But the burden of hooking in to a private API that changes constantly is large.
So maybe we could scope it down a bit. Instead of focusing on blocking in the Ansible plugins, focus on enabling well behaved users with better error handling and early fails for things that we don't like. If we miss one, or if chasing upstream churn is too hard, we can opt not to do it for the next version of Ansible.
But then to make up for the loss in code execution prevention, also take on the work of locking bubblewrap down more and/or adding other container runtimes. That should still be more straight-forward and future-proof than chasing Ansible's APIs.
On Wed, Mar 13, 2019 at 12:42:46PM -0400, Paul Belanger wrote:
On Wed, Mar 13, 2019 at 09:23:10AM -0700, James E. Blair wrote:
I'd like to reset this thread because I think we were getting off-topic.
To summarize, there has been some trepidation, some specific concerns, and some support.
I'm in the cautiously support camp, and see if possible to remove.
Okay, I might have moved directly into the yes camp :)
Reason being, we on ansible-network team have now migrated 100% into zuul for all of our testing. The majority of playbooks we'd like to run from zuul will be using ansible_connection=network_cli along with modules like vyos_config and eos_config.
For now, we are using nested ansible for testing, but we might just be able to use zuul directly, if we are planning on dropping the restriction.
The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway.
The specific concerns are:
- We should ensure that Zookeeper is sufficiently secure that a job
can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient.
- We would be allowing arbitrary code execution which may allow an
attacker access to exploit a vulnerability (e.g., in the kernel) on the host.
I admit, this sounds scary, but if we 99.9% believe bubblewrap is going to provide the best protection, this should be fine. I do think our plugins provide another layer of protection, which helps limit the amount of things we can even run on the executor. So far, the majority of jobs haven't needed access to localhost that I have seen.
I've tried to digest the thread so far, but I may have missed something. Are there any other concerns about regressions which this may introduce?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
On 13.03.19, 17:23, "James E. Blair" corvus@inaugust.com wrote:
I'd like to reset this thread because I think we were getting off-topic.
To summarize, there has been some trepidation, some specific concerns, and some support.
The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway.
The specific concerns are:
1) We should ensure that Zookeeper is sufficiently secure that a job can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient.
2) We would be allowing arbitrary code execution which may allow an attacker access to exploit a vulnerability (e.g., in the kernel) on the host.
What we need to keep in mind is that we must ensure that bwrap drops all privileges the zuul-executor may have. It probably does so but we should double check.
I've tried to digest the thread so far, but I may have missed something. Are there any other concerns about regressions which this may introduce?
-Jim
We also have the (un)trusted mounts setting in the executor which is currently needed so ansible can access windows nodes via winrm. It's afaik not easy to secure those private keys in a similar way like ssh-agent. So those would be freely accessible by the jobs.
I'm currently on the fence about this idea as this means that we're dropping one layer of security. But as already pointed out this layer has many holes in it so it might be really more work maintaining it than it's worth.
- Tobias
On Wed, Mar 13, 2019 at 16:45 Tobias.Henkel@bmw.de wrote:
On 13.03.19, 17:23, "James E. Blair" corvus@inaugust.com wrote:
I'd like to reset this thread because I think we were getting off-topic. To summarize, there has been some trepidation, some specific concerns, and some support. The support tends to come from the perspective that our masking is incomplete enough that we're probably effectively already in this state anyway. The specific concerns are: 1) We should ensure that Zookeeper is sufficiently secure that a job can't just connect to it and start manipulating the data. Tristan and Shrews are looking into that. We should follow up with a definitive answer as to whether the ZK ACLs are sufficient. 2) We would be allowing arbitrary code execution which may allow an attacker access to exploit a vulnerability (e.g., in the kernel) on the host.What we need to keep in mind is that we must ensure that bwrap drops all privileges the zuul-executor may have. It probably does so but we should double check.
Bubblewrap drops capabilities and closes FD by default.
I've tried to digest the thread so far, but I may have missed something. Are there any other concerns about regressions which this may introduce? -JimWe also have the (un)trusted mounts setting in the executor which is currently needed so ansible can access windows nodes via winrm. It's afaik not easy to secure those private keys in a similar way like ssh-agent. So those would be freely accessible by the jobs.
The custom Ansible plugins also prevent access to the ansible fact-cache and cp directory as well as prevent workspace symlink to point at secret which may result in leak.
I'm currently on the fence about this idea as this means that we're dropping one layer of security. But as already pointed out this layer has many holes in it so it might be really more work maintaining it than it's worth.
- Tobias
I'm also on the fence as remote code execution greatly extend the attack surface. Wouldn't it be possible to submit those plugins upstream so that they become an Ansible feature? IIUC the Zuul plugins enable: - command output stream - task blacklist for localhost execution - read/write path whitelist for localhost access
Regards, -Tristan
participants (7)
-
Artem Goncharov
-
Clint Byrum
-
corvus＠inaugust.com
-
David Shrewsbury
-
Paul Belanger
-
Tobias.Henkel＠bmw.de
-
Tristan Cacqueray