Open Stack

Hey Colleen,

Ildiko asked me to summarize StarlingX’s experiences with using Keystone in STX R2.0 .
Specifically summarize StarlingX’s experiences with using Keystone for Kubernetes API Authentication and Docker Registry Authentication.

Greg.


Background:

In STX 1.0

  *   StarlingX managed a Baremetal OpenStack Cloud,
  *   All the OpenStack Services, e.g. neutron, nova, cinder, etc. ...
including StarlingX itself, used Keystone to authenticate,

In STX 2.0

  *   StarlingX changed to manage a Baremetal Kubernetes Cluster,
with a Containerized OpenStack Cloud running on top of Kubernetes.
  *   The solution is now layered
     *   StarlingX Kubernetes
        *   Keystone
        *   StarlingX  ... authenticating with the Platform Keystone
        *   Kubernetes ... authenticating with the Platform Keystone (intent ... but abandoned)
        *   Docker Registry ... authenticating with the Platform Keystone
     *   OPTIONAL Containerized OpenStack
        *   Keystone,
        *   Nova, Neutron, Glance, Cinder, Horizon, Heat
           *   ... all authenticating with the OpenStack Keystone
        *   Barbican, Telemetry (Ceilometer, Panko, Gnocchi, Aodh), Ironic
           *   ... all authenticating with the OpenStack Keystone


Use of Keystone in “StarlingX Kubernetes”:


  *   Ironically we were using Keystone for authentication of a Baremetal Kubernetes Cluster
     *   Normally this would be done for a Kubernetes Cluster running on top of OpenStack
     *   i.e. the Kubernetes Cluster using services of the underlying cloud provider
        *   e.g. using the cloud provider’s authentication services, load balancing services, storage services, etc.
     *   ( This doesn’t have any relevance to this discussion ... just thought I’d point it out. )

  *   StarlingX Services authenticating with Keystone
     *   No issues ... did it in STX 1.0 ... so no changes required.

  *   Docker Registry authenticating with Keystone
     *   This link: https://docs.docker.com/registry/spec/auth/token/
describes how Docker Registry token authentication works
     *   The Authentication/Authorization Service in the above link’s diagram,
MUST be a JSON Web Token Server supporting OAUTH2 interface to docker,
     *   Docker provides an EXAMPLE of such a Token Server in its github:
https://github.com/docker/distribution/tree/master/contrib/token-server ,
which can be copied and updated to handle your preferred authn/z backend.
        *   Docker guy we spoke to was kind of surprised someone hadn’t done this already for keystone.
     *   So StarlingX started with this example and built a
JSON Web Token Server supporting OAUTH2 interface to docker and with a backend talking to Keystone
        *   Located here: https://opendev.org/starlingx/containers/src/branch/master/registry-token-server
     *   ... we are still doing this currently.

  *   Kubernetes API authenticating with Keystone
     *   So we were using the Keystone Authentication and Authorization Webhooks and
                                the Client-Keystone-Auth
defined here:  https://github.com/kubernetes/cloud-provider-openstack
     *   I presented this solution in this presentation at the DENVER OPENSTACK SUMMIT https://www.dropbox.com/s/gnyf3mtyf5janq1/KubernetesMultiTenancyWithKeystone.pptx?dl=0
        *   See slides 6,7,8

     *   This did not work out-of-the box circa early-2019,
we had to fix the following issues:
        *   Had to add Authentication to Armada  (upstreamed)
           *   StarlingX used Armada to manage set of helm charts for openstack
           *   Armada did NOT support kubernetes authentication
           *   StarlingX team added it and upstreamed it
        *   Had to fix lock feature for Authentication in Armada (upstreamed)
           *   After previous upstream commit to support authentication in armada,
           *   A commit went in to Armada to support locking (for atomic transactions I think),
and the commit forgot to take into account authentication,
           *   StarlingX team fixed this commit upstream.
        *   Had issues with expiring tokens with the client-keystone-auth
           *   I believe that the client-auth-plugin was using a cached keystone client that
was holding on to tokens that eventually expired
           *   We put in some workaround for this ... can’t remember details.

     *   Eventually we “generally” had this working ... except for a specific but-easily-repeatable use case, where the
system would hang or fail in client-keystone-auth (being called from the StarlingX systemInventory process)
        *   It was a path in the client-keystone-auth, where the code was trying to refresh the token in the kubernetes client,
        *   Investigation suggested that the client-keystone-auth was not compatible with our version of
eventlet(0.18.4) ... which is a few years old,
        *   There was no way to update the token through “proper” methods. Attempting to delete the client
caused the code to mysteriously freeze. If you tried to set the client to none on 1 line then set it to a new
client in another line, the code would freeze as you try to set the client to none.
        *   we managed to get it to work by digging through the internals of the client and setting a new token that
way, but this was somewhat of a hack.
        *   Since eventlet was used everywhere and we were getting close to STX 2.0 release date,
we felt like it was too much risk to try and upversion it.
     *   So, we abandoned the “Kubernetes API authentication with Keystone” work.

  *   StarlingX Roadmap wrt Kubernetes API Authentication
     *   Given our struggles in getting Keystone working ... a little uneasy in picking it up again,
     *   Also Authentication requirements from StarlingX 2.0 Users of the Kubernetes Layer only,
are asking for things like:
        *   OpenID Connect (OIDC, OAuth2) type authentication,
        *   LDAP
        *   SAML
        *   Integration with specific authentication platforms like:
           *   GitHub,
           *   LinkedIn,
           *   Azure,
           *   Windows Active Directory
     *   I suspect many of these would require development in Keystone to support.
     *   Whereas there are other existing opensource solutions that can provide that now.

Greg.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/edge-computing/attachments/20191004/aa320dd3/attachment-0001.html>