Open Stack

Hi,

On Fri, Oct 4, 2019, at 04:43, Waines, Greg wrote:
>  
> Hey Colleen,
> 
> 
> Ildiko asked me to summarize StarlingX’s experiences with using 
> Keystone in STX R2.0 .
> 
> Specifically summarize StarlingX’s experiences with using Keystone for 
> Kubernetes API Authentication and Docker Registry Authentication.
> 
> 

[snipped]

> 
> *
>  * StarlingX Services authenticating with Keystone    * No issues ... 
> did it in STX 1.0 ... so no changes required.
> 
>  * Docker Registry authenticating with Keystone    * This link: 
> https://docs.docker.com/registry/spec/auth/token/ 
>  describes how Docker Registry token authentication works
>    * The Authentication/Authorization Service in the above link’s 
> diagram, 
>  MUST be a JSON Web Token Server supporting OAUTH2 interface to docker,
>    * Docker provides an EXAMPLE of such a Token Server in its github:
> https://github.com/docker/distribution/tree/master/contrib/token-server 
> ,
>  which can be copied and updated to handle your preferred authn/z 
> backend.      * Docker guy we spoke to was kind of surprised someone 
> hadn’t done this already for keystone.
>    * So StarlingX started with this example and built a 
>  JSON Web Token Server supporting OAUTH2 interface to docker and with a 
> backend talking to Keystone      * Located here:  
> https://opendev.org/starlingx/containers/src/branch/master/registry-token-server 
>    * ... we are still doing this currently.

Keystone has supported JSON Web Tokens since the Stein release:

https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#jws-tokens
http://specs.openstack.org/openstack/keystone-specs/specs/keystone/stein/json-web-tokens.html

Can you say more about why this doesn't meet the needs for Docker Registry? One of the driving factors for this feature was increased interoperability with non-OpenStack services and it would be great to know what we can improve to achieve that goal.

> 
>  * Kubernetes API authenticating with Keystone    * So we were using 
> the Keystone Authentication and Authorization Webhooks and 
>  the Client-Keystone-Auth 
>  defined here: https://github.com/kubernetes/cloud-provider-openstack 
>    * I presented this solution in this presentation at the DENVER 
> OPENSTACK SUMMIT  
> https://www.dropbox.com/s/gnyf3mtyf5janq1/KubernetesMultiTenancyWithKeystone.pptx?dl=0      * See slides 6,7,8
> 
>    * This did not work out-of-the box circa early-2019,
>  we had to fix the following issues:      * Had to add Authentication 
> to Armada (upstreamed)        * StarlingX used Armada to manage set of 
> helm charts for openstack
>        * Armada did NOT support kubernetes authentication 
>        * StarlingX team added it and upstreamed it
>      * Had to fix lock feature for Authentication in Armada 
> (upstreamed)        * After previous upstream commit to support 
> authentication in armada,
>        * A commit went in to Armada to support locking (for atomic 
> transactions I think),
>  and the commit forgot to take into account authentication,
>        * StarlingX team fixed this commit upstream.
>      * Had issues with expiring tokens with the client-keystone-auth    
>     * I believe that the client-auth-plugin was using a cached keystone 
> client that 
>  was holding on to tokens that eventually expired
>        * We put in some workaround for this ... can’t remember details.
> 
>    * Eventually we “generally” had this working ... except for a 
> specific but-easily-repeatable use case, where the 
>  system would hang or fail in client-keystone-auth (being called from 
> the StarlingX systemInventory process)      * It was a path in the 
> client-keystone-auth, where the code was trying to refresh the token in 
> the kubernetes client,
>      * Investigation suggested that the client-keystone-auth was not 
> compatible with our version of 
>  eventlet(0.18.4) ... which is a few years old,
>      * There was no way to update the token through “proper” methods. 
> Attempting to delete the client 
>  caused the code to mysteriously freeze. If you tried to set the client 
> to none on 1 line then set it to a new 
>  client in another line, the code would freeze as you try to set the 
> client to none.
>      * we managed to get it to work by digging through the internals of 
> the client and setting a new token that 
>  way, but this was somewhat of a hack.
>      * Since eventlet was used everywhere and we were getting close to 
> STX 2.0 release date, 
>  we felt like it was too much risk to try and upversion it.
>    * So, we abandoned the “Kubernetes API authentication with Keystone” 
> work.

If I read this correctly, there were bugs and UX issues in the client-keystone-auth tool (which is not an OpenStack tool) that prevented StarlingX from using it. Are there improvements that are needed for the keystone server project itself or to the documentation that would help improve the usability of this client? Are there bug reports raised with the maintainers of that tool that the keystone team could help follow up on?

> 
>  * StarlingX Roadmap wrt Kubernetes API Authentication    * Given our 
> struggles in getting Keystone working ... a little uneasy in picking it 
> up again,
>    * Also Authentication requirements from StarlingX 2.0 Users of the 
> Kubernetes Layer only,
>  are asking for things like:      * OpenID Connect (OIDC, OAuth2) type 
> authentication,
>      * LDAP
>      * SAML

Keystone supports all of these:

https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html
https://docs.openstack.org/keystone/latest/admin/configuration.html#integrate-identity-with-ldap

>      * Integration with specific authentication platforms like:        
> * GitHub,
>        * LinkedIn,
>        * Azure, 
>        * Windows Active Directory 

Keystone doesn't have specific support for any proprietary platforms but if these providers use an open protocol for which a web server auth module exists (eg. mod_auth_openidc, mod_shib, etc) then these can be configured like any other federated identity provider in the above documentation.

>    * I suspect many of these would require development in Keystone to 
> support.
>    * Whereas there are other existing opensource solutions that can 
> provide that now.
> 
> Greg.
> 
>

Thanks for sharing, apologies for responding so late (I don't pay much attention to this mailing list and my mail filter filtered this away even though you CC'd me directly). I would suggest that future discussions about keystone use the openstack-discuss mailing list as that will get the attention of more keystone and OpenStack developers.

Colleen