From Greg.Waines at windriver.com Fri Oct 4 11:43:23 2019 From: Greg.Waines at windriver.com (Waines, Greg) Date: Fri, 4 Oct 2019 11:43:23 +0000 Subject: [Edge-computing] StarlingX's Use of Keystone for Kubernetes Authentication Message-ID: <5D941121-89C6-4C49-9059-E6D2B2296333@windriver.com> Hey Colleen, Ildiko asked me to summarize StarlingX’s experiences with using Keystone in STX R2.0 . Specifically summarize StarlingX’s experiences with using Keystone for Kubernetes API Authentication and Docker Registry Authentication. Greg. Background: In STX 1.0 * StarlingX managed a Baremetal OpenStack Cloud, * All the OpenStack Services, e.g. neutron, nova, cinder, etc. ... including StarlingX itself, used Keystone to authenticate, In STX 2.0 * StarlingX changed to manage a Baremetal Kubernetes Cluster, with a Containerized OpenStack Cloud running on top of Kubernetes. * The solution is now layered * StarlingX Kubernetes * Keystone * StarlingX ... authenticating with the Platform Keystone * Kubernetes ... authenticating with the Platform Keystone (intent ... but abandoned) * Docker Registry ... authenticating with the Platform Keystone * OPTIONAL Containerized OpenStack * Keystone, * Nova, Neutron, Glance, Cinder, Horizon, Heat * ... all authenticating with the OpenStack Keystone * Barbican, Telemetry (Ceilometer, Panko, Gnocchi, Aodh), Ironic * ... all authenticating with the OpenStack Keystone Use of Keystone in “StarlingX Kubernetes”: * Ironically we were using Keystone for authentication of a Baremetal Kubernetes Cluster * Normally this would be done for a Kubernetes Cluster running on top of OpenStack * i.e. the Kubernetes Cluster using services of the underlying cloud provider * e.g. using the cloud provider’s authentication services, load balancing services, storage services, etc. * ( This doesn’t have any relevance to this discussion ... just thought I’d point it out. ) * StarlingX Services authenticating with Keystone * No issues ... did it in STX 1.0 ... so no changes required. * Docker Registry authenticating with Keystone * This link: https://docs.docker.com/registry/spec/auth/token/ describes how Docker Registry token authentication works * The Authentication/Authorization Service in the above link’s diagram, MUST be a JSON Web Token Server supporting OAUTH2 interface to docker, * Docker provides an EXAMPLE of such a Token Server in its github: https://github.com/docker/distribution/tree/master/contrib/token-server , which can be copied and updated to handle your preferred authn/z backend. * Docker guy we spoke to was kind of surprised someone hadn’t done this already for keystone. * So StarlingX started with this example and built a JSON Web Token Server supporting OAUTH2 interface to docker and with a backend talking to Keystone * Located here: https://opendev.org/starlingx/containers/src/branch/master/registry-token-server * ... we are still doing this currently. * Kubernetes API authenticating with Keystone * So we were using the Keystone Authentication and Authorization Webhooks and the Client-Keystone-Auth defined here: https://github.com/kubernetes/cloud-provider-openstack * I presented this solution in this presentation at the DENVER OPENSTACK SUMMIT https://www.dropbox.com/s/gnyf3mtyf5janq1/KubernetesMultiTenancyWithKeystone.pptx?dl=0 * See slides 6,7,8 * This did not work out-of-the box circa early-2019, we had to fix the following issues: * Had to add Authentication to Armada (upstreamed) * StarlingX used Armada to manage set of helm charts for openstack * Armada did NOT support kubernetes authentication * StarlingX team added it and upstreamed it * Had to fix lock feature for Authentication in Armada (upstreamed) * After previous upstream commit to support authentication in armada, * A commit went in to Armada to support locking (for atomic transactions I think), and the commit forgot to take into account authentication, * StarlingX team fixed this commit upstream. * Had issues with expiring tokens with the client-keystone-auth * I believe that the client-auth-plugin was using a cached keystone client that was holding on to tokens that eventually expired * We put in some workaround for this ... can’t remember details. * Eventually we “generally” had this working ... except for a specific but-easily-repeatable use case, where the system would hang or fail in client-keystone-auth (being called from the StarlingX systemInventory process) * It was a path in the client-keystone-auth, where the code was trying to refresh the token in the kubernetes client, * Investigation suggested that the client-keystone-auth was not compatible with our version of eventlet(0.18.4) ... which is a few years old, * There was no way to update the token through “proper” methods. Attempting to delete the client caused the code to mysteriously freeze. If you tried to set the client to none on 1 line then set it to a new client in another line, the code would freeze as you try to set the client to none. * we managed to get it to work by digging through the internals of the client and setting a new token that way, but this was somewhat of a hack. * Since eventlet was used everywhere and we were getting close to STX 2.0 release date, we felt like it was too much risk to try and upversion it. * So, we abandoned the “Kubernetes API authentication with Keystone” work. * StarlingX Roadmap wrt Kubernetes API Authentication * Given our struggles in getting Keystone working ... a little uneasy in picking it up again, * Also Authentication requirements from StarlingX 2.0 Users of the Kubernetes Layer only, are asking for things like: * OpenID Connect (OIDC, OAuth2) type authentication, * LDAP * SAML * Integration with specific authentication platforms like: * GitHub, * LinkedIn, * Azure, * Windows Active Directory * I suspect many of these would require development in Keystone to support. * Whereas there are other existing opensource solutions that can provide that now. Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ildiko at openstack.org Thu Oct 10 18:14:53 2019 From: ildiko at openstack.org (Ildiko Vancsa) Date: Thu, 10 Oct 2019 20:14:53 +0200 Subject: [Edge-computing] PTG planning - ACTION NEEDED Message-ID: <83A61C0F-B642-4CB5-AE43-89AA5724E4E5@openstack.org> Hi, I’m reaching out with a friendly reminder that it is time now to prepare for the upcoming PTG (https://www.openstack.org/ptg/) which will be held in Shanghai, November 6-8. The Edge Computing Group session is scheduled for all day Friday (November 8). We have time to discuss items that are more relevant for the working group’s activities as well as organizing cross-project sessions. I created an etherpad which has a few items there already: https://etherpad.openstack.org/p/PVG-ECG-PTG Please add ‘+1’s next to the items you are interested in discussing as well as comments or new items to talk about. We will continue the planning on the next weekly call next Tuesday (October 15). It is crucial to identify projects to have sessions with to make sure we get a chance to schedule the collaborative sessions. Please let me know if you have any questions. Thanks and Best Regards, Ildikó From colleen at gazlene.net Fri Oct 11 20:43:01 2019 From: colleen at gazlene.net (Colleen Murphy) Date: Fri, 11 Oct 2019 13:43:01 -0700 Subject: [Edge-computing] StarlingX's Use of Keystone for Kubernetes Authentication In-Reply-To: <5D941121-89C6-4C49-9059-E6D2B2296333@windriver.com> References: <5D941121-89C6-4C49-9059-E6D2B2296333@windriver.com> Message-ID: <571fee11-735b-422d-879b-c9218d6807c1@www.fastmail.com> Hi, On Fri, Oct 4, 2019, at 04:43, Waines, Greg wrote: > > Hey Colleen, > > > Ildiko asked me to summarize StarlingX’s experiences with using > Keystone in STX R2.0 . > > Specifically summarize StarlingX’s experiences with using Keystone for > Kubernetes API Authentication and Docker Registry Authentication. > > [snipped] > > * > * StarlingX Services authenticating with Keystone * No issues ... > did it in STX 1.0 ... so no changes required. > > * Docker Registry authenticating with Keystone * This link: > https://docs.docker.com/registry/spec/auth/token/ > describes how Docker Registry token authentication works > * The Authentication/Authorization Service in the above link’s > diagram, > MUST be a JSON Web Token Server supporting OAUTH2 interface to docker, > * Docker provides an EXAMPLE of such a Token Server in its github: > https://github.com/docker/distribution/tree/master/contrib/token-server > , > which can be copied and updated to handle your preferred authn/z > backend. * Docker guy we spoke to was kind of surprised someone > hadn’t done this already for keystone. > * So StarlingX started with this example and built a > JSON Web Token Server supporting OAUTH2 interface to docker and with a > backend talking to Keystone * Located here: > https://opendev.org/starlingx/containers/src/branch/master/registry-token-server > * ... we are still doing this currently. Keystone has supported JSON Web Tokens since the Stein release: https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#jws-tokens http://specs.openstack.org/openstack/keystone-specs/specs/keystone/stein/json-web-tokens.html Can you say more about why this doesn't meet the needs for Docker Registry? One of the driving factors for this feature was increased interoperability with non-OpenStack services and it would be great to know what we can improve to achieve that goal. > > * Kubernetes API authenticating with Keystone * So we were using > the Keystone Authentication and Authorization Webhooks and > the Client-Keystone-Auth > defined here: https://github.com/kubernetes/cloud-provider-openstack > * I presented this solution in this presentation at the DENVER > OPENSTACK SUMMIT > https://www.dropbox.com/s/gnyf3mtyf5janq1/KubernetesMultiTenancyWithKeystone.pptx?dl=0 * See slides 6,7,8 > > * This did not work out-of-the box circa early-2019, > we had to fix the following issues: * Had to add Authentication > to Armada (upstreamed) * StarlingX used Armada to manage set of > helm charts for openstack > * Armada did NOT support kubernetes authentication > * StarlingX team added it and upstreamed it > * Had to fix lock feature for Authentication in Armada > (upstreamed) * After previous upstream commit to support > authentication in armada, > * A commit went in to Armada to support locking (for atomic > transactions I think), > and the commit forgot to take into account authentication, > * StarlingX team fixed this commit upstream. > * Had issues with expiring tokens with the client-keystone-auth > * I believe that the client-auth-plugin was using a cached keystone > client that > was holding on to tokens that eventually expired > * We put in some workaround for this ... can’t remember details. > > * Eventually we “generally” had this working ... except for a > specific but-easily-repeatable use case, where the > system would hang or fail in client-keystone-auth (being called from > the StarlingX systemInventory process) * It was a path in the > client-keystone-auth, where the code was trying to refresh the token in > the kubernetes client, > * Investigation suggested that the client-keystone-auth was not > compatible with our version of > eventlet(0.18.4) ... which is a few years old, > * There was no way to update the token through “proper” methods. > Attempting to delete the client > caused the code to mysteriously freeze. If you tried to set the client > to none on 1 line then set it to a new > client in another line, the code would freeze as you try to set the > client to none. > * we managed to get it to work by digging through the internals of > the client and setting a new token that > way, but this was somewhat of a hack. > * Since eventlet was used everywhere and we were getting close to > STX 2.0 release date, > we felt like it was too much risk to try and upversion it. > * So, we abandoned the “Kubernetes API authentication with Keystone” > work. If I read this correctly, there were bugs and UX issues in the client-keystone-auth tool (which is not an OpenStack tool) that prevented StarlingX from using it. Are there improvements that are needed for the keystone server project itself or to the documentation that would help improve the usability of this client? Are there bug reports raised with the maintainers of that tool that the keystone team could help follow up on? > > * StarlingX Roadmap wrt Kubernetes API Authentication * Given our > struggles in getting Keystone working ... a little uneasy in picking it > up again, > * Also Authentication requirements from StarlingX 2.0 Users of the > Kubernetes Layer only, > are asking for things like: * OpenID Connect (OIDC, OAuth2) type > authentication, > * LDAP > * SAML Keystone supports all of these: https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html https://docs.openstack.org/keystone/latest/admin/configuration.html#integrate-identity-with-ldap > * Integration with specific authentication platforms like: > * GitHub, > * LinkedIn, > * Azure, > * Windows Active Directory Keystone doesn't have specific support for any proprietary platforms but if these providers use an open protocol for which a web server auth module exists (eg. mod_auth_openidc, mod_shib, etc) then these can be configured like any other federated identity provider in the above documentation. > * I suspect many of these would require development in Keystone to > support. > * Whereas there are other existing opensource solutions that can > provide that now. > > Greg. > > Thanks for sharing, apologies for responding so late (I don't pay much attention to this mailing list and my mail filter filtered this away even though you CC'd me directly). I would suggest that future discussions about keystone use the openstack-discuss mailing list as that will get the attention of more keystone and OpenStack developers. Colleen From ildiko at openstack.org Tue Oct 15 14:04:26 2019 From: ildiko at openstack.org (Ildiko Vancsa) Date: Tue, 15 Oct 2019 16:04:26 +0200 Subject: [Edge-computing] Edge call is on now Message-ID: <8F200D2C-CEC5-4E6C-9CBF-9F123B89BCF6@openstack.org> Zoom is fixed. :) https://zoom.us/j/879678938 Thanks, Ildikó From ildiko at openstack.org Tue Oct 29 20:38:43 2019 From: ildiko at openstack.org (Ildiko Vancsa) Date: Tue, 29 Oct 2019 13:38:43 -0700 Subject: [Edge-computing] PTG cross-project session scheduling and plans Message-ID: Hi, As a few people from the group are already on the road and had to skip the meeting today I’m dropping this mail as a reminder to the PTG planning. We have a few cross-project slots that are currently scheduled for Thursday, namely: * Neutron - Thursday 12:00pm - 12:30pm * Cinder - 1:30pm - A few more cross-project sessions scheduled for Friday * Ironic - Friday 1:30pm - * Airship - some time in the afternoon, check right after lunch Beyond the above you can find the currently planned topics here: https://etherpad.openstack.org/p/PVG-ECG-PTG Please reply to this thread asap if any of the cross-project sessions are colliding with other activities. Please add comments to the etherpad or reply to this thread with further questions or comments. Thanks, Ildikó From Arkady.Kanevsky at dell.com Wed Oct 30 14:25:17 2019 From: Arkady.Kanevsky at dell.com (Arkady.Kanevsky at dell.com) Date: Wed, 30 Oct 2019 14:25:17 +0000 Subject: [Edge-computing] PTG cross-project session scheduling and plans In-Reply-To: References: Message-ID: Definitely want to attend airship one. Please, schedule it closer to lunch. Thanks -----Original Message----- From: Ildiko Vancsa Sent: Tuesday, October 29, 2019 3:39 PM To: edge-computing Subject: [Edge-computing] PTG cross-project session scheduling and plans [EXTERNAL EMAIL] Hi, As a few people from the group are already on the road and had to skip the meeting today I’m dropping this mail as a reminder to the PTG planning. We have a few cross-project slots that are currently scheduled for Thursday, namely: * Neutron - Thursday 12:00pm - 12:30pm * Cinder - 1:30pm - A few more cross-project sessions scheduled for Friday * Ironic - Friday 1:30pm - * Airship - some time in the afternoon, check right after lunch Beyond the above you can find the currently planned topics here: https://etherpad.openstack.org/p/PVG-ECG-PTG Please reply to this thread asap if any of the cross-project sessions are colliding with other activities. Please add comments to the etherpad or reply to this thread with further questions or comments. Thanks, Ildikó _______________________________________________ Edge-computing mailing list Edge-computing at lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing From ildiko at openstack.org Wed Oct 30 16:17:53 2019 From: ildiko at openstack.org (Ildiko Vancsa) Date: Wed, 30 Oct 2019 11:17:53 -0500 Subject: [Edge-computing] PTG cross-project session scheduling and plans In-Reply-To: References: Message-ID: Hi Arkady, We have the Ironic session scheduled for right after lunch on Friday. Do you have a hard limit on Friday to calculate with? Thanks, Ildikó > On Oct 30, 2019, at 09:25, Arkady.Kanevsky at dell.com wrote: > > Definitely want to attend airship one. > Please, schedule it closer to lunch. > Thanks > > -----Original Message----- > From: Ildiko Vancsa > Sent: Tuesday, October 29, 2019 3:39 PM > To: edge-computing > Subject: [Edge-computing] PTG cross-project session scheduling and plans > > > [EXTERNAL EMAIL] > > Hi, > > As a few people from the group are already on the road and had to skip the meeting today I’m dropping this mail as a reminder to the PTG planning. > > We have a few cross-project slots that are currently scheduled for Thursday, namely: > * Neutron - Thursday 12:00pm - 12:30pm > * Cinder - 1:30pm - > > A few more cross-project sessions scheduled for Friday > * Ironic - Friday 1:30pm - > * Airship - some time in the afternoon, check right after lunch > > Beyond the above you can find the currently planned topics here: https://etherpad.openstack.org/p/PVG-ECG-PTG > > Please reply to this thread asap if any of the cross-project sessions are colliding with other activities. > > Please add comments to the etherpad or reply to this thread with further questions or comments. > > Thanks, > Ildikó > > > > _______________________________________________ > Edge-computing mailing list > Edge-computing at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing From Arkady.Kanevsky at dell.com Wed Oct 30 18:06:25 2019 From: Arkady.Kanevsky at dell.com (Arkady.Kanevsky at dell.com) Date: Wed, 30 Oct 2019 18:06:25 +0000 Subject: [Edge-computing] PTG cross-project session scheduling and plans In-Reply-To: References: Message-ID: <7f6077c942d64b1e8a01155193b6a83c@AUSX13MPS308.AMER.DELL.COM> Right after lunch will work fine -----Original Message----- From: Ildiko Vancsa Sent: Wednesday, October 30, 2019 11:18 AM To: Kanevsky, Arkady Cc: edge-computing at lists.openstack.org Subject: Re: [Edge-computing] PTG cross-project session scheduling and plans [EXTERNAL EMAIL] Hi Arkady, We have the Ironic session scheduled for right after lunch on Friday. Do you have a hard limit on Friday to calculate with? Thanks, Ildikó > On Oct 30, 2019, at 09:25, Arkady.Kanevsky at dell.com wrote: > > Definitely want to attend airship one. > Please, schedule it closer to lunch. > Thanks > > -----Original Message----- > From: Ildiko Vancsa > Sent: Tuesday, October 29, 2019 3:39 PM > To: edge-computing > Subject: [Edge-computing] PTG cross-project session scheduling and plans > > > [EXTERNAL EMAIL] > > Hi, > > As a few people from the group are already on the road and had to skip the meeting today I’m dropping this mail as a reminder to the PTG planning. > > We have a few cross-project slots that are currently scheduled for Thursday, namely: > * Neutron - Thursday 12:00pm - 12:30pm > * Cinder - 1:30pm - > > A few more cross-project sessions scheduled for Friday > * Ironic - Friday 1:30pm - > * Airship - some time in the afternoon, check right after lunch > > Beyond the above you can find the currently planned topics here: https://etherpad.openstack.org/p/PVG-ECG-PTG > > Please reply to this thread asap if any of the cross-project sessions are colliding with other activities. > > Please add comments to the etherpad or reply to this thread with further questions or comments. > > Thanks, > Ildikó > > > > _______________________________________________ > Edge-computing mailing list > Edge-computing at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing From ildiko at openstack.org Wed Oct 30 18:48:35 2019 From: ildiko at openstack.org (Ildiko Vancsa) Date: Wed, 30 Oct 2019 13:48:35 -0500 Subject: [Edge-computing] PTG cross-project session scheduling and plans In-Reply-To: <7f6077c942d64b1e8a01155193b6a83c@AUSX13MPS308.AMER.DELL.COM> References: <7f6077c942d64b1e8a01155193b6a83c@AUSX13MPS308.AMER.DELL.COM> Message-ID: <5CE9F3B5-8F66-4A90-87DE-8EF8041D30A1@openstack.org> I need to check that with both Airship folks and the Ironic team. Will see what I can do. Thanks, Ildikó > On Oct 30, 2019, at 13:06, Arkady.Kanevsky at dell.com wrote: > > Right after lunch will work fine > > -----Original Message----- > From: Ildiko Vancsa > Sent: Wednesday, October 30, 2019 11:18 AM > To: Kanevsky, Arkady > Cc: edge-computing at lists.openstack.org > Subject: Re: [Edge-computing] PTG cross-project session scheduling and plans > > > [EXTERNAL EMAIL] > > Hi Arkady, > > We have the Ironic session scheduled for right after lunch on Friday. Do you have a hard limit on Friday to calculate with? > > Thanks, > Ildikó > > >> On Oct 30, 2019, at 09:25, Arkady.Kanevsky at dell.com wrote: >> >> Definitely want to attend airship one. >> Please, schedule it closer to lunch. >> Thanks >> >> -----Original Message----- >> From: Ildiko Vancsa >> Sent: Tuesday, October 29, 2019 3:39 PM >> To: edge-computing >> Subject: [Edge-computing] PTG cross-project session scheduling and plans >> >> >> [EXTERNAL EMAIL] >> >> Hi, >> >> As a few people from the group are already on the road and had to skip the meeting today I’m dropping this mail as a reminder to the PTG planning. >> >> We have a few cross-project slots that are currently scheduled for Thursday, namely: >> * Neutron - Thursday 12:00pm - 12:30pm >> * Cinder - 1:30pm - >> >> A few more cross-project sessions scheduled for Friday >> * Ironic - Friday 1:30pm - >> * Airship - some time in the afternoon, check right after lunch >> >> Beyond the above you can find the currently planned topics here: https://etherpad.openstack.org/p/PVG-ECG-PTG >> >> Please reply to this thread asap if any of the cross-project sessions are colliding with other activities. >> >> Please add comments to the etherpad or reply to this thread with further questions or comments. >> >> Thanks, >> Ildikó >> >> >> >> _______________________________________________ >> Edge-computing mailing list >> Edge-computing at lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing > > _______________________________________________ > Edge-computing mailing list > Edge-computing at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing From ben.silverman at cbts.com Tue Oct 15 15:39:02 2019 From: ben.silverman at cbts.com (Silverman, Ben) Date: Tue, 15 Oct 2019 15:39:02 -0000 Subject: [Edge-computing] Edge call is on now In-Reply-To: <8F200D2C-CEC5-4E6C-9CBF-9F123B89BCF6@openstack.org> References: <8F200D2C-CEC5-4E6C-9CBF-9F123B89BCF6@openstack.org> Message-ID: <1588852F-E4E5-4015-B66A-BADA63F532AA@cbts.com> Sorry, been extremely busy, couldn’t make it again today. -b From: Ildiko Vancsa Date: Tuesday, October 15, 2019 at 10:07 AM To: edge-computing Subject: [Edge-computing] Edge call is on now Zoom is fixed. :) https://zoom.us/j/879678938 Thanks, Ildikó _______________________________________________ Edge-computing mailing list Edge-computing at lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing -------------- next part -------------- An HTML attachment was scrubbed... URL: