Hi Zuul team.
Hope all are safe.
I am trying to create a new repo under zuul namespace named swift-proxy.
The new repo would include this code. https://review.opendev.org/c/zuul/zuul/+/708233
Let me explain a little about this PS.
It is a wsgi proxy to proxy the requests like this;
```
ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path
```
This proxy requires clouds.yaml file and CLOUD_NAME envvar for the swift.
Please give the feedback. Thank you.
Open10k8s
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...]
It is a wsgi proxy to proxy the requests like this;
ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path
[...]
A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs?
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...]
A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy Stanley
The original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
/Tobias
Tobias.Henkel@bmw.de writes:
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...] A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy StanleyThe original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
This all makes sense. It also might be useful for multi-tenant situations?
In OpenDev, obviously everything is publicly accessible, so we just allow anonymous access to all of our different log storage backends (which are all either swift or pretending to be swift).
For a private internal Zuul installation, I imagine that the same situation could apply if you have a private object storage system that is only accessible to your company (perhaps by network restriction).
But anything more sophisticated would probably require a proxy like this, or apache/nginx/etc, right?
Is it possible to accomplish this with just Apache/nginx?
I'd love to hear what other log storage/serving configurations people are using. Is anyone doing anything with logs in AWS?
-Jim
First, language problem
Well, seems the code seems more convenient.
For swift, the proxy has 2 processes- service discovery and authentication.
The proxy has to get the endpoint url of swift first and need to authenticate against the openstack.
For other storage backend, we need to follow the processes for that backend.
Such processes can be done by nginx or apache configuration only? even though using lua? seems python is better to realize the logic smoothly.
Second, scope of the project
- We can create one project per one storage backend type. ex: zuul-swift-proxy, zuul-s3-proxy ...
- OR create a one project for all storage backend type, ex: named zuul-storage-proxy. By using config or envvar, can select the storage backend.
Which one seems better?
On 1/25/21 7:33 PM, James E. Blair wrote:
Tobias.Henkel@bmw.de writes:
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...]
A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy Stanley
The original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
This all makes sense. It also might be useful for multi-tenant situations?
In OpenDev, obviously everything is publicly accessible, so we just allow anonymous access to all of our different log storage backends (which are all either swift or pretending to be swift).
For a private internal Zuul installation, I imagine that the same situation could apply if you have a private object storage system that is only accessible to your company (perhaps by network restriction).
But anything more sophisticated would probably require a proxy like this, or apache/nginx/etc, right?
Is it possible to accomplish this with just Apache/nginx?
I'd love to hear what other log storage/serving configurations people are using. Is anyone doing anything with logs in AWS?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
On 1/25/21 7:33 PM, James E. Blair wrote:
Tobias.Henkel@bmw.de writes:
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...] A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy StanleyThe original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
This all makes sense. It also might be useful for multi-tenant situations?
In OpenDev, obviously everything is publicly accessible, so we just allow anonymous access to all of our different log storage backends (which are all either swift or pretending to be swift).
For a private internal Zuul installation, I imagine that the same situation could apply if you have a private object storage system that is only accessible to your company (perhaps by network restriction).
But anything more sophisticated would probably require a proxy like this, or apache/nginx/etc, right?
Is it possible to accomplish this with just Apache/nginx?
I'd love to hear what other log storage/serving configurations people are using. Is anyone doing anything with logs in AWS?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
On Mon, Jan 25, 2021 at 22:35 okozachenko wrote:
First, language problem
Well, seems the code seems more convenient.
For swift, the proxy has 2 processes- service discovery and authentication.
The proxy has to get the endpoint url of swift first and need to authenticate against the openstack.
For other storage backend, we need to follow the processes for that backend.
Such processes can be done by nginx or apache configuration only? even though using lua? seems python is better to realize the logic smoothly.
Second, scope of the project
- We can create one project per one storage backend type. ex:
zuul-swift-proxy, zuul-s3-proxy ...
- OR create a one project for all storage backend type, ex: named
zuul-storage-proxy. By using config or envvar, can select the storage backend.
Which one seems better?
Hi okozachenko,
Thank you for proposing such project. A single zuul-storage-proxy service to serve logs from the different backends supported by the zuul-jobs upload-log roles seems better.
Kind Regards, -Tristan
On 1/25/21 7:33 PM, James E. Blair wrote:
Tobias.Henkel@bmw.de writes:
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...]
A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy Stanley
The original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
This all makes sense. It also might be useful for multi-tenant situations?
In OpenDev, obviously everything is publicly accessible, so we just allow anonymous access to all of our different log storage backends (which are all either swift or pretending to be swift).
For a private internal Zuul installation, I imagine that the same situation could apply if you have a private object storage system that is only accessible to your company (perhaps by network restriction).
But anything more sophisticated would probably require a proxy like this, or apache/nginx/etc, right?
Is it possible to accomplish this with just Apache/nginx?
I'd love to hear what other log storage/serving configurations people are using. Is anyone doing anything with logs in AWS?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
On 1/25/21 7:33 PM, James E. Blair wrote:
Tobias.Henkel@bmw.de writes:
On 25.01.21, 18:08, "Jeremy Stanley" fungi@yuggoth.org wrote:
On 2021-01-25 19:01:41 +0200 (+0200), okozachenko wrote: [...] > It is a wsgi proxy to proxy the requests like this; > > ``` > ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> > SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path > ``` [...] A bit more about *why* it's useful could help the discussion. I'm going to guess it's that some sites may want to present build logs in a way that causes them to look like they're being served directly by the Zuul server, or to avoid having to publicly expose the object storage where those logs are stored, at the cost of funneling all build log access through the proxy server(s) rather than directly to/from object storage. Could this also be useful for sites which want to put read controls on the data in object storage and then rely on tenant-scoped credentials for Zuul to decide who gets to see which logs? -- Jeremy StanleyThe original change is by me so I guess I can explain the reason. We run it in front of swift since our swift backend storage doesn't use the same authentication system as we have for zuul itself. Thus we need a proxy in front of it that does the auth against the user and auths itself against the backend storage. For non-authed zuul's I guess this would also be useful if the backend storage doesn't support anonymous access.
I think this should be potentially multi-backend ready in the future if e.g. s3 should be supported so we should take this into consideration when naming it. E.g zuul-storage-proxy or zuul-logs-proxy.
This all makes sense. It also might be useful for multi-tenant situations?
In OpenDev, obviously everything is publicly accessible, so we just allow anonymous access to all of our different log storage backends (which are all either swift or pretending to be swift).
For a private internal Zuul installation, I imagine that the same situation could apply if you have a private object storage system that is only accessible to your company (perhaps by network restriction).
But anything more sophisticated would probably require a proxy like this, or apache/nginx/etc, right?
Is it possible to accomplish this with just Apache/nginx?
I'd love to hear what other log storage/serving configurations people are using. Is anyone doing anything with logs in AWS?
-Jim
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
Zuul-discuss mailing list Zuul-discuss@lists.zuul-ci.org http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
participants (5)
-
Jeremy Stanley
-
jim＠acmegating.com
-
okozachenko
-
Tobias.Henkel＠bmw.de
-
Tristan Cacqueray