Change
https://review.opendev.org/c/opendev/system-config/+/805933
is using DOCKER_BUILDKIT=1 to build an image (gitea) but has a requirement to pull another image (the new "assets" image, proposed with it's parent https://review.opendev.org/c/opendev/system-config/+/805933/) from the intermediate registry.
This reliably fails as you can see from any of the zuul results. The failure is
--- root@ubuntu-focal-ovh-bhs1-0026222696:/home/zuul/src/opendev.org/opendev/system-config/docker/gitea# DOCKER_BUILDKIT=1 docker build . ... => ERROR [internal] load metadata for docker.io/opendevorg/assets:latest ------
[internal] load metadata for docker.io/opendevorg/assets:latest:
------ failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed commit on ref "config-sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07: failed precondition ---
I enabled debugging on the docker daemon and it produces the following
--- Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.097177192Z" level=debug msg=resolving host=registry-1.docker.io Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.097232892Z" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.0+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/golang/manifests/1.16-buster" Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100016964Z" level=debug msg="fetch response received" host="zuul-jobs.buildset-registry:5000" response.header.content-length=921 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 01 Sep 2021 03:56:36 GMT" response.header.docker-content-digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" response.header.server=CherryPy/18.6.0 response.status="200 OK" url="https://zuul-jobs.buildset-registry:5000/v2/opendevorg/assets/manifests/late..." Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100201992Z" level=debug msg=resolved desc.digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" host="zuul-jobs.buildset-registry:5000" Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100454497Z" level=debug msg=fetch digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" mediatype=application/vnd.docker.distribution.manifest.v2+json size=921 Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.101860659Z" level=debug msg=fetch digest="sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" mediatype=application/vnd.docker.container.image.v1+json size=0 ---
We see digest="sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" has size=0; clarkb helpfully noted that the "unexpected commit digest" e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is actually the sha256 of an empty string. This seems related.
The buildset registry output for this is
--- 2021-09-01 04:10:11,794 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:11,794 INFO registry.api: HEAD manifest _local docker/dockerfile 1.3 2021-09-01 04:10:11,794 ERROR registry.api: Manifest docker/dockerfile 1.3 not found 2021-09-01 04:10:11,794 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:11] "HEAD /v2/docker/dockerfile/manifests/1.3?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,115 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,115 INFO registry.api: HEAD manifest _local library/debian buster-slim 2021-09-01 04:10:12,115 ERROR registry.api: Manifest library/debian buster-slim not found 2021-09-01 04:10:12,116 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/library/debian/manifests/buster-slim?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,119 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,119 INFO registry.api: HEAD manifest _local library/golang 1.16-buster 2021-09-01 04:10:12,120 ERROR registry.api: Manifest library/golang 1.16-buster not found 2021-09-01 04:10:12,120 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/library/golang/manifests/1.16-buster?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,125 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,125 INFO registry.api: HEAD manifest _local opendevorg/assets latest 2021-09-01 04:10:12,126 DEBUG registry.api: Manifest opendevorg/assets latest digest found sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,126 DEBUG registry.api: Retrieved sha256 3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,126 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/opendevorg/assets/manifests/latest?ns=docker.io HTTP/1.1" 200 - "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,128 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,129 INFO registry.api: GET manifest _local opendevorg/assets latest 2021-09-01 04:10:12,129 DEBUG registry.api: Manifest opendevorg/assets latest digest found sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,129 DEBUG registry.api: Retrieved sha256 3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,129 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "GET /v2/opendevorg/assets/manifests/latest?ns=docker.io HTTP/1.1" 200 921 "" "containerd/1.4.0+unknown" ---
There is a file on the host
--- # cat buildkit/content/blobs/sha256/3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "digest": "sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 3122, "digest": "sha256:d400fcf962679c5efdb77b55b187b8afffb53f7342895f0b6f6206cf1fbbac6d" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 1899, "digest": "sha256:1cdb62f96c3d6766a459909f1306f97d174fd4b5fa11a46ecc2f20d8c529db93" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 7671, "digest": "sha256:f7721f02fe31867cdcab1396b7fc132f427ab98fe1f083611b2cc01e3f4fb4a8" } ] ---
I note that the "config" here doesn't have a size -- it seems to me from
https://docs.docker.com/registry/spec/manifest-v2-2/
it should. So I poked at the original opendev-buildset-registry job that the "assets" image as uploaded to
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
That shows this has a size "2329"
--- 2021-08-31 20:36:33,782 INFO registry.api: Get blob _local opendevorg/assets sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07 2021-08-31 20:36:33,782 INFO cherrypy.access.140479630381648: ::ffff:172.17.0.1 - - [31/Aug/2021:20:36:33] "GET /v2/opendevorg/assets/blobs/sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07 HTTP/1.1" 200 2329 "" "containers/5.10.2 (github.com/containers/image)" ---
So, I hand edited in this size to the "config" above; to be clear it now looked like
... "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 2329, "digest": "sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" }, ...
and ... the build started working.
Along with the weird hashing of an empty string, this makes more more convinced it has something to do with this size setting.
How has this config in the manifest not got a size? Is this really the problem? Is it something we've managed to drop in the intermediate registry, or has docker uploaded it incorrectly?
I'd love any suggestions!
-i