Can i convert proxyfied registries ?

[ptempier]

I mean, i tried with estargz, it didn't work, but i was somehow expecting it to work.
Not sure if
it's not supported at all and i am doing something wrong.
Maybe i need a different setup, like an actual proxy project and an accelerated copy of that project.
It's not working and its not supposed to work.
It doesn't work right now, but maybe it will in a future release.

[Desiki-high]

Hi @ptempier. Could you please give more information to help reproduce your issue? Such as your setup steps and what kind of proxyfied registries are used. Acceleration-Service can work well with estargz in the latest harbor.

[ptempier]

Hello

I am not using the latest version, but v2.9.0-6d1ad65c, so maybe that's it.
I tried looking at the changlog but couldn't find anything related to acceld and proxy.
Was the support added in 2.10 ?

harbor-acceld is v0.2.13

The test was doing a pull from a proxified github registry.
The robot could successfully authenticate to pull the image
the accelertor could transform to estargz
but i was getting an nginx permission error (403?) when pushing, it was not appearing in the harbor logs.

Creating an estargz image via kaniko, pushing it in a different projet
then pulling it with a properly configured containerd works as expected.

[Desiki-high]

Can you provide the acceld work log? @ptempier

[Desiki-high]

Actually harbor v2.9 is supported too.

[Desiki-high]

Can you provide the acceld work log? @ptempier

If you can provide the harbor log, it will be better.

[ptempier]

The time it almost worked , i had this error :

error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"

Just tried a docker pull anonyme/hub.docker.com/library/postgres
It went through the proxy cache as per habor log

harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:3a27b8f06bc0cc0b76ab124b8c48bf3177703aedbd9cc28fcebc0e312bcb8c7a artifact create 1/9/24, 12:51 PM
anonymous hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact pull 1/9/24, 12:45 PM
harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact create 1/9/24, 12:45 PM

It went through the webhook

10703 WEBHOOK Success Artifact pushed xxxx 1/9/24, 12:51 PM 1/9/24, 12:51 PM
10702 WEBHOOK Success Artifact pushed xxx 1/9/24, 12:45 PM 1/9/24, 12:45 PM

But then in acceld failed with :

time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

time="2024-01-08T15:40:46.676196062Z" level=info msg="Version: v0.2.13 b82ccee1e73845741b033f8b04ab418fad5b84ef.20240108.0649\n"
time="2024-01-08T15:40:46.698308247Z" level=info msg="[API] HTTP server started on 0.0.0.0:2077"
time="2024-01-08T15:50:26.552634578Z" level=info msg="received webhook request from 172.19.0.1:52386" module=api
time="2024-01-08T15:50:26.581593392Z" level=info msg="POST /api/v1/conversions 200 28.919643ms 602>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:26.606109487Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/rockylinux@sha256:8b5296204bad12e84837c7b4c8b2cdff45bcc92ee5ffaaa3e86ba683f2384b14"
time="2024-01-08T15:50:31.651439807Z" level=info msg="received webhook request from 172.19.0.1:52392" module=api
time="2024-01-08T15:50:31.659627035Z" level=info msg="POST /api/v1/conversions 200 8.161151ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:31.668051229Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:28a16e31b140d750048cd5fadcaed22ac08d0eeb18567f79f822aee1f237b43c"
time="2024-01-08T15:53:45.357608101Z" level=info msg="received webhook request from 172.19.0.1:57896" module=api
time="2024-01-08T15:53:45.366814898Z" level=info msg="POST /api/v1/conversions 200 9.191926ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:53:45.373215343Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/nginx@sha256:5be1749f6a023b14ef069f2bbe1afd9a39295694a490963e527a848e4bc4d442"
time="2024-01-08T15:56:51.580870348Z" level=info msg="received webhook request from 172.19.0.1:47690" module=api
time="2024-01-08T15:56:51.587785057Z" level=info msg="POST /api/v1/conversions 200 6.842051ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:56:51.634318232Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:6f453b1c7bcbc42b8e3d7949d8dfa28a70f8bf86dff277f5909c5e714ee5153a"
time="2024-01-08T16:05:49.7946499Z" level=info msg="received webhook request from 127.0.0.1:34366" module=api
time="2024-01-08T16:05:49.80163495Z" level=info msg="POST /api/v1/conversions?sync=false 200 7.002226ms 135>5bytes 127.0.0.1" module=api
time="2024-01-08T16:05:49.802579063Z" level=info msg="pulling image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:05:52.887395575Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:05:52.888131555Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:05:53.009889983Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:05:53.009965038Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:06:05.439993598Z" level=info msg="pulled image anonyme/hub.docker.com/library/mysql:latest , elapse 15.607270101s" module=converter
time="2024-01-08T16:06:05.440048067Z" level=info msg="converting image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:06:13.090493323Z" level=info msg="received webhook request from 172.19.0.1:51998" module=api
time="2024-01-08T16:06:13.09860536Z" level=info msg="received webhook request from 172.19.0.1:51990" module=api
time="2024-01-08T16:06:13.11688972Z" level=info msg="POST /api/v1/conversions 200 26.377558ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.166796733Z" level=info msg="POST /api/v1/conversions 200 47.731353ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.186198622Z" level=info msg="received webhook request from 172.19.0.1:52006" module=api
time="2024-01-08T16:06:13.232626827Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:b8bfa6bfda24cf129ce2a20ea3fe679ad377840c0c16d7937887233485b3f170"
time="2024-01-08T16:06:13.279097726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:483bf8eb111365bf322a25443d3f96ae0d80829c60f00fb329d8a0de0f21c6e7"
time="2024-01-08T16:06:13.318753293Z" level=info msg="POST /api/v1/conversions 200 132.527925ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.341869273Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:e870e58e0e1f937652982f99cddff85ab2076d217db08732856d22eb334e9e2a"
time="2024-01-08T16:08:48.548941466Z" level=info msg="converted image anonyme/hub.docker.com/library/mysql:latest-esgz , elapse 2m43.080799299s" module=converter
time="2024-01-08T16:08:48.549005034Z" level=info msg="pushing image anonyme/hub.docker.com/library/mysql:latest-esgz" module=converter
time="2024-01-08T16:08:48.587453917Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:08:48.58782373Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:08:48.597293349Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:08:48.597781217Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:08:48.968420375Z" level=error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"
time="2024-01-08T16:40:47.091302038Z" level=info msg="garbage collect, elapse 8.45699ms"
time="2024-01-08T17:33:13.824434821Z" level=info msg="received webhook request from 172.19.0.1:60738" module=api
time="2024-01-08T17:33:13.914808701Z" level=info msg="POST /api/v1/conversions 200 90.366352ms 617>5bytes 172.19.0.1" module=api
time="2024-01-08T17:33:13.921133726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/sameersbn/apt-cacher-ng@sha256:58e74113cfac7e593201444648c105351cbfce7538bfb36dcafdc9479b2aefcc"
time="2024-01-08T17:39:58.249120243Z" level=info msg="received webhook request from 172.19.0.1:45660" module=api
time="2024-01-08T17:39:58.299701694Z" level=info msg="POST /api/v1/conversions 200 50.5531ms 608>5bytes 172.19.0.1" module=api
time="2024-01-08T17:39:58.30843498Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/gitlab/gitlab-runner@sha256:7a267b16df7d05786fa7d76758e5a0dcc34dc6318902fbfa29aaad7ad3cbe1be"
time="2024-01-09T11:45:17.207698977Z" level=info msg="received webhook request from 172.19.0.1:45606" module=api
time="2024-01-09T11:45:17.223161837Z" level=info msg="POST /api/v1/conversions 200 15.467165ms 596>5bytes 172.19.0.1" module=api
time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

[ptempier]

I am thinking maybe i need to add some dependencies to acceld ?
Right now it run inside a docker made from rockylinux:9,
At this moment it's only raw rockylinux, 0 packages added, and accelld copied into /root/

[ptempier]

I added some docker tools inside the container, but it doesn't change anything.
level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/alpinelinux/rsyncd@sha256:6f8b68b4b15a8e6b0abfb7db0e2a765849c77a6104ac248810ff9a9fb97996fb"

The Dockerfile i use :

FROM  anonyme/hub.docker.com/library/rockylinux:9

RUN echo "proxy=http://anonyme:3142" >> /etc/yum.conf
RUN dnf -y install dnf-plugins-core && dnf config-manager --set-disabled '*' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/AppStream/$basearch/os/' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/BaseOS/$basearch/os/'


#add the docker part
RUN dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
RUN dnf -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin


WORKDIR "/root"


RUN dnf install -y wget procps
RUN wget "https://raw.githubusercontent.com/goharbor/acceleration-service/main/misc/config/config.estargz.yaml"

RUN wget "https://github.com/goharbor/acceleration-service/releases/download/v0.2.13/harbor-acceld-v0.2.13-linux-amd64.tgz" && tar -xvzf  ./harbor-acceld-v0.2.13-linux-amd64.tgz
RUN ls -lha
RUN chmod ugo+x ./harbor-acceld/acceld ./harbor-acceld/accelctl


RUN  sed -i 's|hub.harbor.com|anonyme|' ./config.estargz.yaml && \
 sed -i 's|auth_header: header|auth_header: anonyme|' ./config.estargz.yaml && \
 sed -i 's|# auth: YTpiCg==|auth: anonyme|' ./config.estargz.yaml


CMD "./harbor-acceld/acceld --config ./config.estargz.yaml"

[imeoer]

@ptempier You seem to be pushing an image with the name format example.com/namespace/repo@sha256:xxx, how about trying to use the name format example.com/namespace/repo:tag instead?

[ptempier]

@imeoer nope i am doing :

docker pull anonyme/hub.docker.com/library/postgres
anonyme is the harbor registry server
hub.docker.com is the project set as proxy to docker hub
library is the default project space in docker hub
postgres is the image name
my understanding is that docker pull will automatically append the latest is no tag is specfied

So for me it s a very "normal" image pull with a very commonly used registry.

But acceld when receiving such a request respond with :
convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

[imeoer]

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

[Desiki-high]

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

Support image@sha265 can be a new feature? @imeoer .

[imeoer]

Support image@sha265 can be a new feature? @imeoer .

@Desiki-high Yes, it should also support the repo@sha256:xxx image name, but a map rule is needed to transform it to the converted image name, possibly repo@sha256:yyy, however, this makes it inconvenient for users to find the converted image.

[ptempier]

Maybe you need a fix on the harbor side to send the proper image name ?
The client send the query to harbor asking for image:tag
and its harbor who decide that it needs to pull blob in format repo@sha256:xxx
then send this to acceld

So maybe a webhook of acceld type is needed.