Issue using harbor private registries with AWS storage behind harbor as my policy bundle server

[s004pmg]

For me, my setup works as a harbor public repo (if I remove the credential information), but not if I mark the harbor repo private.

I've added all of the Amazon root CA's to my cert bundle reference from the opa-config:

    services:
      oci-registry:
        url: "https://my.oci.server.example/"
        type: oci
        tls:
          ca_cert: ${CA_CERT_FILE}

        credentials:
          bearer:
            scheme: "Basic"
            #scheme: "Bearer"
            token: "username:password"

From the OPA logs, some of the layers seem to come in alright?:

{
  "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
  "level": "debug",
  "msg": "do request",
  "request.header.accept": "application/vnd.oci.empty.v1+json, */*",
  "request.header.user-agent": "containerd/1.7.11+unknown",
  "request.method": "GET",
  "time": "2024-01-12T17:29:32Z",
  "url": "https://my.oci.server.example/path/to/bundle/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
}

But then not, with an HTTP 400:

{
  "digest": "sha256:e967dc5eb7d24250f5309b698221cdb8bc87a6f0f4cd7c0a5461de8a5ce5e1ea",
  "level": "debug",
  "msg": "fetch response received",
  "response.header.content-type": "application/xml",
  "response.header.date": "Fri, 12 Jan 2024 17:29:32 GMT",
  "response.header.server": "AmazonS3",
  "response.header.x-amz-id-2": "c41pck5x....",
  "response.header.x-amz-request-id": "FTA...",
  "response.status": "400 Bad Request",
  "time": "2024-01-12T17:29:32Z",
  "url": "https://my.oci.server.example/path/to/bundle/tag/blobs/sha256:e967dc5eb7d24250f5309b698221cdb8bc87a6f0f4cd7c0a5461de8a5ce5e1ea"
}
{
  "level": "error",
  "msg": "Bundle load failed: failed to pull my.oci.server.example/path/to/bundle:my_tag: download for 'my.oci.server.example/path/to/bundle:my_tag' failed: failed to ingest: copy failed: httpReadSeeker: failed open: unexpected status code https://my.oci.server.example/path/to/bundle/blobs/sha256:e967dc5eb7d24250f5309b698221cdb8bc87a6f0f4cd7c0a5461de8a5ce5e1ea: 400 Bad Request",
  "name": "authz",
  "plugin": "bundle",
  "time": "2024-01-12T17:29:32Z"
}

Nothing in the various Harbor logs seems helpful. My understanding is that the OCIDownloader is a little different than the normal service declaration. I'm hoping that I've just done the configuration wrong here?

[ashutosh-narkar]

The configuration looks ok to me. @gitu and @carabasdaniel have contributed a lot to the OCI downloader and may have some ideas here.

[carabasdaniel]

The OPA configuration looks good. Does it work if you manually pull the manifest from v2 using a simple curl ?

[s004pmg]

Thanks @ashutosh-narkar for the look over and referral, and thanks @carabasdaniel for the suggestion, which led to some interesting things.

By curling at various points along the network path, I realized that I needed to use a k8s cluster internal address to get out of the way of an SSO proxy in the middle, so that probably gets me a step closer. So that's changed now.

The manifest layer comes in just fine. I think what's coming into focus is that the S3 layer may need another header that isn't there:

{
  "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
  "level": "debug",
  "msg": "fetch response received",
  "response.header.content-type": "application/xml",
  "response.header.date": "Thu, 18 Jan 2024 21:32:35 GMT",
  "response.header.server": "AmazonS3",
  "response.header.x-amz-id-2": "8...",
  "response.header.x-amz-request-id": "T...",
  "response.status": "400 Bad Request",
  "time": "2024-01-18T21:32:35Z",
  "url": "https://my.oci.server.example.svc.cluster.local/v2/path/to/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
}

Curling that k8s cluster local address from inside the cluster, and adding -k to ignore any cert validation issues at the moment:

$ curl -k https://my.oci.server.example.svc.cluster.local/v2/path/to/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
{"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}

I'm currently investigating along the lines of whether there needs to be an auth header inserted there for the harbor to S3 connection credentials.

[s004pmg]

We're going to retest this with a harbor upgrade that's hopefully coming in a week or two. I'm hoping that might give different or more specific error messages from within harbor that might give another avenue of investigation.

[s004pmg]

Looks like there are bugs entered with OCI pulls from private repos entered since I mentioned my issue:

open-policy-agent/opa#6590
open-policy-agent/opa#6580

We've had a delay getting a new harbor deployed to retest my original issue, but that's still my plan.

[s004pmg]

Still open for us, but still delayed on our harbor upgrade, will check back in then on this.