[GSoC] refactor: refactor the dependency management with OCI's layered mechanism

[zong-zhe]

Enhancement

KCL is an open-source constraint-based record & functional language mainly used in configuration and policy scenarios. KPM is a package management tool for the KCL language that supports the management of KCL packages in the OCI registry and Git Repo. This topic only applies to third-party dependencies from the OCI registry. Use the layering mechanism in OCI to help KPM implement dependency management of KCL third-party dependencies.

[UtkarshUmre]

Could you please elaborate on the current limitations or challenges faced by the KPM dependency management module that this enhancement aims to address?

[AkashKumar7902]

Hey @zong-zhe,
will there be pretasks ?

[suyiiyii]

Hello
I have a few questions:

1. After reading the idea description, I couldn’t fully understand the pain points it aims to address. I would appreciate further clarification.

Currently, my plan is:

1. Exploring the use of KCL for GitOps as a replacement for Helm.
2. Learning about and experimenting with KPM.
3. Reading relevant source code and OCI specifications to identify possible improvements.

I would love to receive your guidance on this!

[suyiiyii]

Hello everyone,

Recently, I have been researching the KCL package and the OCI specification. Below is my main implementation approach, and I would appreciate your guidance.

Currently, each kcl package is an OCI image, stored as a tar archive. Fetching dependencies means pulling this tar package from the registry and saving it to the .kcl directory.

As we all know, OCI images are layered. Based on this, we can package all third-party dependencies of a custom package as layers within a single image.

To verify the feasibility of this approach, I attempted to use the oras tool for validation.

I was able to package an OCI image with the following manifest:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "artifactType": "application/vnd.oci.image.layer.v1.tar",
    "config": {
        "mediaType": "application/vnd.oci.empty.v1+json",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2,
        "data": "e30="
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar",
            "digest": "sha256:10fc15caf52b9c40f9108b757afa3460e4f8973953277132c7553c65fc6d3450",
            "size": 2506240,
            "annotations": {
                "org.opencontainers.image.title": "k8s_1.31.2.tar"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar",
            "digest": "sha256:d17e408fbd19088c49f9b91341c9452d9fcdb6ddd0274bcf2f08e378088bc911",
            "size": 1768960,
            "annotations": {
                "org.opencontainers.image.title": "knative_0.2.0.tar"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar",
            "digest": "sha256:0af30e70b23898323217c8c92d4b8f2e36422b7a0551338476f329e2a33263cb",
            "size": 253041,
            "annotations": {
                "org.opencontainers.image.title": "my_package_0.1.0.tar"
            }
        }
    ],
    "annotations": {
        "org.kcllang.package.description": "",
        "org.kcllang.package.name": "my_package",
        "org.kcllang.package.sum": "xBZgPsnpaayWnpbhuPQHReeRx28eUHGFoaPeqbct+vs=",
        "org.kcllang.package.version": "0.1.0",
        "org.opencontainers.image.created": "2025-03-17T16:12:16Z"
    }
}

Comparing it to the manifest of an existing package:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "artifactType": "application/vnd.oci.image.layer.v1.tar",
    "config": {
        "mediaType": "application/vnd.oci.empty.v1+json",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2,
        "data": "e30="
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar",
            "digest": "sha256:d17e408fbd19088c49f9b91341c9452d9fcdb6ddd0274bcf2f08e378088bc911",
            "size": 1768960,
            "annotations": {
                "org.opencontainers.image.title": "knative_0.2.0.tar"
            }
        }
    ],
    "annotations": {
        "org.kcllang.package.description": "KCL module for Knative Serving and Eventing",
        "org.kcllang.package.name": "knative",
        "org.kcllang.package.sum": "GVwNctjleycGcWU8nEl5vJF5bliKeLeGK2V3pE/PXgg=",
        "org.kcllang.package.version": "0.2.0",
        "org.opencontainers.image.created": "2025-02-10T02:42:31Z"
    }
}

We can see that by adding both the current user package and its dependencies as layers in an OCI image, we can pull everything at once when fetching dependencies.

Additionally, for the checksum of existing packages (org.kcllang.package.sum), we can consider storing it in the config as a separate JSON file to describe all layers.

This is my current idea, and I would greatly appreciate your guidance. Thank you!