Consolidate go mod dependency versions across repositories

[gman0]

Feature Description

kcp already has a lot of repositories, with a lot of (inter-)dependencies but no way to manage them centrally. This makes it hard to keeping them in sync, and rebasing is also more difficult than it has to be because of that.

We have hack/verify-go-modules.sh script that attempts to check dependency versions against k/k, but it's a rather manual process.

Proposed Solution

Kubernetes has a mono-repository structure with the staging dir in k/k that's replicated into separate repos. Could this be something we want to implement?

Alternative Solutions

Implement a GH action that checks dependencies across the repos in the project.

Want to contribute?

• I would like to work on this issue.

Additional Context

No response

[ntnn]

I like the idea of moving towards a mono repo. Question is if every repo should just be moved into staging or if some should be merged and some into staging. E.g. the embedded etcd could just be moved to pkg/embeddedetcd (without further looking into it :D). client-go on the other hand could be moved into staging.

[gman0]

Exactly, and more questions like what does it then mean for the release cycle of all these "sub-projects"? They'd have to be tied to kcp. For some it makes sense because they need to be in sync anyway. For others, ¯\_(ツ)_/¯

kcp-dev/embeddedetcd#1 is a good example of what happens when a library is split into a separate repo: it depends on the kcp-dev/kcp/sdk module, but kcp-dev/kcp depends on kcp-dev/embeddedetcd so we have to do things like #3377 (review)

[ntnn]

I'd just copy what kube does for the ones that land in staging - if it works there it will work here. Just gotta consider that someone will have to run and maintain the bot (unless that can be piggy backed somewhere).
For everything that doesn't land in staging but is changed to be a module just follows the release cycle of kcp.

[embik]

I'm a little skeptical about a mono repository, the k/k one seems to have a metric ton of release tooling around it because it's not a nice experience.

We do however need to ensure that we stop cyclic dependencies in the downstream repositories.

@xmudrii with your expertise in k/k release engineering, would you recommend kcp to use a mono repo?

[xmudrii]

I'm a little skeptical about a mono repository, the k/k one seems to have a metric ton of release tooling around it because it's not a nice experience.

Not necessary, the process is in fact relatively simple, but the tooling is a little fragile though.

The tl;dr is that you need to run publishing-bot in some cluster to sync between the mono repo and actual (staging) repos. Sometimes, the bot gets broken, but it's usually simply fixed by removing its volumes and re-running it. Other than that, it's relatively low effort to keep the bot running.

The biggest issue IMO is the developer experience, as there are some shenanigans needed to get it working. Out of the box, if you make some change in staging and compile kcp, you might not see those changes, as Go will use the code that's in the actual repositories instead of staging. I think k/k solved it by adopting Go workspaces, e.g. see https://github.com/kubernetes/kubernetes/blob/master/go.work

with your expertise in k/k release engineering, would you recommend kcp to use a mono repo?

Honestly, I can't tell without getting some more context. What do we exactly want to stage?

Side note: we have a maintainer of the publishing-bot here 🙂 (cc @sttts)

[gman0]

The point I'd try to work in is to have a single source of truth for a list of packages known to work well together, and then doing updates in bulk. How we implement that is a separate problem, but I'd like to see if we agree on (somehow) centralizing what deps we use across the modules (and repos), and then work towards that.

If I understand correctly, k/k does this by having a mono-repo structure for most of their sub-projects, with a root go.mod that's redistributed into the respective staged sub-projects. At the moment we have no of such tooling in place, and each repo somehow maintains its deps independently, and hopes that the through transitive dependencies things will be fine. But there have been issues where version skew can become a problem that's hard to debug, or when distributing security fixes into modules, etc.

An alternative to that would be to externalize the go.mod (in whatever shape or form) into a separate repo and prepare scripts/workflows to consult it when updating dependencies. We'd have to find a way to generate the list of deps we want to enforce somehow too. That's why mono-repo sounds like an easier way to do this :) but perhaps easier to start with, for a small set of repos? Although that's also true for mono-repo since the original repos still exist, albeit as a "mirror" of their staged/ home.

---

As to which repos we would want to have managed deps (through staging or having a reference go.mod): I'm not sure. In general, the more the better, as that means less work to manage deps individually. We can start small and enroll more repos in the future. There are some repos that wouldn't be eligible too for sure: e.g. kuberentes fork for obvious reasons, or repos that have release cycle separate from core kcp.

[gman0]

#3373 was an attempt to use kcp-dev/kuberenetes as that source of truth and check at that at least core kcp has its deps in sync. A first step, but not really a solution for the overall problem we have in the rest of our repos.

[embik]

Overall, I think I would be in favour of moving things like client-go and apimachinery into a monorepo. If we nail down the automation based on k/k (given the expertise we have here) we should have less running around cutting releases for repositories that are effectively in sync with kcp-dev/kcp.

[xmudrii]

/assign @xmudrii @gman0