Status of golang support? Seems very slow overall

[HenrikPoulsen]

I saw in the docs that golang support is in the beta phase, however it doesn't really address how usable it is.
I tried it out on a monorepo I have with 10 or so local modules (~80 dependencies and ~200 indirect dependencies, aka pretty small as monorepos go) and it seems very rough around the edges for sure.

The main issue/deal breaker imo is that in my case it takes 20+ minutes to just download and analyze and build all the modules.
It also ends up taking ~90% of the available system memory (of which I have 64GB total) while it's running

Why does it take this long and why does it not, seemingly, use the GOPATH cache for this which already has everything downloaded?
Is this expected and by design?
Is there a config issue on my end? If so how do you debug that?

By comparison go test ./... after doing a go clean --modcache it takes ~10 seconds to pull all the packages. On subsequent runs it takes no measurable time before it starts running the tests. For pantsbuild it takes 60+ seconds on a rerun before it starts running any tests, since it seems to be rechecking all the dependencies again, quite slowly.

Is this the expected behavior or is there some other issue at play here for my specific setup?

[benjyw]

Thanks for starting the discussion. Going to pull @tdyas in, as he's the Golang guru, and is better placed than I am to answer specific technical questions, especially around the performance questions, which do seem pathological, and may fixable. That said, slow "time to first test" is an ongoing issue with Pants, across all backends, and one we are looking at various ways of addressing.

FWIW I think the status of the golang backend is "needs ownership/sponsorship". Anyone can jump in and work on the parts of Pants that they care about, so where we see the most PRs is an indication of the priorities of the more active maintainers (who work on Pants on behalf of their companies). The golang backend sees relatively few such PRs, and is therefore maintained on a best-effort basis by the core maintainers, who likely aren't using it themselves for their own projects, and so aren't exposed to such issues on a personal and regular basis.

We would love to have more contributors to the golang backend. Or alternatively, it is possible to contract projects on it out to freelance members of the community. See https://www.pantsbuild.org/spotlight/service-providers for example.

[frekw]

We're also experiencing this.

After having exported the pants diagnostics and imported it as traces, it seems like the majority of the time is spent in

pants/src/python/pants/backend/go/target_type_rules.py

Line 361 in 6cac603

async def generate_targets_from_go_mod(

[benjyw]

@tdyas this tracks with what we know, right?

[aleclerc]

Any updates here? While generally things are functional, they are far from optimized.

[tdyas]

No updates to be had really. The points that @benjyw made in his prior comment still remain true; namely, that the Go support "needs ownership / sponsorship" and that the current Pants maintainers are not actively developing it. Members of the community remain free to contribute changes whether by doing it themselves or employing (including contracting with) someone else to make the changes.

Personally, I do not see much future for the Go backend unless changes in go's build logic are mirrored in Pants rules on a regular basis (which would require code contributions). For example, test coverage no longer works with Go 1.25 because Pants does not support the "coverage redesign" introduced in Go 1.20. See this comment and the error in this comment.

Alternate approaches are likely more sustainable, e.g. using Pants macros and the shell backend to just invoke the go tool directly. Then there would be no need for the Go backend.

[benjyw]

Yeah, I agree with @tdyas . The currently active contributors aren't using go, so unfortunately it's not getting a lot of love. As an all-volunteer project with no big corporate backer we rely on OSS contributions, and these naturally tend to be in the directions that the contributors care about for their day jobs... As I mentioned above, there are ways to sponsor such development, if it's worth it to you. But of course the ideal situation would be for some folks who are invested in Go and want state-of-the-art build system support for it to jump in and "own" that backend. We'd be happy to ramp such people up.