Automatically set GOMAXPROCS and GOMEMLIMIT based on available resources (container cgroups / system)

[frittentheke]

Feature Description

While k6 itself does take the number of CPUs into account, e.g. at

k6/execution/scheduler.go

Line 268 in f495046

doneInits := e.initVUsConcurrently(subctx, samplesOut, vusToInitialize, runtime.GOMAXPROCS(0), logger)

(planned at

k6/output/cloud/expv2/output.go

Line 116 in f495046

// change the default of cloudapi.MetricPushConcurrency to use GOMAXPROCS(0)

) the value has to be configured manually and explicitly.

So manual adjustments to GOMAXPROCS and GOMEMLIMIT are required to tune K6 to work best with the available resources of the system (hardware, VM, container).

I use k6s on Kubernetes and via the k6-operator and the provided TestRun CustomResource does allow for requests and limits to be configured. While these settings are then applied to the PodSpec, they are not converted or made available to the corresponding golang settings GOMAXPROCS and GOMEMLIMIT.

While this capability could be added to the k6-operator (shout out to @yorugac), to me it makes more sense to enable k6 itself to auto-tune itself to the available resources as it can be run without the k6 operator or non-containerized and will still benefit from these auto-tuning capabilities. There is a similar suggestion for the Grafana Helm chart, also discussing the use of auto-tuning for the options: grafana/helm-charts#3138 (@jnoordsij, @jkroepke)

Therefore I'd like to propose adding automatic config of the Golang runtime via well established golang libs, which are actually
commonly used for other Grafana tools:

GOMAXPROCS

• https://github.com/uber-go/automaxprocs
• https://github.com/search?q=org%3Agrafana+automaxprocs+&type=code

GOMEMLIMIT

• https://github.com/KimMachineGun/automemlimit
• https://github.com/search?q=org%3Agrafana+automemlimit&type=code

There are many examples / MRs for the proposed mechanisms / libraries for other popular tools ....

• GOMAXPROCS for Prometheus Add auto-gomaxprocs optional feature prometheus/prometheus#10498
• GOMEMLIMIT for Prometheus prometheus/prometheus@2c0f9d1
• GOMAXPROCS for Kubernetes Investigating detecting GOMAXPROCS based on cgroup values kubernetes/kubernetes#116997
• https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4142
• ...

Suggested Solution (optional)

No response

Already existing or connected issues / PRs (optional)

No response

[olegbespalov]

@grafana/k6-core, what do you think?

[joanlopez]

@grafana/k6-core, what do you think?

I don't have specific experience with these libraries, but it definitely sounds as something that may worth looking at and trying out.
Although it isn't probably something with really high priority, tbh.

[SuperQ]

We use both (uber-go/automaxprocs and KimMachineGun/automemlimit) in Prometheus. They work very well to control resources.

[codebien]

Note, once Go 1.25 is released, this issue will become less relevant, because the default value for GOMAXPROCS will then consider the cgroup CPU quota.

[frittentheke]

Note, once Go 1.25 is released, this issue will become less relevant, because the default value for GOMAXPROCS will then consider the cgroup CPU quota.

True, but this does not cover GOMEMLIMIT (discussed in golang/go#75164 though).
Until then I believe https://github.com/KimMachineGun/automemlimit would be a nice and simple addition.