Unable to run Testground pex-convergence with 48 nodes

[aratz-lasa]

Description

When running convergence test case in Testground with more than 48 nodes, the execution fails. There are times during the execution that nodes are not able to dial another peer.

Testground command: testground run single --plan=casm --testcase="pex-convergence" --runner=local:docker --builder=docker:go --instances=48

Error output is:

failed to dial QmbwgXDBnCvDC1XYP7SzfckdScYWFpQNhPUJQmEPgcQwsK:
  * [/ip4/192.18.0.41/tcp/46245] dial tcp4 192.18.0.41:46245: connect: network is unreachable
  * [/ip4/16.0.0.41/tcp/46245] dial tcp4 0.0.0.0:34309->16.0.0.41:46245: i/o timeout
  * [/ip4/127.0.0.1/tcp/46245] dial tcp4 0.0.0.0:34309->127.0.0.1:46245: i/o timeout

Ideas

The main idea is that the network and containers get overloaded. Convergence test case makes use of redis barriers in every iteration, in order to syncrhonize nodes. However, redis barriers are known to have a big overhead.

[aratz-lasa]

In order to test whether barriers present too much overhead, a new branch with no barriers was created. The branch can be found at no-barriers branch.

However, it also raises errors, but this time due to name resolution:

failed while signalling entry to state finished: dial tcp: lookup testground-redis: Temporary failure in name resolutiongoroutine 1 [running]:
runtime/debug.Stack(0xecfdd5, 0x2, 0xc0002479c0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/testground/sdk-go/runtime.(*RunEnv).RecordCrash(0xc00021a000, 0xdb9420, 0xc0005045c0)
	/go/pkg/mod/github.com/testground/sdk-go@v0.2.7/runtime/runenv_events.go:245 +0xa5
github.com/testground/sdk-go/run.invoke.func3(0xc00021a000)
	/go/pkg/mod/github.com/testground/sdk-go@v0.2.7/run/invoker.go:135 +0x65
panic(0xdb9420, 0xc0005045c0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/testground/sdk-go/run.invoke(0xc00021a000, 0xd92620, 0x102e0b0)
	/go/pkg/mod/github.com/testground/sdk-go@v0.2.7/run/invoker.go:173 +0x505
github.com/testground/sdk-go/run.InvokeMap(0xc0000f9e38)
	/go/pkg/mod/github.com/testground/sdk-go@v0.2.7/run/invoker.go:65 +0xa9
main.main()
	/plan/main.go:14 +0x145

[aratz-lasa]

Another test case was created in order to check whether name resolution problem arise because of overloading the system. The test simply runs N nodes, and all except one node, call a Redis barrier. Preliminary tests show that if 200-500 nodes are used, Testground raises name resolution errors.

Error output:
{"ts":1636108935146165512,"msg":"failed while getting barriers; iteration skipped","group_id":"single","run_id":"c62ggeeqmkrnfmls1p40","process":"barriers","error":"dial tcp: lookup testground-redis: Temporary failure in name resolution"}

and also

failed to send batch to InfluxDB; attempt 1; err: Post "http://testground-influxdb:8086/write?consistency=&db=testground&precision=ns&rp=": dial tcp: lookup testground-influxdb: Temporary failure in name resolution

Command to run new test case is: testground run single --plan=casm --testcase="dns-test" --runner=local:docker --builder=docker:go --instances=500 .