op-devstack: resolve race conditions arising from `dsl.EOA` abuse

[joshklop]

Acceptance tests often assume that they have complete ownership of a *dsl.EOA, when in fact multiple may be using the same account at the same time. There are three ways to solve this:

1. Strengthen the *dsl.EOA abstraction to assume concurrent ownership of the onchain account is possible. This primarily impacts funding and nonce management.
2. Run all tests serially and assume the account cannot be used concurrently outside of acceptance tests.
3. Stop writing acceptance tests as normal Go unit tests. Make a custom test runner instead. Then we can compile a single program that contains all of the tests, and runs them in parallel or in sequence. EOA ownership could be regulated through a master wallet object that is injected into each test.

I'm currently leaning toward option 3: it's probably not even that hard to do. It should also help substantially decrease test times in CI, since we should be able to parallelize almost all of our acceptance tests. We can do Option 2 as well to solve problems where transactions sent by a previous owner of an EOA affect its current owner.

[pcw109550]

Some tests cannot be parallelized by its nature even if we use the custom runner, for example some acceptance tests must have ELP2P enabled or disabled.

Also for "custom test runner", is this what op-acceptor is for? For example

# Instead of:
go test ./acceptance-tests/...

# Use:
op-acceptor --testdir ./acceptance-tests/...

For

EOA ownership could be regulated through a master wallet object that is injected into each test.

op-acceptor may set env variable and tell the op-devstack to use the external wallet object, and may be smart enough to use different wallet per tests. For example, DEVNET_EXPECT_PRECONDITIONS_MET.

[joshklop]

Some tests cannot be parallelized by its nature even if we use the custom runner, for example some acceptance tests must have ELP2P enabled or disabled.

Yeah after sleeping on it I think the idea is not quite as powerful as I initially thought.

Also for "custom test runner", is this what op-acceptor is for?

op-acceptor wraps the go test runner. Because the go test tool compiles and runs each package as a separate binary, we cannot synchronize EOA use across packages.

[ajsutton]

Acceptance tests often assume that they have complete ownership of a *dsl.EOA, when in fact multiple may be using the same account at the same time.

This is a perfectly reasonable assumption - you are going to be in a world of hurt if you try to use the same EOA from two parallel processes - it's just impossible to sort out the nonces and other effects on balance etc. We should be using the faucets to fund new, unique wallets on request for tests so they are always completely isolated.