nonce for the sequential transactions

[novaknole]

Hi Richard, @ricmoo

I am having something like this:

let a = contract1.function1()
let b = contract2.function2()

I've not used await on any of them yet.

Question 1: does the nonce get set on them without await ing ? meaning that if I do await a && await b now, will one of them fail because nonce was already set above on each one of them ? or does the nonce get set(getting it from getTransacionCount) when I do the await a ?

Question 2: If the answer to the above qeuestion is that nonce gets set when the await is used, then it means that if I call await a && await b together, their nonces will be the same. I guess, one way is that I don't call b unless I waited for tx.wait() on the a so that it's been mined which means calling await b after that will get new nonce. everything correct ?

Question 3: what If I still want to do await a && await b right away ? is the NonceManager only solution. ?it says it's not stable and is experimental ? is it worth using it ?

[ricmoo]

Heya!

So, keep in mind that the library cannot detect await and is just syntactic sugar that does all the .then/.catch stuff in a way that appears to be sequential.

That said, if you do not await, the flurry of asynchronous tasks involved in computing a transaction do not have any guarantee on their order, so if you prepare and send multiple transactions there is no way to know for sure what order the various calls internally will occur.

Part of the async operations a contract will perform is using the signer.getTransactionCount("pending") to compute the nonce, but due too the eventually consistent nature of Ether, there is no guarantee that two sequential calls to signer.getTransactionCount("pending") will return different values if a transaction was submitted between the two calls, and some backends do not honour the "pending" block tag at all (as that involves transaction pool inspection, which can be a problem for large clusters of nodes).

A call to contract.function() will not resolve until the transaction has been prepared and sent to the network, so once resolved it will have some nonce.

If you are attempting to send multiple transactions without waiting for a transaction to be mined, you most certainly should use the NonceManager or for small numbers of transactions, manual nonce management is also a good option.

The main reason for the NonceManager remaining in experimental is that it doesn't currently support re-broadcasting. For a couple transaction, a node will keep the transactions (if it believes they are likely to be accepted into a block) in the mempool. But it is a DoS attack vector to keep too many in the memory pool, so if you try sending a lot of transactions at once (for example, even 10 is likely too many), the first 2-3 might get mined, but those beyond that are simply dropped on the floor and need to be resent to the network. This will be added in the future, watching each block for mined transactions and rebroadcasting transactions past what was mined. A callback is also required for handling workflows if a transaction fails to recover...

Another caveat is that gasLimit uses estimateGas if left null, which can also lead to issues without waiting for transactions to be mined. For example, if your current balance is 0 ERC-20 tokens, and want to approve 10 tokens, to then immediately send 10 tokens; calling estimateGas on the transfer 10 tokens will fail (since from the estimateGas' point of view the balance is still 0. In these cases, you are best to specify your gasLimit explicitly.

One more thing to keep in mind though, is that if the order of the transactions matter, you probably want to manage the transaction flow more carefully, in case transaction A fails, you may require some sort of clean-up or intervention before transaction B.

Basically, stacking multiple transactions is complicated. I'm crossing my fingers we get Rich Transactions soon. :)

I don't know if this really helped answer your questions directly, but hopefully it explains some of the complications.

[novaknole]

@ricmoo Thanks a lot for this detailed answer. Really appreciated. Yes, Rich Transactions is exactly what I need. Somehow, something doesn't make sense and I will ask again.

Question 1:

So as far as I understood, the below sets nonce on them and most definatelly, the nonces will be the same.

let a = contract1.function1()
let b = contract2.function2()

Which makes me believe that below will fail.

await a;
await b;

but you mention that

A call to contract.function() will not resolve until the transaction has been prepared and sent to the network, so once resolved it will have some nonce.

Which seems like that contract1.function() shouldn't be setting a nonce yet, unless we do
contract1.function().then(() => { console.log("nonce is set, tx is sent to the network");.. but I think this is sending to the network, meaning that the contract2.function2().then(() => {} before sending to the network would grab the nonce like this - signer.getTransactionCount("pending"). why is not this considering the fact to return + 1 nonce. we use pending keyword, so it should be returning the increased nonce, because contract1.function() was already sent to the network's mempool.

---

Question 2:

Would your newest JSONBatchRPCProvider be a good solution ?
case 1: Would this handle sending transactions one after another ? meaning that second shouldn't get executed until the first one is mined, third one shouldn't get executed, until the second one is mined and so on. Would JSONBatchRPCProvider handle this ?
case 2: I guess, sending transactions sequentially without waiting for each other, this is something JSONBatchRPCProvider wouldn't handle, due to nonce being set in a wrong way, right ? I think for this case, NonceManager is the only solution.

What do you think ?

[ricmoo]

Question 1

Imagine the following two cases:

1. A = contract.function1(); B = contract.function2(); await A; await B;

The tx A will eventually request a nonce in its async chain and tx B will eventually request a nonce. But they are simply requests being made to the network, there is no way to be sure which will occur first and more importantly which will resolve first. They are just fetches in some big async pool.

2. A = await contract.function1(); B = await contract.function2();

Now, the value of A will eventually request a nonce. And B won't get a chance until A is complete (got a nonce) and sent to the network. So, B will have its nonce set after A has been submitted. (Note: these events are likely milliseconds apart)

But, it is important though to understand what "pending" means. It means, "please check the mempool and try to guess what the current nonce is based on the mempool".

Firstly, not all nodes will even look at the mempool (some may not even have access to one, such as light clients), so some backends will just use "latest", regardless of what you do.

But even if a backend supports a mempool, imagine a system like INFURA, with 1000's of nodes across multiple data centres geographically isolated from each other. When you make a request for A, you are assigned a random node in a random data center and given the nonce X. Now you send the transaction, with nonce X to another random node in a random data center; and INFURA begins spreading this across the mempools of the 1000's of nodes. Now you request a nonce for B, which is assigned to a random node in a random data center; you might happen to hit a node that has an updated mempool and get X+1, or you might get assigned a node that does not and get X again.

This problem is further exacerbated by the fact that there are multiple backends (you might request Alchemy for X, send to Etherscan and ask INFURA for X+1). And each node may have its own custom rules for what it considers "likely to succeed", so the mempools can vary widely.

The mempool is not a reliable source of data, and has no economic guarantees.

Question 2

You would have to experiment with this.

If you are thinking you could split this into two transactions A: [ X = getTransactionCount ] and B: [ sendTransaction, X+1 = getTransactionCount ], that might work but would mean you are limited to only using JSON-RPC endpoints, and still have no guarantee that the backend would always honour mempool scanning; it might work now, it might work for a bunch of requests, but there is no way to be assured it would always work on all environments. The mempool affords a lot of leeway to implementors and configuration.

Recommendation

In general, I would recommend handling the nonce your self (or via the NonceManager). If you have a UI, a good example to look at is the ENS manager or Uniswap interface for how to handle multiple transactions.

Does that help?

[novaknole]

Yes, that helps. A couple of quick points i will add..

const provider = new ethers.providers.InfuraProvider("rinkeby");
const wallet =  new ethers.Wallet("your-private-key")
const signer = wallet.connect(provider);


const address = '0x2e89bc4194cc477438cc935a0464f06415ac5845'
const abi = ["function store(uint256) public"]
const contract = new ethers.Contract(address, abi, signer);

Now,

async function test() {
    await contract.store(1)
    await contract.store(2)
    console.log("good1");
}

This will always succeed.

and this will always fail:

async function test() {
    const a =  contract.store(1)
    const b =  contract.store(2)

    await a;
    await b;
    console.log("good1");
}

Correct ?

I noticed you also use getTransactionCount('pending') here -

ethers.js/packages/abstract-signer/src.ts/index.ts

Line 205 in ce8f1e4

if (tx.nonce == null) { tx.nonce = this.getTransactionCount("pending"); }

So, it depends on what provider I pass to ethers.js. If I pass provider that don't support pending mempool, then it will use latest and I have to deal with problems by myself.

Correct ?

About question 2.

If I pass the provider that is mine, meaning it's a node which is full node, then the problem about mempool scanning doesn't appear. because I always request nonce from my node and send tx through my node. Correct ?

Thanks a ton for all the answers.

[ricmoo]

and this will always fail:

I don't understand what you mean by always fail? Without the awaits, it could work just fine. Unless you mean because the nonce won't increment, but they could by chance increment, and if you use something like NonceManager, it will always work, you just might not get the transactions in the order A then B.

So, it depends on what provider I pass to ethers.js. If I pass provider that don't support pending mempool, then it will use latest and I have to deal with problems by myself.

Yes. But NonceManager (a Signer) handles that for you, regardless of what Provider you use.

because I always request nonce from my node and send tx through my node

That's up to your node implementation to decide.

For example, nodes running in light-client mode may not work as you expect.

Also, if the private key in Geth is not managed by that node (e.g. you use a Wallet instance and the private key is not installed in Geth), you will have similar limits on the transaction pool queued backlog for the from address.

My guess is that this will likely work, but it is not guaranteed to. I would also recommend against relying on this; if your code relies on exactly a single node as the gatekeeper, in the future when you change to something else it could start failing, and by then you'd forget this assumption you made back in the day. :)

[novaknole]

@ricmoo

async function test() {
    const a =  contract.store(1)
    const b =  contract.store(2)

    await a;
    await b;
    console.log("good1");
}

I tested this and it always fails with replacement fee too low.. How the above code might sometimes work ? I saw the

ethers.js/packages/abstract-signer/src.ts/index.ts

Line 122 in ce8f1e4

sendTransaction(transaction: Deferrable<TransactionRequest>): Promise<TransactionResponse> {

and I can see that even when I type contract.store(1) without any awaits, it executes sendTransaction which populatesTransaction and figures out the nonce, but it doesn't send the tx to the network unless i do the await. So the above code will produce the same nonce for a and b all time..

How is this not the case ?

[ricmoo]

That is because the nonce is the same, so it looks like you are trying to replace the transaction, which requires a large bump in gasPrice.

If you specified separate nonces for them (via overrides) or used a Signer that manages the nonce itself, it would work just fine. Or if by chance, the sendTransaction of A completes before the getTransactionCount() gets called, it might work.

So, for example these would succeed:

const noncePromise = signer.getTransactionCount();

const a = contract.store(1, { nonce: noncePromise });
const b = contract.store(1, { nonce: (noncePromise).then((n) => (n+1)) });

await a;
await b;

contract = new Contract(address, abi, new NonceManager(signer));

const a = contract.store(1);
const b = contract.strore(2);

await a;
await b;

Also, it is possible the order of async operations could just succeed by accident if the order of the event loop resulted in A.getTransactionCount(); A.send(); B.getTransactionCount(); on a node with an effective mempool. It is random, but it could. Also, never rely on that. :p

So, there is nothing inherently wrong with dropping the awaits, but it depends on the situation. It's a wonderful and complex tapestry. :)

[novaknole]

@ricmoo

I think we are misunderstanding each other.

You mention this:

Also, it is possible the order of async operations could just succeed by accident if the order of the event loop resulted in A.getTransactionCount(); A.send(); B.getTransactionCount();

How can this happen in this case?

const a = contract.store(1);
const b = contract.strore(2);

await a;
await b;

contract.store(1) will call sendTransaction on your library. It then calls populateTransaction which has getTransactionCount('pending'). Let's say this returns 1. That's it. transaction has been constructed, but hasn't been sent to the network. Now, contract.store(2) gets called since it's the below line. Now, the same thing happens, and getTransactionCount('pending') will again return 1 because contract.store(1)has not been sent to network.

This is why I think the above code never succeds. Could you provide an example of when this order might happen ? A.getTransactionCount(); A.send(); B.getTransactionCount(); ?

[ricmoo]

This is hard to type, but I will try to graph it out here. Assuming:

a = contract.store(1);
b = contract.store(2);

We will call these requests A and B (respective to a and b above). Here is an example path through the event loop that can succeed, at each loop time step:

(keep in mind each request to a provider hits the network, which means that there can be an arbitrary duration for the command to execute; could be 0, or could be large, especially when the the backend is returning 429 responses, so pending operations in the event loop can literally resolve in any order after any amount of time)

Step 1: [
  A, B
]

Step 2: [
  A: [
    A.nonce = signer.getTransactionCount,
    A.gasPrice = provider.getGasPrice,
    A.gasLimit = provider.estiamteGas,
  ],
  B: [
    B.nonce = signer.getTransactionCount,
    B.gasPrice = provider.getGasPrice,
    B.gasLimit = provider.estiamteGas,
  ]
]

Step 3: [
  A: [
    A.nonce = signer.getTransactionCount,
    A.gasPrice = provider.getGasPrice,
    A.gasLimit = provider.estiamteGas,
  ],
  B.send = provider.sendTransaction,
]

Step 4: [
  A: [
    A.nonce = signer.getTransactionCount,
    A.gasPrice = provider.getGasPrice,
    A.gasLimit = provider.estiamteGas,
  ]
]

// Note B is now complete and awaiting on it would pass.

Step 5: [
  A: [
    A.nonce = signer.getTransactionCount, // this may see the updated value in the mempool now
    A.gasPrice = provider.getGasPrice,
    A.gasLimit = provider.estiamteGas,
  ]
]

Step 6: [
  A.send = provider.sendTransaction
]

Step 7: [
]

// The event loop is now drained

// B got nonce and A got nonce + 1, by luck... Also, prolly not what was desired

// These will now pass
await a;
await b;

It's hard to narrate the the ebbs and flows of the event loop, but hopefully that helps. :)

[novaknole]

@ricmoo

Thanks Richard for all the help. needed a fresh air. Hahah. I looked at your code and now it makes sense.

I thought that in the function you were awaiting each async functions, but turns out you don't await and just call them which means that contract.store(1) ends right away and all the async function it calls inside go into event loop.

Thanks a lot again.

Have a great weekend.