Call Data Encoding

[Cryptverse]

Ethers Version

5.5.4

Search Terms

bytes encoding

Describe the Problem

I have been having a hard time using the high-level transaction methods. It seems when I populate a function that uses a bytes argument for call-back data it does not encode it properly. I have resulted to manually encoding the data in order to get the data to be reflected correctly on Etherscan.

It seems like if data is passed to a contract populate transaction function that contains bytes, the bytes aren't properly encoded. There may be an easy way to do this but I have ended up writing my own encoding and decoding functions for bytes data. This is very inefficient because many implementations need to be developed in order to deal with different functions calls whereas much fewer would be needed if the populate transaction function would work as expected.

If this truly is a bug then I'm sure I could be of assistance in adding this functionality as I have already developed my own methods to achieve the correct encodings.

Code Snippet

No response

Contract ABI

function swap(uint amount0Out, uint amount1Out, address to, bytes calldata data) external lock

Errors

No response

Environment

No response

Environment (Other)

node

[ricmoo]

Can you please provide code examples of what you are passing in, what you are getting and what you are expecting?

[Cryptverse]

const populatedTransaction = contract.populateTransaction.flash(recipient, amount0, amount1, bytesData)
const data = populatedTransaction.data
const parsedTransactionData = contract.interface.parseTransaction({data})
console.log(parseTransactionData)
/*...
recipient: 'address is fine',
    amount0: BigNumber { _hex: '0x00', _isBigNumber: true },
    amount1: BigNumber { _hex: '0x1acc97c07fffffff', _isBigNumber: true },
    data: '0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000...
I guess the data is also fine above, as well, since the parser is supposed to return the arguments as they were passed in.
*/ 
console,log(data)

/*
0x490e6cbc0000000000000000000000000000addressIsFine000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001acc97c07fffffff00000000000000000000000000000000000000000000
0000000000000000008000000000000000000000000000000000000000000000000000000000000002e0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001500000000000000000000000000000000000000000000000000000000000000000000000002a90...
*/
//I've never looked directly at the data like this before and I can see the encoding is done properly here. So, my issue might be more 
//related to bytes decoding and, perhaps, this may be a too specific use-case for ethers to have to resolve. However, the issue I've had to 
//solve came from when I tried to decode this data as a uint256[] like so:


const decoded data = defaultABICoder.decode(['uint256[]', [data])
//throws: Argument of type 'string[]' is not assignable to parameter of type 'BytesLike'.
//Type 'string[]' is not assignable to type 'string'.

The 'bytesData' was a uint256[] that was encoded before being passed to the flash populator. I haven't succeeded in decoding 'bytesData' back into a uint256[] using ethers. Yet, I require being able to decode all of 'data' into a uint256[]. I've succeeded in doing so by writing a splitDataHexString function. However, I had thought that this would have been something that ethers would be able to do. Perhaps, I am using it incorrectly or, as I've mentioned, this might be too specific a use case.

To prevent any confusion or fix my incorrect labelling of types, I mean a BigNumber[] or string[] when I say uint256[]. Thanks for the fast reply @ricmoo.

[ricmoo]

Please be patient. You have to understand I put anywhere from 200-300 hours a month into ethers, and before committing to adding (and more importantly, maintaining) a feature, I need to be convinced of its general usefulness.

I am not convinced this is something that should be added to ethers, especially since it is easily added as a function where needed and the use-case (which I still do not understand; a bytes and a uint[] are the exact same length, there is no gas savings and a bytes is directly supported by Solidity for decoding and preserves all internal linkage). It is also a function which you have already written (I'd add boundary checks though):

// I have not tested this; expect typos
function split(data) {
  const length = hexDataLength(data);
  if (length % 32) { logger.throwArgumentError("bad length", "data", data); }
  return Array.from(new Array(length / 32), (_, i) => hexDataSlice(data, i * 32, (i + 1) * 32));
}

I don't know what you are really asking then. It sounds like you have solved the problem. Please do not attack members of the community that are just trying to help; @zemse is very knowledgable and one of the very few people I have trusted as a collaborator. I have also read and re-read this thread and it is still confusing to me too. The use-case is important if it is functionality you want incorporated into ethers.

[Cryptverse]

Yes, we are discussing another topic now but that is perfectly okay. I see you are still looking for a good enough reason for this operation to be relevant enough to be considered for an ethers utility. I've come to this point because I try to use as few external dependencies as I can and have seen that using too many, unnecessarily, can lead to many problems. Making use of ethers' utilities has normally made developing a much smoother process for me and it has actually been rare for me to not be able to solve a problem, using ethers.

Naturally, I'd end up going straight to the ethers docs and I'd be able to find a simple solution by using the ethers utilities and various libraries. The byte manipulation utilities was a very daring library for me to explore at first, since I am relatively new to this, but I managed to wrap my head around it and make use of it where I needed. It was just surprising to me, that I was unable to solve this specific issue at hand, because I had expected myself to be able to create a simple solution, as I have done many times before, using ethers' built-in utilities. It just felt like it was missing! So, much to the point that I had thought I must have done something wrong but as it turns out, perhaps, this function truly is missing. Perhaps, the applicability is not obvious as of yet but I have a strong feeling that my case is not the only one and I just feel that having this functionality would just be in the interest of greater utility in the ethers library.

Just to be clear, I've started out specifying uint256[] but see that you refer to it as uint[] and if my understanding is correct, those are exactly the same things so I hope we are on the same page there. Furthermore, according to my understanding, a bytes type is really a dynamic array where each element is a single byte or a uint8, for example. Because of this, I do not understand what you mean by bytes and a uint[] having exactly the same length. I'd agree that a bytes and a uint8[] would have the same length but not a bytes and a uint[]. I've learned this because I had to do the coding manually and when I had to encode an encoded uint[], the uint[] offset and length were present because that was already coded but when I had to encode it again (please don't tell me that I shouldn't be doing this. I know it is correct because I get the correct output), I had to add two to the length (the offset and length on the encoded uint[]) and then multiply by the number of bytes in a uint (there are 32 bytes in a uint) and this would give me the correct value for the length of the bytes in the coding. The offset for the bytes pretty much works the same way as the encoding for the uint[].

So, I do not agree that bytes and uint[] are the same length. Typically, for the same data the bytes encoding would have a length that is the uint[] encoding length multiplied by 32. A uint[] is also directly supported by Solidity for decoding and also preserves all internal linkage but that is not the point here. The point here is that this decoding is not directly supported by ethers. And, yes I have solved the problem so this issue has turned more into a suggestion and quite the stimulating discussion. This stuff is already hard enough to grasp as it is so I did not mean to make things that much more confusing, which becomes the case when getting references to unrelated topics and concepts that should obviously already be understood to be known based on the complexity of the discussion at hand.

The bound check for the splitDataHexString is a good idea. I hadn't had the need to implement it yet since the data I've been splitting has always been byteCount % splitSize === 0. I do think the argument for the size in bytes of each element will be useful because, although I've only been using 32 bytes so far, I'm sure splits of other sizes have many other use cases. Another approach, apart from the bounds-check, would be to just pad the uneven data in the last element. This can be done either according to type or according to a third function parameter that specifies the side and, when omitted, would fall-back to the bounds-check.

I am not sure how close this function is to the internal coders, apart from the actual coding that it does not work on, but I do imagine that the coding functions use something very similar when parsing data and one is not too far away from building their own coders when using a splitDataHexString function. So, either it is being used internally but would benefit developers if exposed or a splitDataHexString might be a better method to use for parsing data for encoding and decoding. I could be very wrong there, though, but if it were the case then there would only need to be one function to fulfil these purposes.

So, just to clarify, ethers is working properly as I'm sure you are all aware. Decoding the data into a uint[] actually fails as it should because it is, technically, not encoded as a uint[]. It works in solidity because it is able to easily expose only the bytes data and, hence, it is easily decoded into a uint[]. However, there is no way to expose the bytes data like this in ethers because all the arguments need to be encoded together in order to preserve the correct bytes encoding and, when decoding the data according to the types it was encoded as, the bytes encoding is lost, which defeats the purpose because the encoding needs to be nested in order to be decoded correctly during execution. So, for this use-case, more advanced byte-manipulation techniques are required and this is my whole motivation behind why I think a utility like this would be a valuable one for ethers to have. Simply put, more complex encoding-nesting operations requires more advanced byte manipulation functions that ethers does not have nor do the standard encoding methods serve this purpose and I am telling you what kind of function helps with exactly this.

Alright, it's been good. Thank you all for your time and I wish you the best of luck in your development work.

[ricmoo]

A bytes is not the same thing as a bytes1[] as you think, which I think is where the misunderstanding is. A bytes is tightly packed, the same way a string is; if you have 640 bytes of data, it will be 32 bytes offset + 32 bytes length (i.e. 640) + 640 bytes for a total of 704 bytes; using your format of a uint[] (which, yes is an alias for uint256[]) is 32 bytes offset + 32 bytes length (i.e. 20) + 20 * 32 bytes per slot for a total of 704. They are the exact same length, except a bytes is portable and implicitly supported by Solidity, ethers and any other Ethereum library.

[ricmoo]

Also, without the bounds check it will produce corrupt data for anything that is not exactly congruent to 32 mod 0 bytes long, as the last word would have numeric padding injected implicitly (by converting through a number), while the correct representation would require bytes padding; so it would move any padding bytes of the last word from the right to the left.

[Cryptverse]

Yes, that makes sense that data would be the same length in bytes whether it is in bytes or uint[]. I may have not explained, as best I should have, my point of the similarities between bytes, bytes1[], uint8[], etc. What I'd tried to point out is that their elements require the same number of bits for representation and that is where they are the same size. Perhaps, you were talking about the amount of memory they use when being stored and that bytes and uint[] uses the exact same amount of memory and I agree. Whereas, bytes and uint8[] would take up different amounts of memory regardless of requiring the same number of bits for representation. I'd also like to point out that even though they may take up different amounts of memory, their size (as in number of elements) is the same.

However, I was not arguing this point because I was talking about the 'length' data that the offset points to when data is encoded. I think it is important to think of it this way because this is the actual length of the data in the context of its execution and also illustrates a horrible example of how gas could be saved where a for loop would have to iterate 32 times more than if the data was decoded as a uint[]. This might not happen in practice too often so it's a bad reason for uint[] being more gas-efficient that bytes. A better reason relates more to my case where the uint[] actually holds its own encoding for the data it stores and the logic that executes the data runs in a for loop that iterates over the uint[]. So, for this case, it seems that bytes encoding is an unnecessary step that is only conformed to because of the bytes call data convention. The uint[] stores encoded data of any type as well as bytes data that is actually nested encodings of different sequences of itself. These different sequences are encoded as such so that execution data is decoded properly when sent through nested call-backs. So, yes @ricmoo, I agree with you that bytes is a great type to use but, for this case, bytes encoding is not enough and when the bytes encoding itself needs to be encoded then encoding it as uint[] has been a brilliant encoding to use.