Decoding a struct from provider.getStorageAt

[bmateus]

I have a struct

struct MyStruct {
address address;
uint16 offset;
uint16 size;
}

that is stored in contract memory as:
"0x0000000000000000035f0689b7b8ac0d4cb526ff06a3f70a13d20652c7f45082"

Is there a way to decode this back into a struct with ethers?

I've tried:
ethers.utils.defaultAbiCoder.decode(['address, uint16, uint16'], data);
ethers.utils.defaultAbiCoder.decode(['tuple(address, uint16, uint16)'], data);
but get "Error: data out-of-bounds (length=32, offset=64, code=BUFFER_OVERRUN, version=abi/5.5.0)"
like ethers thinks these need to be in 2 slots?

[scherroman]

I've also been wondering the same thing. I couldn't find a single example of decoding a packed storage slot value anywhere, either for ethers.js, web3.js or otherwise. Closest article I found was here but they don't show their work for the packed slot example. Seems like a pretty important thing to be able to do easily to audit smart contracts that make certain storage variables internal/private, which are often important to the behavior of the contract.

I couldn't get this to work with decode as it appears to assume the hex value is unpacked, whereas in reality some slot values are packed when neighboring data types are small enough, like in your example above. I discovered the default encoder/decoder apparently aren't meant to encode/decode packed slot values when I tried using them on two uint32s and got this:

const { ethers } = require('ethers')

let encodedUints = ethers.utils.defaultAbiCoder.encode(
    ['uint32', 'uint32'],
    [1, 2]
)
console.log(encodedUints)
// 0x00000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000002

console.log(
    ethers.utils.defaultAbiCoder.decode(['uint32', 'uint32'], encodedUints)
)
// [ 1, 2 ]

The closest thing I found related to packed values was the ethers.utils.solidityPack function, but there's no corresponding decode/unpack function.

In my case I spent the better part of a day trying to figure out how to decode some of the internal variables Chainlink uses in their ETH/USD price feed aggregator contract, like the threshold used, which is the threshold for the "Current bound assumed on number of faulty/dishonest oracles participating". Probably important to know what that's set to if you're relying on it and want to ensure it's configured properly right? After looking into how storage values are packed into 32 byte (256 bit) slots, I plugged all the source code from etherscan into Remix, compiled it, found its storage slot under the storage layout compilation details, plugged it into ether.js' getStorageAt to get the slot value, and since the threshold variable is part of a packed struct, ended up manually segmenting the slot value to extract each item in the struct and decode them separately as if they were each in a slot of their own like so:

const ETH_USD_PRICE_FEED_CONTRACT_ADDRESS = '0x37bC7498f4FF12C19678ee8fE19d713b87F6a9e6'
const THRESHOLD_STORAGE_SLOT = 42
let storageSlotValue = await provider.getStorageAt(
    ETH_USD_PRICE_FEED_CONTRACT_ADDRESS,
    THRESHOLD_STORAGE_SLOT
)

// Location of threshold variable is in a packed internal struct:

/*
struct HotVars {
    bytes16 latestConfigDigest;
    uint40 latestEpochAndRound;
    uint8 threshold;
    uint32 latestAggregatorRoundId;
}
HotVars internal s_hotVars;
*/

// s_hotVars storage slot value
// 0x000000000000000081170a0001ff2c029f88d5f42679b75c4a0244716f695ef3

// Since we know slot values are 32 bytes and the hex representation is 64 characters, each byte is 2 characters long
// So bytes16 -> 16 bytes -> 32 characters
// uint40 -> 40 bits / 8 bits = 5 bytes -> 10 characters
// uint8 -> 8 bits / 8 bits = 1 byte -> 2 characters
// uint32 -> 32 bits / 8 bits = 4 bytes -> 8 characters

// segmented from right to left by types bytes16, uint40, uint8, uint32
// 0x000000000000|00008117|0a|0001ff2c02|9f88d5f42679b75c4a0244716f695ef3

// template '0x0000000000000000000000000000000000000000000000000000000000000000'

// latestConfigDigest = 0x00000000000000000000000000000000
// value = '0x000000000000000000000000000000009f88d5f42679b75c4a0244716f695ef3'
// types = ['bytes16']

// latestEpochAndRound = 33500162
// value = '0x0000000000000000000000000000000000000000000000000000000001ff2c02'
// types = ['uint40']

// threshold = 10
value = '0x000000000000000000000000000000000000000000000000000000000000000a'
types = ['uint8']

// latestAggregatorRoundId = 33047
// value = '0x0000000000000000000000000000000000000000000000000000000000008117'
// types = ['uint32']

console.log(ethers.utils.defaultAbiCoder.decode(types, value))
// [ 10 ]

Anyhow that's certainly error prone and it would be better to write a function to do this. I'm surprised there appear to be no direct utility functions for this built in to these tools, as it's fairly common to encounter this scenario, and they're the only tools I see recommended whenever someone asks how to read storage values directly from a deployed smart contract.

Even if you're not purposefully trying to obfuscate the code to introduce a vulnerability, the default way smart contracts are written still makes it quite difficult to understand exactly how many deployed contracts work by only exposing public state variables that users/viewers are meant to query, while hiding private/internal variables that are meant to be used for implementation details. The value of a private/internal state variable can completely change how a contract works, so IMO it should be just as easy to query private/internal variables, just while having them scoped under a private/internal/hidden namespace (or something like a dropdown for explorer interfaces like etherscan) to maintain the clean public user interface.

All in all, you have to 1. find the exact smart contract source code, often across multiple files, 2. copy them all over and compile them through Remix or a command line tool to find out the exact storage slot of the variable you're looking for (its incredibly time consuming and error prone try to count storage slots manually for even smaller sized contracts), 3. run some code to query for the storage slot value, and then on top of that, 4. manually parse the value or write custom code to decode it if it's a packed slot. Really puts a dent in the common perception that "anyone can view and audit the smart contracts" to figure out what they do. If you're a non-technical user, forget it.

Anyways maybe there is actually an easy way to do this and It's just my relative ignorance at the moment. I think I'll open a stack overflow question about this to hopefully get some further input from the community.

[ricmoo]

For very specific situations, you may be able to create a function that decides a storage slot, but that could easily break between instances of Solidity. The packing solidity uses internally is baked into the contract at compile time, which is one reason giving access to another contract’s storage could be dangerous. Two versions of solidity may store the internal representation differently, or even the same version of solidity could (theoretically) store different internal representations based on the optimizer settings.

For example, the way solidity stores bytes and strings in storage is completely unrelated to how bytes are encoded as ABI; since most strings are short, special sentinel values and byte offsets are used so that strings 31 bytes or less are crammed into a single storage slot, while ABI coding would require 2.

And future versions of solidity are free to change up the game however they see fit, as storage is intended to be more or less private.

So it is very discouraged to try writing any generic function to decode this data; a better solution is to add a method to the contract that acts as a getter that returns the necessary struct, as this means the contract (at compile time) will correctly account for any optimizer and version quirks when unpacking the data from storage.

Does that make sense?

[ricmoo]

(also note that packed values in Solidity cannot be unpacked; it is a lossy process that is one-way, so there is no way to decode them; for example if the output was effectively [ string, string ] and the input was “helloworld” there is no way to tell if it should be [ “hell”, “oworld” ] or [ “hello”, “world” ])

[scherroman]

Thanks @ricmoo, this was very helpful to understand what's going on!

Though I've often heard that no data on deployed smart contracts are "truly" private and can be uncovered if you dig deep enough (even if only imperfectly sometimes like you showed above because packing is lossy). Without the source code I imagine it would be especially hard to uncover specific private/internal state variables, but perhaps they could be uncovered in some cases by decompiling the bytecode to find out the types and offsets for different slots and brute forcing it. Makes me think that it would be all around better to be able to also easily query even internal/private state similarly to public state, to give no illusions that it's actually private, but just a way of being selective about what's exposed in the public interface for the contract. Similar to how python has privacy by convention, where you mark private variables with an underscore, but consumers can still read it if they choose to. Perhaps there are tradeoffs there that would make this undesirable though

[ricmoo]

There is definitely no such thing as private variables (well, except blinded and variables hidden via zk-snarks), but their value should be treated as opaque.

I was going to exactly mention the point regarding the underscore; that is very much how data in storage should be thought of. Reading those variables may very well not be what you expect without internal knowledge of the code (e.g. negative values could represent something completely unrelated) and writing to them will likely break code; so it is best not to mess with them and only use the authorized access control provided by the higher level class. :)

Using code inspection you can absolutely determine what those variables hold, but the content isn’t standardized or consistent, so my only point is that generic tools (like ethers) can’t in general determine what they are.