Skip to content

Add Memory utility library #5189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jul 10, 2025
Merged

Add Memory utility library #5189

merged 27 commits into from
Jul 10, 2025

Conversation

ernestognw
Copy link
Member

@ernestognw ernestognw commented Sep 4, 2024

Picked from #5094

PR Checklist

  • Tests
  • Documentation
  • Changeset entry (run npx changeset add)

Copy link

changeset-bot bot commented Sep 4, 2024

🦋 Changeset detected

Latest commit: f03d149

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
openzeppelin-solidity Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@ernestognw ernestognw requested review from Amxx and cairoeth September 4, 2024 17:35
@ernestognw ernestognw marked this pull request as ready for review September 4, 2024 17:35
@Amxx
Copy link
Collaborator

Amxx commented Sep 5, 2024

IMO this PR should include the usage of this lib

@ernestognw ernestognw requested a review from Amxx September 5, 2024 18:21
Co-authored-by: Hadrien Croubois <hadrien.croubois@gmail.com>
@Amxx Amxx added this to the 5.4 milestone Mar 6, 2025
@Amxx
Copy link
Collaborator

Amxx commented Mar 6, 2025

Its sad we missed that in 5.3, as it is not controversial. Lets fix conflicts and make sure it gets into 5.4

Co-authored-by: Arr00 <13561405+arr00@users.noreply.github.com>
@Amxx Amxx modified the milestones: 5.4, 5.5 Jun 5, 2025
@ernestognw ernestognw marked this pull request as draft June 8, 2025 19:15
@ernestognw ernestognw marked this pull request as ready for review June 9, 2025 00:08
@ernestognw ernestognw mentioned this pull request Jun 9, 2025
3 tasks
@ernestognw ernestognw mentioned this pull request Jun 9, 2025
3 tasks
}
}

/// @dev Returns a `Pointer` to the content of a `bytes` buffer. Skips the length word.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything from here onward (line 33 to 103) was recently added with no discussion about the usecase. I was personally happy with having utils to "cleanup" memory manually, which AFAIK was the original idea behind this PR.

Some of the things here ressemble `Bytes._unsafeReadBytesOffset, which was left private (and not internal) because we could not guarantee the memory safety. This library goes way furter, with assembly marked memory safe when its not something we know (depends on the user input).

Using the copy function, its quite easy to break stuff by writting to the wrong locations (for example to 0x60).

Copy link
Member Author

@ernestognw ernestognw Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The background for these changes is that they're used by the RLP library and we previously kept private:
0332ffe#diff-0399261b2cd954212dda0147b86044380b879ab21e1d1b89c564cb48f3552216L304-L328

I agree with your take. However I think it's worth pursuing this pattern since it composes pretty well imo (as long as we add proper WARNINGS and docs). See how it looks in some places in RLP:

function readRawBytes(Item memory item) internal pure returns (bytes memory) {
    uint256 itemLength = item.length;
    bytes memory result = new bytes(itemLength);
    result.contentPointer().copy(item.ptr, itemLength);

    return result;
}

I would be fine removing these functions. Want to hear your thoughts on the pattern

Copy link

@0xClandestine 0xClandestine Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than exposing these in Memory when they're only used in the RLP library, maybe make them private within the RLP library?

Would also consider constraining destPtr to be greater than fmp offset.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amxx Worth noting the original proveth implementation does essentially the same, but manually, without mcopy.

  /*
  * @param src Pointer to source
  * @param dest Pointer to destination
  * @param len Amount of memory to copy from the source
  */
  function copy(uint src, uint dest, uint len) private pure {
      if (len == 0) return;

      // copy as many word sizes as possible
      for (; len >= WORD_SIZE; len -= WORD_SIZE) {
          assembly {
              mstore(dest, mload(src))
          }

          src += WORD_SIZE;
          dest += WORD_SIZE;
      }

      // left over bytes. Mask is used to remove unwanted bytes from the word
      uint mask = 256 ** (WORD_SIZE - len) - 1;
      assembly {
          let srcpart := and(mload(src), not(mask)) // zero out src
          let destpart := and(mload(dest), mask) // retrieve the bytes
          mstore(dest, or(destpart, srcpart))
      }
  }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed all new controversial functions. Opened #5792 instead. Also updated the RLP PR (#5680) accordingly as suggested @0xClandestine

@ernestognw ernestognw requested a review from a team as a code owner July 4, 2025 21:59
@ernestognw ernestognw requested a review from Amxx July 9, 2025 18:33
Co-authored-by: Arr00 <13561405+arr00@users.noreply.github.com>
Co-authored-by: Arr00 <13561405+arr00@users.noreply.github.com>
arr00
arr00 previously approved these changes Jul 9, 2025
@Amxx
Copy link
Collaborator

Amxx commented Jul 9, 2025

Should be covered by #5094

@Amxx Amxx merged commit 21cd7e8 into OpenZeppelin:master Jul 10, 2025
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants