VM v5 Release Planning

[holgerd77]

Now with the monorepo transition being in its latests phases we can start on planning the next major VM release, various things are on the door for this. I'll try to collect here what I can think of and we can further discuss, if we have some agreement on further additions please just add to the list below:

TODO List

• VM: Init method to prevent race conditions PR Add initialization method to prevent race conditions #665 by @rumkin, review/merge , @evertonfraga
• VM: Opcode refactor PR Refactor opcodes #664 by @rumkin, review/merge, @cgewecke, @alcuadrado
• VM: Integrate Block v3.0.0 and new Blockchain versions, @evertonfraga
  • Block library TypeScript v3.0.0 (see New release v3.0.0 ethereumjs-block#74, work already done, just needs release PR + npm publishing), PR Block release v3.0.0 #682, @evertonfraga
  • Blockchain: Integrate Block v3.0.0, promisified Blockchain version Promisify blockchain and ethash packages #833 , @evertonfraga @ryanio
• VM: Integrate MPT release v4.0.0, @ryanio, PR Update merkle-patricia-tree to v4, move account trie-related methods to StateManager #787
  • Block: integrate into Block library - eventually along v3.0.0 Block release v3.0.0 #682, @evertonfraga
  • Account: eventually integrate into Account library (devDependencies for tests), @ryanio, update: dependency removed, PR Update merkle-patricia-tree to v4, move account trie-related methods to StateManager #787
• VM: Extract the state manager to have its own package, see section in VM README for context, @ryanio, update: decided (missing reference, can't find) to keep it integrated for now
  • Integrate promises into StateManager, Integrate promises into StateManager #719
  • Add Interface for StateManager, StateManager interface #763
• All: Update ethereumjs-util to v7.0.2, Update ethereumjs-util to v7 #748 (double-check if all occurences updated!)
• All: Update to a a new ethereumjs-config version integrating e.g. Upgrade primary build target to ES2017, add browser build ethereumjs-config#28 (ES2017 build target, separate browser build)
• Analyze and fix speed deterioration of MPT v4 library, see Analyze speed deterioration of v4 merkle-patricia-tree#124, @ryanio
  • Add simple benchmarking tool to VM, PR VM: add simple benchmarking tool for mainnet blocks #794, @s1na, @ryanio, @holgerd77
  • Run benchmarks on pre-deteriorated, post-deteriorated and fixed MPT and compare
  • Compare with MPTs own benchmarks and see if consistent
• Finish all Hardfork backports for VM, @jochem-brouwer
  • SpuriousDragon HF support, SpuriousDragon HF Support #791, @holgerd77
  • TangerineWhistle HF support, TangerineWhistle HF Support #807, @holgerd77, @jochem-brouwer
  • Finalize and review Homestead support PR, [VM] fix blockchain tests and add Homestead support #815, @jochem-brouwer, all
  • Finalize and review chainstart (Frontier) support PR, Add frontier support #828, @jochem-brouwer, all
  • Optional but nice (shouldn't bee too much extra work and then we are really complete): add DAO HF, all
• Add Berlin HF EIPs
  • Finalize and review BLS precompile EIP PR, VM: Implement EIP 2537 (BLS12-381 precompiles) #785, @jochem, all -> Postponed/skipped on release due to EIP HF integration uncertainties due to EVM384 alternative proposal
  • Review, eventually finalize subroutine EIP PR, Implement EIP-2315: subroutines #754, @cgewecke, all
• Set VM default HF to Istanbul (MuirGlacier is problematic since not available on all testnets (e.g. Rinkeby which would lead to unexpected behavior if people just change the network without further thinking about this (which is likely))

Be Happy! 🦄 😄 🌻

Discussion

• VM: Do we finally want to expose the reworked Interpreter API Sina @s1na introduced along the v4.0.0 release ( see "EVM Modularization and Structural Refactoring" section as well as linked PR within the section)? Related: VM: Run transaction with custom EVM/Interpreter / API Discussion #636
  • Update: agreement that there is currently still too much movement around this, so won't be integrated in v5 but rather postponed to a subsequent major version release
• VM, merkle-patricia-tree: Do we want to also wait for the TypeScript / Promisification v4.0.0 release of the merkle tree library to add here or do we want to give this a bit more time (this would also need an update on the Block and eventually the Account (only dev dependency) library)? @ryanio
  • -> Update: MPT release is imminent, so this can be integrated, have moved to TODO

Notes

1. Ownership: I've assigned team members already involved or I thought fitting to the tasks, if you can't/don't want to take over just replace with OPEN or something.

2. Date: Just as some orientation, I would roughly target a date within a range of 2-3 weeks for a possible release, let's take April 22, Wednesday as a starter, let me know if you have comments on that

Anything else? Something missing? Other suggestions?

[holgerd77]

Just some notes, before I forget, will work this better into the description above and generally update the issue list early next week.

• Work here turns out to be likely to extensive to match the April 22 date in a couple of days, we'll likely have to move here by at least I would say 2 weeks
• MPT promisification PR by @ryanio is merged (yeah 🎉, great work) and a v4 release is relatively imminent Release - v4.0.0 merkle-patricia-tree#111, so we can likely take a MPT update in
• @cgewecke has committed to do some work on the ethereumjs-util library and finalize Update secp2561 ECDSA dependency to v4.0.0 ethereumjs-util#228, wonder if we should also take this in (would be also a major release on the util library) since this would significantly improve the installation process, which is currently really a pain regarding the crypto packages and source for many complaints. Not sure, Chris will you be able to work on this relatively soon (next week optimally?)?

We generally have to look at the dependency tree (by @evertonfraga) really closely regarding all these things to not miss some updates.

[cgewecke]

Chris will you be able to work on this relatively soon (next week optimally?)?

@holgerd77 Yes, I will start working on this Monday, April 20.

[kumavis]

some ideas from earlier days of ethereumjs-*

• merkle-patricia-tree if you make getting the current state root of a merkle-patricia-tree an async op, you can put off ALL hash calculations for new values until the root is read. if the root is never read, you saved yourself a bunch of work
• vm if you make the storage backend easily pluggable, you can run the vm off of rpc requests
  https://github.com/ethereumjs/ethereumjs-vm/blob/0f10fcfdbc55ceadbff6adeb54bca6df393ef62f/lib/hooked.js#L77-L83 This is hugely useful for anyone who wants to run the vm while accessing state remotely. sad to see support for it was dropped.

basicaclly scifi:

• vm: out-of-order execution. (not speculative execution or branch prediction) a significant change, but if you make all values you push onto the stack to be promises, you get great perf wins like parallel storage reads (huge for zero and light clients), and you let the js engine figure out the causality requirements. to really do it correctly you need to build a funky mutex machine around memory read/writes, but it sounds like a fun challenge. I started playing with this over the summer and you can make iterative progress while maintaining passing tests.
  e13503b...dynamic-execution#diff-f42577b259c8466289ed92b49e2a212c

[kumavis]

worth noting that ganache's forked blockchain feature also uses a json rpc client for state reads, this is how they are hooking in https://github.com/trufflesuite/ganache-core/blob/a5f43a4d666c82bce9da51f9ce1d15b6561e24f3/lib/utils/forkedblockchain.js#L89-L93

this usecase should be supported without having to override underscored properties/methods

[s1na]

Wrote down some thoughts re making EVM (Interpreter and EEI) public here. Probably makes sense to keep them private for this release.

[holgerd77]

For notice: have substantially updated the initial TODO list here to keep track with our latest discussions around various questions.