London update 9.0.0: EIP1559

The recent tests changes:

Merge PRs with London EIP1559 tests.
Update test format to support EIP1559 transaction format (+change in test RPC json object)
Regenerate all tests using London rules.

Announcement

The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.5

Test format

State Tests

Treat legacy blockheader as 1559 with baseFee 10 on London
Tests are recalculated with transaction gasPrice set to 10

Blockchain Tests

Tests are recalculated with transaction gasPrice set to 1000
Tests with gasPrice 0 are not included in London (as minimum baseFee is 7)

State Tests | BlockchainTests | TransitionTests

• the env section now could have

currentBaseFee: 0x3B9ACA00

indicating a eip1559 genesis.

• transaction section eip1559 support could now have

  maxFeePerGas: '0x12A05F200'
  maxPriorityFeePerGas: '2'

instead of gasPrice
PR example: #826

BlockchainTests | TransitionTests

• the blockHeader now could have

baseFee: 0x3B9ACA00

indicating a eip1559 block.
In blockchain fillers blocks section, the blockHeader (overwrite) section has a new field

               "blockHeader" : {
                   "gasLimit" : "600000",
                   "remove" : [ "baseFee" ]
               },

"remove" that indicates an array of fields to be removed from the client returned structure. making it possible to switch mined block into legacy and eip1559 back and forth. see transition test example.

Test Regeneration

Update all:
Regenerate all tests using London rules.
Treat legacy blockheader as 1559 with baseFee 10 on London rules
GStateTests transaction gasPrice set to 10
BCTests transaction gasPrice set to 1000
PR #857

• Documentation update
  PR #838
  PR #849

• EIP1559 coverage
  Blockchain tests, gasTarget, gasPrice, transaction rules
  PR #837

• EIP1559 tests example
  PR #826

• EIP 3541 coverage
  Check that contracts creation starting with 0xef is rejected
  PR #835
  PR #852

• Enable BaseFee opcode support check in all opcodes test
  PR #841

• EIP 3529 additional coverage
  refund gas reduction
  PR #850

Upcoming changes:

Deprecate "transaction" section in filled GeneralStateTests #828
London EIP tests
Besu retesteth RPC support (web vmtracing support)
Move legacy tests into separate repo (too much space used)
OE wrapper support on retesteth.ethdevops.io. FAILED

Berlin update 8.0.5: Moving to London

The recent tests changes:

Discussion that affects all custom test runners: #828 issue
save the space, deprecate general transaction vector parsing and signing (execution speed up).

- WARNING: it will break all the custom test executor tools!

Announcement

The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.4

Test format

none

Test Regeneration

Update all:
none

• Fixed a bug caused by the higher price of a call in Berlin
  PR #833

• More tests around EIP2930 on "manual" account creation
  PR #834

• Calling a contract which has an undefined opcode as first byte of code
  PR #836

• Deploy contracts with CREATE1/CREATE2 with all possble first bytes
  PR #843

• Check for correct rollback of init code with side effects
  This adds a test that checks that exceptional abort during init code correctly rolls back side effects.
  PR #844

• Disagble python script checking tests format
  As the format check is done by retesteth already
  PR #840

Upcoming changes:

• Fillers and Filled

State Tests | BlockchainTests | TransitionTests

• the env section now could have

currentBaseFee: 0x3B9ACA00

indicating a eip1559 genesis.

• transaction section eip1559 support could now have

  maxFeePerGas: '0x12A05F200'
  maxPriorityFeePerGas: '2'

instead of gasPrice
PR example: #826

BlockchainTests | TransitionTests

• the blockHeader now could have

baseFee: 0x3B9ACA00

indicating a eip1559 block.
In blockchain fillers blocks section, the blockHeader (overwrite) section has a new field

               "blockHeader" : {
                   "gasLimit" : "600000",
                   "remove" : [ "baseFee" ]
               },

"remove" that indicates an array of fields to be removed from the client returned structure. making it possible to switch mined block into legacy and eip1559 back and forth. see transition test example.

Discuss #828 issue
London EIP tests
Besu retesteth RPC support (web vmtracing support)
OE wrapper support on retesteth.ethdevops.io. FAILED

Berlin update 8.0.4: Post fork tests

The recent tests changes:

Discussion that affects all custom test runners: #828 issue
save the space, deprecate general transaction vector parsing and signing.

Announcement

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.3

Test format

• Fillers
  none

• Filled
  none

Test Regeneration

Update all:
none

• Documentation update
  PR #830

• Check memory opcodes with and without enough gas
  PR #832

• Precompiles and access list tests
  More tests around precompiles as the OE fork happened around it
  PR #829

• Access list test texample
  PR #825

• More coverage for eip2929 edge cases
  OOG for cold access with EIP2929
  PR #824

Upcoming changes:

Discuss #828 issue
EIP1559, typed transaction 02, according tests format change
London EIP tests
Besu retesteth RPC support (web vmtracing support)
OE wrapper support on retesteth.ethdevops.io

Berlin update 8.0.3: Consensus issues

The recent tests changes:

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.2

Test format

./BlockchainTests/ValidBlocks/VMTests removed
./src/VMTestsFiller moved to ./src/LegacyTests/Constantinople
./VMTests moved to ./LegacyTests/Constantinople/VMTests

• Fillers
  none

• Filled
  none

Test Regeneration

Update all:
none

• Consensus issue random test 648
  This produces a consensus error, on nethermind, versus besu/geth/oe.
  PR #822

• Consensus issue random test 649
  A crash in nethermind, where the codecopy produced an error when called with length 0. The same error was also in CALLDATACOPY, CODECOPY, EXTCODECOPY and RETURNDATACOPY
  PR #822

• Consensus issue random test 650
  This one was a crash in besu, that crashed when calling the modexp precompile with base and mod length 0 and a big exponent.
  PR #822

• Move original VMTests into LegacyTests
  PR #818

• More test examples with amazing retesteth features
  PR #819
  PR #820

• Cosmetic change requested by besu
  Typed transaction exported as 01 + tx.bytes instead of rlp prefix
  PR #821

Upcoming changes:

EIP1559, typed transaction 02, according tests format change
London EIP tests
Besu retesteth RPC support (web vmtracing support)

Berlin update 8.0.2: Precompile check

The recent tests changes:

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.1

Test format

New tests in GeneralStateTests/VMTests folder converted.

• Fillers
  none

• Filled
  none

Test Regeneration

Update all:
none

• Precompile availability check
  Checks that a precompile address is enabled or disabled.
  PR #811

• Migrate VMTests into StateTests format
  PR #806

• Fix issue #459
  More tests around division operator
  PR #807

• Fix issue #585
  More tests for RETURNDATASIZE after CALL/CREATE reverts
  PR #808

• Fix issue #675
  Cosmetic change
  PR #810

• Basic operations with non const arguments
  PR #813

• Fix issue #364
  PR #814

Upcoming changes:

• old VMTests and it's blockchain form are to be removed.

Berlin update fix1

The recent tests changes:

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.0

Test format

• Fillers
  tx.none : "auto" is fixed to work with 64bits in blockchain tests
  Nonce > 2**64 - 1 is not allowed now. same as block gasLimit

• Filled
  New field in state tests txbytes containing transaction rlp or rlp ( tx.type + tx.bytes)

Test Regeneration

Update all:
PR #805

• Migrate VMTests into StateTests format
  PR #802

• Fix the nonce bug in extcodehash test
  PR #804

• More tests around revert opcode
  PR #803

Upcoming changes:

• old VMTests and it's blockchain form are to be removed.

Berlin update

The recent tests changes:

Includes changes of releases previously marked as pre-release:
Changes of: https://github.com/ethereum/tests/releases/tag/7.0.2
Changes of: https://github.com/ethereum/tests/releases/tag/v7.0.1

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/

Test format

• Fillers
  From pre releases a new folder under GeneralStateTests - VMTests containing reworked VMTests into state format

• Filled
  State tests post section contains new field txbytes with a transaction rlp. So no need to build a transaction from arrays and do the signing operation and to avoid confusion with transaction indexing. This field is yet to appear in new tests with the test regeneration.

Test Regeneration

Update all:

• Migrate VMTests in StateTests format
  The VMTests are being reviwed and reworked and now translated by Ori into StateTest format
  PR #791

• EIP-2315 (https://eips.ethereum.org/EIPS/eip-2315)
  Canceled. Rewoking the tests.
  PR #798
  PR #796

• Close issue #581 More EXTCODEHASH Tests
  PR #799

• Close issue #348 Revert after log operation
  PR #794
  PR #795

• Quick stack tests update:
  PR #790

Upcoming changes:

• old VMTests and it's blockchain form are to be removed.

AccessLists update

The recent tests changes:

Access lists tests, Underflow tests, documentation update
reminder of a web interface for test debugging: http://retesteth.ethdevops.io/web/

!!! quick fix for failing stack tests !!!
#790

Test format

• Fillers
  accessList support in transaction in generalState test fillers and blockchain test fillers.

• Filled
  in General State Tests transaction is expanded with optional field "accessLists" which size is 1 to 1 to "data" array. meaning each tx.data could have an accessList. {} - empty access list null - absent access list (legacy transaction) {access list json} - access list as defined by geth t8ntool. chainID is assumed 1 everywhere.

• In blockchain test transaction could have accessList. rlp of transaction is 01 + rlp (chainid, .... tx data fields) as described by EIP2930

• new test folder stEIP2930 in GeneralStateTests and BlockchainTests/GeneralStateTests

Test Regeneration

Update:
#774 - EIP2930 tests
#786 - Underflow tests
#788 - Documentation update
#784 - Documentation update

Upcoming changes:

• GeneralStateTest filled format update to include transaction rlp so that would be no need to calculate the signature from secret key. (#772)
• old VMTests and it's blockchain form are to be removed.

YoloV3 update

The recent tests changes:

EIPS covered with Berlin test configuration (geth YOLOv2, YOLOv3)

Test format

• The VMTests are being reworked as subSuite of GeneralStateTests.
  Same subsuite in BlockchainTests/GeneralStateTests/VMTests

• Fillers
  gtest fillers now support ":label somelabel " prefix in tx.data
  in expect section it is also possible to select transaction data indexes by label
  Filled
  tx.index -> label map is exported in gtest "_info" section for debug. is optional to parse it or not.

Test Regeneration

Update all:
PR #723
PR #775

• EIP2929 (https://eips.ethereum.org/EIPS/eip-2929)
  Gas cost increases for state access opcodes
  PR #760
  PR #756
  PR #745

• EIP2537 (https://eips.ethereum.org/EIPS/eip-2537)
  Precompile for BLS12-381 curve operations
  PR #713
  REMOVED: #776

• EIP2315 (https://eips.ethereum.org/EIPS/eip-2315)
  Simple Subroutines for the EVM
  PR #693
  PR #685

• EIP2565 (https://eips.ethereum.org/EIPS/eip-2565)
  ModExp Gas Cost
  existing test regeneration will test out the changes
  PR: #775
  PR: #779
  PR: #777

• VMTests rework under GeneralStateTests
  PR: #771

Upcoming changes:

• EIP2718 - EIP2930 (https://eips.ethereum.org/EIPS/eip-2718, https://eips.ethereum.org/EIPS/eip-2930)
  (Typed Transaction Envelope, Optional access lists)
  New test format (access list test example): PR: https://github.com/ethereum/tests/pull/772/files
  Access list tests PR: #774

• old VMTests and it's blockchain form are to be removed.

v7.0.0 - Pre-Berlin HF Release

This is the last Ethereum tests release with a pre-Berlin HF state, so it provides some stable ground to upgrade client test runners against to be ready for a Berlin hardfork integration.

Tests Added

StateTests / BlockchainTests

EIP-2200

Added state tests (also as BlockchainTests) to validate the EIP-1706/EIP-2200 out of gas condition,
specifically aimed at validating less than or equals to the stipend handling, see GeneralStateTests/stSStoreTest/sstore_gasLeft.json, PR #649

DIV/SDIV/MOD/SDIV by Zero

Added state tests (also as BlockchainTests) checking DIV/SDIV/MOD/SDIV by zero, see GeneralStateTests/stSolidityTest/ByZero.json, PR #647

Stack Validity of SWAP

Added tests checking stack validity of SWAP, see GeneralStateTests/stStackTests/stackOverflowSWAP.json and [..]/stacksanitySWAP.json, PR #647

EXTCODEHASH Empty Account

Added more EXTCODEHASH state tests (also as BlockchainTests) of nonexistent and post suicide accounts, see GeneralStateTests/stExtCodeHash/callToNonExistent.json,
[..]/callToSuicideThenExtcodehash.json and [..]/createEmptyThenExtcodehash.json, PR #654

SELFBALANCE in different Call Types

Added a composite state test (also in BlockchainTests) for SELFBALANCE that will do all the exisitng test inside of CALL, DELEGATECALL, and CALLCODE calls,
see GeneralStateTests/stSelfBalance/selfBalanceCallTypes.json, PR #671

Retesteth Unit Tests

Introduced retesteth unitTests in BlockchainTests/InvalidBlocks/bcExpectSection/filling_unexpectedAccount.json, PR #676

Difficulty Tests

Added difficulty tests for EIP-2384 (Muir Glacier Difficulty Bomb Delay), PR #662

Keystore Tests

Added KeyStore test based on MyCrypto file in order to ensure that the generated file is compatible with MyCrypto and Metamask, PR #665

RLP Tests

Added RLP invalid tests for insufficient bytes, see RLPTests/invalidRLPTest.json, PR #614

Test Format Changes

BlockChainTests

LegacyTests

BlockChain tests up to ConstantinopleFix (aka Petersburg) now also have been moved to the [LegacyTests/] folder, the main test folders on this release only
contain the Istanbul tests, PR #648

VMTests -> BlockchainTests

VMTests have been converted to BlockchainTests and can now be found in BlockchainTests/ValidBlocks/VMTests/, PR #680

PostState Correction

On some LegacyTests postState has been corrected to postStateHash, you can use this repo search
to see which tests are affected. This might need some adoption of your test runner.

InvalidBlockChain Tests: Expected Block Exceptions

Major blockchain test refactoring, and regeneration on geth + retesteth, see files diff for an impression on the format changes (InvalidBlockchainTests). PR #672

Retesteth

• Add Transition genesis retesteth configurations to default geth configs, PR #646
• Updated default mining reward config for transition nets in retesteth configs to avoid retesteth error when filling the state tests, PR #650
• Use additional forks in retesteth configs. Additional forks used in TransitionTests and not automatically used in StateTests/BlockchainTests, PR #651
• Enable Istanbul in aleth's retesteth configs, PR #656
• Fix ripemd160 precompile name in retesteth configs, PR #659
• Fix ECADD and ECMUL precompiles in Istanbul retesteth config, PR #660
• Add ChainID to Istanbul retesteth config, PR #661
• Remove retesteth configs, configs are now stored in retesteth repo. PR #666

Test Fixes

• Reduce 50k bytes code in quadratic complexity tests to 20k bytes. Resolves issue #657, see GeneralStateTests/stQuadraticComplexityTest/Call[*]KbytesContract[*].json, PR #658 and PR #669
• Minor changes to the test filler format around field strictness, affected tests
  BlockchainTests/InvalidBlocks/bcExpectSection/filling_wrongStorage2.json,
  GeneralStateTests/stExtCodeHash/*, GeneralStateTests/stRandom/ (3),
  GeneralStateTests/stReturnDataTest/subcallReturnMoreThenExpected.json,
  GeneralStateTests/stStaticCall/static_call_value_inherit.json,
  GeneralStateTests/stTransactionTest/Opcodes_TransactionInit.json,
  PR #677
• Upgrade and regenerate invalid RLP blockchain tests, PR #673
• Fix test fillers affected by the CompareStates bug, affected tests
  BlockchainTests/ValidBlocks/bcStateTests/randomStatetest377.json,
  GeneralStateTests/stCallCreateCallCodeTest/Call1024PreCalls.json,
  GeneralStateTests/stChangedEIP150/Call1024PreCalls.json,
  GeneralStateTests/stCreate2/create2checkFieldsInInitcode.json,
  GeneralStateTests/stDelegatecallTestHomestead/Call1024PreCalls.json,
  GeneralStateTests/stRandom/randomStatetest349.json,
  GeneralStateTests/stRandom2/randomStatetest578.json,
  GeneralStateTests/stRefundTest/*,
  GeneralStateTests/stStaticCall/*,
  PR #678
• Fixed blockgaslimit in CALLBlake2f_MaxRouds, see GeneralStateTests/stTimeConsuming/CALLBlake2f_MaxRounds.json, PR #679
• Removed bcExpectSection in BlockchainTests from public tests, PR #684