From 2ed8e186df2270ac52996c083aac3c12c7dece3f Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 18 Mar 2025 18:49:33 +0000 Subject: [PATCH 01/33] docs: Add first checklist templates --- .../eip_testing_checklist_template.md | 65 +++++++++++++++++++ docs/checklist_templates/index.md | 4 ++ 2 files changed, 69 insertions(+) create mode 100644 docs/checklist_templates/eip_testing_checklist_template.md create mode 100644 docs/checklist_templates/index.md diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md new file mode 100644 index 00000000000..e3f51cad947 --- /dev/null +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -0,0 +1,65 @@ +# EIP Execution Layer Testing Checklist Templates + +Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features: + +## New Opcode + +The EIP introduces one or more new opcodes to the EVM. + +- [ ] Memory expansion + - [ ] Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: + - [ ] Zero bytes expansion + - [ ] Single byte expansion + - [ ] 31 bytes expansion + - [ ] 32 bytes expansion + - [ ] 33 bytes expansion + - [ ] 64 or more bytes expansion +- [ ] Stack over/underflows + - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more, verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. +- [ ] Execution context + - [ ] Normal call + - [ ] Static call + - [ ] Verify exeptional abort if the opcode attempts to modify the code, storage or balance of an account + - [ ] Delegate call + - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. + - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. + - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. + - [ ] Code call + - [ ] Initcode + - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation: + - [ ] Initcode of a contract creating transaction. + - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). + - [ ] Transaction context + - [ ] If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. + - [ ] Block context + - [ ] If opcode changes behavior depending on particular block properties, test using multiple values for each property. + - [ ] EOF Container Context + - [ ] If opcode changes behavior depending on particular EOF container properties, test using multiple values for each property. +- [ ] Return data + - [ ] Verify proper return data buffer modification if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: + - [ ] At current call context. + - [ ] At parent call context. +- [ ] Gas usage + - [ ] Normal operation gas usage + - [ ] Verify gas affectation of each parameter value consumed by the opcode. + - [ ] Memory expansion + - [ ] Verify that the memory expansion correctly follows the gas calculation + - [ ] Out-of-gas during opcode execution + - [ ] Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exeptional abort. + - [ ] Out-of-gas during memory expansion + - [ ] Verify that the expansion of memory can result in out-of-gas exeptional abort. +- [ ] Terminating opcode + - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios: + - [ ] Top-level call termination + - [ ] Sub-level call termination + - [ ] Initcode termination +- [ ] Exeptional Abort + - [ ] Verify behavior that is supposed to cause an exeptional abort, other than stack over or underflow, or out-of-gas errors. +- [ ] Data portion + - [ ] If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). +- [ ] Contract creation + - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. + - [ ] Verify that contract is not created in case of: + - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. + - [ ] Contract creation would result in an address collision with an existing contract or eoa-delegated address. diff --git a/docs/checklist_templates/index.md b/docs/checklist_templates/index.md new file mode 100644 index 00000000000..e5427b5faf3 --- /dev/null +++ b/docs/checklist_templates/index.md @@ -0,0 +1,4 @@ +# Testing Checklist Templates + +This page includes checklist templates that must be followed to guarantee a minimum base testing coverage of new EIP feature. + From cf08fea5716528d6093ca7138dbb3399f8d63ace Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 18 Mar 2025 19:54:06 +0000 Subject: [PATCH 02/33] changes: Precompiles --- .../eip_testing_checklist_template.md | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index e3f51cad947..f3e92faa5a0 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -2,10 +2,20 @@ Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features: +## General + +- [ ] Code coverage: + - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exeptions being unreachable code lines. + - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. +- [ ] Fuzzing: + - [ ] TBD + ## New Opcode The EIP introduces one or more new opcodes to the EVM. +### Test Vectors + - [ ] Memory expansion - [ ] Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: - [ ] Zero bytes expansion @@ -63,3 +73,86 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify that contract is not created in case of: - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - [ ] Contract creation would result in an address collision with an existing contract or eoa-delegated address. +- [ ] Fork transition + - [ ] Verify that the opcode results in exeptional abort if executed before its activation fork. + - [ ] Verify that the opcode results in invalid EOF container if attempted to deploy before its activation fork. + +### Framework Changes + +- [ ] Add opcode to `src/ethereum_test_vm/opcode.py` +- [ ] Add opcode to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New Precompile + +### Test Vectors + +- [ ] Call contexts: + - [ ] Normal call to precompile from contract + - [ ] Delegate call to precompile from contract + - [ ] Static call to precompile from contract + - [ ] Code call to precompile from contract + - [ ] Precompile as transaction entry-point + - [ ] Call from initcode + - [ ] Contract creating transaction + - [ ] Contract creating opcode +- [ ] Inputs + - [ ] Verify combinations of valid inputs to the precompile: + - [ ] Verify interesting edge values given the precompile functionality. + - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties. + - [ ] Verify all zeros input. + - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. +- [ ] Input lengths: + - [ ] Zero-length calldata. + - [ ] Precompile has static-length input: + - [ ] Correct static-length calldata + - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. + - [ ] Precompile has dynamic-length input: + - [ ] Verify correct precompile execution for valid lengths, given different inputs. + - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. +- [ ] Gas usage: + - [ ] Precompile has constant gas usage + - [ ] Verify exact gas consumption + - [ ] Verify exact gas consumption minus one results in out-of-gas error. + - [ ] Precompile has dynamic gas usage + - [ ] Verify exact gas consumption, given different valid inputs. + - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. +- [ ] Fork transition: + - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. + - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. + + +### Framework Changes + +- [ ] Add precompile address to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New System Contract + +### Test Vectors + +### Framework Changes + +## New Block Header Field + +### Test Vectors + +### Framework Changes + +## New Block Body Field + +### Test Vectors + +### Framework Changes + +## New Transaction Type + +### Test Vectors + +### Framework Changes + +## Intrinsic Gas Cost Changes + +### Test Vectors + +### Framework Changes From 1558fa5ae702b834884144fb4baa2edea958f1ab Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 18 Mar 2025 20:19:36 +0000 Subject: [PATCH 03/33] changes: System contracts --- .../eip_testing_checklist_template.md | 81 ++++++++++++++----- 1 file changed, 62 insertions(+), 19 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index f3e92faa5a0..577c4e6e7fd 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -1,13 +1,13 @@ # EIP Execution Layer Testing Checklist Templates -Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features: +Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features ## General -- [ ] Code coverage: +- [ ] Code coverage - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exeptions being unreachable code lines. - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. -- [ ] Fuzzing: +- [ ] Fuzzing - [ ] TBD ## New Opcode @@ -17,7 +17,7 @@ The EIP introduces one or more new opcodes to the EVM. ### Test Vectors - [ ] Memory expansion - - [ ] Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: + - [ ] Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes - [ ] Zero bytes expansion - [ ] Single byte expansion - [ ] 31 bytes expansion @@ -37,7 +37,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. - [ ] Code call - [ ] Initcode - - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation: + - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - [ ] Initcode of a contract creating transaction. - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). - [ ] Transaction context @@ -47,7 +47,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] EOF Container Context - [ ] If opcode changes behavior depending on particular EOF container properties, test using multiple values for each property. - [ ] Return data - - [ ] Verify proper return data buffer modification if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: + - [ ] Verify proper return data buffer modification if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected - [ ] At current call context. - [ ] At parent call context. - [ ] Gas usage @@ -60,7 +60,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Out-of-gas during memory expansion - [ ] Verify that the expansion of memory can result in out-of-gas exeptional abort. - [ ] Terminating opcode - - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios: + - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios - [ ] Top-level call termination - [ ] Sub-level call termination - [ ] Initcode termination @@ -70,7 +70,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). - [ ] Contract creation - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - - [ ] Verify that contract is not created in case of: + - [ ] Verify that contract is not created in case of - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - [ ] Contract creation would result in an address collision with an existing contract or eoa-delegated address. - [ ] Fork transition @@ -86,7 +86,7 @@ The EIP introduces one or more new opcodes to the EVM. ### Test Vectors -- [ ] Call contexts: +- [ ] Call contexts - [ ] Normal call to precompile from contract - [ ] Delegate call to precompile from contract - [ ] Static call to precompile from contract @@ -96,29 +96,31 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Contract creating transaction - [ ] Contract creating opcode - [ ] Inputs - - [ ] Verify combinations of valid inputs to the precompile: + - [ ] Verify combinations of valid inputs to the precompile - [ ] Verify interesting edge values given the precompile functionality. - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. -- [ ] Input lengths: +- [ ] Input Lengths - [ ] Zero-length calldata. - - [ ] Precompile has static-length input: + - [ ] Precompile has static-length input - [ ] Correct static-length calldata - - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. - - [ ] Precompile has dynamic-length input: + - [ ] Calldata too short, where the value represents a correct but truncated input to the precompile. + - [ ] Calldata too long, where the value represents a correct input to the precompile with padded zeros. + - [ ] Precompile has dynamic-length input - [ ] Verify correct precompile execution for valid lengths, given different inputs. - - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. -- [ ] Gas usage: + - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the precompile. + - [ ] Calldata too long, given different inputs, where the value represents a correct input to the precompile with padded zeros. +- [ ] Gas usage - [ ] Precompile has constant gas usage - [ ] Verify exact gas consumption - [ ] Verify exact gas consumption minus one results in out-of-gas error. - [ ] Precompile has dynamic gas usage - [ ] Verify exact gas consumption, given different valid inputs. - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. -- [ ] Fork transition: +- [ ] Excessive Gas Cases + - [ ] Verify spending all block gas in calls to the precompile (100 million gas or more). +- [ ] Fork transition - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. @@ -131,8 +133,49 @@ The EIP introduces one or more new opcodes to the EVM. ### Test Vectors +- [ ] Call contexts + - [ ] Normal call to system contract from contract + - [ ] Delegate call to system contract from contract + - [ ] Static call to system contract from contract + - [ ] Code call to system contract from contract + - [ ] System contract as transaction entry-point + - [ ] Call from initcode + - [ ] Contract creating transaction + - [ ] Contract creating opcode +- [ ] Inputs + - [ ] Verify combinations of valid inputs to the system contract + - [ ] Verify interesting edge values given the system contract functionality. + - [ ] Verify all zeros input. + - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. +- [ ] Input Lengths + - [ ] Zero-length calldata. + - [ ] System contract has static-length input + - [ ] Correct static-length calldata + - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. + - [ ] System contract has dynamic-length input + - [ ] Verify correct System contract execution for valid lengths, given different inputs. + - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. +- [ ] Excessive Gas Cases + - [ ] If possible, simulate a scenario where the execution of the contract at the end of the block execution by the system address would result in excessive gas usage (100 million gas or more). + - [ ] Verify spending all block gas in calls to system contract (100 million gas or more). +- [ ] System Contract Deployment + - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed. + - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. +- [ ] Contract Variations + - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior +- [ ] Contract Substitution + - [ ] Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): + - [ ] Modified return value lengths + - [ ] Modify emitted logs +- [ ] Fork transition + - [ ] Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). + ### Framework Changes +- [ ] Add system contract address to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` + ## New Block Header Field ### Test Vectors From 391f89fd34f893fb3eb7a77ed57730de37cc7801 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 18 Mar 2025 20:24:26 +0000 Subject: [PATCH 04/33] nit --- docs/checklist_templates/eip_testing_checklist_template.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 577c4e6e7fd..639d2498bea 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -1,6 +1,6 @@ -# EIP Execution Layer Testing Checklist Templates +# EIP Execution Layer Testing Checklist Template -Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features +Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features. ## General From 7a3a23df76b4463e5ac4bd2bc6dff4e8205e2725 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 18 Mar 2025 20:56:54 +0000 Subject: [PATCH 05/33] changes: Add (incomplete) transaction tests --- .../eip_testing_checklist_template.md | 39 +++++++++++++++++-- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 639d2498bea..adbb7f81268 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -80,7 +80,7 @@ The EIP introduces one or more new opcodes to the EVM. ### Framework Changes - [ ] Add opcode to `src/ethereum_test_vm/opcode.py` -- [ ] Add opcode to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` ## New Precompile @@ -127,7 +127,7 @@ The EIP introduces one or more new opcodes to the EVM. ### Framework Changes -- [ ] Add precompile address to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` ## New System Contract @@ -174,28 +174,59 @@ The EIP introduces one or more new opcodes to the EVM. ### Framework Changes -- [ ] Add system contract address to relevant methods in the fork it is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` ## New Block Header Field ### Test Vectors +**TBD** + ### Framework Changes +**TBD** + + ## New Block Body Field ### Test Vectors +**TBD** + ### Framework Changes +**TBD** + + ## New Transaction Type ### Test Vectors +- [ ] Intrinsic Gas Costs + - [ ] Transaction validity + - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction. + - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction. +- [ ] Encoding Tests + - [ ] Verify correct transaction rejection due to incorrect field sizes +- [ ] RPC Tests* + - [ ] Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type. + + +* Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. + ### Framework Changes -## Intrinsic Gas Cost Changes +- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. + +## Gas Cost Changes ### Test Vectors +**TBD** + ### Framework Changes + +- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. +- [ ] Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. +- [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. +- [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. From 46689bacf5113202113abb55b871cb400aab6fc3 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 21 Mar 2025 19:52:38 +0000 Subject: [PATCH 06/33] More changes --- .../eip_testing_checklist_template.md | 92 +++++++++++++++---- 1 file changed, 73 insertions(+), 19 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index adbb7f81268..4edd1c3b1b2 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -64,6 +64,10 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Top-level call termination - [ ] Sub-level call termination - [ ] Initcode termination +- [ ] Out-of-bounds checks + - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: + - [ ] Max value for each parameter + - [ ] Max value + 1 for each parameter - [ ] Exeptional Abort - [ ] Verify behavior that is supposed to cause an exeptional abort, other than stack over or underflow, or out-of-gas errors. - [ ] Data portion @@ -101,6 +105,10 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. +- [ ] Out-of-bounds checks + - [ ] Verify if the precompile has out-of-bounds conditions in its inputs and verify: + - [ ] Max value for each input + - [ ] Max value + 1 for each input - [ ] Input Lengths - [ ] Zero-length calldata. - [ ] Precompile has static-length input @@ -147,6 +155,10 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify interesting edge values given the system contract functionality. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. +- [ ] Out-of-bounds checks + - [ ] Verify if the system contract has out-of-bounds conditions in its inputs and verify: + - [ ] Max value for each input + - [ ] Max value + 1 for each input - [ ] Input Lengths - [ ] Zero-length calldata. - [ ] System contract has static-length input @@ -175,48 +187,69 @@ The EIP introduces one or more new opcodes to the EVM. ### Framework Changes - [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -## New Block Header Field +## New Transaction Type ### Test Vectors -**TBD** +- [ ] Intrinsic Gas Costs + - [ ] Transaction validity: For each new field that affects the intrinsic gas cost of the transaction: + - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. + - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. +- [ ] Encoding Tests + - [ ] Verify correct transaction rejection due to incorrect field sizes +- [ ] RPC Tests + - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type + - [ ] Verify `eth_sendRawTransaction` using `execute` +- [ ] Out-of-bounds checks + - [ ] Verify if the transaction has out-of-bounds conditions in its fields and verify: + - [ ] Max value for each field + - [ ] Max value + 1 for each field +- [ ] Contract creation + - [ ] Verify that the transaction can create new contracts, or that it fails to do so if it's not allowed +- [ ] Sender account modifications + - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented by one after the transaction is included in a block + - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) after the transaction is included in a block +- [ ] Fork transition + - [ ] Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. + + +* Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. ### Framework Changes -**TBD** +- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. +- [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). -## New Block Body Field +## New Block Header Field ### Test Vectors -**TBD** +- [ ] Genesis value + - [ ] Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that the value is correctly modified for the current and subsequent blocks as expected, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Fork transition + - [ ] Verify initial value of the field at the first block of the activation fork. + - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. + - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. + ### Framework Changes **TBD** - -## New Transaction Type +## New Block Body Field ### Test Vectors -- [ ] Intrinsic Gas Costs - - [ ] Transaction validity - - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction. - - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction. -- [ ] Encoding Tests - - [ ] Verify correct transaction rejection due to incorrect field sizes -- [ ] RPC Tests* - - [ ] Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type. - - -* Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. +**TBD** ### Framework Changes -- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. +**TBD** ## Gas Cost Changes @@ -230,3 +263,24 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. - [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. - [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. + +## Blob Count Changes + +### Test Vectors + +- [ ] Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. + +### Framework Changes + +- [ ] Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. + +## New Execution Layer Request + +### Test Vectors + +- [ ] Cross-Request-Type Interaction + - [ ] Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations + +### Framework Changes + +- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. \ No newline at end of file From 039ccc519861cbb0b0530ca8ca0ca668ff796f04 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Mon, 24 Mar 2025 12:31:44 +0000 Subject: [PATCH 07/33] Add more encoding tests --- docs/checklist_templates/eip_testing_checklist_template.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 4edd1c3b1b2..072215f702f 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -197,8 +197,11 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Transaction validity: For each new field that affects the intrinsic gas cost of the transaction: - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. -- [ ] Encoding Tests +- [ ] Encoding Tests (RLP, SSZ) - [ ] Verify correct transaction rejection due to incorrect field sizes + - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing + - [ ] Verify correct transaction rejection if the transaction type contains extra fields + - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field - [ ] RPC Tests - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - [ ] Verify `eth_sendRawTransaction` using `execute` From 59bccaf484d51007471f78ddb311fe6982057124 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Mon, 31 Mar 2025 18:27:44 +0000 Subject: [PATCH 08/33] More cases --- .../eip_testing_checklist_template.md | 54 ++++++++++++++++--- 1 file changed, 47 insertions(+), 7 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 072215f702f..f1693ad9013 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -193,30 +193,70 @@ The EIP introduces one or more new opcodes to the EVM. ### Test Vectors -- [ ] Intrinsic Gas Costs - - [ ] Transaction validity: For each new field that affects the intrinsic gas cost of the transaction: +- [ ] Intrinsic Validity + - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. + - [ ] Max fee per gas: + - [ ] Verify the transaction (and the block it is included in) is invalid if its max-priority-fee-per-gas value is lower than the max-fee-per-gas. + - [ ] Verify the transaction (and the block it is included in) is invalid if its max-fee-per-gas value is lower than the blocks base-fee-per-gas. + - [ ] Chain ID: + - [ ] Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. + - [ ] Nonce: + - [ ] Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. + - [ ] To: + - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. + - [ ] Value: + - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. + - [ ] Data: + - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas gost. +- [ ] Signature: + - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: + - [ ] V, R, S represent a value that is inside of the field but outside of the curve. + - [ ] V (yParity) is any of the following invalid values: + - [ ] `2` + - [ ] `27` (Type-0 transaction valid value) + - [ ] `28` (Type-0 transaction valid value) + - [ ] `35` (Type-0 replay-protected transaction valid value for chain id 1) + - [ ] `36` (Type-0 replay-protected transaction valid value for chain id 1) + - [ ] `2**8-1` + - [ ] R is any of the following invalid values: + - [ ] `0` + - [ ] `SECP256K1N-1` + - [ ] `SECP256K1N` + - [ ] `SECP256K1N+1` + - [ ] `2**256-1` + - [ ] `2**256` + - [ ] S is any of the following invalid values: + - [ ] `0` + - [ ] `SECP256K1N//2-1` + - [ ] `SECP256K1N//2` + - [ ] `SECP256K1N//2+1` + - [ ] `SECP256K1N-1` + - [ ] `SECP256K1N` + - [ ] `SECP256K1N+1` + - [ ] `2**256-1` + - [ ] `2**256` + - [ ] `SECP256K1N - S` of a valid signature - [ ] Encoding Tests (RLP, SSZ) - [ ] Verify correct transaction rejection due to incorrect field sizes - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - [ ] Verify correct transaction rejection if the transaction type contains extra fields - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field -- [ ] RPC Tests - - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - - [ ] Verify `eth_sendRawTransaction` using `execute` - [ ] Out-of-bounds checks - [ ] Verify if the transaction has out-of-bounds conditions in its fields and verify: - [ ] Max value for each field - [ ] Max value + 1 for each field - [ ] Contract creation - - [ ] Verify that the transaction can create new contracts, or that it fails to do so if it's not allowed + - [ ] Verify that the transaction can create new contracts if the transaction type supports it. - [ ] Sender account modifications - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented by one after the transaction is included in a block - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) after the transaction is included in a block - [ ] Fork transition - [ ] Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. - +- [ ] RPC Tests + - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type + - [ ] Verify `eth_sendRawTransaction` using `execute` * Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. From 6d9747e41d01eecb5604e63f2601a59b5c9727c6 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Mon, 31 Mar 2025 22:47:20 +0000 Subject: [PATCH 09/33] More test vectors --- .../eip_testing_checklist_template.md | 37 ++++++++++++++----- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index f1693ad9013..be2e71fe57f 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -23,19 +23,25 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] 31 bytes expansion - [ ] 32 bytes expansion - [ ] 33 bytes expansion - - [ ] 64 or more bytes expansion + - [ ] 64 bytes expansion + - [ ] 2**32-1 bytes expansion + - [ ] 2**32 bytes expansion + - [ ] 2**64-1 bytes expansion + - [ ] 2**64 bytes expansion + - [ ] 2**256-1 bytes expansion - [ ] Stack over/underflows - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more, verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. - [ ] Execution context - - [ ] Normal call - - [ ] Static call - - [ ] Verify exeptional abort if the opcode attempts to modify the code, storage or balance of an account - - [ ] Delegate call + - [ ] CALL + - [ ] STATICCALL + - [ ] Verify exeptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. + - [ ] Verify subcalls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exeptional abort behavior. + - [ ] DELEGATECALL - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. - - [ ] Code call + - [ ] CALLCODE - [ ] Initcode - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - [ ] Initcode of a contract creating transaction. @@ -47,7 +53,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] EOF Container Context - [ ] If opcode changes behavior depending on particular EOF container properties, test using multiple values for each property. - [ ] Return data - - [ ] Verify proper return data buffer modification if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected + - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: - [ ] At current call context. - [ ] At parent call context. - [ ] Gas usage @@ -64,6 +70,12 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Top-level call termination - [ ] Sub-level call termination - [ ] Initcode termination + - [ ] If the terminating opcode is meant to rollback the executing call frame, verify the following events are properly rolled back: + - [ ] Balance changes + - [ ] Storage changes + - [ ] Contract creations + - [ ] Nonce increments + - [ ] Log events - [ ] Out-of-bounds checks - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: - [ ] Max value for each parameter @@ -74,12 +86,14 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). - [ ] Contract creation - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - - [ ] Verify that contract is not created in case of + - [ ] Verify that contract is not created in case of: - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - - [ ] Contract creation would result in an address collision with an existing contract or eoa-delegated address. + - [ ] Creation would result in an address collision with an existing contract or eoa-delegated address. + - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. - [ ] Fork transition - [ ] Verify that the opcode results in exeptional abort if executed before its activation fork. - [ ] Verify that the opcode results in invalid EOF container if attempted to deploy before its activation fork. + - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. ### Framework Changes @@ -99,6 +113,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Call from initcode - [ ] Contract creating transaction - [ ] Contract creating opcode + - [ ] Set code delegated address (no precompile logic executed) - [ ] Inputs - [ ] Verify combinations of valid inputs to the precompile - [ ] Verify interesting edge values given the precompile functionality. @@ -131,6 +146,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Fork transition - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. + - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. ### Framework Changes @@ -273,7 +289,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Genesis value - [ ] Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. - [ ] Value behavior - - [ ] Verify, given multiple initial values, that the value is correctly modified for the current and subsequent blocks as expected, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. - [ ] Fork transition - [ ] Verify initial value of the field at the first block of the activation fork. - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. From db7e952be90b87837644de2ef7a2cc0ae7dc759b Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Mon, 31 Mar 2025 23:17:32 +0000 Subject: [PATCH 10/33] More review comments --- .../eip_testing_checklist_template.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index be2e71fe57f..f1919aab819 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -226,6 +226,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. - [ ] Data: - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas gost. + - [ ] Sender balance: + - [ ] Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. - [ ] Signature: - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: - [ ] V, R, S represent a value that is inside of the field but outside of the curve. @@ -254,8 +256,17 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] `2**256-1` - [ ] `2**256` - [ ] `SECP256K1N - S` of a valid signature -- [ ] Encoding Tests (RLP, SSZ) - - [ ] Verify correct transaction rejection due to incorrect field sizes +- Transaction-Scoped Attributes/Variables + - Verify attributes that can be read in the EVM from transaction fields. + - Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts). +- [ ] Encoding (RLP, SSZ) + - [ ] Verify correct transaction rejection due to incorrect field sizes: + - [ ] Add leading zero byte + - [ ] Remove single byte from fixed-byte-length fields + - [ ] If the transaction contains a new field that is a list, verify: + - [ ] Zero-element list + - [ ] Max count list + - [ ] Max count plus one list - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - [ ] Verify correct transaction rejection if the transaction type contains extra fields - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field @@ -266,7 +277,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Contract creation - [ ] Verify that the transaction can create new contracts if the transaction type supports it. - [ ] Sender account modifications - - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented by one after the transaction is included in a block + - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanic that bumps the nonce by more than one). - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) after the transaction is included in a block - [ ] Fork transition - [ ] Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. From f7742b114c4dba8b7579c62e1e62e6fcf7ac7928 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 1 Apr 2025 16:30:25 +0000 Subject: [PATCH 11/33] Add asymetrical stack tests --- .../checklist_templates/eip_testing_checklist_template.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index f1919aab819..46397770782 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -29,9 +29,11 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] 2**64-1 bytes expansion - [ ] 2**64 bytes expansion - [ ] 2**256-1 bytes expansion -- [ ] Stack over/underflows - - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more, verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. +- [ ] Stack + - [ ] Overflows/Underflows + - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more, verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. + - [ ] If opcode performs stack operations different of simple pop->push, test for these operations on an asymmetrical stack. - [ ] Execution context - [ ] CALL - [ ] STATICCALL From 6fb207fe8bcd8a6c0f1b7e395465c10f0c33dcf4 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 1 Apr 2025 17:53:37 +0000 Subject: [PATCH 12/33] Add some IDs --- .../eip_testing_checklist_template.md | 79 ++++++++++--------- 1 file changed, 40 insertions(+), 39 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 46397770782..aa343b09a9c 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -10,25 +10,26 @@ Depending on the changes introduced by an EIP, the following template is the min - [ ] Fuzzing - [ ] TBD -## New Opcode +## New Opcode The EIP introduces one or more new opcodes to the EVM. -### Test Vectors - -- [ ] Memory expansion - - [ ] Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes - - [ ] Zero bytes expansion - - [ ] Single byte expansion - - [ ] 31 bytes expansion - - [ ] 32 bytes expansion - - [ ] 33 bytes expansion - - [ ] 64 bytes expansion - - [ ] 2**32-1 bytes expansion - - [ ] 2**32 bytes expansion - - [ ] 2**64-1 bytes expansion - - [ ] 2**64 bytes expansion - - [ ] 2**256-1 bytes expansion +### Test Vectors + +- [ ] Memory expansion: Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: + - [ ] Zero bytes expansion + - [ ] Zero-offset + - [ ] 2**256-1 offset + - [ ] Single byte expansion + - [ ] 31 bytes expansion + - [ ] 32 bytes expansion + - [ ] 33 bytes expansion + - [ ] 64 bytes expansion + - [ ] 2**32-1 bytes expansion + - [ ] 2**32 bytes expansion + - [ ] 2**64-1 bytes expansion + - [ ] 2**64 bytes expansion + - [ ] 2**256-1 bytes expansion - [ ] Stack - [ ] Overflows/Underflows - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. @@ -97,14 +98,14 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify that the opcode results in invalid EOF container if attempted to deploy before its activation fork. - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. -### Framework Changes +### Framework Changes -- [ ] Add opcode to `src/ethereum_test_vm/opcode.py` -- [ ] Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add opcode to `src/ethereum_test_vm/opcode.py` +- [ ] Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -## New Precompile +## New Precompile -### Test Vectors +### Test Vectors - [ ] Call contexts - [ ] Normal call to precompile from contract @@ -151,13 +152,13 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. -### Framework Changes +### Framework Changes - [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -## New System Contract +## New System Contract -### Test Vectors +### Test Vectors - [ ] Call contexts - [ ] Normal call to system contract from contract @@ -202,14 +203,14 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Fork transition - [ ] Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). -### Framework Changes +### Framework Changes - [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - [ ] Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -## New Transaction Type +## New Transaction Type -### Test Vectors +### Test Vectors - [ ] Intrinsic Validity - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: @@ -289,15 +290,15 @@ The EIP introduces one or more new opcodes to the EVM. * Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. -### Framework Changes +### Framework Changes - [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. - [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). -## New Block Header Field +## New Block Header Field -### Test Vectors +### Test Vectors - [ ] Genesis value - [ ] Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. @@ -310,11 +311,11 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. -### Framework Changes +### Framework Changes **TBD** -## New Block Body Field +## New Block Body Field ### Test Vectors @@ -324,7 +325,7 @@ The EIP introduces one or more new opcodes to the EVM. **TBD** -## Gas Cost Changes +## Gas Cost Changes ### Test Vectors @@ -337,23 +338,23 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. - [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. -## Blob Count Changes +## Blob Count Changes -### Test Vectors +### Test Vectors - [ ] Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. -### Framework Changes +### Framework Changes - [ ] Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. -## New Execution Layer Request +## New Execution Layer Request -### Test Vectors +### Test Vectors - [ ] Cross-Request-Type Interaction - [ ] Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations -### Framework Changes +### Framework Changes - [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. \ No newline at end of file From 8b0fc046e6844ad103c3940b1684729cd57145b3 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 1 Apr 2025 19:57:51 +0000 Subject: [PATCH 13/33] Add serialization items --- docs/checklist_templates/eip_testing_checklist_template.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index aa343b09a9c..2ba226cc6c7 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -273,6 +273,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - [ ] Verify correct transaction rejection if the transaction type contains extra fields - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field + - [ ] Verify transaction is correctly rejected if the serialized bytes object is truncated + - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes - [ ] Out-of-bounds checks - [ ] Verify if the transaction has out-of-bounds conditions in its fields and verify: - [ ] Max value for each field From fd8dc8c78afb5e045855a4237d9783968d4d6d80 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 1 Apr 2025 20:11:53 +0000 Subject: [PATCH 14/33] Add more serialization items --- docs/checklist_templates/eip_testing_checklist_template.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 2ba226cc6c7..96bbe2826d1 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -272,9 +272,10 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Max count plus one list - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - [ ] Verify correct transaction rejection if the transaction type contains extra fields - - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field - [ ] Verify transaction is correctly rejected if the serialized bytes object is truncated - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes + - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field, plus: + - [ ] Verify transaction rejection if the serializable field is incorrectly encoded as bytes instead of using the correct encoding. - [ ] Out-of-bounds checks - [ ] Verify if the transaction has out-of-bounds conditions in its fields and verify: - [ ] Max value for each field From 8cb1b4df8f2c9176880a4a017f4942f96f8db827 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Wed, 2 Apr 2025 15:28:09 +0000 Subject: [PATCH 15/33] Add gas refunds category --- .../eip_testing_checklist_template.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 96bbe2826d1..e0cc2a5de5c 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -341,6 +341,16 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. - [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. +## Gas Refunds Changes + +### Test Vectors + +**TBD** + +### Framework Changes + +**TBD** + ## Blob Count Changes ### Test Vectors From 2fef85c1845415430ba2c9ff377509861a972fd4 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 11 Apr 2025 17:15:59 +0000 Subject: [PATCH 16/33] docs: Add post-mortems --- docs/navigation.md | 1 + docs/writing_tests/post_mortems.md | 52 ++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 docs/writing_tests/post_mortems.md diff --git a/docs/navigation.md b/docs/navigation.md index ecf1217c0b1..9cbf12b8153 100644 --- a/docs/navigation.md +++ b/docs/navigation.md @@ -16,6 +16,7 @@ * [Code Standards](writing_tests/code_standards.md) * [Exception Tests](writing_tests/exception_tests.md) * [Tutorial: Adding a State Test](writing_tests/tutorials/state_transition.md) + * [Post-mortems](writing_tests/post_mortems.md) * [Filling Tests](filling_tests/index.md) * [Getting Started](filling_tests/getting_started.md) * [Filling Tests at a Prompt](filling_tests/filling_tests_command_line.md) diff --git a/docs/writing_tests/post_mortems.md b/docs/writing_tests/post_mortems.md new file mode 100644 index 00000000000..36f291df35d --- /dev/null +++ b/docs/writing_tests/post_mortems.md @@ -0,0 +1,52 @@ +# Post-Mortems of Missed Test Scenarios + +This page contains a collection of post-mortem analyses for test cases that were not initially identified by the testing process. + +The objective is to document and learn from missed scenarios — including those that were caught on the client side due to consensus issue, client developer raised issues, external reviewers, or external bug bounties — in order to improve test coverage and reduce the likelihood of similar omissions in the future. + +Each entry must include an explanation of why the test case was missed plus the changes that were implemented in documentation or framework to improve the testing process. + +## List + +## TEMPLATE +## Date - Title - Fork + +### Description + +Provide a concise summary of the issue, how it was discovered, emphasizing the how it relates to the specifications and testing. + +*Example:* + +> A consensus-breaking issue was found during the bug-bounty phase of the Pectra fork specifically in the EIP-2537, which involved calling the BLS pairing precompile using two special points: the infinity point and a point that is outside of the BLS12-381 curve. +> The specification correctly specified the behavior of the precompile when one of these inputs was used, but it did not specify the behavior of the combined input. + +### Root Cause Analysis + +Explain why this scenario was not covered by the test suite. Consider whether it was due to ambiguous specification wording, gaps in test generation logic, overlooked edge cases, or incorrect assumptions about expected behavior. + +*Consider prompting questions:* + +- Was the behavior implied but not explicitly stated in the specification? +- Was the area considered low-risk or assumed covered elsewhere? +- Were there limitations in the current test generation tools or processes? +- Was there any different type of testing that could have caught the issue at an earlier stage? (Fuzzing, property based testing) + +### Steps Taken To Avoid Recurrence + +List the actions taken to reduce the chance of this type of miss happening again. E.g. procedure changes, checklist updates, review practices, framework improvements. + +### Implemented Test Case + +IDs of the tests added that now cover the missed scenario and link to the documentation page where they are included. + +*Example:* + +- [`tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm.py::test_invalid\[fork_Prague-state_test---bls_g1_truncated_input-\]`](https://eest.ethereum.org/main/tests/prague/eip2537_bls_12_381_precompiles/test_bls12_g1msm/test_invalid/) + +### Framework/Documentation Changes + +Note any modifications that were introduced in the framework and/or documentation to prevent similar misses. + +*Example:* +- Updated EIP checklist to include testing combinations of interesting points related to the elliptic-curve under test, and all combinations between them. + From 0eb4cb074d74700faee0d0e8b162c710e920f16d Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 11 Apr 2025 17:21:22 +0000 Subject: [PATCH 17/33] fix(github): Add the post-mortem docs to the PR template --- .github/PULL_REQUEST_TEMPLATE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index aaecb4738aa..28bae9617ee 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -14,3 +14,4 @@ - [ ] Tests: A PR with removal of converted JSON/YML tests from [ethereum/tests](/ethereum/tests) have been opened. - [ ] Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65. - [ ] Tests: Ran `mkdocs serve` locally and verified the auto-generated docs for new tests in the [Test Case Reference](https://eest.ethereum.org/main/tests/) are correctly formatted. +- [ ] Tests: For PRs implementing a missed test case, update the [post-mortem document](/ethereum/execution-spec-tests/blob/main/docs/writing_tests/post_mortems.md) to add an entry the list. From c01ffd354942d3d05446b308a3dd931b7d0606e0 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 17:15:14 +0000 Subject: [PATCH 18/33] fix: Remove references to EOF --- docs/checklist_templates/eip_testing_checklist_template.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index e0cc2a5de5c..c2217d79b6d 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -53,8 +53,6 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - [ ] Block context - [ ] If opcode changes behavior depending on particular block properties, test using multiple values for each property. - - [ ] EOF Container Context - - [ ] If opcode changes behavior depending on particular EOF container properties, test using multiple values for each property. - [ ] Return data - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: - [ ] At current call context. @@ -95,7 +93,6 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. - [ ] Fork transition - [ ] Verify that the opcode results in exeptional abort if executed before its activation fork. - - [ ] Verify that the opcode results in invalid EOF container if attempted to deploy before its activation fork. - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. ### Framework Changes From a9b5f23bf85adc9f1beb26700f9961b439fcf7c1 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 19:42:22 +0000 Subject: [PATCH 19/33] fix: Review comments Co-authored-by: danceratopz Co-authored-by: spencer-tb Co-authored-by: Jochem Brouwer --- .../eip_testing_checklist_template.md | 116 ++++++++++++++---- 1 file changed, 95 insertions(+), 21 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index c2217d79b6d..e749c37ab8c 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -6,6 +6,7 @@ Depending on the changes introduced by an EIP, the following template is the min - [ ] Code coverage - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exeptions being unreachable code lines. + - [ ] Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. - [ ] Fuzzing - [ ] TBD @@ -29,16 +30,25 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] 2**32 bytes expansion - [ ] 2**64-1 bytes expansion - [ ] 2**64 bytes expansion - - [ ] 2**256-1 bytes expansion + - [ ] 2**256-1 bytes expansion (Should always result in Out-of-gas) - [ ] Stack - [ ] Overflows/Underflows - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more, verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. - - [ ] If opcode performs stack operations different of simple pop->push, test for these operations on an asymmetrical stack. + - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. + - [ ] If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations: + - [ ] Operation on different stack heights: + - [ ] Zero (Potential stack-underflow) + - [ ] Odd + - [ ] Even + - [ ] For each variable `n` of the opcode that accesses the nth stack item, test `n` being: + - [ ] Top stack item + - [ ] Bottom stack item + - [ ] Middle stack item - [ ] Execution context - [ ] CALL - [ ] STATICCALL - [ ] Verify exeptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. + - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). - [ ] Verify subcalls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exeptional abort behavior. - [ ] DELEGATECALL - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. @@ -49,6 +59,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - [ ] Initcode of a contract creating transaction. - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). + - [ ] Verify opcode behavior on re-entry using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). + - [ ] Set-code delegated account: Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. - [ ] Transaction context - [ ] If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - [ ] Block context @@ -58,14 +70,12 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] At current call context. - [ ] At parent call context. - [ ] Gas usage - - [ ] Normal operation gas usage - - [ ] Verify gas affectation of each parameter value consumed by the opcode. - - [ ] Memory expansion - - [ ] Verify that the memory expansion correctly follows the gas calculation - - [ ] Out-of-gas during opcode execution - - [ ] Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exeptional abort. - - [ ] Out-of-gas during memory expansion - - [ ] Verify that the expansion of memory can result in out-of-gas exeptional abort. + - [ ] Normal operation gas usage: Verify gas affectation of each parameter value consumed by the opcode. + - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation + - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exeptional abort. + - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exeptional abort. + - [ ] Order-of-operations: + - [ ] If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of eachother) and create a case for success and another for OOG (with a 1-gas-difference). - [ ] Terminating opcode - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios - [ ] Top-level call termination @@ -102,6 +112,8 @@ The EIP introduces one or more new opcodes to the EVM. ## New Precompile +The EIP introduces one or more new precompiles. + ### Test Vectors - [ ] Call contexts @@ -117,9 +129,18 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Inputs - [ ] Verify combinations of valid inputs to the precompile - [ ] Verify interesting edge values given the precompile functionality. - - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties. + - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. + - [ ] Verify combinations of invalid inputs to the precompile. + - [ ] Inputs that fail specific mathematical or cryptographic validity checks. + - [ ] Inputs that are malformed/corrupted. +- [ ] Value Transfer + - [ ] If the precompile requires a minimum value with the calls to it, either constant or depending on a formula, verify: + - [ ] Calls with the required value amount minus one, expect failure. + - [ ] Calls with the exact required amount, expect success. + - [ ] Calls with extra value than the required amount, expect success. + - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. - [ ] Out-of-bounds checks - [ ] Verify if the precompile has out-of-bounds conditions in its inputs and verify: - [ ] Max value for each input @@ -142,7 +163,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify exact gas consumption, given different valid inputs. - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. - [ ] Excessive Gas Cases - - [ ] Verify spending all block gas in calls to the precompile (100 million gas or more). + - [ ] Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). - [ ] Fork transition - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. @@ -153,6 +174,20 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +## Removed Precompile + +The EIP removes one or more precompiles from the existing list of precompiles. + +### Test Vectors + +- [ ] Fork boundary + - [ ] Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. + - [ ] Verify the account is warm up until the last block before the fork is active, and becomes cold afterwards. + +### Framework Changes + +- [ ] Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` + ## New System Contract ### Test Vectors @@ -171,6 +206,15 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify interesting edge values given the system contract functionality. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. + - [ ] Verify combinations of invalid inputs to the precompile. + - [ ] Inputs that fail specific validity checks. + - [ ] Inputs that are malformed/corrupted. +- [ ] Value Transfer + - [ ] If the system contract requires a minimum value with the calls to it, either constant or depending on a formula, verify: + - [ ] Calls with the required value amount minus one, expect failure. + - [ ] Calls with the exact required amount, expect success. + - [ ] Calls with extra value than the required amount, expect success. + - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. - [ ] Out-of-bounds checks - [ ] Verify if the system contract has out-of-bounds conditions in its inputs and verify: - [ ] Max value for each input @@ -191,6 +235,7 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] System Contract Deployment - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed. - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. + - [ ] Contract Variations - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior - [ ] Contract Substitution @@ -258,7 +303,10 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] `SECP256K1N - S` of a valid signature - Transaction-Scoped Attributes/Variables - Verify attributes that can be read in the EVM from transaction fields. - - Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts). + - Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. + - Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts): + - Persist throughout the entire transaction. + - Reset on subsequent transactions in the same block. - [ ] Encoding (RLP, SSZ) - [ ] Verify correct transaction rejection due to incorrect field sizes: - [ ] Add leading zero byte @@ -280,8 +328,18 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Contract creation - [ ] Verify that the transaction can create new contracts if the transaction type supports it. - [ ] Sender account modifications - - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanic that bumps the nonce by more than one). - - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) after the transaction is included in a block + - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). + - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). +- [ ] Block Level Interactions + - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: + - [ ] New transaction type is the single transaction in the block: + - [ ] Rejected if `tx.gas_limit == block.gas_limit + 1` + - [ ] Accepted if `tx.gas_limit == block.gas_limit` + - [ ] New transaction type is the last transaction of a block with two transactions: + - [ ] Accepted if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit` + - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` + - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. + - [ ] Verify a block with all transactions types including the new type is executed correctly. - [ ] Fork transition - [ ] Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. - [ ] RPC Tests @@ -303,8 +361,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Genesis value - [ ] Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. - [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. - [ ] Fork transition - [ ] Verify initial value of the field at the first block of the activation fork. - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. @@ -313,17 +371,33 @@ The EIP introduces one or more new opcodes to the EVM. ### Framework Changes -**TBD** +- [ ] Add the new header field to the relevant objects: + - [ ] `ethereum_test_fixtures.FixtureHeader` + - [ ] `ethereum_test_fixtures.FixtureExecutionPayload` + - [ ] `ethereum_test_specs.Header` +- [ ] Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. ## New Block Body Field ### Test Vectors -**TBD** +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Fork transition + - [ ] Verify that a block containing the new block body field before the activation of the fork is invalid. + - [ ] Verify that a block lacking the new block field at the activation of the fork is invalid. ### Framework Changes -**TBD** +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Add the new body field to the relevant objects: + - [ ] `ethereum_test_fixtures.FixtureBlockBase` + - [ ] `ethereum_test_fixtures.FixtureEngineNewPayload` + - [ ] `ethereum_test_specs.Block` +- [ ] Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. ## Gas Cost Changes From 7b97a89c88070904c1b8e21a83bac643a4f3e931 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 21:59:46 +0000 Subject: [PATCH 20/33] Add gas tests --- .../eip_testing_checklist_template.md | 25 ++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index e749c37ab8c..7ba34c90405 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -403,7 +403,11 @@ The EIP removes one or more precompiles from the existing list of precompiles. ### Test Vectors -**TBD** +- [ ] Gas Usage: Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. +- [ ] Out-of-gas: Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. +- [ ] Fork transition: Verify gas costs are: + - [ ] Unaffected before the fork activation block. + - [ ] Updated on and after fork activation block. ### Framework Changes @@ -416,11 +420,26 @@ The EIP removes one or more precompiles from the existing list of precompiles. ### Test Vectors -**TBD** +- [ ] Refund calculation: Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios: + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT + 1` + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT` + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT - 1` +- [ ] Exceptional Abort: + - [ ] If the operation causing the refund can be reverted, verify the refund is not applied if the following cases: + - [ ] `REVERT` + - [ ] Out-of-gas + - [ ] Invalid opcode + - [ ] `REVERT` of an upper call frame + - [ ] If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases: + - [ ] `REVERT` at the top call frame + - [ ] Out-of-gas at the top call frame + - [ ] Invalid opcode at the top call frame +- [ ] Cross-Functional Test: Verify the following tests are updated to support the new type of refunds: + - [ ] `tests/prague/eip7623_increase_calldata_cost/test_refunds.py` ### Framework Changes -**TBD** +N/A ## Blob Count Changes From 7e72f52abd81576091ec14cc86e3bb9922a66f0e Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 22:08:47 +0000 Subject: [PATCH 21/33] Simplify bullet points --- .../eip_testing_checklist_template.md | 78 ++++++++----------- 1 file changed, 33 insertions(+), 45 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 7ba34c90405..4528b926687 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -61,10 +61,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). - [ ] Verify opcode behavior on re-entry using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). - [ ] Set-code delegated account: Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. - - [ ] Transaction context - - [ ] If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - - [ ] Block context - - [ ] If opcode changes behavior depending on particular block properties, test using multiple values for each property. + - [ ] Transaction context: If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. + - [ ] Block context: If opcode changes behavior depending on particular block properties, test using multiple values for each property. - [ ] Return data - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: - [ ] At current call context. @@ -74,8 +72,9 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exeptional abort. - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exeptional abort. - - [ ] Order-of-operations: - - [ ] If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of eachother) and create a case for success and another for OOG (with a 1-gas-difference). + - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of eachother) and the following cases for each: + - [ ] Success using the exact amount of gas required for the stipend. + - [ ] OOG with a 1-gas-difference from the gas required for the stipend. - [ ] Terminating opcode - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios - [ ] Top-level call termination @@ -91,10 +90,8 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: - [ ] Max value for each parameter - [ ] Max value + 1 for each parameter -- [ ] Exeptional Abort - - [ ] Verify behavior that is supposed to cause an exeptional abort, other than stack over or underflow, or out-of-gas errors. -- [ ] Data portion - - [ ] If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). +- [ ] Exeptional Abort: Verify behavior that is supposed to cause an exeptional abort, other than stack over or underflow, or out-of-gas errors. +- [ ] Data portion: If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). - [ ] Contract creation - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - [ ] Verify that contract is not created in case of: @@ -162,8 +159,7 @@ The EIP introduces one or more new precompiles. - [ ] Precompile has dynamic gas usage - [ ] Verify exact gas consumption, given different valid inputs. - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. -- [ ] Excessive Gas Cases - - [ ] Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). +- [ ] Excessive Gas Cases: Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). - [ ] Fork transition - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. @@ -203,7 +199,7 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Contract creating opcode - [ ] Inputs - [ ] Verify combinations of valid inputs to the system contract - - [ ] Verify interesting edge values given the system contract functionality. + - [ ] Verify interesting boundary values given the system contract functionality. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - [ ] Verify combinations of invalid inputs to the precompile. @@ -237,13 +233,15 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. - [ ] Contract Variations - - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior -- [ ] Contract Substitution - - [ ] Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): - - [ ] Modified return value lengths - - [ ] Modify emitted logs -- [ ] Fork transition - - [ ] Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). + - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior, + - [ ] Verify execution of a variation that causes an exception. + - [ ] Verify execution of a variation that consumes: + - [ ] 30,000,000 million gas exactly, execution should be successful. + - [ ] 30,000,001 million gas exactly, execution should fail. +- [ ] Contract Substitution: Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): + - [ ] Modified return value lengths + - [ ] Modify emitted logs +- [ ] Fork transition: Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). ### Framework Changes @@ -258,21 +256,15 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. - - [ ] Max fee per gas: - - [ ] Verify the transaction (and the block it is included in) is invalid if its max-priority-fee-per-gas value is lower than the max-fee-per-gas. - - [ ] Verify the transaction (and the block it is included in) is invalid if its max-fee-per-gas value is lower than the blocks base-fee-per-gas. - - [ ] Chain ID: - - [ ] Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. - - [ ] Nonce: - - [ ] Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. - - [ ] To: - - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. - - [ ] Value: - - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. - - [ ] Data: - - [ ] Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas gost. - - [ ] Sender balance: - - [ ] Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. + - [ ] Max fee per gas: Verify the transaction (and the block it is included in) is invalid if: + - [ ] Its max-priority-fee-per-gas value is lower than the max-fee-per-gas. + - [ ] Its max-fee-per-gas value is lower than the blocks base-fee-per-gas. + - [ ] Chain ID: Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. + - [ ] Nonce: Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. + - [ ] To: Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. + - [ ] Value: Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. + - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas gost. + - [ ] Sender balance: Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. - [ ] Signature: - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: - [ ] V, R, S represent a value that is inside of the field but outside of the curve. @@ -321,12 +313,10 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field, plus: - [ ] Verify transaction rejection if the serializable field is incorrectly encoded as bytes instead of using the correct encoding. -- [ ] Out-of-bounds checks - - [ ] Verify if the transaction has out-of-bounds conditions in its fields and verify: - - [ ] Max value for each field - - [ ] Max value + 1 for each field -- [ ] Contract creation - - [ ] Verify that the transaction can create new contracts if the transaction type supports it. +- [ ] Out-of-bounds checks: Verify if the transaction has out-of-bounds conditions in its fields and verify: + - [ ] Max value for each field + - [ ] Max value + 1 for each field +- [ ] Contract creation: Verify that the transaction can create new contracts if the transaction type supports it. - [ ] Sender account modifications - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). @@ -340,8 +330,7 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. - [ ] Verify a block with all transactions types including the new type is executed correctly. -- [ ] Fork transition - - [ ] Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. +- [ ] Fork transition: Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. - [ ] RPC Tests - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - [ ] Verify `eth_sendRawTransaction` using `execute` @@ -358,8 +347,7 @@ The EIP removes one or more precompiles from the existing list of precompiles. ### Test Vectors -- [ ] Genesis value - - [ ] Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. +- [ ] Genesis value: Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. - [ ] Value behavior - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. From 417da2389d424ebb2ac6198b780b3d4b2593a06e Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 22:24:45 +0000 Subject: [PATCH 22/33] fix(docs): tox --- .../eip_testing_checklist_template.md | 54 +++++++++---------- docs/checklist_templates/index.md | 1 - docs/writing_tests/post_mortems.md | 3 +- whitelist.txt | 9 ++++ 4 files changed, 36 insertions(+), 31 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 4528b926687..7d5d97aa950 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -5,7 +5,7 @@ Depending on the changes introduced by an EIP, the following template is the min ## General - [ ] Code coverage - - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exeptions being unreachable code lines. + - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. - [ ] Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. - [ ] Fuzzing @@ -32,9 +32,9 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] 2**64 bytes expansion - [ ] 2**256-1 bytes expansion (Should always result in Out-of-gas) - [ ] Stack - - [ ] Overflows/Underflows - - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exeptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exeptional abort then stack has 1 less item than the minimum stack height expected. + - [ ] Overflow/Underflow + - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. - [ ] If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations: - [ ] Operation on different stack heights: - [ ] Zero (Potential stack-underflow) @@ -47,9 +47,9 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Execution context - [ ] CALL - [ ] STATICCALL - - [ ] Verify exeptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. + - [ ] Verify exceptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). - - [ ] Verify subcalls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exeptional abort behavior. + - [ ] Verify sub-calls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exceptional abort behavior. - [ ] DELEGATECALL - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. @@ -64,15 +64,15 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Transaction context: If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - [ ] Block context: If opcode changes behavior depending on particular block properties, test using multiple values for each property. - [ ] Return data - - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unnaffected: + - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected: - [ ] At current call context. - [ ] At parent call context. - [ ] Gas usage - [ ] Normal operation gas usage: Verify gas affectation of each parameter value consumed by the opcode. - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation - - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exeptional abort. - - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exeptional abort. - - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of eachother) and the following cases for each: + - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. + - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exceptional abort. + - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the following cases for each: - [ ] Success using the exact amount of gas required for the stipend. - [ ] OOG with a 1-gas-difference from the gas required for the stipend. - [ ] Terminating opcode @@ -90,16 +90,16 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: - [ ] Max value for each parameter - [ ] Max value + 1 for each parameter -- [ ] Exeptional Abort: Verify behavior that is supposed to cause an exeptional abort, other than stack over or underflow, or out-of-gas errors. +- [ ] Exceptional Abort: Verify behavior that is supposed to cause an exceptional abort, other than stack over or underflow, or out-of-gas errors. - [ ] Data portion: If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). - [ ] Contract creation - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - [ ] Verify that contract is not created in case of: - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - - [ ] Creation would result in an address collision with an existing contract or eoa-delegated address. + - [ ] Creation would result in an address collision with an existing contract or EOA-delegated address. - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. - [ ] Fork transition - - [ ] Verify that the opcode results in exeptional abort if executed before its activation fork. + - [ ] Verify that the opcode results in exceptional abort if executed before its activation fork. - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. ### Framework Changes @@ -130,8 +130,8 @@ The EIP introduces one or more new precompiles. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific mathematical or cryptographic validity checks. - - [ ] Inputs that are malformed/corrupted. + - [ ] Inputs that fail specific mathematical or cryptographic validity checks. + - [ ] Inputs that are malformed/corrupted. - [ ] Value Transfer - [ ] If the precompile requires a minimum value with the calls to it, either constant or depending on a formula, verify: - [ ] Calls with the required value amount minus one, expect failure. @@ -165,7 +165,6 @@ The EIP introduces one or more new precompiles. - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. - ### Framework Changes - [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` @@ -203,8 +202,8 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Verify all zeros input. - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific validity checks. - - [ ] Inputs that are malformed/corrupted. + - [ ] Inputs that fail specific validity checks. + - [ ] Inputs that are malformed/corrupted. - [ ] Value Transfer - [ ] If the system contract requires a minimum value with the calls to it, either constant or depending on a formula, verify: - [ ] Calls with the required value amount minus one, expect failure. @@ -229,9 +228,8 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] If possible, simulate a scenario where the execution of the contract at the end of the block execution by the system address would result in excessive gas usage (100 million gas or more). - [ ] Verify spending all block gas in calls to system contract (100 million gas or more). - [ ] System Contract Deployment - - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed. + - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. - - [ ] Contract Variations - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior, - [ ] Verify execution of a variation that causes an exception. @@ -263,7 +261,7 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Nonce: Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. - [ ] To: Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. - [ ] Value: Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. - - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas gost. + - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas cost. - [ ] Sender balance: Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. - [ ] Signature: - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: @@ -321,28 +319,27 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). - [ ] Block Level Interactions - - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: + - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: - [ ] New transaction type is the single transaction in the block: - [ ] Rejected if `tx.gas_limit == block.gas_limit + 1` - [ ] Accepted if `tx.gas_limit == block.gas_limit` - [ ] New transaction type is the last transaction of a block with two transactions: - [ ] Accepted if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit` - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` - - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. - - [ ] Verify a block with all transactions types including the new type is executed correctly. + - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. + - [ ] Verify a block with all transactions types including the new type is executed correctly. - [ ] Fork transition: Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. - [ ] RPC Tests - - [ ] * Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type + - [ ] *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - [ ] Verify `eth_sendRawTransaction` using `execute` -* Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. +*Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. ### Framework Changes - [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. - [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). - ## New Block Header Field ### Test Vectors @@ -356,7 +353,6 @@ The EIP removes one or more precompiles from the existing list of precompiles. - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. - ### Framework Changes - [ ] Add the new header field to the relevant objects: @@ -448,4 +444,4 @@ N/A ### Framework Changes -- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. \ No newline at end of file +- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. diff --git a/docs/checklist_templates/index.md b/docs/checklist_templates/index.md index e5427b5faf3..ec798641680 100644 --- a/docs/checklist_templates/index.md +++ b/docs/checklist_templates/index.md @@ -1,4 +1,3 @@ # Testing Checklist Templates This page includes checklist templates that must be followed to guarantee a minimum base testing coverage of new EIP feature. - diff --git a/docs/writing_tests/post_mortems.md b/docs/writing_tests/post_mortems.md index 36f291df35d..6e2db8e0cef 100644 --- a/docs/writing_tests/post_mortems.md +++ b/docs/writing_tests/post_mortems.md @@ -9,6 +9,7 @@ Each entry must include an explanation of why the test case was missed plus the ## List ## TEMPLATE + ## Date - Title - Fork ### Description @@ -48,5 +49,5 @@ IDs of the tests added that now cover the missed scenario and link to the docume Note any modifications that were introduced in the framework and/or documentation to prevent similar misses. *Example:* -- Updated EIP checklist to include testing combinations of interesting points related to the elliptic-curve under test, and all combinations between them. +- Updated EIP checklist to include testing combinations of interesting points related to the elliptic-curve under test, and all combinations between them. diff --git a/whitelist.txt b/whitelist.txt index cdfa4e6a735..c84a9baae1e 100644 --- a/whitelist.txt +++ b/whitelist.txt @@ -94,6 +94,7 @@ controlflow cp CPUs crypto +cryptographic currentframe customizations Customizations @@ -295,6 +296,8 @@ mixins mixhash mkdocs mkdocstrings +mortem +mortems msm mypy ctx @@ -334,6 +337,7 @@ parseable passthrough pathlib pdb +Pectra perf permalink petersburg @@ -395,6 +399,7 @@ secp256k1 secp256k1n selfbalance ser +serializable servable setenv setitem @@ -412,6 +417,7 @@ spencertb splitext squidfunk src +ssz stackoverflow StateTest StateTestFiller @@ -436,6 +442,7 @@ sudo sysconfig t8n tamasfe +tbd terminalreporter testability TestAddress @@ -470,6 +477,7 @@ ubuntu ukiyo uid uncomment +underflow undersize unlink usr @@ -509,6 +517,7 @@ xpassed yaml YAML yml +yParity yul zfill From a2a94394aa895010702a7d2affc4db75f29763dc Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 22:43:29 +0000 Subject: [PATCH 23/33] Add more ids --- .../eip_testing_checklist_template.md | 125 +++++++++--------- 1 file changed, 62 insertions(+), 63 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 7d5d97aa950..a0997799e03 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -2,14 +2,13 @@ Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features. -## General +## General -- [ ] Code coverage - - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. - - [ ] Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. - - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. -- [ ] Fuzzing - - [ ] TBD +- [ ] Code coverage + - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. + - [ ] Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. + - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. +- [ ] Fuzzing ## New Opcode @@ -363,85 +362,85 @@ The EIP removes one or more precompiles from the existing list of precompiles. ## New Block Body Field -### Test Vectors +### Test Vectors -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Fork transition - - [ ] Verify that a block containing the new block body field before the activation of the fork is invalid. - - [ ] Verify that a block lacking the new block field at the activation of the fork is invalid. +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Fork transition + - [ ] Verify that a block containing the new block body field before the activation of the fork is invalid. + - [ ] Verify that a block lacking the new block field at the activation of the fork is invalid. -### Framework Changes +### Framework Changes -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Add the new body field to the relevant objects: - - [ ] `ethereum_test_fixtures.FixtureBlockBase` - - [ ] `ethereum_test_fixtures.FixtureEngineNewPayload` - - [ ] `ethereum_test_specs.Block` -- [ ] Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Add the new body field to the relevant objects: + - [ ] `ethereum_test_fixtures.FixtureBlockBase` + - [ ] `ethereum_test_fixtures.FixtureEngineNewPayload` + - [ ] `ethereum_test_specs.Block` +- [ ] Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. ## Gas Cost Changes -### Test Vectors +### Test Vectors -- [ ] Gas Usage: Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. -- [ ] Out-of-gas: Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. -- [ ] Fork transition: Verify gas costs are: - - [ ] Unaffected before the fork activation block. - - [ ] Updated on and after fork activation block. +- [ ] Gas Usage: Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. +- [ ] Out-of-gas: Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. +- [ ] Fork transition: Verify gas costs are: + - [ ] Unaffected before the fork activation block. + - [ ] Updated on and after fork activation block. -### Framework Changes +### Framework Changes -- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. -- [ ] Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. -- [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. -- [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. +- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. +- [ ] Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. +- [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. +- [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. ## Gas Refunds Changes -### Test Vectors - -- [ ] Refund calculation: Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios: - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT + 1` - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT` - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT - 1` -- [ ] Exceptional Abort: - - [ ] If the operation causing the refund can be reverted, verify the refund is not applied if the following cases: - - [ ] `REVERT` - - [ ] Out-of-gas - - [ ] Invalid opcode - - [ ] `REVERT` of an upper call frame - - [ ] If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases: - - [ ] `REVERT` at the top call frame - - [ ] Out-of-gas at the top call frame - - [ ] Invalid opcode at the top call frame -- [ ] Cross-Functional Test: Verify the following tests are updated to support the new type of refunds: - - [ ] `tests/prague/eip7623_increase_calldata_cost/test_refunds.py` - -### Framework Changes +### Test Vectors + +- [ ] Refund calculation: Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios: + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT + 1` + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT` + - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT - 1` +- [ ] Exceptional Abort: + - [ ] If the operation causing the refund can be reverted, verify the refund is not applied if the following cases: + - [ ] `REVERT` + - [ ] Out-of-gas + - [ ] Invalid opcode + - [ ] `REVERT` of an upper call frame + - [ ] If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases: + - [ ] `REVERT` at the top call frame + - [ ] Out-of-gas at the top call frame + - [ ] Invalid opcode at the top call frame +- [ ] Cross-Functional Test: Verify the following tests are updated to support the new type of refunds: + - [ ] `tests/prague/eip7623_increase_calldata_cost/test_refunds.py` + +### Framework Changes N/A ## Blob Count Changes -### Test Vectors +### Test Vectors -- [ ] Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. +- [ ] Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. -### Framework Changes +### Framework Changes -- [ ] Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. +- [ ] Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. ## New Execution Layer Request -### Test Vectors +### Test Vectors -- [ ] Cross-Request-Type Interaction - - [ ] Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations +- [ ] Cross-Request-Type Interaction + - [ ] Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations -### Framework Changes +### Framework Changes -- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. +- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. From 5384ae85360e27bd2352ab7fbeea31c21c533ce5 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 22:49:38 +0000 Subject: [PATCH 24/33] Add more ids --- .../eip_testing_checklist_template.md | 328 +++++++++--------- 1 file changed, 163 insertions(+), 165 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index a0997799e03..5f6f69027fc 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -19,87 +19,86 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Memory expansion: Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: - [ ] Zero bytes expansion - [ ] Zero-offset - - [ ] 2**256-1 offset - - [ ] Single byte expansion - - [ ] 31 bytes expansion - - [ ] 32 bytes expansion - - [ ] 33 bytes expansion - - [ ] 64 bytes expansion - - [ ] 2**32-1 bytes expansion - - [ ] 2**32 bytes expansion - - [ ] 2**64-1 bytes expansion - - [ ] 2**64 bytes expansion - - [ ] 2**256-1 bytes expansion (Should always result in Out-of-gas) -- [ ] Stack - - [ ] Overflow/Underflow - - [ ] If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - - [ ] If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. - - [ ] If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations: - - [ ] Operation on different stack heights: - - [ ] Zero (Potential stack-underflow) - - [ ] Odd - - [ ] Even - - [ ] For each variable `n` of the opcode that accesses the nth stack item, test `n` being: - - [ ] Top stack item - - [ ] Bottom stack item - - [ ] Middle stack item -- [ ] Execution context - - [ ] CALL - - [ ] STATICCALL - - [ ] Verify exceptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. - - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). - - [ ] Verify sub-calls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exceptional abort behavior. - - [ ] DELEGATECALL - - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. - - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. - - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. - - [ ] CALLCODE - - [ ] Initcode - - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - - [ ] Initcode of a contract creating transaction. - - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). - - [ ] Verify opcode behavior on re-entry using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). - - [ ] Set-code delegated account: Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. - - [ ] Transaction context: If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - - [ ] Block context: If opcode changes behavior depending on particular block properties, test using multiple values for each property. -- [ ] Return data - - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected: - - [ ] At current call context. - - [ ] At parent call context. -- [ ] Gas usage - - [ ] Normal operation gas usage: Verify gas affectation of each parameter value consumed by the opcode. - - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation - - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. - - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exceptional abort. - - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the following cases for each: - - [ ] Success using the exact amount of gas required for the stipend. - - [ ] OOG with a 1-gas-difference from the gas required for the stipend. -- [ ] Terminating opcode - - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios - - [ ] Top-level call termination - - [ ] Sub-level call termination - - [ ] Initcode termination - - [ ] If the terminating opcode is meant to rollback the executing call frame, verify the following events are properly rolled back: - - [ ] Balance changes - - [ ] Storage changes - - [ ] Contract creations - - [ ] Nonce increments - - [ ] Log events -- [ ] Out-of-bounds checks - - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: - - [ ] Max value for each parameter - - [ ] Max value + 1 for each parameter -- [ ] Exceptional Abort: Verify behavior that is supposed to cause an exceptional abort, other than stack over or underflow, or out-of-gas errors. -- [ ] Data portion: If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). -- [ ] Contract creation - - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - - [ ] Verify that contract is not created in case of: - - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - - [ ] Creation would result in an address collision with an existing contract or EOA-delegated address. - - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. -- [ ] Fork transition - - [ ] Verify that the opcode results in exceptional abort if executed before its activation fork. - - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. + - [ ] 2**256-1 offset + - [ ] Single byte expansion + - [ ] 31 bytes expansion + - [ ] 32 bytes expansion + - [ ] 33 bytes expansion + - [ ] 64 bytes expansion + - [ ] 2**32-1 bytes expansion + - [ ] 2**32 bytes expansion + - [ ] 2**64-1 bytes expansion + - [ ] 2**64 bytes expansion + - [ ] 2**256-1 bytes expansion (Should always result in Out-of-gas) +- [ ] Stack + - [ ] Overflow: If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + - [ ] Underflow: If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. + - [ ] If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations: + - [ ] Operation on different stack heights: + - [ ] Zero (Potential stack-underflow) + - [ ] Odd + - [ ] Even + - [ ] For each variable `n` of the opcode that accesses the nth stack item, test `n` being: + - [ ] Top stack item + - [ ] Bottom stack item + - [ ] Middle stack item +- [ ] Execution context + - [ ] CALL + - [ ] STATICCALL + - [ ] Verify exceptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. + - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). + - [ ] Verify sub-calls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exceptional abort behavior. + - [ ] DELEGATECALL + - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. + - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. + - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. + - [ ] CALLCODE + - [ ] Initcode + - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation + - [ ] Initcode of a contract creating transaction. + - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). + - [ ] Verify opcode behavior on re-entry using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). + - [ ] Set-code delegated account: Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. + - [ ] Transaction context: If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. + - [ ] Block context: If opcode changes behavior depending on particular block properties, test using multiple values for each property. +- [ ] Return data + - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected: + - [ ] At current call context. + - [ ] At parent call context. +- [ ] Gas usage + - [ ] Normal operation gas usage: Verify gas affectation of each parameter value consumed by the opcode. + - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation + - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. + - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exceptional abort. + - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the following cases for each: + - [ ] Success using the exact amount of gas required for the stipend. + - [ ] OOG with a 1-gas-difference from the gas required for the stipend. +- [ ] Terminating opcode + - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios + - [ ] Top-level call termination + - [ ] Sub-level call termination + - [ ] Initcode termination + - [ ] If the terminating opcode is meant to rollback the executing call frame, verify the following events are properly rolled back: + - [ ] Balance changes + - [ ] Storage changes + - [ ] Contract creations + - [ ] Nonce increments + - [ ] Log events +- [ ] Out-of-bounds checks + - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: + - [ ] Max value for each parameter + - [ ] Max value + 1 for each parameter +- [ ] Exceptional Abort: Verify behavior that is supposed to cause an exceptional abort, other than stack over or underflow, or out-of-gas errors. +- [ ] Data portion: If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). +- [ ] Contract creation + - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. + - [ ] Verify that contract is not created in case of: + - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. + - [ ] Creation would result in an address collision with an existing contract or EOA-delegated address. + - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. +- [ ] Fork transition + - [ ] Verify that the opcode results in exceptional abort if executed before its activation fork. + - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. ### Framework Changes @@ -249,95 +248,94 @@ The EIP removes one or more precompiles from the existing list of precompiles. ### Test Vectors -- [ ] Intrinsic Validity - - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: - - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. - - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. - - [ ] Max fee per gas: Verify the transaction (and the block it is included in) is invalid if: - - [ ] Its max-priority-fee-per-gas value is lower than the max-fee-per-gas. - - [ ] Its max-fee-per-gas value is lower than the blocks base-fee-per-gas. - - [ ] Chain ID: Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. - - [ ] Nonce: Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. - - [ ] To: Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. - - [ ] Value: Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. - - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas cost. - - [ ] Sender balance: Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. -- [ ] Signature: - - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: - - [ ] V, R, S represent a value that is inside of the field but outside of the curve. - - [ ] V (yParity) is any of the following invalid values: - - [ ] `2` - - [ ] `27` (Type-0 transaction valid value) - - [ ] `28` (Type-0 transaction valid value) - - [ ] `35` (Type-0 replay-protected transaction valid value for chain id 1) - - [ ] `36` (Type-0 replay-protected transaction valid value for chain id 1) - - [ ] `2**8-1` - - [ ] R is any of the following invalid values: - - [ ] `0` - - [ ] `SECP256K1N-1` - - [ ] `SECP256K1N` - - [ ] `SECP256K1N+1` - - [ ] `2**256-1` - - [ ] `2**256` - - [ ] S is any of the following invalid values: - - [ ] `0` - - [ ] `SECP256K1N//2-1` - - [ ] `SECP256K1N//2` - - [ ] `SECP256K1N//2+1` - - [ ] `SECP256K1N-1` - - [ ] `SECP256K1N` - - [ ] `SECP256K1N+1` - - [ ] `2**256-1` - - [ ] `2**256` - - [ ] `SECP256K1N - S` of a valid signature -- Transaction-Scoped Attributes/Variables - - Verify attributes that can be read in the EVM from transaction fields. - - Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. - - Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts): - - Persist throughout the entire transaction. - - Reset on subsequent transactions in the same block. -- [ ] Encoding (RLP, SSZ) - - [ ] Verify correct transaction rejection due to incorrect field sizes: - - [ ] Add leading zero byte - - [ ] Remove single byte from fixed-byte-length fields - - [ ] If the transaction contains a new field that is a list, verify: - - [ ] Zero-element list - - [ ] Max count list - - [ ] Max count plus one list - - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - - [ ] Verify correct transaction rejection if the transaction type contains extra fields - - [ ] Verify transaction is correctly rejected if the serialized bytes object is truncated - - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes - - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field, plus: - - [ ] Verify transaction rejection if the serializable field is incorrectly encoded as bytes instead of using the correct encoding. -- [ ] Out-of-bounds checks: Verify if the transaction has out-of-bounds conditions in its fields and verify: - - [ ] Max value for each field - - [ ] Max value + 1 for each field -- [ ] Contract creation: Verify that the transaction can create new contracts if the transaction type supports it. -- [ ] Sender account modifications - - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). - - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). -- [ ] Block Level Interactions - - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: - - [ ] New transaction type is the single transaction in the block: - - [ ] Rejected if `tx.gas_limit == block.gas_limit + 1` - - [ ] Accepted if `tx.gas_limit == block.gas_limit` - - [ ] New transaction type is the last transaction of a block with two transactions: - - [ ] Accepted if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit` - - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` - - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. - - [ ] Verify a block with all transactions types including the new type is executed correctly. -- [ ] Fork transition: Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. -- [ ] RPC Tests - - [ ] *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - - [ ] Verify `eth_sendRawTransaction` using `execute` +- [ ] Intrinsic Validity + - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: + - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. + - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. + - [ ] Max fee per gas: Verify the transaction (and the block it is included in) is invalid if: + - [ ] Its max-priority-fee-per-gas value is lower than the max-fee-per-gas. + - [ ] Its max-fee-per-gas value is lower than the blocks base-fee-per-gas. + - [ ] Chain ID: Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. + - [ ] Nonce: Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. + - [ ] To: Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. + - [ ] Value: Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. + - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas cost. + - [ ] Sender balance: Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. +- [ ] Signature: + - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: + - [ ] V, R, S represent a value that is inside of the field but outside of the curve. + - [ ] V (yParity) is any of the following invalid values: + - [ ] `2` + - [ ] `27` (Type-0 transaction valid value) + - [ ] `28` (Type-0 transaction valid value) + - [ ] `35` (Type-0 replay-protected transaction valid value for chain id 1) + - [ ] `36` (Type-0 replay-protected transaction valid value for chain id 1) + - [ ] `2**8-1` + - [ ] R is any of the following invalid values: + - [ ] `0` + - [ ] `SECP256K1N-1` + - [ ] `SECP256K1N` + - [ ] `SECP256K1N+1` + - [ ] `2**256-1` + - [ ] `2**256` + - [ ] S is any of the following invalid values: + - [ ] `0` + - [ ] `SECP256K1N//2-1` + - [ ] `SECP256K1N//2` + - [ ] `SECP256K1N//2+1` + - [ ] `SECP256K1N-1` + - [ ] `SECP256K1N` + - [ ] `SECP256K1N+1` + - [ ] `2**256-1` + - [ ] `2**256` + - [ ] `SECP256K1N - S` of a valid signature +- [ ] Transaction-Scoped Attributes/Variables + - [ ] Verify attributes that can be read in the EVM from transaction fields. + - [ ] Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. + - [ ] Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts): + - [ ] Persist throughout the entire transaction. + - [ ] Reset on subsequent transactions in the same block. +- [ ] Encoding (RLP, SSZ) + - [ ] Verify correct transaction rejection due to incorrect field sizes: + - [ ] Add leading zero byte + - [ ] Remove single byte from fixed-byte-length fields + - [ ] If the transaction contains a new field that is a list, verify: + - [ ] Zero-element list + - [ ] Max count list + - [ ] Max count plus one list + - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing + - [ ] Verify correct transaction rejection if the transaction type contains extra fields + - [ ] Verify transaction is correctly rejected if the serialized bytes object is truncated + - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes + - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field, plus: + - [ ] Verify transaction rejection if the serializable field is incorrectly encoded as bytes instead of using the correct encoding. +- [ ] Out-of-bounds checks: Verify if the transaction has out-of-bounds conditions in its fields and verify: + - [ ] Max value for each field + - [ ] Max value + 1 for each field +- [ ] Contract creation: Verify that the transaction can create new contracts if the transaction type supports it. +- [ ] Sender account modifications + - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). + - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). +- [ ] Block Level Interactions + - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: + - [ ] Rejected if `tx.gas_limit == block.gas_limit + 1` + - [ ] Accepted if `tx.gas_limit == block.gas_limit` + - [ ] New transaction type is the last transaction of a block with two transactions: + - [ ] Accepted if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit` + - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` + - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. + - [ ] Verify a block with all transactions types including the new type is executed correctly. +- [ ] Fork transition: Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. +- [ ] RPC Tests + - [ ] *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type + - [ ] Verify `eth_sendRawTransaction` using `execute` *Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. ### Framework Changes -- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. -- [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). +- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. +- [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). ## New Block Header Field From 4f3753cffc0e1e0f0017b0d0fca349061420440b Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 22:51:35 +0000 Subject: [PATCH 25/33] Add more ids --- .../eip_testing_checklist_template.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 5f6f69027fc..cfe68971901 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -341,22 +341,22 @@ The EIP removes one or more precompiles from the existing list of precompiles. ### Test Vectors -- [ ] Genesis value: Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Fork transition - - [ ] Verify initial value of the field at the first block of the activation fork. - - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. - - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. - -### Framework Changes - -- [ ] Add the new header field to the relevant objects: - - [ ] `ethereum_test_fixtures.FixtureHeader` - - [ ] `ethereum_test_fixtures.FixtureExecutionPayload` - - [ ] `ethereum_test_specs.Header` -- [ ] Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. +- [ ] Genesis value: Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. +- [ ] Value behavior + - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. + - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. +- [ ] Fork transition + - [ ] Verify initial value of the field at the first block of the activation fork. + - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. + - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. + +### Framework Changes + +- [ ] Add the new header field to the relevant objects: + - [ ] `ethereum_test_fixtures.FixtureHeader` + - [ ] `ethereum_test_fixtures.FixtureExecutionPayload` + - [ ] `ethereum_test_specs.Header` +- [ ] Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. ## New Block Body Field From da5638e4512e406b9308c95181a4faea7eb0945d Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 23:00:52 +0000 Subject: [PATCH 26/33] Add more ids --- .../eip_testing_checklist_template.md | 246 +++++++++--------- 1 file changed, 124 insertions(+), 122 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index cfe68971901..75426ddaef7 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -43,16 +43,16 @@ The EIP introduces one or more new opcodes to the EVM. - [ ] Bottom stack item - [ ] Middle stack item - [ ] Execution context - - [ ] CALL - - [ ] STATICCALL + - [ ] `CALL` + - [ ] `STATICCALL` - [ ] Verify exceptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). - - [ ] Verify sub-calls using other opcodes (e.g. CALL, DELEGATECALL, etc) also results in the same exceptional abort behavior. - - [ ] DELEGATECALL + - [ ] Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. + - [ ] `DELEGATECALL` - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. - - [ ] CALLCODE + - [ ] `CALLCODE` - [ ] Initcode - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - [ ] Initcode of a contract creating transaction. @@ -111,138 +111,140 @@ The EIP introduces one or more new precompiles. ### Test Vectors -- [ ] Call contexts - - [ ] Normal call to precompile from contract - - [ ] Delegate call to precompile from contract - - [ ] Static call to precompile from contract - - [ ] Code call to precompile from contract - - [ ] Precompile as transaction entry-point - - [ ] Call from initcode - - [ ] Contract creating transaction - - [ ] Contract creating opcode - - [ ] Set code delegated address (no precompile logic executed) -- [ ] Inputs - - [ ] Verify combinations of valid inputs to the precompile - - [ ] Verify interesting edge values given the precompile functionality. - - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). - - [ ] Verify all zeros input. - - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific mathematical or cryptographic validity checks. - - [ ] Inputs that are malformed/corrupted. -- [ ] Value Transfer - - [ ] If the precompile requires a minimum value with the calls to it, either constant or depending on a formula, verify: - - [ ] Calls with the required value amount minus one, expect failure. - - [ ] Calls with the exact required amount, expect success. - - [ ] Calls with extra value than the required amount, expect success. - - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. -- [ ] Out-of-bounds checks - - [ ] Verify if the precompile has out-of-bounds conditions in its inputs and verify: - - [ ] Max value for each input - - [ ] Max value + 1 for each input -- [ ] Input Lengths - - [ ] Zero-length calldata. - - [ ] Precompile has static-length input - - [ ] Correct static-length calldata - - [ ] Calldata too short, where the value represents a correct but truncated input to the precompile. - - [ ] Calldata too long, where the value represents a correct input to the precompile with padded zeros. - - [ ] Precompile has dynamic-length input - - [ ] Verify correct precompile execution for valid lengths, given different inputs. - - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the precompile. - - [ ] Calldata too long, given different inputs, where the value represents a correct input to the precompile with padded zeros. -- [ ] Gas usage - - [ ] Precompile has constant gas usage - - [ ] Verify exact gas consumption - - [ ] Verify exact gas consumption minus one results in out-of-gas error. - - [ ] Precompile has dynamic gas usage - - [ ] Verify exact gas consumption, given different valid inputs. - - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. -- [ ] Excessive Gas Cases: Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). -- [ ] Fork transition - - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. - - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. +- [ ] Call contexts + - [ ] `CALL` + - [ ] `DELEGATECALL` + - [ ] `STATICCALL` + - [ ] If the precompile is stateful, meaning calling it affects its storage, verify that using `STATICCALL` to call it results in exceptional abort. + - [ ] `CALLCODE` + - [ ] Precompile as transaction entry-point + - [ ] Call from Initcode + - [ ] Contract creating transaction + - [ ] Contract creating opcode + - [ ] Set code delegated address (no precompile logic executed) +- [ ] Inputs + - [ ] Verify combinations of valid inputs to the precompile + - [ ] Verify interesting edge values given the precompile functionality. + - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). + - [ ] Verify all zeros input. + - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. + - [ ] Verify combinations of invalid inputs to the precompile. + - [ ] Inputs that fail specific mathematical or cryptographic validity checks. + - [ ] Inputs that are malformed/corrupted. +- [ ] Value Transfer + - [ ] If the precompile requires a minimum value with the calls to it, either constant or depending on a formula, verify: + - [ ] Calls with the required value amount minus one, expect failure. + - [ ] Calls with the exact required amount, expect success. + - [ ] Calls with extra value than the required amount, expect success. + - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. +- [ ] Out-of-bounds checks + - [ ] Verify if the precompile has out-of-bounds conditions in its inputs and verify: + - [ ] Max value for each input + - [ ] Max value + 1 for each input +- [ ] Input Lengths + - [ ] Zero-length calldata. + - [ ] Precompile has static-length input + - [ ] Correct static-length calldata + - [ ] Calldata too short, where the value represents a correct but truncated input to the precompile. + - [ ] Calldata too long, where the value represents a correct input to the precompile with padded zeros. + - [ ] Precompile has dynamic-length input + - [ ] Verify correct precompile execution for valid lengths, given different inputs. + - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the precompile. + - [ ] Calldata too long, given different inputs, where the value represents a correct input to the precompile with padded zeros. +- [ ] Gas usage + - [ ] Precompile has constant gas usage + - [ ] Verify exact gas consumption + - [ ] Verify exact gas consumption minus one results in out-of-gas error. + - [ ] Precompile has dynamic gas usage + - [ ] Verify exact gas consumption, given different valid inputs. + - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. +- [ ] Excessive Gas Cases: Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). +- [ ] Fork transition + - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. + - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. + - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. ### Framework Changes -- [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -## Removed Precompile +## Removed Precompile The EIP removes one or more precompiles from the existing list of precompiles. -### Test Vectors +### Test Vectors -- [ ] Fork boundary - - [ ] Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. - - [ ] Verify the account is warm up until the last block before the fork is active, and becomes cold afterwards. +- [ ] Fork boundary + - [ ] Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. + - [ ] Verify the account is warm up until the last block before the fork is active, and becomes cold afterwards. -### Framework Changes +### Framework Changes -- [ ] Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` +- [ ] Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` ## New System Contract ### Test Vectors -- [ ] Call contexts - - [ ] Normal call to system contract from contract - - [ ] Delegate call to system contract from contract - - [ ] Static call to system contract from contract - - [ ] Code call to system contract from contract - - [ ] System contract as transaction entry-point - - [ ] Call from initcode - - [ ] Contract creating transaction - - [ ] Contract creating opcode -- [ ] Inputs - - [ ] Verify combinations of valid inputs to the system contract - - [ ] Verify interesting boundary values given the system contract functionality. - - [ ] Verify all zeros input. - - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific validity checks. - - [ ] Inputs that are malformed/corrupted. -- [ ] Value Transfer - - [ ] If the system contract requires a minimum value with the calls to it, either constant or depending on a formula, verify: - - [ ] Calls with the required value amount minus one, expect failure. - - [ ] Calls with the exact required amount, expect success. - - [ ] Calls with extra value than the required amount, expect success. - - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. -- [ ] Out-of-bounds checks - - [ ] Verify if the system contract has out-of-bounds conditions in its inputs and verify: - - [ ] Max value for each input - - [ ] Max value + 1 for each input -- [ ] Input Lengths - - [ ] Zero-length calldata. - - [ ] System contract has static-length input - - [ ] Correct static-length calldata - - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. - - [ ] System contract has dynamic-length input - - [ ] Verify correct System contract execution for valid lengths, given different inputs. - - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. -- [ ] Excessive Gas Cases - - [ ] If possible, simulate a scenario where the execution of the contract at the end of the block execution by the system address would result in excessive gas usage (100 million gas or more). - - [ ] Verify spending all block gas in calls to system contract (100 million gas or more). -- [ ] System Contract Deployment - - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). - - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. -- [ ] Contract Variations - - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior, - - [ ] Verify execution of a variation that causes an exception. - - [ ] Verify execution of a variation that consumes: - - [ ] 30,000,000 million gas exactly, execution should be successful. - - [ ] 30,000,001 million gas exactly, execution should fail. -- [ ] Contract Substitution: Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): - - [ ] Modified return value lengths - - [ ] Modify emitted logs -- [ ] Fork transition: Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). - -### Framework Changes - -- [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -- [ ] Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Call contexts + - [ ] `CALL` + - [ ] `DELEGATECALL` + - [ ] `STATICCALL` + - [ ] If the system contract is stateful, meaning calling it affects its storage, verify that using `STATICCALL` to call it results in exceptional abort. + - [ ] `CALLCODE` + - [ ] System contract as transaction entry-point + - [ ] Call from Initcode + - [ ] Contract creating transaction + - [ ] Contract creating opcode +- [ ] Inputs + - [ ] Verify combinations of valid inputs to the system contract + - [ ] Verify interesting boundary values given the system contract functionality. + - [ ] Verify all zeros input. + - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. + - [ ] Verify combinations of invalid inputs to the precompile. + - [ ] Inputs that fail specific validity checks. + - [ ] Inputs that are malformed/corrupted. +- [ ] Value Transfer + - [ ] If the system contract requires a minimum value with the calls to it, either constant or depending on a formula, verify: + - [ ] Calls with the required value amount minus one, expect failure. + - [ ] Calls with the exact required amount, expect success. + - [ ] Calls with extra value than the required amount, expect success. + - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. +- [ ] Out-of-bounds checks + - [ ] Verify if the system contract has out-of-bounds conditions in its inputs and verify: + - [ ] Max value for each input + - [ ] Max value + 1 for each input +- [ ] Input Lengths + - [ ] Zero-length calldata. + - [ ] System contract has static-length input + - [ ] Correct static-length calldata + - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. + - [ ] System contract has dynamic-length input + - [ ] Verify correct System contract execution for valid lengths, given different inputs. + - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. + - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. +- [ ] Excessive Gas Cases + - [ ] If possible, simulate a scenario where the execution of the contract at the end of the block execution by the system address would result in excessive gas usage (100 million gas or more). + - [ ] Verify spending all block gas in calls to system contract (100 million gas or more). +- [ ] System Contract Deployment + - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). + - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. +- [ ] Contract Variations + - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior, + - [ ] Verify execution of a variation that causes an exception. + - [ ] Verify execution of a variation that consumes: + - [ ] 30,000,000 million gas exactly, execution should be successful. + - [ ] 30,000,001 million gas exactly, execution should fail. +- [ ] Contract Substitution: Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): + - [ ] Modified return value lengths + - [ ] Modify emitted logs +- [ ] Fork transition: Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). + +### Framework Changes + +- [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- [ ] Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` ## New Transaction Type From 736c73c6cb7dbe899baf967392ceae7ac9bd3572 Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 9 May 2025 23:02:13 +0000 Subject: [PATCH 27/33] fix: tox --- whitelist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/whitelist.txt b/whitelist.txt index c84a9baae1e..b35ba3dc29c 100644 --- a/whitelist.txt +++ b/whitelist.txt @@ -419,6 +419,7 @@ squidfunk src ssz stackoverflow +stateful StateTest StateTestFiller StateTests From 56877b2c57241cdf9c5084af75ff66774dd91b7e Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 13 May 2025 22:20:10 +0000 Subject: [PATCH 28/33] refactor(docs): Convert EIP checklist to table --- .../eip_testing_checklist_template.md | 1470 ++++++++++++----- 1 file changed, 1074 insertions(+), 396 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 75426ddaef7..65be6692971 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -2,445 +2,1123 @@ Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features. -## General +## General -- [ ] Code coverage - - [ ] Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. - - [ ] Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. - - [ ] Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. -- [ ] Fuzzing +#### Code coverage -## New Opcode +| ID | Description | Status | Tests | +|-|-|-|-| +| `general/code_coverage/eels` | Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. | | | +| `general/code_coverage/test_coverage` | Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. | | | +| `general/code_coverage/second_client` | Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. | | | + +#### Fuzzing + +## New Opcode The EIP introduces one or more new opcodes to the EVM. -### Test Vectors - -- [ ] Memory expansion: Verify that the opcode execution results in the correct memory expansion, being by offset or size or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). Test at least the following memory expansion sizes: - - [ ] Zero bytes expansion - - [ ] Zero-offset - - [ ] 2**256-1 offset - - [ ] Single byte expansion - - [ ] 31 bytes expansion - - [ ] 32 bytes expansion - - [ ] 33 bytes expansion - - [ ] 64 bytes expansion - - [ ] 2**32-1 bytes expansion - - [ ] 2**32 bytes expansion - - [ ] 2**64-1 bytes expansion - - [ ] 2**64 bytes expansion - - [ ] 2**256-1 bytes expansion (Should always result in Out-of-gas) -- [ ] Stack - - [ ] Overflow: If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - - [ ] Underflow: If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. - - [ ] If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations: - - [ ] Operation on different stack heights: - - [ ] Zero (Potential stack-underflow) - - [ ] Odd - - [ ] Even - - [ ] For each variable `n` of the opcode that accesses the nth stack item, test `n` being: - - [ ] Top stack item - - [ ] Bottom stack item - - [ ] Middle stack item -- [ ] Execution context - - [ ] `CALL` - - [ ] `STATICCALL` - - [ ] Verify exceptional abort if the opcode is banned for static contexts or if it attempts to modify the code, storage or balance of an account. - - [ ] If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). - - [ ] Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. - - [ ] `DELEGATECALL` - - [ ] If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. - - [ ] If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. - - [ ] If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. - - [ ] `CALLCODE` - - [ ] Initcode - - [ ] Verify opcode behaves as expected when running during the initcode phase of contract creation - - [ ] Initcode of a contract creating transaction. - - [ ] Initcode of a contract creating opcode (including itself if opcode creates a contract). - - [ ] Verify opcode behavior on re-entry using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). - - [ ] Set-code delegated account: Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. - - [ ] Transaction context: If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - - [ ] Block context: If opcode changes behavior depending on particular block properties, test using multiple values for each property. -- [ ] Return data - - [ ] Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected: - - [ ] At current call context. - - [ ] At parent call context. -- [ ] Gas usage - - [ ] Normal operation gas usage: Verify gas affectation of each parameter value consumed by the opcode. - - [ ] Memory expansion: Verify that the memory expansion correctly follows the gas calculation - - [ ] Out-of-gas during opcode execution: Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. - - [ ] Out-of-gas during memory expansion: Verify that the expansion of memory can result in out-of-gas exceptional abort. - - [ ] Order-of-operations: If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the following cases for each: - - [ ] Success using the exact amount of gas required for the stipend. - - [ ] OOG with a 1-gas-difference from the gas required for the stipend. -- [ ] Terminating opcode - - [ ] If an opcode is terminating, meaning it results in the current call context to end execution, test the following scenarios - - [ ] Top-level call termination - - [ ] Sub-level call termination - - [ ] Initcode termination - - [ ] If the terminating opcode is meant to rollback the executing call frame, verify the following events are properly rolled back: - - [ ] Balance changes - - [ ] Storage changes - - [ ] Contract creations - - [ ] Nonce increments - - [ ] Log events -- [ ] Out-of-bounds checks - - [ ] Verify if the opcode has out-of-bounds conditions in its parameters and verify: - - [ ] Max value for each parameter - - [ ] Max value + 1 for each parameter -- [ ] Exceptional Abort: Verify behavior that is supposed to cause an exceptional abort, other than stack over or underflow, or out-of-gas errors. -- [ ] Data portion: If an opcode has data portion that affects its behavior, verify checklist items with multiple interesting values (E.g. if data portion size is 1 byte, use at least 0x00, 0x01, 0x7F, 0xFF). -- [ ] Contract creation - - [ ] Verify contract is created at the expected address given multiple inputs to the opcode parameters. - - [ ] Verify that contract is not created in case of: - - [ ] Out-of-gas when available gas is less than minimum contract creation stipend. - - [ ] Creation would result in an address collision with an existing contract or EOA-delegated address. - - [ ] Verify recursive contract creation using the opcode: Factory contract uses the opcode, and initcode calls back to factory contract. -- [ ] Fork transition - - [ ] Verify that the opcode results in exceptional abort if executed before its activation fork. - - [ ] Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. - -### Framework Changes - -- [ ] Add opcode to `src/ethereum_test_vm/opcode.py` -- [ ] Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## New Precompile +### Test Vectors + +#### Memory expansion + +If the opcode execution can expand the memory size, either by writing to memory or reading from an offset that exceeds current memory, or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/mem_exp/zero_bytes_zero_offset` | Zero bytes expansion with zero-offset. | | | +| `new_opcode/test/mem_exp/zero_bytes_max_offset` | Zero bytes expansion with 2**256-1 offset. | | | +| `new_opcode/test/mem_exp/single_byte` | Single byte expansion. | | | +| `new_opcode/test/mem_exp/31_bytes` | 31 bytes expansion. | | | +| `new_opcode/test/mem_exp/32_bytes` | 32 bytes expansion. | | | +| `new_opcode/test/mem_exp/33_bytes` | 33 bytes expansion. | | | +| `new_opcode/test/mem_exp/64_bytes` | 64 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_32_minus_one_bytes` | `2**32-1` bytes expansion. | | | +| `new_opcode/test/mem_exp/2_32_bytes` | 2**32 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_64_minus_one_bytes` | 2**64-1 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_64_bytes` | 2**64 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_256_minus_one_bytes` | 2**256-1 bytes expansion (Should always result in Out-of-gas). | | | + +#### Stack + +##### Stack Overflow + +If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/stack_overflow` | Stack Overflow. | | | + +##### Stack Underflow + +If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/stack_underflow` | Stack Underflow. | | | + +##### Stack Complex Operations + +If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/stack_complex_operations/stack_heights/zero` | Operation on an empty stack (Potential stack-underflow). | | | +| `new_opcode/test/stack_complex_operations/stack_heights/odd` | Operation on a stack with an odd height. | | | +| `new_opcode/test/stack_complex_operations/stack_heights/even` | Operation on a stack with an even height. | | | + +##### Stack Manipulation With Data Portion Variables + +If the opcode contains variables in its data portion, for each variable `n` of the opcode that accesses the nth stack item, test `n` being. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/stack_complex_operations/variable_n/top` | Top stack item. | | | +| `new_opcode/test/stack_complex_operations/variable_n/bottom` | Bottom stack item. | | | +| `new_opcode/test/stack_complex_operations/variable_n/middle` | Middle stack item. | | | + +#### Execution context + +Evaluate opcode's behavior in different execution contexts. + +##### `CALL` + +Verify opcode operation in a subcall frame originated from a `CALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/call` | `CALL`. | | | + +##### `STATICCALL` + +Verify opcode operation in a subcall frame originated from a `STATICCALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/staticcall/ban_check` | Verify exceptional abort if the opcode attempts to modify the code, storage or balance of an account. | | | +| `new_opcode/test/execution_context/staticcall/ban_no_modification` | If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). | | | +| `new_opcode/test/execution_context/staticcall/sub_calls` | Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. | | | + +##### `DELEGATECALL` + +Verify opcode operation in a subcall frame originated from a `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/delegatecall` | `DELEGATECALL`. | | | +| `new_opcode/test/execution_context/delegatecall/storage` | If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. | | | +| `new_opcode/test/execution_context/delegatecall/balance` | If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. | | | +| `new_opcode/test/execution_context/delegatecall/code` | If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. | | | + +##### `CALLCODE` + +Verify opcode operation in a subcall frame originated from a `CALLCODE` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/callcode` | `CALLCODE`. | | | + +##### Initcode + +Verify opcode behaves as expected when running during the initcode phase of contract creation. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/initcode/behavior` | Initcode operation. | | | +| `new_opcode/test/execution_context/initcode/behavior/tx` | Initcode of a contract creating transaction. | | | +| `new_opcode/test/execution_context/initcode/behavior/opcode` | Initcode of a contract creating opcode (including itself if opcode creates a contract). | | | +| `new_opcode/test/execution_context/initcode/reentry` | Initcode re-entry: using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). | | | + +##### Set-code Delegated Account + +Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/set_code` | Set-code delegated account. | | | + +##### Transaction Context + +If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/tx_context` | Transaction-context dependent opcode. | | | + +##### Block Context + +If opcode changes behavior depending on particular block properties, test using multiple values for each property. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/execution_context/block_context` | Block-context dependent opcode. | | | + +#### Return data + +Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/return_data/buffer/current` | Return buffer at current call context. | | | +| `new_opcode/test/return_data/buffer/parent` | Return buffer at parent call context. | | | + +#### Gas Usage + +##### Normal Operation + +Verify gas usage affectation of each stack argument or memory input consumed by the opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/gas_usage/normal` | Normal operation gas usage. | | | + +##### Memory Expansion + +Verify that the memory expansion correctly follows the gas calculation. + +| `new_opcode/test/gas_usage/memory_expansion` | Memory expansion. | | | + +##### Out-Of-Gas + +Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/gas_usage/out_of_gas_execution` | Out-of-gas due to opcode inputs. | | | +| `new_opcode/test/gas_usage/out_of_gas_memory` | Out-of-gas due to memory expansion. | | | + +##### Order-of-operations + +If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the listed cases for each. + +| `new_opcode/test/gas_usage/order_of_operations/exact` | Success using the exact amount of gas required for the stipend. | | | +| `new_opcode/test/gas_usage/order_of_operations/oog` | OOG with a 1-gas-difference from the gas required for the stipend. | | | + +#### Terminating opcode + +If an opcode is terminating, meaning it results in the current call context to end execution. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/terminating/scenarios/top_level` | Top-level call termination. | | | +| `new_opcode/test/terminating/scenarios/sub_level` | Sub-level call termination. | | | +| `new_opcode/test/terminating/scenarios/initcode` | Initcode termination. | | | + +#### Aborting/Reverting opcode + +If the terminating opcode is meant to rollback the executing call frame, verify the listed events are properly rolled back. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/terminating/rollback/balance` | Balance changes. | | | +| `new_opcode/test/terminating/rollback/storage` | Storage changes. | | | +| `new_opcode/test/terminating/rollback/contracts` | Contract creations. | | | +| `new_opcode/test/terminating/rollback/nonce` | Nonce increments. | | | +| `new_opcode/test/terminating/rollback/logs` | Log events. | | | + +#### Out-of-bounds checks + +If the opcode has out-of-bounds conditions in its parameters/inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/out_of_bounds/verify/max` | Max value for each parameter. | | | +| `new_opcode/test/out_of_bounds/verify/max_plus_one` | Max value + 1 for each parameter. | | | + +#### Exceptional Abort + +If the opcode has conditions, either inputs or execution context states, that should cause exceptional abort and are different than out-of-gas or stack overflow or underflow. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/exceptional_abort` | Exceptional abort conditions. | | | + +#### Data portion + +If an opcode has a data portion, meaning the `N` bytes following the opcode in the contract bytecode are skipped from execution. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/data_portion/all_zeros` | All zeros data portion. | | | +| `new_opcode/test/data_portion/max_value` | Max value data portion (`2**N-1` where `N` is the bit size of the data portion). | | | +| `new_opcode/test/data_portion/jump` | Jump into the data portion. | | | + + +#### Contract creation + +If the opcode execution results in the creation of a new contract in the state. + +##### Correct Creation + +Verify contract is created at the expected address given multiple inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/contract_creation/address` | Address calculation. | | | + +##### Creation Failure + +The contract creation fails given the listed conditions. + +| `new_opcode/test/contract_creation/failure/oog` | Out-of-gas when available gas is less than minimum contract creation stipend. | | | +| `new_opcode/test/contract_creation/failure/insufficient_value` | Opcode has a value parameter and the caller does not have enough funds. | | | +| `new_opcode/test/contract_creation/failure/collision` | Creation would result in an address collision with an existing contract or EOA-delegated address. | | | + +##### Recursive Contract Creation + +Opcode is used to attempt to recreate a contract that is currently mid-creation by a previous call of the same opcode. + +| `new_opcode/test/contract_creation/recursive` | Recursive contract creation using the opcode. | | | + +#### Fork transition + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_opcode/test/fork_transition/invalid` | Exceptional abort if executed before its activation fork/after its deactivation fork. | | | +| `new_opcode/test/fork_transition/at` | Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. | | | + +### Framework Changes + +| ID | Description | Status | Tests | +|-|-|-|-| +- Add opcode to `src/ethereum_test_vm/opcode.py` +- Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New Precompile The EIP introduces one or more new precompiles. -### Test Vectors - -- [ ] Call contexts - - [ ] `CALL` - - [ ] `DELEGATECALL` - - [ ] `STATICCALL` - - [ ] If the precompile is stateful, meaning calling it affects its storage, verify that using `STATICCALL` to call it results in exceptional abort. - - [ ] `CALLCODE` - - [ ] Precompile as transaction entry-point - - [ ] Call from Initcode - - [ ] Contract creating transaction - - [ ] Contract creating opcode - - [ ] Set code delegated address (no precompile logic executed) -- [ ] Inputs - - [ ] Verify combinations of valid inputs to the precompile - - [ ] Verify interesting edge values given the precompile functionality. - - [ ] If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). - - [ ] Verify all zeros input. - - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific mathematical or cryptographic validity checks. - - [ ] Inputs that are malformed/corrupted. -- [ ] Value Transfer - - [ ] If the precompile requires a minimum value with the calls to it, either constant or depending on a formula, verify: - - [ ] Calls with the required value amount minus one, expect failure. - - [ ] Calls with the exact required amount, expect success. - - [ ] Calls with extra value than the required amount, expect success. - - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. -- [ ] Out-of-bounds checks - - [ ] Verify if the precompile has out-of-bounds conditions in its inputs and verify: - - [ ] Max value for each input - - [ ] Max value + 1 for each input -- [ ] Input Lengths - - [ ] Zero-length calldata. - - [ ] Precompile has static-length input - - [ ] Correct static-length calldata - - [ ] Calldata too short, where the value represents a correct but truncated input to the precompile. - - [ ] Calldata too long, where the value represents a correct input to the precompile with padded zeros. - - [ ] Precompile has dynamic-length input - - [ ] Verify correct precompile execution for valid lengths, given different inputs. - - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the precompile. - - [ ] Calldata too long, given different inputs, where the value represents a correct input to the precompile with padded zeros. -- [ ] Gas usage - - [ ] Precompile has constant gas usage - - [ ] Verify exact gas consumption - - [ ] Verify exact gas consumption minus one results in out-of-gas error. - - [ ] Precompile has dynamic gas usage - - [ ] Verify exact gas consumption, given different valid inputs. - - [ ] Verify exact gas consumption minus one results in out-of-gas error, given different valid inputs. -- [ ] Excessive Gas Cases: Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as max amount). -- [ ] Fork transition - - [ ] Verify that calling the precompile before its activation fork results in a valid call even for inputs that are expected to be invalid for the precompile. - - [ ] Verify that calling the precompile before its activation fork with zero gas results in a valid call. - - [ ] Verify precompile address becomes warm on and after the fork activation block, but not prior. - -### Framework Changes - -- [ ] Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## Removed Precompile +### Test Vectors + +#### Call contexts + +Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. + +##### `CALL` + +Verify precompile operation when called using the `CALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/normal` | `CALL`. | | | + +##### `DELEGATECALL` + +Verify precompile operation when called using the `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/delegate` | `DELEGATECALL`. | | | + +##### `STATICCALL` + +Verify precompile operation when called using the `STATICCALL` opcode. + +If the precompile is stateful, meaning calling it affects its storage, this call must result in exceptional abort. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/static` | `STATICCALL`. | | | + +##### `CALLCODE` + +Verify precompile operation when called using the `CALLCODE` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/callcode` | `CALLCODE`. | | | + +##### Transaction Entry-point + +Verify precompile behavior when it's used as `tx.to`. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/tx_entry` | Precompile as transaction entry-point. | | | + +##### Initcode call + +Verify calling the opcode during initcode execution of a new contract. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | +| `new_precompile/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | + + +##### Precompile as Set-code Delegated Address + +Test setting the precompile as a set-code delegated address, and verify no precompile logic is executed. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/call_contexts/set_code` | Set code delegated address. | | | + +#### Inputs + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/inputs/valid` | Verify combinations of valid inputs to the precompile. | | | +| `new_precompile/test/inputs/valid/boundary` | Verify all boundary values given the precompile functionality. | | | +| `new_precompile/test/inputs/valid/crypto` | If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). | | | +| `new_precompile/test/inputs/all_zeros` | Verify all zeros input. | | | +| `new_precompile/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | +| `new_precompile/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | +| `new_precompile/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | +| `new_precompile/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | + +#### Value Transfer + +##### Minimum Fee Precompile + +If the precompile requires a minimum value (fee) to execute, either constant or depending on a formula. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | +| `new_precompile/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | +| `new_precompile/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | + +##### No-Fee Precompile + +If the precompile does not require any minimum value (fee) to execute. + +| `new_precompile/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | + +#### Out-of-bounds checks + +If the precompile has out-of-bounds conditions in its inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/out_of_bounds/max` | Max value for each input. | | | +| `new_precompile/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | + +#### Input Lengths + +Verify different combinations of input lengths to the precompile, ensuring the correct minimum fee (if any) is provided. + +##### Zero-length Input + +Regardless of the input requirements for the precompile. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/input_lengths/zero` | Zero-length calldata. | | | + +##### Static Required Input Length + +If the precompile has a static required input length. + +| `new_precompile/test/input_lengths/static/correct` | Correct static-length calldata. | | | +| `new_precompile/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | +| `new_precompile/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | + +##### Dynamic Required Input Length + +If the precompile has a variable required input-length based on a formula, test all listed scenarios given different input lengths. + +| `new_precompile/test/input_lengths/dynamic/valid` | Verify correct precompile execution for valid lengths. | | | +| `new_precompile/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | +| `new_precompile/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | + +#### Gas usage + +##### Constant Gas Cost + +If the precompile always charges the same gas cost regardless of input conditions. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | +| `new_precompile/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Variable Gas Cost + +If the precompile charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | +| `new_precompile/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Excessive Gas + +Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as reference max amount). + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/excessive_gas_usage` | Excessive gas usage. | | | + +#### Fork transition + +##### Pre-fork Block Call + +Verify that calling the precompile before its activation fork results in a valid call, even for inputs that are expected to be invalid for the precompile, or a zero-gas call. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/fork_transition/before/invalid_input` | Invalid input call. | | | +| `new_precompile/test/fork_transition/before/zero_gas` | Zero-gas call. | | | + + +##### Cold/Warm Precompile Address State + +Verify the cold/warm state of the precompile address depending on the fork activation. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_precompile/test/fork_transition/before/cold` | Precompile address is cold by default before the fork activation. | | | +| `new_precompile/test/fork_transition/after/warm` | Precompile address is warm by default after the fork activation. | | | + +### Framework Changes + +- Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## Removed Precompile The EIP removes one or more precompiles from the existing list of precompiles. -### Test Vectors - -- [ ] Fork boundary - - [ ] Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. - - [ ] Verify the account is warm up until the last block before the fork is active, and becomes cold afterwards. - -### Framework Changes - -- [ ] Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` - -## New System Contract - -### Test Vectors - -- [ ] Call contexts - - [ ] `CALL` - - [ ] `DELEGATECALL` - - [ ] `STATICCALL` - - [ ] If the system contract is stateful, meaning calling it affects its storage, verify that using `STATICCALL` to call it results in exceptional abort. - - [ ] `CALLCODE` - - [ ] System contract as transaction entry-point - - [ ] Call from Initcode - - [ ] Contract creating transaction - - [ ] Contract creating opcode -- [ ] Inputs - - [ ] Verify combinations of valid inputs to the system contract - - [ ] Verify interesting boundary values given the system contract functionality. - - [ ] Verify all zeros input. - - [ ] Verify 2^N-1 where N is a single or multiple valid bit-lengths. - - [ ] Verify combinations of invalid inputs to the precompile. - - [ ] Inputs that fail specific validity checks. - - [ ] Inputs that are malformed/corrupted. -- [ ] Value Transfer - - [ ] If the system contract requires a minimum value with the calls to it, either constant or depending on a formula, verify: - - [ ] Calls with the required value amount minus one, expect failure. - - [ ] Calls with the exact required amount, expect success. - - [ ] Calls with extra value than the required amount, expect success. - - [ ] If the system contract does not require a minimum value embedded in the calls to it, verify sending value does not cause an exception, unless otherwise specified by the EIP. -- [ ] Out-of-bounds checks - - [ ] Verify if the system contract has out-of-bounds conditions in its inputs and verify: - - [ ] Max value for each input - - [ ] Max value + 1 for each input -- [ ] Input Lengths - - [ ] Zero-length calldata. - - [ ] System contract has static-length input - - [ ] Correct static-length calldata - - [ ] Calldata too short, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, where the value represents a correct input to the contract with padded zeros. - - [ ] System contract has dynamic-length input - - [ ] Verify correct System contract execution for valid lengths, given different inputs. - - [ ] Calldata too short, given different inputs, where the value represents a correct but truncated input to the contract. - - [ ] Calldata too long, given different inputs, where the value represents a correct input to the contract with padded zeros. -- [ ] Excessive Gas Cases - - [ ] If possible, simulate a scenario where the execution of the contract at the end of the block execution by the system address would result in excessive gas usage (100 million gas or more). - - [ ] Verify spending all block gas in calls to system contract (100 million gas or more). -- [ ] System Contract Deployment - - [ ] Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). - - [ ] Verify deployment transaction results in the system contract being deployed to the expected address. -- [ ] Contract Variations - - [ ] Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior, - - [ ] Verify execution of a variation that causes an exception. - - [ ] Verify execution of a variation that consumes: - - [ ] 30,000,000 million gas exactly, execution should be successful. - - [ ] 30,000,001 million gas exactly, execution should fail. -- [ ] Contract Substitution: Substitute the contract to modify its behavior when called by the system address (at the end of the block execution): - - [ ] Modified return value lengths - - [ ] Modify emitted logs -- [ ] Fork transition: Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). - -### Framework Changes - -- [ ] Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -- [ ] Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## New Transaction Type - -### Test Vectors - -- [ ] Intrinsic Validity - - [ ] Gas Limit: For each new field that affects the intrinsic gas cost of the transaction: - - [ ] Verify the transaction (and the block it is included in) is valid by providing the exact intrinsic gas as `gas_limit` value to the transaction with all multiple combinations of values to the field. - - [ ] Verify the transaction (and the block it is included in) is invalid by providing the exact intrinsic gas minus one as `gas_limit` value to the transaction with all multiple combinations of values to the field. - - [ ] Max fee per gas: Verify the transaction (and the block it is included in) is invalid if: - - [ ] Its max-priority-fee-per-gas value is lower than the max-fee-per-gas. - - [ ] Its max-fee-per-gas value is lower than the blocks base-fee-per-gas. - - [ ] Chain ID: Verify the transaction (and the block it is included in) is invalid if its chain-id value does not match the network configuration. - - [ ] Nonce: Verify the transaction (and the block it is included in) is invalid if its nonce value does not match the account's current nonce. - - [ ] To: Verify the transaction (and the block it is included in) is invalid if the transaction type does not allow contract creation and the to-address field is empty. - - [ ] Value: Verify the transaction (and the block it is included in) is invalid if the transaction contains a value of 1 and the account does not contain enough funds to cover the intrinsic transaction cost plus 1. - - [ ] Data: Verify the transaction (and the block it is included in) is invalid if the transaction contains enough data so the data floor cost is higher than the intrinsic gas cost and the gas_limit is equal to the intrinsic gas cost. - - [ ] Sender balance: Verify the transaction (and the block it is included in) is invalid when the sender account does not have enough balance to cover the gas limit multiplied by the max fee per gas. -- [ ] Signature: - - [ ] Verify the transaction is correctly rejected if it contains an invalid signature: - - [ ] V, R, S represent a value that is inside of the field but outside of the curve. - - [ ] V (yParity) is any of the following invalid values: - - [ ] `2` - - [ ] `27` (Type-0 transaction valid value) - - [ ] `28` (Type-0 transaction valid value) - - [ ] `35` (Type-0 replay-protected transaction valid value for chain id 1) - - [ ] `36` (Type-0 replay-protected transaction valid value for chain id 1) - - [ ] `2**8-1` - - [ ] R is any of the following invalid values: - - [ ] `0` - - [ ] `SECP256K1N-1` - - [ ] `SECP256K1N` - - [ ] `SECP256K1N+1` - - [ ] `2**256-1` - - [ ] `2**256` - - [ ] S is any of the following invalid values: - - [ ] `0` - - [ ] `SECP256K1N//2-1` - - [ ] `SECP256K1N//2` - - [ ] `SECP256K1N//2+1` - - [ ] `SECP256K1N-1` - - [ ] `SECP256K1N` - - [ ] `SECP256K1N+1` - - [ ] `2**256-1` - - [ ] `2**256` - - [ ] `SECP256K1N - S` of a valid signature -- [ ] Transaction-Scoped Attributes/Variables - - [ ] Verify attributes that can be read in the EVM from transaction fields. - - [ ] Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. - - [ ] Verify values or variables that are persistent through the execution of the transaction (transient storage, warm/cold accounts): - - [ ] Persist throughout the entire transaction. - - [ ] Reset on subsequent transactions in the same block. -- [ ] Encoding (RLP, SSZ) - - [ ] Verify correct transaction rejection due to incorrect field sizes: - - [ ] Add leading zero byte - - [ ] Remove single byte from fixed-byte-length fields - - [ ] If the transaction contains a new field that is a list, verify: - - [ ] Zero-element list - - [ ] Max count list - - [ ] Max count plus one list - - [ ] Verify correct transaction rejection if the fields particular to the new transaction types are missing - - [ ] Verify correct transaction rejection if the transaction type contains extra fields - - [ ] Verify transaction is correctly rejected if the serialized bytes object is truncated - - [ ] Verify transaction is correctly rejected if the serialized bytes object has extra bytes - - [ ] If the transaction contains fields with new serializable types, perform all previous tests on the new type/field, plus: - - [ ] Verify transaction rejection if the serializable field is incorrectly encoded as bytes instead of using the correct encoding. -- [ ] Out-of-bounds checks: Verify if the transaction has out-of-bounds conditions in its fields and verify: - - [ ] Max value for each field - - [ ] Max value + 1 for each field -- [ ] Contract creation: Verify that the transaction can create new contracts if the transaction type supports it. -- [ ] Sender account modifications - - [ ] Verify that the sender account of the new transaction type transaction has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). - - [ ] Verify that the sender account of the new transaction type transaction has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). -- [ ] Block Level Interactions - - [ ] Verify the new transaction type and the following accept/reject behavior depending on interactions with the block gas limit: - - [ ] Rejected if `tx.gas_limit == block.gas_limit + 1` - - [ ] Accepted if `tx.gas_limit == block.gas_limit` - - [ ] New transaction type is the last transaction of a block with two transactions: - - [ ] Accepted if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit` - - [ ] Rejected if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)` - - [ ] Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. - - [ ] Verify a block with all transactions types including the new type is executed correctly. -- [ ] Fork transition: Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. -- [ ] RPC Tests - - [ ] *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type - - [ ] Verify `eth_sendRawTransaction` using `execute` +### Test Vectors + +#### Fork transition + +##### Precompile Operation + +Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `removed_precompile/test/fork_transition/operational` | Precompile operation on fork activation. | | | + +##### Cold/Warm Precompile Address State + +Verify the cold/warm state of the precompile address depending on the fork activation. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `removed_precompile/test/fork_transition/before/warm` | Precompile address is warm by default before the fork activation. | | | +| `removed_precompile/test/fork_transition/after/cold` | Precompile address is cold by default after the fork activation. | | | + +### Framework Changes + +- Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` + +## New System Contract + +### Test Vectors + +#### Call contexts + +Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. + +##### `CALL` + +Verify system contract operation when called using the `CALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/normal` | `CALL`. | | | + +##### `DELEGATECALL` + +Verify system contract operation when called using the `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/delegate` | `DELEGATECALL`. | | | + +##### `STATICCALL` + +Verify system contract operation when called using the `STATICCALL` opcode. + +If the system contract is stateful, meaning calling it affects its storage, this call must result in exceptional abort. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/static` | `STATICCALL`. | | | + +##### `CALLCODE` + +Verify system contract operation when called using the `CALLCODE` opcode. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/callcode` | `CALLCODE`. | | | + +##### Transaction Entry-point + +Verify system contract behavior when it's used as `tx.to`. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/tx_entry` | System Contract as transaction entry-point. | | | + +##### Initcode call + +Verify calling the opcode during initcode execution of a new contract. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | +| `new_system_contract/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | + + +##### System Contract as Set-code Delegated Address + +Test setting the system contract as a set-code delegated address, and verify no system contract side-effects are triggered, even if the actual system contract logic is executed. + +If the system contract requires specific storage pre-conditions to be set for proper execution (e.g. if the contract contains a safety mechanism that prevents it from executing prior to the fork activation), ensure the delegated account has the proper values in the storage to guarantee the correct logic is executed. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/call_contexts/set_code` | Set code delegated address. | | | + +#### Inputs + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/inputs/valid` | Verify combinations of valid inputs to the system contract. | | | +| `new_system_contract/test/inputs/boundary` | Verify all boundary values given the system contract functionality. | | | +| `new_system_contract/test/inputs/all_zeros` | Verify all zeros input. | | | +| `new_system_contract/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | +| `new_system_contract/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | +| `new_system_contract/test/inputs/invalid/checks` | Inputs that fail validity checks. | | | +| `new_system_contract/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | +| `new_system_contract/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | + +#### Value Transfer + +##### Minimum Fee System Contract + +If the system contract requires a minimum value (fee) to execute, either constant or depending on a formula. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | +| `new_system_contract/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | +| `new_system_contract/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | + +##### No-Fee System Contract + +If the system contract does not require any minimum value (fee) to execute. + +| `new_system_contract/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | + +#### Out-of-bounds checks + +If the system contract has out-of-bounds conditions in its inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/out_of_bounds/max` | Max value for each input. | | | +| `new_system_contract/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | + +#### Input Lengths + +Verify different combinations of input lengths to the system contract, ensuring the correct minimum fee (if any) is provided. + +##### Zero-length Input + +Regardless of the input requirements for the system contract. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/input_lengths/zero` | Zero-length calldata. | | | + + +##### Static Required Input Length + +If the system contract has a static required input length. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/input_lengths/static/correct` | Correct static-length calldata. | | | +| `new_system_contract/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | +| `new_system_contract/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | + +##### Dynamic Required Input Length + +If the system contract has a variable required input-length based on a formula, test all listed scenarios given different input lengths. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/input_lengths/dynamic/valid` | Verify correct system contract execution for valid lengths. | | | +| `new_system_contract/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | +| `new_system_contract/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | + + + +#### Gas usage + +##### Constant Gas Cost + +If the system contract always charges the same gas cost regardless of input conditions. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | +| `new_system_contract/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Variable Gas Cost + +If the system contract charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | +| `new_system_contract/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +#### Excessive Gas Cases + +##### Excessive Gas Usage During Block Execution + +Verify spending all block gas in calls to system contract (Use `Environment().gas_limit` as reference max amount). + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/excessive_gas/block_gas` | Exhaust block gas limit. | | | + + +##### Excessive Gas Usage During System Call + +If possible, produce a scenario where, given all transactions executed within the block result in the execution of the contract by the system address would result in excessive gas usage. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/excessive_gas/system_call` | Excessive gas on system call. | | | + + +#### System Contract Deployment + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/deployment/missing` | Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). | | | +| `new_system_contract/test/deployment/address` | Verify deployment transaction results in the system contract being deployed to the expected address. | | | + +#### Contract Variations + +Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/contract_variations/networks` | Different network contract variations. | | | + +#### Contract Substitution + +Substitute the default system contract with a mock contract to modify its behavior when called by the system address (at the end of the block execution). + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/contract_substitution/return_lengths` | Modified return value lengths. | | | +| `new_system_contract/test/contract_substitution/logs` | Modified to emit logs or modified logs. | | | +| `new_system_contract/test/contract_substitution/exception` | Modified to cause an exception (e.g. invalid opcode). | | | +| `new_system_contract/test/contract_substitution/gas_limit_success` | Modified to consume 30,000,000 million gas exactly, execution should be successful. | | | +| `new_system_contract/test/contract_substitution/gas_limit_failure` | Modified to consume 30,000,001 million gas exactly, execution should fail. | | | + +#### Fork transition + +Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_system_contract/test/fork_transition/call_before_fork` | Call system contract before fork. | | | + +### Framework Changes + +- Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New Transaction Type + +### Test Vectors + +#### Intrinsic Validity + +Verify the transaction (and the block it is included in) is valid or invalid given the intrinsic validity of the new transaction type, depending on each test scenario. + +For each new field that affects the intrinsic gas cost of the transaction verify all listed scenarios. + +##### Gas Limit + +Note: Data floor gas cost affects the intrinsic validity of all transaction types, so it must be taken into account when designing all test scenarios. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/gas_limit/exact` | Provide the exact intrinsic gas as `gas_limit` value to the transaction. | | | +| `new_transaction_type/test/intrinsic_validity/gas_limit/insufficient` | Provide the exact intrinsic gas minus one as `gas_limit` value to the transaction. | | | + +##### Gas Fee + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_lower_than_max_fee` | Invalid if `tx.max_priority_fee_per_gas < tx.max_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_equal_to_max_fee` | Valid if `tx.max_priority_fee_per_gas == tx.max_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/base_lower` | Invalid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/base_equal` | Valid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | + +##### Chain ID + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/chain_id` | Invalid if `tx.chain_id != network.chain_id`. | | | + +##### Nonce + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/nonce_minus_one` | Invalid if `tx.nonce == sender.nonce - 1`. | | | +| `new_transaction_type/test/intrinsic_validity/nonce_plus_one` | Invalid if `tx.nonce == sender.nonce + 1`. | | | +| `new_transaction_type/test/intrinsic_validity/nonce_exact` | Valid if `tx.nonce == sender.nonce`. | | | + +##### To + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/to` | Valid/Invalid if `tx.to == None`, depending on the EIP.| | | + +##### Value + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/value_non_zero_insufficient_balance` | Invalid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | +| `new_transaction_type/test/intrinsic_validity/value_non_zero_sufficient_balance` | Valid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) + 1`. | | | +| `new_transaction_type/test/intrinsic_validity/value_zero_insufficient_balance` | Invalid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) - 1`. | | | +| `new_transaction_type/test/intrinsic_validity/value_zero_sufficient_balance` | Valid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | + +##### Data + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Invalid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == tx.intrinsic_gas_cost`. | | | +| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Valid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == data_floor_cost(len(tx.data))`. | | | + + +#### Signature. + +Verify the transaction is correctly rejected if it contains an invalid signature. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/signature/invalid/field_outside_curve` | V, R, S represent a value that is inside of the field but outside of the curve. | | | + +##### V + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/signature/invalid/v/2` | `2`. | | | +| `new_transaction_type/test/signature/invalid/v/27` | `27` (Type-0 transaction valid value). | | | +| `new_transaction_type/test/signature/invalid/v/28` | `28` (Type-0 transaction valid value). | | | +| `new_transaction_type/test/signature/invalid/v/35` | `35` (Type-0 replay-protected transaction valid value for chain id 1). | | | +| `new_transaction_type/test/signature/invalid/v/36` | `36` (Type-0 replay-protected transaction valid value for chain id 1). | | | +| `new_transaction_type/test/signature/invalid/v/max` | `2**8-1`. | | | + +##### R + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/signature/invalid/r/0` | `0`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n_minus_one` | `SECP256K1N-1`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n` | `SECP256K1N`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n_plus_one` | `SECP256K1N+1`. | | | +| `new_transaction_type/test/signature/invalid/r/max_minus_one` | `2**256-1`. | | | +| `new_transaction_type/test/signature/invalid/r/max` | `2**256`. | | | + +##### S + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/signature/invalid/s/0` | `0`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_minus_one` | `SECP256K1N//2-1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half` | `SECP256K1N//2`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_plus_one` | `SECP256K1N//2+1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_minus_one` | `SECP256K1N-1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n` | `SECP256K1N`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_plus_one` | `SECP256K1N+1`. | | | +| `new_transaction_type/test/signature/invalid/s/max_minus_one` | `2**256-1`. | | | +| `new_transaction_type/test/signature/invalid/s/max` | `2**256`. | | | +| `new_transaction_type/test/signature/invalid/s/complement` | `SECP256K1N - S` of a valid signature. | | | + +#### Transaction Attributes Readable From EVM + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/tx_scoped_attributes/read` | Verify attributes that can be read in the EVM from transaction fields. | | | +| `new_transaction_type/test/tx_scoped_attributes/older_tx_types` | Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. | | | + +#### Transaction-Scoped Persistent Values + +Verify values or variables that are persistent through the execution of the transaction (e.g. transient storage, warm/cold accounts). + +| `new_transaction_type/test/tx_scoped_attributes/persistent/throughout` | Persist throughout the entire transaction. | | | +| `new_transaction_type/test/tx_scoped_attributes/persistent/reset` | Reset on subsequent transactions in the same block. | | | + +#### Encoding (RLP, SSZ) + +Verify correct transaction rejection in all listed scenarios. + +##### Field Sizes + +Verify all listed scenarios for each transaction field. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/encoding/field_sizes/leading_zero` | Add leading zero byte. | | | +| `new_transaction_type/test/encoding/field_sizes/remove_byte` | Remove single byte from fixed-byte-length fields. | | | + +##### Fields of List Type + +Verify for each transaction field that is of type list. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/encoding/list_field/zero` | Zero-element list (Failure depending on EIP). | | | +| `new_transaction_type/test/encoding/list_field/max` | Max count list. | | | +| `new_transaction_type/test/encoding/list_field/max_plus_one` | Max count plus one list. | | | + +##### Extra/Missing Fields + +| `new_transaction_type/test/encoding/missing_fields` | Any fields particular to the new transaction types are missing. | | | +| `new_transaction_type/test/encoding/extra_fields` | Transaction contains extra fields. | | | + +##### Serialization Corruption + +| `new_transaction_type/test/encoding/truncated` | Serialized bytes object is truncated by one byte. | | | +| `new_transaction_type/test/encoding/extra_bytes` | Serialized bytes object has one extra byte. | | | + +##### Serializable Fields + +Verify for each serializable field, all previous tests plus following listed scenario. + +| `new_transaction_type/test/encoding/new_types/incorrect_encoding` | Serializable field is encoded as bytes instead of using the correct encoding (e.g. list in the case of RLP). | | | + +#### Out-of-bounds checks + +Verify if the transaction has out-of-bounds conditions in its fields and verify. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/out_of_bounds/max` | Max value for each field. | | | +| `new_transaction_type/test/out_of_bounds/max_plus_one` | Max value + 1 for each field. | | | + +#### Contract creation + +Verify that the transaction can create new contracts if the transaction type supports it. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/contract_creation` | Create contracts using new transaction type. | | | + +#### Sender account modifications + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/sender_account/nonce` | Sender account has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). | | | +| `new_transaction_type/test/sender_account/balance` | Sender account has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). | | | + +#### Block Level Interactions + +##### Single Transaction In Block + +Verify a block where the new transaction type is the sole transaction contained in the block. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/block_interactions/single_tx/invalid` | Invalid if `tx.gas_limit == block.gas_limit + 1`. | | | +| `new_transaction_type/test/block_interactions/single_tx/valid` | Valid if `tx.gas_limit == block.gas_limit`. | | | + + +##### Two Transactions In Block + +Verify a block where the new transaction type is the last transaction contained in a block with two transactions. + +| `new_transaction_type/test/block_interactions/last_tx/valid` | Valid if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit`. | | | +| `new_transaction_type/test/block_interactions/last_tx/invalid` | Invalid if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)`. | | | + +##### EIP-7825 + +Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. + +| `new_transaction_type/test/block_interactions/eip7825/invalid` | Exceeds EIP-7825 gas limit by one. | | | +| `new_transaction_type/test/block_interactions/eip7825/valid` | Gas limit is exactly the EIP-7825 gas limit. | | | + +##### Mixed transactions + +Verify a block with all transactions types including the new type is executed correctly. + +| `new_transaction_type/test/block_interactions/mixed_txs` | Mixed transactions. | | | + +#### Fork transition + +Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_transaction_type/test/fork_transition/before` | New transaction type included before fork activation block. | | | + +#### RPC Tests + +- *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type. +- `new_transaction_type/test/rpc/send_raw` | Verify `eth_sendRawTransaction` using `execute`. *Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. -### Framework Changes +### Framework Changes + +- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. +- Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). -- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. -- [ ] Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). +## New Block Header Field -## New Block Header Field +### Test Vectors -### Test Vectors +#### Genesis value -- [ ] Genesis value: Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Fork transition - - [ ] Verify initial value of the field at the first block of the activation fork. - - [ ] Verify that a block containing the new header field before the activation of the fork is invalid. - - [ ] Verify that a block lacking the new header field at the activation of the fork is invalid. +Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. -### Framework Changes +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_block_header_field/test/genesis` | New block header field at genesis. | | | -- [ ] Add the new header field to the relevant objects: - - [ ] `ethereum_test_fixtures.FixtureHeader` - - [ ] `ethereum_test_fixtures.FixtureExecutionPayload` - - [ ] `ethereum_test_specs.Header` -- [ ] Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. +#### Value behavior -## New Block Body Field +Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. -### Test Vectors +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_block_header_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | +| `new_block_header_field/test/value_behavior/reject` | Block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Fork transition - - [ ] Verify that a block containing the new block body field before the activation of the fork is invalid. - - [ ] Verify that a block lacking the new block field at the activation of the fork is invalid. +#### Fork transition -### Framework Changes +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_block_header_field/test/fork_transition/initial` | Verify initial value of the field at the first block of the activation fork. | | | +| `new_block_header_field/test/fork_transition/before` | Verify that a block containing the new header field before the activation of the fork is invalid. | | | +| `new_block_header_field/test/fork_transition/after` | Verify that a block lacking the new header field at the activation of the fork is invalid. | | | -- [ ] Value behavior - - [ ] Verify, given multiple initial values, that a block is accepted if the value is correctly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. - - [ ] Verify, given multiple initial values, that a block is rejected if the value is incorrectly modified for the current block, depending on the circumstances that affect the value as defined in the EIP. -- [ ] Add the new body field to the relevant objects: - - [ ] `ethereum_test_fixtures.FixtureBlockBase` - - [ ] `ethereum_test_fixtures.FixtureEngineNewPayload` - - [ ] `ethereum_test_specs.Block` -- [ ] Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. +### Framework Changes -## Gas Cost Changes +- Add the new header field to the relevant objects: + - `ethereum_test_fixtures.FixtureHeader`. + - `ethereum_test_fixtures.FixtureExecutionPayload`. + - `ethereum_test_specs.Header`. +- Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. -### Test Vectors +## New Block Body Field -- [ ] Gas Usage: Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. -- [ ] Out-of-gas: Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. -- [ ] Fork transition: Verify gas costs are: - - [ ] Unaffected before the fork activation block. - - [ ] Updated on and after fork activation block. +### Test Vectors -### Framework Changes +#### Value behavior -- [ ] Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. -- [ ] Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. -- [ ] Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. -- [ ] Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. +Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. -## Gas Refunds Changes +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_block_body_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | +| `new_block_body_field/test/value_behavior/reject` | Block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | -### Test Vectors +#### Fork transition -- [ ] Refund calculation: Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios: - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT + 1` - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT` - - [ ] `refund == gas_used // MAX_REFUND_QUOTIENT - 1` -- [ ] Exceptional Abort: - - [ ] If the operation causing the refund can be reverted, verify the refund is not applied if the following cases: - - [ ] `REVERT` - - [ ] Out-of-gas - - [ ] Invalid opcode - - [ ] `REVERT` of an upper call frame - - [ ] If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases: - - [ ] `REVERT` at the top call frame - - [ ] Out-of-gas at the top call frame - - [ ] Invalid opcode at the top call frame -- [ ] Cross-Functional Test: Verify the following tests are updated to support the new type of refunds: - - [ ] `tests/prague/eip7623_increase_calldata_cost/test_refunds.py` +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_block_body_field/test/fork_transition/before` | Verify that a block containing the new block body field before the activation of the fork is invalid. | | | +| `new_block_body_field/test/fork_transition/after` | Verify that a block lacking the new block field at the activation of the fork is invalid. | | | -### Framework Changes +### Framework Changes + +- Add the new body field to the relevant objects. + - `ethereum_test_fixtures.FixtureBlockBase`. + - `ethereum_test_fixtures.FixtureEngineNewPayload`. + - `ethereum_test_specs.Block`. +- Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. + +## Gas Cost Changes + +### Test Vectors + +#### Gas Usage +Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. + +#### Out-of-gas +Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. + +#### Fork transition + +Verify gas costs are. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_cost_changes/test/fork_transition/before` | Unaffected before the fork activation block. | | | +| `gas_cost_changes/test/fork_transition/after` | Updated on and after fork activation block. | | | + +### Framework Changes + +- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. +- Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. +- Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. +- Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. + +## Gas Refunds Changes + +### Test Vectors + +#### Refund calculation + +Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_refunds_changes/test/refund_calculation/over` | `refund == gas_used // MAX_REFUND_QUOTIENT + 1`. | | | +| `gas_refunds_changes/test/refund_calculation/exact` | `refund == gas_used // MAX_REFUND_QUOTIENT`. | | | +| `gas_refunds_changes/test/refund_calculation/under` | `refund == gas_used // MAX_REFUND_QUOTIENT - 1`. | | | + +#### Exceptional Abort. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_refunds_changes/test/exceptional_abort/revertable` | If the operation causing the refund can be reverted, verify the refund is not applied if the following cases:. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/revert` | `REVERT`. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/out_of_gas` | Out-of-gas. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/invalid_opcode` | Invalid opcode. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/upper_revert` | `REVERT` of an upper call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable` | If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases:. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/revert` | `REVERT` at the top call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/out_of_gas` | Out-of-gas at the top call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/invalid_opcode` | Invalid opcode at the top call frame. | | | + +#### Cross-Functional Test + +Verify the following tests are updated to support the new type of refunds. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_refunds_changes/test/cross_functional/calldata_cost` | `tests/prague/eip7623_increase_calldata_cost/test_refunds.py`. | | | + +### Framework Changes N/A -## Blob Count Changes +## Blob Count Changes + +### Test Vectors -### Test Vectors +#### Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. -- [ ] Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. +### Framework Changes -### Framework Changes +- Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. -- [ ] Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. +## New Execution Layer Request -## New Execution Layer Request +### Test Vectors -### Test Vectors +#### Cross-Request-Type Interaction -- [ ] Cross-Request-Type Interaction - - [ ] Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations +| ID | Description | Status | Tests | +|-|-|-|-| +| `new_execution_layer_request/test/cross_request_type/update` | Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations. | | | -### Framework Changes +### Framework Changes -- [ ] Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. +- Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. From d0466b7a283c18d9ab9e7cf7c69fab49f7190d5d Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Tue, 13 May 2025 22:25:21 +0000 Subject: [PATCH 29/33] fixes --- .../eip_testing_checklist_template.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md index 65be6692971..f46e5c89598 100644 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ b/docs/checklist_templates/eip_testing_checklist_template.md @@ -1038,19 +1038,29 @@ Verify, given multiple initial values, that a block is accepted or rejected depe ### Test Vectors #### Gas Usage + Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_cost_changes/test/gas_updates_measurement` | Measure updated gas costs. | | | + #### Out-of-gas + Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. +| ID | Description | Status | Tests | +|-|-|-|-| +| `gas_cost_changes/test/out_of_gas` | Out-Of-Gas with new gas prices. | | | + #### Fork transition -Verify gas costs are. +Verify gas costs are updated at the fork transition boundary. | ID | Description | Status | Tests | |-|-|-|-| -| `gas_cost_changes/test/fork_transition/before` | Unaffected before the fork activation block. | | | -| `gas_cost_changes/test/fork_transition/after` | Updated on and after fork activation block. | | | +| `gas_cost_changes/test/fork_transition/before` | Costs unaffected before the fork activation block. | | | +| `gas_cost_changes/test/fork_transition/after` | Costs are updated on and after fork activation block. | | | ### Framework Changes @@ -1103,7 +1113,13 @@ N/A ### Test Vectors -#### Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. +#### Existing Test Updates + +Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. + +| ID | Description | Status | Tests | +|-|-|-|-| +| `blob_count_changes/test/eip4844_blobs_changes` | Updates to `tests/cancun/eip4844_blobs` were applied correctly. | | | ### Framework Changes From 50aaa31714b2b1c91f7e09c19206b470b8d67f4e Mon Sep 17 00:00:00 2001 From: danceratopz Date: Wed, 14 May 2025 18:33:25 +0200 Subject: [PATCH 30/33] Eip checklist suggestions (#1601) * chore(docs): auto-format doc, in particular tables, with prettier * chore(docs): remove periods from sub-titles * chore(docs): fix indentation in unordered lists * chore(docs): add missing md table headers * refactor(docs): indent navigation ready for more deeply nested entries * docs: add testing checklists to docs/writing; include in html * docs: update whitelist * docs: disable MD001 for eip testing checklist --- .../eip_testing_checklist_template.md | 1140 ---------------- docs/navigation.md | 98 +- .../eip_testing_checklist_template.md | 1157 +++++++++++++++++ .../checklist_templates/index.md | 0 whitelist.txt | 3 +- 5 files changed, 1209 insertions(+), 1189 deletions(-) delete mode 100644 docs/checklist_templates/eip_testing_checklist_template.md create mode 100644 docs/writing_tests/checklist_templates/eip_testing_checklist_template.md rename docs/{ => writing_tests}/checklist_templates/index.md (100%) diff --git a/docs/checklist_templates/eip_testing_checklist_template.md b/docs/checklist_templates/eip_testing_checklist_template.md deleted file mode 100644 index f46e5c89598..00000000000 --- a/docs/checklist_templates/eip_testing_checklist_template.md +++ /dev/null @@ -1,1140 +0,0 @@ -# EIP Execution Layer Testing Checklist Template - -Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features. - -## General - -#### Code coverage - -| ID | Description | Status | Tests | -|-|-|-|-| -| `general/code_coverage/eels` | Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. | | | -| `general/code_coverage/test_coverage` | Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. | | | -| `general/code_coverage/second_client` | Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. | | | - -#### Fuzzing - -## New Opcode - -The EIP introduces one or more new opcodes to the EVM. - -### Test Vectors - -#### Memory expansion - -If the opcode execution can expand the memory size, either by writing to memory or reading from an offset that exceeds current memory, or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/mem_exp/zero_bytes_zero_offset` | Zero bytes expansion with zero-offset. | | | -| `new_opcode/test/mem_exp/zero_bytes_max_offset` | Zero bytes expansion with 2**256-1 offset. | | | -| `new_opcode/test/mem_exp/single_byte` | Single byte expansion. | | | -| `new_opcode/test/mem_exp/31_bytes` | 31 bytes expansion. | | | -| `new_opcode/test/mem_exp/32_bytes` | 32 bytes expansion. | | | -| `new_opcode/test/mem_exp/33_bytes` | 33 bytes expansion. | | | -| `new_opcode/test/mem_exp/64_bytes` | 64 bytes expansion. | | | -| `new_opcode/test/mem_exp/2_32_minus_one_bytes` | `2**32-1` bytes expansion. | | | -| `new_opcode/test/mem_exp/2_32_bytes` | 2**32 bytes expansion. | | | -| `new_opcode/test/mem_exp/2_64_minus_one_bytes` | 2**64-1 bytes expansion. | | | -| `new_opcode/test/mem_exp/2_64_bytes` | 2**64 bytes expansion. | | | -| `new_opcode/test/mem_exp/2_256_minus_one_bytes` | 2**256-1 bytes expansion (Should always result in Out-of-gas). | | | - -#### Stack - -##### Stack Overflow - -If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/stack_overflow` | Stack Overflow. | | | - -##### Stack Underflow - -If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/stack_underflow` | Stack Underflow. | | | - -##### Stack Complex Operations - -If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/stack_complex_operations/stack_heights/zero` | Operation on an empty stack (Potential stack-underflow). | | | -| `new_opcode/test/stack_complex_operations/stack_heights/odd` | Operation on a stack with an odd height. | | | -| `new_opcode/test/stack_complex_operations/stack_heights/even` | Operation on a stack with an even height. | | | - -##### Stack Manipulation With Data Portion Variables - -If the opcode contains variables in its data portion, for each variable `n` of the opcode that accesses the nth stack item, test `n` being. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/stack_complex_operations/variable_n/top` | Top stack item. | | | -| `new_opcode/test/stack_complex_operations/variable_n/bottom` | Bottom stack item. | | | -| `new_opcode/test/stack_complex_operations/variable_n/middle` | Middle stack item. | | | - -#### Execution context - -Evaluate opcode's behavior in different execution contexts. - -##### `CALL` - -Verify opcode operation in a subcall frame originated from a `CALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/call` | `CALL`. | | | - -##### `STATICCALL` - -Verify opcode operation in a subcall frame originated from a `STATICCALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/staticcall/ban_check` | Verify exceptional abort if the opcode attempts to modify the code, storage or balance of an account. | | | -| `new_opcode/test/execution_context/staticcall/ban_no_modification` | If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). | | | -| `new_opcode/test/execution_context/staticcall/sub_calls` | Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. | | | - -##### `DELEGATECALL` - -Verify opcode operation in a subcall frame originated from a `DELEGATECALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/delegatecall` | `DELEGATECALL`. | | | -| `new_opcode/test/execution_context/delegatecall/storage` | If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. | | | -| `new_opcode/test/execution_context/delegatecall/balance` | If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. | | | -| `new_opcode/test/execution_context/delegatecall/code` | If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. | | | - -##### `CALLCODE` - -Verify opcode operation in a subcall frame originated from a `CALLCODE` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/callcode` | `CALLCODE`. | | | - -##### Initcode - -Verify opcode behaves as expected when running during the initcode phase of contract creation. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/initcode/behavior` | Initcode operation. | | | -| `new_opcode/test/execution_context/initcode/behavior/tx` | Initcode of a contract creating transaction. | | | -| `new_opcode/test/execution_context/initcode/behavior/opcode` | Initcode of a contract creating opcode (including itself if opcode creates a contract). | | | -| `new_opcode/test/execution_context/initcode/reentry` | Initcode re-entry: using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). | | | - -##### Set-code Delegated Account - -Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/set_code` | Set-code delegated account. | | | - -##### Transaction Context - -If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/tx_context` | Transaction-context dependent opcode. | | | - -##### Block Context - -If opcode changes behavior depending on particular block properties, test using multiple values for each property. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/execution_context/block_context` | Block-context dependent opcode. | | | - -#### Return data - -Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/return_data/buffer/current` | Return buffer at current call context. | | | -| `new_opcode/test/return_data/buffer/parent` | Return buffer at parent call context. | | | - -#### Gas Usage - -##### Normal Operation - -Verify gas usage affectation of each stack argument or memory input consumed by the opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/gas_usage/normal` | Normal operation gas usage. | | | - -##### Memory Expansion - -Verify that the memory expansion correctly follows the gas calculation. - -| `new_opcode/test/gas_usage/memory_expansion` | Memory expansion. | | | - -##### Out-Of-Gas - -Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/gas_usage/out_of_gas_execution` | Out-of-gas due to opcode inputs. | | | -| `new_opcode/test/gas_usage/out_of_gas_memory` | Out-of-gas due to memory expansion. | | | - -##### Order-of-operations - -If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the listed cases for each. - -| `new_opcode/test/gas_usage/order_of_operations/exact` | Success using the exact amount of gas required for the stipend. | | | -| `new_opcode/test/gas_usage/order_of_operations/oog` | OOG with a 1-gas-difference from the gas required for the stipend. | | | - -#### Terminating opcode - -If an opcode is terminating, meaning it results in the current call context to end execution. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/terminating/scenarios/top_level` | Top-level call termination. | | | -| `new_opcode/test/terminating/scenarios/sub_level` | Sub-level call termination. | | | -| `new_opcode/test/terminating/scenarios/initcode` | Initcode termination. | | | - -#### Aborting/Reverting opcode - -If the terminating opcode is meant to rollback the executing call frame, verify the listed events are properly rolled back. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/terminating/rollback/balance` | Balance changes. | | | -| `new_opcode/test/terminating/rollback/storage` | Storage changes. | | | -| `new_opcode/test/terminating/rollback/contracts` | Contract creations. | | | -| `new_opcode/test/terminating/rollback/nonce` | Nonce increments. | | | -| `new_opcode/test/terminating/rollback/logs` | Log events. | | | - -#### Out-of-bounds checks - -If the opcode has out-of-bounds conditions in its parameters/inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/out_of_bounds/verify/max` | Max value for each parameter. | | | -| `new_opcode/test/out_of_bounds/verify/max_plus_one` | Max value + 1 for each parameter. | | | - -#### Exceptional Abort - -If the opcode has conditions, either inputs or execution context states, that should cause exceptional abort and are different than out-of-gas or stack overflow or underflow. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/exceptional_abort` | Exceptional abort conditions. | | | - -#### Data portion - -If an opcode has a data portion, meaning the `N` bytes following the opcode in the contract bytecode are skipped from execution. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/data_portion/all_zeros` | All zeros data portion. | | | -| `new_opcode/test/data_portion/max_value` | Max value data portion (`2**N-1` where `N` is the bit size of the data portion). | | | -| `new_opcode/test/data_portion/jump` | Jump into the data portion. | | | - - -#### Contract creation - -If the opcode execution results in the creation of a new contract in the state. - -##### Correct Creation - -Verify contract is created at the expected address given multiple inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/contract_creation/address` | Address calculation. | | | - -##### Creation Failure - -The contract creation fails given the listed conditions. - -| `new_opcode/test/contract_creation/failure/oog` | Out-of-gas when available gas is less than minimum contract creation stipend. | | | -| `new_opcode/test/contract_creation/failure/insufficient_value` | Opcode has a value parameter and the caller does not have enough funds. | | | -| `new_opcode/test/contract_creation/failure/collision` | Creation would result in an address collision with an existing contract or EOA-delegated address. | | | - -##### Recursive Contract Creation - -Opcode is used to attempt to recreate a contract that is currently mid-creation by a previous call of the same opcode. - -| `new_opcode/test/contract_creation/recursive` | Recursive contract creation using the opcode. | | | - -#### Fork transition - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_opcode/test/fork_transition/invalid` | Exceptional abort if executed before its activation fork/after its deactivation fork. | | | -| `new_opcode/test/fork_transition/at` | Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. | | | - -### Framework Changes - -| ID | Description | Status | Tests | -|-|-|-|-| -- Add opcode to `src/ethereum_test_vm/opcode.py` -- Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## New Precompile - -The EIP introduces one or more new precompiles. - -### Test Vectors - -#### Call contexts - -Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. - -##### `CALL` - -Verify precompile operation when called using the `CALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/normal` | `CALL`. | | | - -##### `DELEGATECALL` - -Verify precompile operation when called using the `DELEGATECALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/delegate` | `DELEGATECALL`. | | | - -##### `STATICCALL` - -Verify precompile operation when called using the `STATICCALL` opcode. - -If the precompile is stateful, meaning calling it affects its storage, this call must result in exceptional abort. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/static` | `STATICCALL`. | | | - -##### `CALLCODE` - -Verify precompile operation when called using the `CALLCODE` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/callcode` | `CALLCODE`. | | | - -##### Transaction Entry-point - -Verify precompile behavior when it's used as `tx.to`. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/tx_entry` | Precompile as transaction entry-point. | | | - -##### Initcode call - -Verify calling the opcode during initcode execution of a new contract. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | -| `new_precompile/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | - - -##### Precompile as Set-code Delegated Address - -Test setting the precompile as a set-code delegated address, and verify no precompile logic is executed. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/call_contexts/set_code` | Set code delegated address. | | | - -#### Inputs - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/inputs/valid` | Verify combinations of valid inputs to the precompile. | | | -| `new_precompile/test/inputs/valid/boundary` | Verify all boundary values given the precompile functionality. | | | -| `new_precompile/test/inputs/valid/crypto` | If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). | | | -| `new_precompile/test/inputs/all_zeros` | Verify all zeros input. | | | -| `new_precompile/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | -| `new_precompile/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | -| `new_precompile/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | -| `new_precompile/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | - -#### Value Transfer - -##### Minimum Fee Precompile - -If the precompile requires a minimum value (fee) to execute, either constant or depending on a formula. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | -| `new_precompile/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | -| `new_precompile/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | - -##### No-Fee Precompile - -If the precompile does not require any minimum value (fee) to execute. - -| `new_precompile/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | - -#### Out-of-bounds checks - -If the precompile has out-of-bounds conditions in its inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/out_of_bounds/max` | Max value for each input. | | | -| `new_precompile/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | - -#### Input Lengths - -Verify different combinations of input lengths to the precompile, ensuring the correct minimum fee (if any) is provided. - -##### Zero-length Input - -Regardless of the input requirements for the precompile. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/input_lengths/zero` | Zero-length calldata. | | | - -##### Static Required Input Length - -If the precompile has a static required input length. - -| `new_precompile/test/input_lengths/static/correct` | Correct static-length calldata. | | | -| `new_precompile/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | -| `new_precompile/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | - -##### Dynamic Required Input Length - -If the precompile has a variable required input-length based on a formula, test all listed scenarios given different input lengths. - -| `new_precompile/test/input_lengths/dynamic/valid` | Verify correct precompile execution for valid lengths. | | | -| `new_precompile/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | -| `new_precompile/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | - -#### Gas usage - -##### Constant Gas Cost - -If the precompile always charges the same gas cost regardless of input conditions. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | -| `new_precompile/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | - -##### Variable Gas Cost - -If the precompile charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | -| `new_precompile/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | - -##### Excessive Gas - -Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as reference max amount). - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/excessive_gas_usage` | Excessive gas usage. | | | - -#### Fork transition - -##### Pre-fork Block Call - -Verify that calling the precompile before its activation fork results in a valid call, even for inputs that are expected to be invalid for the precompile, or a zero-gas call. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/fork_transition/before/invalid_input` | Invalid input call. | | | -| `new_precompile/test/fork_transition/before/zero_gas` | Zero-gas call. | | | - - -##### Cold/Warm Precompile Address State - -Verify the cold/warm state of the precompile address depending on the fork activation. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_precompile/test/fork_transition/before/cold` | Precompile address is cold by default before the fork activation. | | | -| `new_precompile/test/fork_transition/after/warm` | Precompile address is warm by default after the fork activation. | | | - -### Framework Changes - -- Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## Removed Precompile - -The EIP removes one or more precompiles from the existing list of precompiles. - -### Test Vectors - -#### Fork transition - -##### Precompile Operation - -Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `removed_precompile/test/fork_transition/operational` | Precompile operation on fork activation. | | | - -##### Cold/Warm Precompile Address State - -Verify the cold/warm state of the precompile address depending on the fork activation. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `removed_precompile/test/fork_transition/before/warm` | Precompile address is warm by default before the fork activation. | | | -| `removed_precompile/test/fork_transition/after/cold` | Precompile address is cold by default after the fork activation. | | | - -### Framework Changes - -- Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` - -## New System Contract - -### Test Vectors - -#### Call contexts - -Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. - -##### `CALL` - -Verify system contract operation when called using the `CALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/normal` | `CALL`. | | | - -##### `DELEGATECALL` - -Verify system contract operation when called using the `DELEGATECALL` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/delegate` | `DELEGATECALL`. | | | - -##### `STATICCALL` - -Verify system contract operation when called using the `STATICCALL` opcode. - -If the system contract is stateful, meaning calling it affects its storage, this call must result in exceptional abort. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/static` | `STATICCALL`. | | | - -##### `CALLCODE` - -Verify system contract operation when called using the `CALLCODE` opcode. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/callcode` | `CALLCODE`. | | | - -##### Transaction Entry-point - -Verify system contract behavior when it's used as `tx.to`. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/tx_entry` | System Contract as transaction entry-point. | | | - -##### Initcode call - -Verify calling the opcode during initcode execution of a new contract. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | -| `new_system_contract/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | - - -##### System Contract as Set-code Delegated Address - -Test setting the system contract as a set-code delegated address, and verify no system contract side-effects are triggered, even if the actual system contract logic is executed. - -If the system contract requires specific storage pre-conditions to be set for proper execution (e.g. if the contract contains a safety mechanism that prevents it from executing prior to the fork activation), ensure the delegated account has the proper values in the storage to guarantee the correct logic is executed. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/call_contexts/set_code` | Set code delegated address. | | | - -#### Inputs - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/inputs/valid` | Verify combinations of valid inputs to the system contract. | | | -| `new_system_contract/test/inputs/boundary` | Verify all boundary values given the system contract functionality. | | | -| `new_system_contract/test/inputs/all_zeros` | Verify all zeros input. | | | -| `new_system_contract/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | -| `new_system_contract/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | -| `new_system_contract/test/inputs/invalid/checks` | Inputs that fail validity checks. | | | -| `new_system_contract/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | -| `new_system_contract/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | - -#### Value Transfer - -##### Minimum Fee System Contract - -If the system contract requires a minimum value (fee) to execute, either constant or depending on a formula. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | -| `new_system_contract/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | -| `new_system_contract/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | - -##### No-Fee System Contract - -If the system contract does not require any minimum value (fee) to execute. - -| `new_system_contract/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | - -#### Out-of-bounds checks - -If the system contract has out-of-bounds conditions in its inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/out_of_bounds/max` | Max value for each input. | | | -| `new_system_contract/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | - -#### Input Lengths - -Verify different combinations of input lengths to the system contract, ensuring the correct minimum fee (if any) is provided. - -##### Zero-length Input - -Regardless of the input requirements for the system contract. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/input_lengths/zero` | Zero-length calldata. | | | - - -##### Static Required Input Length - -If the system contract has a static required input length. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/input_lengths/static/correct` | Correct static-length calldata. | | | -| `new_system_contract/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | -| `new_system_contract/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | - -##### Dynamic Required Input Length - -If the system contract has a variable required input-length based on a formula, test all listed scenarios given different input lengths. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/input_lengths/dynamic/valid` | Verify correct system contract execution for valid lengths. | | | -| `new_system_contract/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | -| `new_system_contract/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | - - - -#### Gas usage - -##### Constant Gas Cost - -If the system contract always charges the same gas cost regardless of input conditions. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | -| `new_system_contract/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | - -##### Variable Gas Cost - -If the system contract charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | -| `new_system_contract/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | - -#### Excessive Gas Cases - -##### Excessive Gas Usage During Block Execution - -Verify spending all block gas in calls to system contract (Use `Environment().gas_limit` as reference max amount). - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/excessive_gas/block_gas` | Exhaust block gas limit. | | | - - -##### Excessive Gas Usage During System Call - -If possible, produce a scenario where, given all transactions executed within the block result in the execution of the contract by the system address would result in excessive gas usage. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/excessive_gas/system_call` | Excessive gas on system call. | | | - - -#### System Contract Deployment - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/deployment/missing` | Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). | | | -| `new_system_contract/test/deployment/address` | Verify deployment transaction results in the system contract being deployed to the expected address. | | | - -#### Contract Variations - -Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/contract_variations/networks` | Different network contract variations. | | | - -#### Contract Substitution - -Substitute the default system contract with a mock contract to modify its behavior when called by the system address (at the end of the block execution). - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/contract_substitution/return_lengths` | Modified return value lengths. | | | -| `new_system_contract/test/contract_substitution/logs` | Modified to emit logs or modified logs. | | | -| `new_system_contract/test/contract_substitution/exception` | Modified to cause an exception (e.g. invalid opcode). | | | -| `new_system_contract/test/contract_substitution/gas_limit_success` | Modified to consume 30,000,000 million gas exactly, execution should be successful. | | | -| `new_system_contract/test/contract_substitution/gas_limit_failure` | Modified to consume 30,000,001 million gas exactly, execution should fail. | | | - -#### Fork transition - -Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_system_contract/test/fork_transition/call_before_fork` | Call system contract before fork. | | | - -### Framework Changes - -- Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` -- Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` - -## New Transaction Type - -### Test Vectors - -#### Intrinsic Validity - -Verify the transaction (and the block it is included in) is valid or invalid given the intrinsic validity of the new transaction type, depending on each test scenario. - -For each new field that affects the intrinsic gas cost of the transaction verify all listed scenarios. - -##### Gas Limit - -Note: Data floor gas cost affects the intrinsic validity of all transaction types, so it must be taken into account when designing all test scenarios. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/gas_limit/exact` | Provide the exact intrinsic gas as `gas_limit` value to the transaction. | | | -| `new_transaction_type/test/intrinsic_validity/gas_limit/insufficient` | Provide the exact intrinsic gas minus one as `gas_limit` value to the transaction. | | | - -##### Gas Fee - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_lower_than_max_fee` | Invalid if `tx.max_priority_fee_per_gas < tx.max_fee_per_gas`. | | | -| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_equal_to_max_fee` | Valid if `tx.max_priority_fee_per_gas == tx.max_fee_per_gas`. | | | -| `new_transaction_type/test/intrinsic_validity/max_fee/base_lower` | Invalid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | -| `new_transaction_type/test/intrinsic_validity/max_fee/base_equal` | Valid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | - -##### Chain ID - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/chain_id` | Invalid if `tx.chain_id != network.chain_id`. | | | - -##### Nonce - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/nonce_minus_one` | Invalid if `tx.nonce == sender.nonce - 1`. | | | -| `new_transaction_type/test/intrinsic_validity/nonce_plus_one` | Invalid if `tx.nonce == sender.nonce + 1`. | | | -| `new_transaction_type/test/intrinsic_validity/nonce_exact` | Valid if `tx.nonce == sender.nonce`. | | | - -##### To - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/to` | Valid/Invalid if `tx.to == None`, depending on the EIP.| | | - -##### Value - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/value_non_zero_insufficient_balance` | Invalid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | -| `new_transaction_type/test/intrinsic_validity/value_non_zero_sufficient_balance` | Valid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) + 1`. | | | -| `new_transaction_type/test/intrinsic_validity/value_zero_insufficient_balance` | Invalid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) - 1`. | | | -| `new_transaction_type/test/intrinsic_validity/value_zero_sufficient_balance` | Valid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | - -##### Data - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Invalid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == tx.intrinsic_gas_cost`. | | | -| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Valid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == data_floor_cost(len(tx.data))`. | | | - - -#### Signature. - -Verify the transaction is correctly rejected if it contains an invalid signature. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/signature/invalid/field_outside_curve` | V, R, S represent a value that is inside of the field but outside of the curve. | | | - -##### V - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/signature/invalid/v/2` | `2`. | | | -| `new_transaction_type/test/signature/invalid/v/27` | `27` (Type-0 transaction valid value). | | | -| `new_transaction_type/test/signature/invalid/v/28` | `28` (Type-0 transaction valid value). | | | -| `new_transaction_type/test/signature/invalid/v/35` | `35` (Type-0 replay-protected transaction valid value for chain id 1). | | | -| `new_transaction_type/test/signature/invalid/v/36` | `36` (Type-0 replay-protected transaction valid value for chain id 1). | | | -| `new_transaction_type/test/signature/invalid/v/max` | `2**8-1`. | | | - -##### R - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/signature/invalid/r/0` | `0`. | | | -| `new_transaction_type/test/signature/invalid/r/secp256k1n_minus_one` | `SECP256K1N-1`. | | | -| `new_transaction_type/test/signature/invalid/r/secp256k1n` | `SECP256K1N`. | | | -| `new_transaction_type/test/signature/invalid/r/secp256k1n_plus_one` | `SECP256K1N+1`. | | | -| `new_transaction_type/test/signature/invalid/r/max_minus_one` | `2**256-1`. | | | -| `new_transaction_type/test/signature/invalid/r/max` | `2**256`. | | | - -##### S - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/signature/invalid/s/0` | `0`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_minus_one` | `SECP256K1N//2-1`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n_half` | `SECP256K1N//2`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_plus_one` | `SECP256K1N//2+1`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n_minus_one` | `SECP256K1N-1`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n` | `SECP256K1N`. | | | -| `new_transaction_type/test/signature/invalid/s/secp256k1n_plus_one` | `SECP256K1N+1`. | | | -| `new_transaction_type/test/signature/invalid/s/max_minus_one` | `2**256-1`. | | | -| `new_transaction_type/test/signature/invalid/s/max` | `2**256`. | | | -| `new_transaction_type/test/signature/invalid/s/complement` | `SECP256K1N - S` of a valid signature. | | | - -#### Transaction Attributes Readable From EVM - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/tx_scoped_attributes/read` | Verify attributes that can be read in the EVM from transaction fields. | | | -| `new_transaction_type/test/tx_scoped_attributes/older_tx_types` | Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. | | | - -#### Transaction-Scoped Persistent Values - -Verify values or variables that are persistent through the execution of the transaction (e.g. transient storage, warm/cold accounts). - -| `new_transaction_type/test/tx_scoped_attributes/persistent/throughout` | Persist throughout the entire transaction. | | | -| `new_transaction_type/test/tx_scoped_attributes/persistent/reset` | Reset on subsequent transactions in the same block. | | | - -#### Encoding (RLP, SSZ) - -Verify correct transaction rejection in all listed scenarios. - -##### Field Sizes - -Verify all listed scenarios for each transaction field. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/encoding/field_sizes/leading_zero` | Add leading zero byte. | | | -| `new_transaction_type/test/encoding/field_sizes/remove_byte` | Remove single byte from fixed-byte-length fields. | | | - -##### Fields of List Type - -Verify for each transaction field that is of type list. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/encoding/list_field/zero` | Zero-element list (Failure depending on EIP). | | | -| `new_transaction_type/test/encoding/list_field/max` | Max count list. | | | -| `new_transaction_type/test/encoding/list_field/max_plus_one` | Max count plus one list. | | | - -##### Extra/Missing Fields - -| `new_transaction_type/test/encoding/missing_fields` | Any fields particular to the new transaction types are missing. | | | -| `new_transaction_type/test/encoding/extra_fields` | Transaction contains extra fields. | | | - -##### Serialization Corruption - -| `new_transaction_type/test/encoding/truncated` | Serialized bytes object is truncated by one byte. | | | -| `new_transaction_type/test/encoding/extra_bytes` | Serialized bytes object has one extra byte. | | | - -##### Serializable Fields - -Verify for each serializable field, all previous tests plus following listed scenario. - -| `new_transaction_type/test/encoding/new_types/incorrect_encoding` | Serializable field is encoded as bytes instead of using the correct encoding (e.g. list in the case of RLP). | | | - -#### Out-of-bounds checks - -Verify if the transaction has out-of-bounds conditions in its fields and verify. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/out_of_bounds/max` | Max value for each field. | | | -| `new_transaction_type/test/out_of_bounds/max_plus_one` | Max value + 1 for each field. | | | - -#### Contract creation - -Verify that the transaction can create new contracts if the transaction type supports it. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/contract_creation` | Create contracts using new transaction type. | | | - -#### Sender account modifications - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/sender_account/nonce` | Sender account has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). | | | -| `new_transaction_type/test/sender_account/balance` | Sender account has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). | | | - -#### Block Level Interactions - -##### Single Transaction In Block - -Verify a block where the new transaction type is the sole transaction contained in the block. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/block_interactions/single_tx/invalid` | Invalid if `tx.gas_limit == block.gas_limit + 1`. | | | -| `new_transaction_type/test/block_interactions/single_tx/valid` | Valid if `tx.gas_limit == block.gas_limit`. | | | - - -##### Two Transactions In Block - -Verify a block where the new transaction type is the last transaction contained in a block with two transactions. - -| `new_transaction_type/test/block_interactions/last_tx/valid` | Valid if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit`. | | | -| `new_transaction_type/test/block_interactions/last_tx/invalid` | Invalid if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)`. | | | - -##### EIP-7825 - -Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. - -| `new_transaction_type/test/block_interactions/eip7825/invalid` | Exceeds EIP-7825 gas limit by one. | | | -| `new_transaction_type/test/block_interactions/eip7825/valid` | Gas limit is exactly the EIP-7825 gas limit. | | | - -##### Mixed transactions - -Verify a block with all transactions types including the new type is executed correctly. - -| `new_transaction_type/test/block_interactions/mixed_txs` | Mixed transactions. | | | - -#### Fork transition - -Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_transaction_type/test/fork_transition/before` | New transaction type included before fork activation block. | | | - -#### RPC Tests - -- *Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type. -- `new_transaction_type/test/rpc/send_raw` | Verify `eth_sendRawTransaction` using `execute`. - -*Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. - -### Framework Changes - -- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. -- Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). - -## New Block Header Field - -### Test Vectors - -#### Genesis value - -Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_block_header_field/test/genesis` | New block header field at genesis. | | | - -#### Value behavior - -Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_block_header_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | -| `new_block_header_field/test/value_behavior/reject` | Block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | - -#### Fork transition - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_block_header_field/test/fork_transition/initial` | Verify initial value of the field at the first block of the activation fork. | | | -| `new_block_header_field/test/fork_transition/before` | Verify that a block containing the new header field before the activation of the fork is invalid. | | | -| `new_block_header_field/test/fork_transition/after` | Verify that a block lacking the new header field at the activation of the fork is invalid. | | | - -### Framework Changes - -- Add the new header field to the relevant objects: - - `ethereum_test_fixtures.FixtureHeader`. - - `ethereum_test_fixtures.FixtureExecutionPayload`. - - `ethereum_test_specs.Header`. -- Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. - -## New Block Body Field - -### Test Vectors - -#### Value behavior - -Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_block_body_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | -| `new_block_body_field/test/value_behavior/reject` | Block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | - -#### Fork transition - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_block_body_field/test/fork_transition/before` | Verify that a block containing the new block body field before the activation of the fork is invalid. | | | -| `new_block_body_field/test/fork_transition/after` | Verify that a block lacking the new block field at the activation of the fork is invalid. | | | - -### Framework Changes - -- Add the new body field to the relevant objects. - - `ethereum_test_fixtures.FixtureBlockBase`. - - `ethereum_test_fixtures.FixtureEngineNewPayload`. - - `ethereum_test_specs.Block`. -- Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. - -## Gas Cost Changes - -### Test Vectors - -#### Gas Usage - -Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_cost_changes/test/gas_updates_measurement` | Measure updated gas costs. | | | - -#### Out-of-gas - -Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_cost_changes/test/out_of_gas` | Out-Of-Gas with new gas prices. | | | - -#### Fork transition - -Verify gas costs are updated at the fork transition boundary. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_cost_changes/test/fork_transition/before` | Costs unaffected before the fork activation block. | | | -| `gas_cost_changes/test/fork_transition/after` | Costs are updated on and after fork activation block. | | | - -### Framework Changes - -- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. -- Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. -- Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. -- Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. - -## Gas Refunds Changes - -### Test Vectors - -#### Refund calculation - -Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_refunds_changes/test/refund_calculation/over` | `refund == gas_used // MAX_REFUND_QUOTIENT + 1`. | | | -| `gas_refunds_changes/test/refund_calculation/exact` | `refund == gas_used // MAX_REFUND_QUOTIENT`. | | | -| `gas_refunds_changes/test/refund_calculation/under` | `refund == gas_used // MAX_REFUND_QUOTIENT - 1`. | | | - -#### Exceptional Abort. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_refunds_changes/test/exceptional_abort/revertable` | If the operation causing the refund can be reverted, verify the refund is not applied if the following cases:. | | | -| `gas_refunds_changes/test/exceptional_abort/revertable/revert` | `REVERT`. | | | -| `gas_refunds_changes/test/exceptional_abort/revertable/out_of_gas` | Out-of-gas. | | | -| `gas_refunds_changes/test/exceptional_abort/revertable/invalid_opcode` | Invalid opcode. | | | -| `gas_refunds_changes/test/exceptional_abort/revertable/upper_revert` | `REVERT` of an upper call frame. | | | -| `gas_refunds_changes/test/exceptional_abort/non_revertable` | If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases:. | | | -| `gas_refunds_changes/test/exceptional_abort/non_revertable/revert` | `REVERT` at the top call frame. | | | -| `gas_refunds_changes/test/exceptional_abort/non_revertable/out_of_gas` | Out-of-gas at the top call frame. | | | -| `gas_refunds_changes/test/exceptional_abort/non_revertable/invalid_opcode` | Invalid opcode at the top call frame. | | | - -#### Cross-Functional Test - -Verify the following tests are updated to support the new type of refunds. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `gas_refunds_changes/test/cross_functional/calldata_cost` | `tests/prague/eip7623_increase_calldata_cost/test_refunds.py`. | | | - -### Framework Changes - -N/A - -## Blob Count Changes - -### Test Vectors - -#### Existing Test Updates - -Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. - -| ID | Description | Status | Tests | -|-|-|-|-| -| `blob_count_changes/test/eip4844_blobs_changes` | Updates to `tests/cancun/eip4844_blobs` were applied correctly. | | | - -### Framework Changes - -- Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. - -## New Execution Layer Request - -### Test Vectors - -#### Cross-Request-Type Interaction - -| ID | Description | Status | Tests | -|-|-|-|-| -| `new_execution_layer_request/test/cross_request_type/update` | Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations. | | | - -### Framework Changes - -- Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. diff --git a/docs/navigation.md b/docs/navigation.md index 9cbf12b8153..b90b652237e 100644 --- a/docs/navigation.md +++ b/docs/navigation.md @@ -1,58 +1,60 @@ * [Home](index.md) * Getting Started - * [Installation](getting_started/installation.md) - * [Installation Troubleshooting](getting_started/installation_troubleshooting.md) - * [VS Code Setup](getting_started/setup_vs_code.md) - * [Repository Overview](getting_started/repository_overview.md) - * [Getting Help](getting_started/getting_help.md) + * [Installation](getting_started/installation.md) + * [Installation Troubleshooting](getting_started/installation_troubleshooting.md) + * [VS Code Setup](getting_started/setup_vs_code.md) + * [Repository Overview](getting_started/repository_overview.md) + * [Getting Help](getting_started/getting_help.md) * [Writing Tests](writing_tests/index.md) - * [Adding a New Test](writing_tests/adding_a_new_test.md) - * [Types of Test](writing_tests/types_of_tests.md) - * [Writing a New Test](writing_tests/writing_a_new_test.md) - * [Test Markers](writing_tests/test_markers.md) - * [Referencing an EIP Spec Version](writing_tests/reference_specification.md) - * [Verifying Changes Locally](writing_tests/verifying_changes.md) - * [Code Standards](writing_tests/code_standards.md) - * [Exception Tests](writing_tests/exception_tests.md) - * [Tutorial: Adding a State Test](writing_tests/tutorials/state_transition.md) - * [Post-mortems](writing_tests/post_mortems.md) + * [Adding a New Test](writing_tests/adding_a_new_test.md) + * [Types of Test](writing_tests/types_of_tests.md) + * [Writing a New Test](writing_tests/writing_a_new_test.md) + * [Test Markers](writing_tests/test_markers.md) + * [Verifying Changes Locally](writing_tests/verifying_changes.md) + * [Code Standards](writing_tests/code_standards.md) + * [Exception Tests](writing_tests/exception_tests.md) + * [Referencing an EIP Spec Version](writing_tests/reference_specification.md) + * [Testing Checklist Templates](writing_tests/checklist_templates/index.md) + * [EIP Execution Layer Testing Checklist Template](writing_tests/checklist_templates/eip_testing_checklist_template.md) + * [Post-mortems](writing_tests/post_mortems.md) + * [Tutorial: Adding a State Test](writing_tests/tutorials/state_transition.md) * [Filling Tests](filling_tests/index.md) - * [Getting Started](filling_tests/getting_started.md) - * [Filling Tests at a Prompt](filling_tests/filling_tests_command_line.md) - * [Filling Tests in VS Code](filling_tests/filling_tests_vs_code.md) - * [Filling Tests for Features Under Development](filling_tests/filling_tests_dev_fork.md) - * [Transition Tool Support](filling_tests/transition_tool_support.md) - * [Debugging Transition Tools](filling_tests/debugging_t8n_tools.md) + * [Getting Started](filling_tests/getting_started.md) + * [Filling Tests at a Prompt](filling_tests/filling_tests_command_line.md) + * [Filling Tests in VS Code](filling_tests/filling_tests_vs_code.md) + * [Filling Tests for Features Under Development](filling_tests/filling_tests_dev_fork.md) + * [Transition Tool Support](filling_tests/transition_tool_support.md) + * [Debugging Transition Tools](filling_tests/debugging_t8n_tools.md) * [Consuming Tests](consuming_tests/index.md) - * [State Tests](consuming_tests/state_test.md) - * [Blockchain Tests](consuming_tests/blockchain_test.md) - * [Blockchain Engine Tests](consuming_tests/blockchain_test_engine.md) - * [EOF Tests](consuming_tests/eof_test.md) - * [Transaction Tests](consuming_tests/transaction_test.md) - * [Common Types](consuming_tests/common_types.md) - * [Exceptions](consuming_tests/exceptions.md) + * [State Tests](consuming_tests/state_test.md) + * [Blockchain Tests](consuming_tests/blockchain_test.md) + * [Blockchain Engine Tests](consuming_tests/blockchain_test_engine.md) + * [EOF Tests](consuming_tests/eof_test.md) + * [Transaction Tests](consuming_tests/transaction_test.md) + * [Common Types](consuming_tests/common_types.md) + * [Exceptions](consuming_tests/exceptions.md) * [Executing Tests](executing_tests/index.md) * [Developer Doc](dev/index.md) - * [Managing Configurations](dev/configurations.md) - * [Interactive Library Usage](dev/interactive_usage.md) - * [Generating Documentation](dev/docs.md) - * [Documenting CLI Commands](dev/documenting_clis.md) - * [Coding Style](dev/coding_style.md) - * [Logging](dev/logging.md) - * [Enabling Precommit Checks](dev/precommit.md) - * [Running Github Actions Locally](dev/test_actions_locally.md) - * [Porting Legacy Tests](dev/porting_legacy_tests.md) + * [Managing Configurations](dev/configurations.md) + * [Interactive Library Usage](dev/interactive_usage.md) + * [Generating Documentation](dev/docs.md) + * [Documenting CLI Commands](dev/documenting_clis.md) + * [Coding Style](dev/coding_style.md) + * [Logging](dev/logging.md) + * [Enabling Precommit Checks](dev/precommit.md) + * [Running Github Actions Locally](dev/test_actions_locally.md) + * [Porting Legacy Tests](dev/porting_legacy_tests.md) * [Changelog](CHANGELOG.md) * [Library Reference](library/index.md) - * [EEST CLI Tools](library/cli/index.md) - * [Ethereum Test Base Types Package](library/ethereum_test_base_types.md) - * [Ethereum Test Exceptions Package](library/ethereum_test_exceptions.md) - * [Ethereum Test Fixtures Package](library/ethereum_test_fixtures.md) - * [Ethereum Test Forks Package](library/ethereum_test_forks.md) - * [Ethereum Test Specs Package](library/ethereum_test_specs.md) - * [Ethereum Test Tools Package](library/ethereum_test_tools.md) - * [Ethereum Test Types Package](library/ethereum_test_types.md) - * [Ethereum Test VM Package](library/ethereum_test_vm.md) - * [Ethereum CLIs Package](library/ethereum_clis.md) - * [Pytest Plugins](library/pytest_plugins/index.md) + * [EEST CLI Tools](library/cli/index.md) + * [Ethereum Test Base Types Package](library/ethereum_test_base_types.md) + * [Ethereum Test Exceptions Package](library/ethereum_test_exceptions.md) + * [Ethereum Test Fixtures Package](library/ethereum_test_fixtures.md) + * [Ethereum Test Forks Package](library/ethereum_test_forks.md) + * [Ethereum Test Specs Package](library/ethereum_test_specs.md) + * [Ethereum Test Tools Package](library/ethereum_test_tools.md) + * [Ethereum Test Types Package](library/ethereum_test_types.md) + * [Ethereum Test VM Package](library/ethereum_test_vm.md) + * [Ethereum CLIs Package](library/ethereum_clis.md) + * [Pytest Plugins](library/pytest_plugins/index.md) diff --git a/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md new file mode 100644 index 00000000000..76f0fa1d4bc --- /dev/null +++ b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md @@ -0,0 +1,1157 @@ + +# EIP Execution Layer Testing Checklist Template + +Depending on the changes introduced by an EIP, the following template is the minimum baseline to guarantee test coverage of the Execution Layer features. + +## General + +#### Code coverage + +| ID | Description | Status | Tests | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ----- | +| `general/code_coverage/eels` | Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. | | | +| `general/code_coverage/test_coverage` | Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. | | | +| `general/code_coverage/second_client` | Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. | | | + +#### Fuzzing + +## New Opcode + +The EIP introduces one or more new opcodes to the EVM. + +### Test Vectors + +#### Memory expansion + +If the opcode execution can expand the memory size, either by writing to memory or reading from an offset that exceeds current memory, or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ---------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/mem_exp/zero_bytes_zero_offset` | Zero bytes expansion with zero-offset. | | | +| `new_opcode/test/mem_exp/zero_bytes_max_offset` | Zero bytes expansion with 2\*\*256-1 offset. | | | +| `new_opcode/test/mem_exp/single_byte` | Single byte expansion. | | | +| `new_opcode/test/mem_exp/31_bytes` | 31 bytes expansion. | | | +| `new_opcode/test/mem_exp/32_bytes` | 32 bytes expansion. | | | +| `new_opcode/test/mem_exp/33_bytes` | 33 bytes expansion. | | | +| `new_opcode/test/mem_exp/64_bytes` | 64 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_32_minus_one_bytes` | `2**32-1` bytes expansion. | | | +| `new_opcode/test/mem_exp/2_32_bytes` | 2\*\*32 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_64_minus_one_bytes` | 2\*\*64-1 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_64_bytes` | 2\*\*64 bytes expansion. | | | +| `new_opcode/test/mem_exp/2_256_minus_one_bytes` | 2\*\*256-1 bytes expansion (Should always result in Out-of-gas). | | | + +#### Stack + +##### Stack Overflow + +If the opcode pushes one or more items to the stack, and the opcode pushes more elements than it pops, verify that the opcode execution results in exceptional abort when pushing elements to the stack would result in the stack having more than 1024 elements. + +| ID | Description | Status | Tests | +| -------------------------------- | --------------- | ------ | ----- | +| `new_opcode/test/stack_overflow` | Stack Overflow. | | | + +##### Stack Underflow + +If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. + +| ID | Description | Status | Tests | +| --------------------------------- | ---------------- | ------ | ----- | +| `new_opcode/test/stack_underflow` | Stack Underflow. | | | + +##### Stack Complex Operations + +If opcode performs stack operations more complex than simple pop/push (e.g. the opcode has a data portion that specifies a variable to access a specific stack element), perform the following test combinations. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------- | -------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/stack_complex_operations/stack_heights/zero` | Operation on an empty stack (Potential stack-underflow). | | | +| `new_opcode/test/stack_complex_operations/stack_heights/odd` | Operation on a stack with an odd height. | | | +| `new_opcode/test/stack_complex_operations/stack_heights/even` | Operation on a stack with an even height. | | | + +##### Stack Manipulation With Data Portion Variables + +If the opcode contains variables in its data portion, for each variable `n` of the opcode that accesses the nth stack item, test `n` being. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------ | ------------------ | ------ | ----- | +| `new_opcode/test/stack_complex_operations/variable_n/top` | Top stack item. | | | +| `new_opcode/test/stack_complex_operations/variable_n/bottom` | Bottom stack item. | | | +| `new_opcode/test/stack_complex_operations/variable_n/middle` | Middle stack item. | | | + +#### Execution context + +Evaluate opcode's behavior in different execution contexts. + +##### `CALL` + +Verify opcode operation in a subcall frame originated from a `CALL` opcode. + +| ID | Description | Status | Tests | +| ---------------------------------------- | ----------- | ------ | ----- | +| `new_opcode/test/execution_context/call` | `CALL`. | | | + +##### `STATICCALL` + +Verify opcode operation in a subcall frame originated from a `STATICCALL` opcode. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ----- | +| `new_opcode/test/execution_context/staticcall/ban_check` | Verify exceptional abort if the opcode attempts to modify the code, storage or balance of an account. | | | +| `new_opcode/test/execution_context/staticcall/ban_no_modification` | If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). | | | +| `new_opcode/test/execution_context/staticcall/sub_calls` | Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. | | | + +##### `DELEGATECALL` + +Verify opcode operation in a subcall frame originated from a `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/execution_context/delegatecall` | `DELEGATECALL`. | | | +| `new_opcode/test/execution_context/delegatecall/storage` | If the opcode modifies the storage of the account currently executing it, verify that only the account that is delegating execution is the one that has its storage modified. | | | +| `new_opcode/test/execution_context/delegatecall/balance` | If the opcode modifies the balance of the account currently executing it, verify that only the account that is delegating execution is the one that has its balance modified. | | | +| `new_opcode/test/execution_context/delegatecall/code` | If the opcode modifies the code of the account currently executing it, verify that only the account that is delegating execution is the one that has its code modified. | | | + +##### `CALLCODE` + +Verify opcode operation in a subcall frame originated from a `CALLCODE` opcode. + +| ID | Description | Status | Tests | +| -------------------------------------------- | ----------- | ------ | ----- | +| `new_opcode/test/execution_context/callcode` | `CALLCODE`. | | | + +##### Initcode + +Verify opcode behaves as expected when running during the initcode phase of contract creation. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/execution_context/initcode/behavior` | Initcode operation. | | | +| `new_opcode/test/execution_context/initcode/behavior/tx` | Initcode of a contract creating transaction. | | | +| `new_opcode/test/execution_context/initcode/behavior/opcode` | Initcode of a contract creating opcode (including itself if opcode creates a contract). | | | +| `new_opcode/test/execution_context/initcode/reentry` | Initcode re-entry: using the same initcode and same address (e.g. CREATE2->REVERT->CREATE2). | | | + +##### Set-code Delegated Account + +Verify opcode operations are applied to the set-code account and do not affect the address that is the target of the delegation. + +| ID | Description | Status | Tests | +| -------------------------------------------- | --------------------------- | ------ | ----- | +| `new_opcode/test/execution_context/set_code` | Set-code delegated account. | | | + +##### Transaction Context + +If opcode changes behavior depending on particular transaction properties, test using multiple values for each property. + +| ID | Description | Status | Tests | +| ---------------------------------------------- | ------------------------------------- | ------ | ----- | +| `new_opcode/test/execution_context/tx_context` | Transaction-context dependent opcode. | | | + +##### Block Context + +If opcode changes behavior depending on particular block properties, test using multiple values for each property. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ------------------------------- | ------ | ----- | +| `new_opcode/test/execution_context/block_context` | Block-context dependent opcode. | | | + +#### Return data + +Verify proper return data buffer overwriting if the opcode is meant to interact with it, otherwise verify that the return data buffer is unaffected. + +| ID | Description | Status | Tests | +| -------------------------------------------- | -------------------------------------- | ------ | ----- | +| `new_opcode/test/return_data/buffer/current` | Return buffer at current call context. | | | +| `new_opcode/test/return_data/buffer/parent` | Return buffer at parent call context. | | | + +#### Gas Usage + +##### Normal Operation + +Verify gas usage affectation of each stack argument or memory input consumed by the opcode. + +| ID | Description | Status | Tests | +| ---------------------------------- | --------------------------- | ------ | ----- | +| `new_opcode/test/gas_usage/normal` | Normal operation gas usage. | | | + +##### Memory Expansion + +Verify that the memory expansion correctly follows the gas calculation. + +| ID | Description | Status | Tests | +| -------------------------------------------- | ----------------- | ------ | ----- | +| `new_opcode/test/gas_usage/memory_expansion` | Memory expansion. | | | + +##### Out-Of-Gas + +Verify that attempting to execute the opcode when gas available is 1 less than the required gas results in exceptional abort. + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ----------------------------------- | ------ | ----- | +| `new_opcode/test/gas_usage/out_of_gas_execution` | Out-of-gas due to opcode inputs. | | | +| `new_opcode/test/gas_usage/out_of_gas_memory` | Out-of-gas due to memory expansion. | | | + +##### Order-of-operations + +If the opcode requires different gas stipends for other operations (e.g. contract creation, cold/warm account access), create one case for each operation (ideally independent of each other) and the listed cases for each. + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | ------------------------------------------------------------------ | ------ | ----- | +| `new_opcode/test/gas_usage/order_of_operations/exact` | Success using the exact amount of gas required for the stipend. | | | +| `new_opcode/test/gas_usage/order_of_operations/oog` | OOG with a 1-gas-difference from the gas required for the stipend. | | | + +#### Terminating opcode + +If an opcode is terminating, meaning it results in the current call context to end execution. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | --------------------------- | ------ | ----- | +| `new_opcode/test/terminating/scenarios/top_level` | Top-level call termination. | | | +| `new_opcode/test/terminating/scenarios/sub_level` | Sub-level call termination. | | | +| `new_opcode/test/terminating/scenarios/initcode` | Initcode termination. | | | + +#### Aborting/Reverting opcode + +If the terminating opcode is meant to rollback the executing call frame, verify the listed events are properly rolled back. + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ------------------- | ------ | ----- | +| `new_opcode/test/terminating/rollback/balance` | Balance changes. | | | +| `new_opcode/test/terminating/rollback/storage` | Storage changes. | | | +| `new_opcode/test/terminating/rollback/contracts` | Contract creations. | | | +| `new_opcode/test/terminating/rollback/nonce` | Nonce increments. | | | +| `new_opcode/test/terminating/rollback/logs` | Log events. | | | + +#### Out-of-bounds checks + +If the opcode has out-of-bounds conditions in its parameters/inputs. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | --------------------------------- | ------ | ----- | +| `new_opcode/test/out_of_bounds/verify/max` | Max value for each parameter. | | | +| `new_opcode/test/out_of_bounds/verify/max_plus_one` | Max value + 1 for each parameter. | | | + +#### Exceptional Abort + +If the opcode has conditions, either inputs or execution context states, that should cause exceptional abort and are different than out-of-gas or stack overflow or underflow. + +| ID | Description | Status | Tests | +| ----------------------------------- | ----------------------------- | ------ | ----- | +| `new_opcode/test/exceptional_abort` | Exceptional abort conditions. | | | + +#### Data portion + +If an opcode has a data portion, meaning the `N` bytes following the opcode in the contract bytecode are skipped from execution. + +| ID | Description | Status | Tests | +| ---------------------------------------- | -------------------------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/data_portion/all_zeros` | All zeros data portion. | | | +| `new_opcode/test/data_portion/max_value` | Max value data portion (`2**N-1` where `N` is the bit size of the data portion). | | | +| `new_opcode/test/data_portion/jump` | Jump into the data portion. | | | + +#### Contract creation + +If the opcode execution results in the creation of a new contract in the state. + +##### Correct Creation + +Verify contract is created at the expected address given multiple inputs. + +| ID | Description | Status | Tests | +| ------------------------------------------- | -------------------- | ------ | ----- | +| `new_opcode/test/contract_creation/address` | Address calculation. | | | + +##### Creation Failure + +The contract creation fails given the listed conditions. + +| ID | Description | Status | Tests | +| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/contract_creation/failure/oog` | Out-of-gas when available gas is less than minimum contract creation stipend. | | | +| `new_opcode/test/contract_creation/failure/insufficient_value` | Opcode has a value parameter and the caller does not have enough funds. | | | +| `new_opcode/test/contract_creation/failure/collision` | Creation would result in an address collision with an existing contract or EOA-delegated address. | | | + +##### Recursive Contract Creation + +Opcode is used to attempt to recreate a contract that is currently mid-creation by a previous call of the same opcode. + +| ID | Description | Status | Tests | +| --------------------------------------------- | --------------------------------------------- | ------ | ----- | +| `new_opcode/test/contract_creation/recursive` | Recursive contract creation using the opcode. | | | + +#### Fork transition + +| ID | Description | Status | Tests | +| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_opcode/test/fork_transition/invalid` | Exceptional abort if executed before its activation fork/after its deactivation fork. | | | +| `new_opcode/test/fork_transition/at` | Verify correct opcode behavior at transition block, in the case of opcodes which behavior depends on current or parent block information. | | | + +### Framework Changes + +| ID | Description | Status | Tests | +| --- | ----------- | ------ | ----- | + +- Add opcode to `src/ethereum_test_vm/opcode.py` +- Add opcode to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New Precompile + +The EIP introduces one or more new precompiles. + +### Test Vectors + +#### Call contexts + +Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. + +##### `CALL` + +Verify precompile operation when called using the `CALL` opcode. + +| ID | Description | Status | Tests | +| ------------------------------------------ | ----------- | ------ | ----- | +| `new_precompile/test/call_contexts/normal` | `CALL`. | | | + +##### `DELEGATECALL` + +Verify precompile operation when called using the `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +| -------------------------------------------- | --------------- | ------ | ----- | +| `new_precompile/test/call_contexts/delegate` | `DELEGATECALL`. | | | + +##### `STATICCALL` + +Verify precompile operation when called using the `STATICCALL` opcode. + +If the precompile is stateful, meaning calling it affects its storage, this call must result in exceptional abort. + +| ID | Description | Status | Tests | +| ------------------------------------------ | ------------- | ------ | ----- | +| `new_precompile/test/call_contexts/static` | `STATICCALL`. | | | + +##### `CALLCODE` + +Verify precompile operation when called using the `CALLCODE` opcode. + +| ID | Description | Status | Tests | +| -------------------------------------------- | ----------- | ------ | ----- | +| `new_precompile/test/call_contexts/callcode` | `CALLCODE`. | | | + +##### Transaction Entry-point + +Verify precompile behavior when it's used as `tx.to`. + +| ID | Description | Status | Tests | +| -------------------------------------------- | -------------------------------------- | ------ | ----- | +| `new_precompile/test/call_contexts/tx_entry` | Precompile as transaction entry-point. | | | + +##### Initcode call + +Verify calling the opcode during initcode execution of a new contract. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ---------------------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | +| `new_precompile/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | + +##### Precompile as Set-code Delegated Address + +Test setting the precompile as a set-code delegated address, and verify no precompile logic is executed. + +| ID | Description | Status | Tests | +| -------------------------------------------- | --------------------------- | ------ | ----- | +| `new_precompile/test/call_contexts/set_code` | Set code delegated address. | | | + +#### Inputs + +| ID | Description | Status | Tests | +| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/inputs/valid` | Verify combinations of valid inputs to the precompile. | | | +| `new_precompile/test/inputs/valid/boundary` | Verify all boundary values given the precompile functionality. | | | +| `new_precompile/test/inputs/valid/crypto` | If precompile performs cryptographic operations, verify behavior on all inputs that have special cryptographic properties (e.g. infinity points as inputs, or input values that result in infinity points returned). | | | +| `new_precompile/test/inputs/all_zeros` | Verify all zeros input. | | | +| `new_precompile/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | +| `new_precompile/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | +| `new_precompile/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | +| `new_precompile/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | + +#### Value Transfer + +##### Minimum Fee Precompile + +If the precompile requires a minimum value (fee) to execute, either constant or depending on a formula. + +| ID | Description | Status | Tests | +| ---------------------------------------------- | ---------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | +| `new_precompile/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | +| `new_precompile/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | + +##### No-Fee Precompile + +If the precompile does not require any minimum value (fee) to execute. + +| `new_precompile/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | + +#### Out-of-bounds checks + +If the precompile has out-of-bounds conditions in its inputs. + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ----------------------------- | ------ | ----- | +| `new_precompile/test/out_of_bounds/max` | Max value for each input. | | | +| `new_precompile/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | + +#### Input Lengths + +Verify different combinations of input lengths to the precompile, ensuring the correct minimum fee (if any) is provided. + +##### Zero-length Input + +Regardless of the input requirements for the precompile. + +| ID | Description | Status | Tests | +| ---------------------------------------- | --------------------- | ------ | ----- | +| `new_precompile/test/input_lengths/zero` | Zero-length calldata. | | | + +##### Static Required Input Length + +If the precompile has a static required input length. + +| ID | Description | Status | Tests | +| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/input_lengths/static/correct` | Correct static-length calldata. | | | +| `new_precompile/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | +| `new_precompile/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | + +##### Dynamic Required Input Length + +If the precompile has a variable required input-length based on a formula, test all listed scenarios given different input lengths. + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/input_lengths/dynamic/valid` | Verify correct precompile execution for valid lengths. | | | +| `new_precompile/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the precompile. | | | +| `new_precompile/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the precompile with padded zeros. | | | + +#### Gas usage + +##### Constant Gas Cost + +If the precompile always charges the same gas cost regardless of input conditions. + +| ID | Description | Status | Tests | +| ---------------------------------------------- | ------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | +| `new_precompile/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Variable Gas Cost + +If the precompile charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. + +| ID | Description | Status | Tests | +| --------------------------------------------- | ------------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | +| `new_precompile/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Excessive Gas + +Verify spending all block gas in calls to the precompile (Use `Environment().gas_limit` as reference max amount). + +| ID | Description | Status | Tests | +| ----------------------------------------- | -------------------- | ------ | ----- | +| `new_precompile/test/excessive_gas_usage` | Excessive gas usage. | | | + +#### Fork transition + +##### Pre-fork Block Call + +Verify that calling the precompile before its activation fork results in a valid call, even for inputs that are expected to be invalid for the precompile, or a zero-gas call. + +| ID | Description | Status | Tests | +| ---------------------------------------------------------- | ------------------- | ------ | ----- | +| `new_precompile/test/fork_transition/before/invalid_input` | Invalid input call. | | | +| `new_precompile/test/fork_transition/before/zero_gas` | Zero-gas call. | | | + +##### Cold/Warm Precompile Address State + +Verify the cold/warm state of the precompile address depending on the fork activation. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ----------------------------------------------------------------- | ------ | ----- | +| `new_precompile/test/fork_transition/before/cold` | Precompile address is cold by default before the fork activation. | | | +| `new_precompile/test/fork_transition/after/warm` | Precompile address is warm by default after the fork activation. | | | + +### Framework Changes + +- Add precompile address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## Removed Precompile + +The EIP removes one or more precompiles from the existing list of precompiles. + +### Test Vectors + +#### Fork transition + +##### Precompile Operation + +Verify that the precompile remains operational up until the last block before the fork is active, and behaves as an account with empty code afterwards. + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | ---------------------------------------- | ------ | ----- | +| `removed_precompile/test/fork_transition/operational` | Precompile operation on fork activation. | | | + +##### Cold/Warm Precompile Address State + +Verify the cold/warm state of the precompile address depending on the fork activation. + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | ----------------------------------------------------------------- | ------ | ----- | +| `removed_precompile/test/fork_transition/before/warm` | Precompile address is warm by default before the fork activation. | | | +| `removed_precompile/test/fork_transition/after/cold` | Precompile address is cold by default after the fork activation. | | | + +### Framework Changes + +- Remove the precompile address from relevant methods in the fork where the EIP is removed in `src/ethereum_test_forks/forks/forks.py` + +## New System Contract + +### Test Vectors + +#### Call contexts + +Evaluate precompile behavior when called using different call opcodes or called from different execution contexts. + +##### `CALL` + +Verify system contract operation when called using the `CALL` opcode. + +| ID | Description | Status | Tests | +| ----------------------------------------------- | ----------- | ------ | ----- | +| `new_system_contract/test/call_contexts/normal` | `CALL`. | | | + +##### `DELEGATECALL` + +Verify system contract operation when called using the `DELEGATECALL` opcode. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | --------------- | ------ | ----- | +| `new_system_contract/test/call_contexts/delegate` | `DELEGATECALL`. | | | + +##### `STATICCALL` + +Verify system contract operation when called using the `STATICCALL` opcode. + +If the system contract is stateful, meaning calling it affects its storage, this call must result in exceptional abort. + +| ID | Description | Status | Tests | +| ----------------------------------------------- | ------------- | ------ | ----- | +| `new_system_contract/test/call_contexts/static` | `STATICCALL`. | | | + +##### `CALLCODE` + +Verify system contract operation when called using the `CALLCODE` opcode. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ----------- | ------ | ----- | +| `new_system_contract/test/call_contexts/callcode` | `CALLCODE`. | | | + +##### Transaction Entry-point + +Verify system contract behavior when it's used as `tx.to`. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ------------------------------------------- | ------ | ----- | +| `new_system_contract/test/call_contexts/tx_entry` | System Contract as transaction entry-point. | | | + +##### Initcode call + +Verify calling the opcode during initcode execution of a new contract. + +| ID | Description | Status | Tests | +| -------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/call_contexts/initcode/CREATE` | Call from Initcode initiated from a CREATE/CREATE2 opcode. | | | +| `new_system_contract/test/call_contexts/initcode/tx` | Call from Initcode initiated from a contract-creating transaction (`tx.to==None`). | | | + +##### System Contract as Set-code Delegated Address + +Test setting the system contract as a set-code delegated address, and verify no system contract side-effects are triggered, even if the actual system contract logic is executed. + +If the system contract requires specific storage pre-conditions to be set for proper execution (e.g. if the contract contains a safety mechanism that prevents it from executing prior to the fork activation), ensure the delegated account has the proper values in the storage to guarantee the correct logic is executed. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | --------------------------- | ------ | ----- | +| `new_system_contract/test/call_contexts/set_code` | Set code delegated address. | | | + +#### Inputs + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ------------------------------------------------------------------------ | ------ | ----- | +| `new_system_contract/test/inputs/valid` | Verify combinations of valid inputs to the system contract. | | | +| `new_system_contract/test/inputs/boundary` | Verify all boundary values given the system contract functionality. | | | +| `new_system_contract/test/inputs/all_zeros` | Verify all zeros input. | | | +| `new_system_contract/test/inputs/max_values` | Verify 2^N-1 where N is a single or multiple valid bit-lengths. | | | +| `new_system_contract/test/inputs/invalid` | Verify combinations of invalid inputs to the precompile. | | | +| `new_system_contract/test/inputs/invalid/checks` | Inputs that fail validity checks. | | | +| `new_system_contract/test/inputs/invalid/crypto` | Inputs that fail specific mathematical or cryptographic validity checks. | | | +| `new_system_contract/test/inputs/invalid/corrupted` | Inputs that are malformed/corrupted. | | | + +#### Value Transfer + +##### Minimum Fee System Contract + +If the system contract requires a minimum value (fee) to execute, either constant or depending on a formula. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ---------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/value_transfer/fee/under` | Calls with the required value amount minus one, expect failure. | | | +| `new_system_contract/test/value_transfer/fee/exact` | Calls with the exact required amount, expect success. | | | +| `new_system_contract/test/value_transfer/fee/over` | Calls with extra value than the required amount, expect success. | | | + +##### No-Fee System Contract + +If the system contract does not require any minimum value (fee) to execute. + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/value_transfer/no_fee` | Sending non-zero value does not cause an exception (unless otherwise specified by the EIP). | | | + +#### Out-of-bounds checks + +If the system contract has out-of-bounds conditions in its inputs. + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | ----------------------------- | ------ | ----- | +| `new_system_contract/test/out_of_bounds/max` | Max value for each input. | | | +| `new_system_contract/test/out_of_bounds/max_plus_one` | Max value + 1 for each input. | | | + +#### Input Lengths + +Verify different combinations of input lengths to the system contract, ensuring the correct minimum fee (if any) is provided. + +##### Zero-length Input + +Regardless of the input requirements for the system contract. + +| ID | Description | Status | Tests | +| --------------------------------------------- | --------------------- | ------ | ----- | +| `new_system_contract/test/input_lengths/zero` | Zero-length calldata. | | | + +##### Static Required Input Length + +If the system contract has a static required input length. + +| ID | Description | Status | Tests | +| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------ | ----- | +| `new_system_contract/test/input_lengths/static/correct` | Correct static-length calldata. | | | +| `new_system_contract/test/input_lengths/static/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | +| `new_system_contract/test/input_lengths/static/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | + +##### Dynamic Required Input Length + +If the system contract has a variable required input-length based on a formula, test all listed scenarios given different input lengths. + +| ID | Description | Status | Tests | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------ | ----- | +| `new_system_contract/test/input_lengths/dynamic/valid` | Verify correct system contract execution for valid lengths. | | | +| `new_system_contract/test/input_lengths/dynamic/too_short` | Calldata too short, where the value represents a correct but truncated input to the contract. | | | +| `new_system_contract/test/input_lengths/dynamic/too_long` | Calldata too long, where the value represents a correct input to the contract with padded zeros. | | | + +#### Gas usage + +##### Constant Gas Cost + +If the system contract always charges the same gas cost regardless of input conditions. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/gas_usage/constant/exact` | Verify exact gas consumption. | | | +| `new_system_contract/test/gas_usage/constant/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +##### Variable Gas Cost + +If the system contract charges variable gas cost given input conditions, test all listed scenarios using multiple different valid inputs. + +| ID | Description | Status | Tests | +| -------------------------------------------------- | ------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/gas_usage/dynamic/exact` | Verify exact gas consumption. | | | +| `new_system_contract/test/gas_usage/dynamic/oog` | Verify exact gas consumption minus one results in out-of-gas error. | | | + +#### Excessive Gas Cases + +##### Excessive Gas Usage During Block Execution + +Verify spending all block gas in calls to system contract (Use `Environment().gas_limit` as reference max amount). + +| ID | Description | Status | Tests | +| -------------------------------------------------- | ------------------------ | ------ | ----- | +| `new_system_contract/test/excessive_gas/block_gas` | Exhaust block gas limit. | | | + +##### Excessive Gas Usage During System Call + +If possible, produce a scenario where, given all transactions executed within the block result in the execution of the contract by the system address would result in excessive gas usage. + +| ID | Description | Status | Tests | +| ---------------------------------------------------- | ----------------------------- | ------ | ----- | +| `new_system_contract/test/excessive_gas/system_call` | Excessive gas on system call. | | | + +#### System Contract Deployment + +| ID | Description | Status | Tests | +| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/deployment/missing` | Verify block execution behavior after fork activation if the system contract has not been deployed (Depending on the EIP, block could be invalid). | | | +| `new_system_contract/test/deployment/address` | Verify deployment transaction results in the system contract being deployed to the expected address. | | | + +#### Contract Variations + +Verify execution of the different variations of the contract for different networks (if any) results in the expected behavior. + +| ID | Description | Status | Tests | +| ------------------------------------------------------- | -------------------------------------- | ------ | ----- | +| `new_system_contract/test/contract_variations/networks` | Different network contract variations. | | | + +#### Contract Substitution + +Substitute the default system contract with a mock contract to modify its behavior when called by the system address (at the end of the block execution). + +| ID | Description | Status | Tests | +| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------ | ----- | +| `new_system_contract/test/contract_substitution/return_lengths` | Modified return value lengths. | | | +| `new_system_contract/test/contract_substitution/logs` | Modified to emit logs or modified logs. | | | +| `new_system_contract/test/contract_substitution/exception` | Modified to cause an exception (e.g. invalid opcode). | | | +| `new_system_contract/test/contract_substitution/gas_limit_success` | Modified to consume 30,000,000 million gas exactly, execution should be successful. | | | +| `new_system_contract/test/contract_substitution/gas_limit_failure` | Modified to consume 30,000,001 million gas exactly, execution should fail. | | | + +#### Fork transition + +Verify calling the system contract before its activation fork results in correct behavior (depends on the system contract implementation). + +| ID | Description | Status | Tests | +| ----------------------------------------------------------- | --------------------------------- | ------ | ----- | +| `new_system_contract/test/fork_transition/call_before_fork` | Call system contract before fork. | | | + +### Framework Changes + +- Add system contract address to relevant methods in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` +- Add system contract bytecode to the returned value of `pre_allocation_blockchain` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` + +## New Transaction Type + +### Test Vectors + +#### Intrinsic Validity + +Verify the transaction (and the block it is included in) is valid or invalid given the intrinsic validity of the new transaction type, depending on each test scenario. + +For each new field that affects the intrinsic gas cost of the transaction verify all listed scenarios. + +##### Gas Limit + +Note: Data floor gas cost affects the intrinsic validity of all transaction types, so it must be taken into account when designing all test scenarios. + +| ID | Description | Status | Tests | +| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/gas_limit/exact` | Provide the exact intrinsic gas as `gas_limit` value to the transaction. | | | +| `new_transaction_type/test/intrinsic_validity/gas_limit/insufficient` | Provide the exact intrinsic gas minus one as `gas_limit` value to the transaction. | | | + +##### Gas Fee + +| ID | Description | Status | Tests | +| -------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_lower_than_max_fee` | Invalid if `tx.max_priority_fee_per_gas < tx.max_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/max_priority_equal_to_max_fee` | Valid if `tx.max_priority_fee_per_gas == tx.max_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/base_lower` | Invalid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | +| `new_transaction_type/test/intrinsic_validity/max_fee/base_equal` | Valid if `tx.max_fee_per_gas < block.base_fee_per_gas`. | | | + +##### Chain ID + +| ID | Description | Status | Tests | +| ------------------------------------------------------- | --------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/chain_id` | Invalid if `tx.chain_id != network.chain_id`. | | | + +##### Nonce + +| ID | Description | Status | Tests | +| -------------------------------------------------------------- | ------------------------------------------ | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/nonce_minus_one` | Invalid if `tx.nonce == sender.nonce - 1`. | | | +| `new_transaction_type/test/intrinsic_validity/nonce_plus_one` | Invalid if `tx.nonce == sender.nonce + 1`. | | | +| `new_transaction_type/test/intrinsic_validity/nonce_exact` | Valid if `tx.nonce == sender.nonce`. | | | + +##### To + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/to` | Valid/Invalid if `tx.to == None`, depending on the EIP. | | | + +##### Value + +| ID | Description | Status | Tests | +| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/value_non_zero_insufficient_balance` | Invalid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | +| `new_transaction_type/test/intrinsic_validity/value_non_zero_sufficient_balance` | Valid if `tx.value == 1` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) + 1`. | | | +| `new_transaction_type/test/intrinsic_validity/value_zero_insufficient_balance` | Invalid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price) - 1`. | | | +| `new_transaction_type/test/intrinsic_validity/value_zero_sufficient_balance` | Valid if `tx.value == 0` and `account.balance == (tx.max_fee_per_gas * tx.gas_price)`. | | | + +##### Data + +| ID | Description | Status | Tests | +| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Invalid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == tx.intrinsic_gas_cost`. | | | +| `new_transaction_type/test/intrinsic_validity/data_floor_above_intrinsic_gas_cost` | Valid if `data_floor_cost(len(tx.data)) > tx.intrinsic_gas_cost` and `tx.gas_limit == data_floor_cost(len(tx.data))`. | | | + +#### Signature + +Verify the transaction is correctly rejected if it contains an invalid signature. + +| ID | Description | Status | Tests | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/signature/invalid/field_outside_curve` | V, R, S represent a value that is inside of the field but outside of the curve. | | | + +##### V + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ---------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/signature/invalid/v/2` | `2`. | | | +| `new_transaction_type/test/signature/invalid/v/27` | `27` (Type-0 transaction valid value). | | | +| `new_transaction_type/test/signature/invalid/v/28` | `28` (Type-0 transaction valid value). | | | +| `new_transaction_type/test/signature/invalid/v/35` | `35` (Type-0 replay-protected transaction valid value for chain id 1). | | | +| `new_transaction_type/test/signature/invalid/v/36` | `36` (Type-0 replay-protected transaction valid value for chain id 1). | | | +| `new_transaction_type/test/signature/invalid/v/max` | `2**8-1`. | | | + +##### R + +| ID | Description | Status | Tests | +| -------------------------------------------------------------------- | --------------- | ------ | ----- | +| `new_transaction_type/test/signature/invalid/r/0` | `0`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n_minus_one` | `SECP256K1N-1`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n` | `SECP256K1N`. | | | +| `new_transaction_type/test/signature/invalid/r/secp256k1n_plus_one` | `SECP256K1N+1`. | | | +| `new_transaction_type/test/signature/invalid/r/max_minus_one` | `2**256-1`. | | | +| `new_transaction_type/test/signature/invalid/r/max` | `2**256`. | | | + +##### S + +| ID | Description | Status | Tests | +| ------------------------------------------------------------------------- | -------------------------------------- | ------ | ----- | +| `new_transaction_type/test/signature/invalid/s/0` | `0`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_minus_one` | `SECP256K1N//2-1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half` | `SECP256K1N//2`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_half_plus_one` | `SECP256K1N//2+1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_minus_one` | `SECP256K1N-1`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n` | `SECP256K1N`. | | | +| `new_transaction_type/test/signature/invalid/s/secp256k1n_plus_one` | `SECP256K1N+1`. | | | +| `new_transaction_type/test/signature/invalid/s/max_minus_one` | `2**256-1`. | | | +| `new_transaction_type/test/signature/invalid/s/max` | `2**256`. | | | +| `new_transaction_type/test/signature/invalid/s/complement` | `SECP256K1N - S` of a valid signature. | | | + +#### Transaction Attributes Readable From EVM + +| ID | Description | Status | Tests | +| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/tx_scoped_attributes/read` | Verify attributes that can be read in the EVM from transaction fields. | | | +| `new_transaction_type/test/tx_scoped_attributes/older_tx_types` | Verify attributes specific to the new transaction type that can be read in the EVM behave correctly on older transaction types. | | | + +#### Transaction-Scoped Persistent Values + +Verify values or variables that are persistent through the execution of the transaction (e.g. transient storage, warm/cold accounts). + +| `new_transaction_type/test/tx_scoped_attributes/persistent/throughout` | Persist throughout the entire transaction. | | | +| `new_transaction_type/test/tx_scoped_attributes/persistent/reset` | Reset on subsequent transactions in the same block. | | | + +#### Encoding (RLP, SSZ) + +Verify correct transaction rejection in all listed scenarios. + +##### Field Sizes + +Verify all listed scenarios for each transaction field. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------- | ------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/encoding/field_sizes/leading_zero` | Add leading zero byte. | | | +| `new_transaction_type/test/encoding/field_sizes/remove_byte` | Remove single byte from fixed-byte-length fields. | | | + +##### Fields of List Type + +Verify for each transaction field that is of type list. + +| ID | Description | Status | Tests | +| ------------------------------------------------------------ | --------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/encoding/list_field/zero` | Zero-element list (Failure depending on EIP). | | | +| `new_transaction_type/test/encoding/list_field/max` | Max count list. | | | +| `new_transaction_type/test/encoding/list_field/max_plus_one` | Max count plus one list. | | | + +##### Extra/Missing Fields + +| ID | Description | Status | Tests | +| --------------------------------------------------- | --------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/encoding/missing_fields` | Any fields particular to the new transaction types are missing. | | | +| `new_transaction_type/test/encoding/extra_fields` | Transaction contains extra fields. | | | + +##### Serialization Corruption + +| ID | Description | Status | Tests | +| ------------------------------------------------ | ------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/encoding/truncated` | Serialized bytes object is truncated by one byte. | | | +| `new_transaction_type/test/encoding/extra_bytes` | Serialized bytes object has one extra byte. | | | + +##### Serializable Fields + +Verify for each serializable field, all previous tests plus following listed scenario. + +| ID | Description | Status | Tests | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------ | ----- | +| `new_transaction_type/test/encoding/new_types/incorrect_encoding` | Serializable field is encoded as bytes instead of using the correct encoding (e.g. list in the case of RLP). | | | + +#### Out-of-bounds checks + +Verify if the transaction has out-of-bounds conditions in its fields and verify. + +| ID | Description | Status | Tests | +| ------------------------------------------------------ | ----------------------------- | ------ | ----- | +| `new_transaction_type/test/out_of_bounds/max` | Max value for each field. | | | +| `new_transaction_type/test/out_of_bounds/max_plus_one` | Max value + 1 for each field. | | | + +#### Contract creation + +Verify that the transaction can create new contracts if the transaction type supports it. + +| ID | Description | Status | Tests | +| --------------------------------------------- | -------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/contract_creation` | Create contracts using new transaction type. | | | + +#### Sender account modifications + +| ID | Description | Status | Tests | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/sender_account/nonce` | Sender account has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). | | | +| `new_transaction_type/test/sender_account/balance` | Sender account has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). | | | + +#### Block Level Interactions + +##### Single Transaction In Block + +Verify a block where the new transaction type is the sole transaction contained in the block. + +| ID | Description | Status | Tests | +| ---------------------------------------------------------------- | ------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/block_interactions/single_tx/invalid` | Invalid if `tx.gas_limit == block.gas_limit + 1`. | | | +| `new_transaction_type/test/block_interactions/single_tx/valid` | Valid if `tx.gas_limit == block.gas_limit`. | | | + +##### Two Transactions In Block + +Verify a block where the new transaction type is the last transaction contained in a block with two transactions. + +| ID | Description | Status | Tests | +| -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/block_interactions/last_tx/valid` | Valid if `block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit`. | | | +| `new_transaction_type/test/block_interactions/last_tx/invalid` | Invalid if `(block.txs[0].gas_used + block.txs[1].gas_limit == block.gas_limit + 1) and (block.txs[0].gas_used < block.gas_limit)`. | | | + +##### EIP-7825 + +Verify a transaction of the new type is rejected if its gas limit exceeds the [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825) gas limit for the current fork. + +| ID | Description | Status | Tests | +| -------------------------------------------------------------- | -------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/block_interactions/eip7825/invalid` | Exceeds EIP-7825 gas limit by one. | | | +| `new_transaction_type/test/block_interactions/eip7825/valid` | Gas limit is exactly the EIP-7825 gas limit. | | | + +##### Mixed transactions + +Verify a block with all transactions types including the new type is executed correctly. + +| ID | Description | Status | Tests | +| -------------------------------------------------------- | ------------------- | ------ | ----- | +| `new_transaction_type/test/block_interactions/mixed_txs` | Mixed transactions. | | | + +#### Fork transition + +Verify that a block prior to fork activation where the new transaction type is introduced and containing the new transaction type is invalid. + +| ID | Description | Status | Tests | +| -------------------------------------------------- | ----------------------------------------------------------- | ------ | ----- | +| `new_transaction_type/test/fork_transition/before` | New transaction type included before fork activation block. | | | + +#### RPC Tests + +- \*Verify `eth_estimateGas` behavior for different valid combinations of the new transaction type. +- `new_transaction_type/test/rpc/send_raw` | Verify `eth_sendRawTransaction` using `execute`. + +\*Tests must be added to [`execution-apis`](https://github.com/ethereum/execution-apis) repository. + +### Framework Changes + +- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py`, adding the appropriate new fields that the transaction introduced and the logic to the intrinsic gas cost calculation, if any. +- Add the transaction type number to `tx_types` response in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` (If applicable add also to `contract_creating_tx_types`). + +## New Block Header Field + +### Test Vectors + +#### Genesis value + +Verify, if possible, that the value can be set at genesis if the network starting fork is the activation fork, and that clients can consume such genesis. + +| ID | Description | Status | Tests | +| ------------------------------------- | ---------------------------------- | ------ | ----- | +| `new_block_header_field/test/genesis` | New block header field at genesis. | | | + +#### Value behavior + +Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ----- | +| `new_block_header_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | +| `new_block_header_field/test/value_behavior/reject` | Block is rejected if the value is modified (using `block.rlp_modifier`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | + +#### Fork transition + +| ID | Description | Status | Tests | +| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_block_header_field/test/fork_transition/initial` | Verify initial value of the field at the first block of the activation fork. | | | +| `new_block_header_field/test/fork_transition/before` | Verify that a block containing the new header field before the activation of the fork is invalid. | | | +| `new_block_header_field/test/fork_transition/after` | Verify that a block lacking the new header field at the activation of the fork is invalid. | | | + +### Framework Changes + +- Add the new header field to the relevant objects: + - `ethereum_test_fixtures.FixtureHeader`. + - `ethereum_test_fixtures.FixtureExecutionPayload`. + - `ethereum_test_specs.Header`. +- Add the appropriate `header_*_required` fork method to `BaseFork` in `ethereum_test_forks`. + +## New Block Body Field + +### Test Vectors + +#### Value behavior + +Verify, given multiple initial values, that a block is accepted or rejected depending on the correct expected value for the current block. + +| ID | Description | Status | Tests | +| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_block_body_field/test/value_behavior/accept` | Block is accepted if the value is the correct expected for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | +| `new_block_body_field/test/value_behavior/reject` | Block is rejected if the value is modified (using appropriate `block`) to an incorrect value for the current block, depending on the circumstances that affect the value as defined in the EIP. | | | + +#### Fork transition + +| ID | Description | Status | Tests | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_block_body_field/test/fork_transition/before` | Verify that a block containing the new block body field before the activation of the fork is invalid. | | | +| `new_block_body_field/test/fork_transition/after` | Verify that a block lacking the new block field at the activation of the fork is invalid. | | | + +### Framework Changes + +- Add the new body field to the relevant objects. + - `ethereum_test_fixtures.FixtureBlockBase`. + - `ethereum_test_fixtures.FixtureEngineNewPayload`. + - `ethereum_test_specs.Block`. +- Modify `ethereum_test_specs.BlockchainTest` filling behavior to account for the new block field. + +## Gas Cost Changes + +### Test Vectors + +#### Gas Usage + +Measure and store the gas usage during the operations affected by the gas cost changes and verify the updated behavior. + +| ID | Description | Status | Tests | +| ----------------------------------------------- | -------------------------- | ------ | ----- | +| `gas_cost_changes/test/gas_updates_measurement` | Measure updated gas costs. | | | + +#### Out-of-gas + +Verify the operations affected by the gas cost changes can run out-of-gas with the updated limits. + +| ID | Description | Status | Tests | +| ---------------------------------- | ------------------------------- | ------ | ----- | +| `gas_cost_changes/test/out_of_gas` | Out-Of-Gas with new gas prices. | | | + +#### Fork transition + +Verify gas costs are updated at the fork transition boundary. + +| ID | Description | Status | Tests | +| ---------------------------------------------- | ----------------------------------------------------- | ------ | ----- | +| `gas_cost_changes/test/fork_transition/before` | Costs unaffected before the fork activation block. | | | +| `gas_cost_changes/test/fork_transition/after` | Costs are updated on and after fork activation block. | | | + +### Framework Changes + +- Modify `transaction_intrinsic_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects intrinsic gas cost calculation. +- Modify `transaction_data_floor_cost_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects calldata floor cost. +- Modify `memory_expansion_gas_calculator` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects memory expansion gas cost calculation. +- Modify `gas_costs` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects specific opcode gas costs. + +## Gas Refunds Changes + +### Test Vectors + +#### Refund calculation + +Verify that the refund does not exceed `gas_used // MAX_REFUND_QUOTIENT` (`MAX_REFUND_QUOTIENT==5` in [EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)) in the following scenarios. + +| ID | Description | Status | Tests | +| --------------------------------------------------- | ------------------------------------------------ | ------ | ----- | +| `gas_refunds_changes/test/refund_calculation/over` | `refund == gas_used // MAX_REFUND_QUOTIENT + 1`. | | | +| `gas_refunds_changes/test/refund_calculation/exact` | `refund == gas_used // MAX_REFUND_QUOTIENT`. | | | +| `gas_refunds_changes/test/refund_calculation/under` | `refund == gas_used // MAX_REFUND_QUOTIENT - 1`. | | | + +#### Exceptional Abort + +| ID | Description | Status | Tests | +| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `gas_refunds_changes/test/exceptional_abort/revertable` | If the operation causing the refund can be reverted, verify the refund is not applied if the following cases:. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/revert` | `REVERT`. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/out_of_gas` | Out-of-gas. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/invalid_opcode` | Invalid opcode. | | | +| `gas_refunds_changes/test/exceptional_abort/revertable/upper_revert` | `REVERT` of an upper call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable` | If the operation causing the refund cannot be reverted (e.g. in the case of a transaction-scoped operation such as authorization refunds in EIP-7702), verify the refund is still applied even in the following cases:. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/revert` | `REVERT` at the top call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/out_of_gas` | Out-of-gas at the top call frame. | | | +| `gas_refunds_changes/test/exceptional_abort/non_revertable/invalid_opcode` | Invalid opcode at the top call frame. | | | + +#### Cross-Functional Test + +Verify the following tests are updated to support the new type of refunds. + +| ID | Description | Status | Tests | +| --------------------------------------------------------- | -------------------------------------------------------------- | ------ | ----- | +| `gas_refunds_changes/test/cross_functional/calldata_cost` | `tests/prague/eip7623_increase_calldata_cost/test_refunds.py`. | | | + +### Framework Changes + +N/A + +## Blob Count Changes + +### Test Vectors + +#### Existing Test Updates + +Verify tests in `tests/cancun/eip4844_blobs` were correctly and automatically updated to take into account the new blob count values at the new fork activation block. + +| ID | Description | Status | Tests | +| ----------------------------------------------- | --------------------------------------------------------------- | ------ | ----- | +| `blob_count_changes/test/eip4844_blobs_changes` | Updates to `tests/cancun/eip4844_blobs` were applied correctly. | | | + +### Framework Changes + +- Modify `blob_base_fee_update_fraction`, `target_blobs_per_block`, `max_blobs_per_block` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` if the EIP affects any of the values returned by each function. + +## New Execution Layer Request + +### Test Vectors + +#### Cross-Request-Type Interaction + +| ID | Description | Status | Tests | +| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | +| `new_execution_layer_request/test/cross_request_type/update` | Update `tests/prague/eip7685_general_purpose_el_requests` tests to include the new request type in the tests combinations. | | | + +### Framework Changes + +- Increment `max_request_type` in the fork where the EIP is introduced in `src/ethereum_test_forks/forks/forks.py` to the new maximum request type number after the EIP is activated. diff --git a/docs/checklist_templates/index.md b/docs/writing_tests/checklist_templates/index.md similarity index 100% rename from docs/checklist_templates/index.md rename to docs/writing_tests/checklist_templates/index.md diff --git a/whitelist.txt b/whitelist.txt index b35ba3dc29c..951f003d044 100644 --- a/whitelist.txt +++ b/whitelist.txt @@ -996,4 +996,5 @@ uncomp isogeny codomain nametag -CFI'd \ No newline at end of file +CFI'd +opcode's \ No newline at end of file From 8465ccab87c791e953c67daae2f8192037da4c50 Mon Sep 17 00:00:00 2001 From: danceratopz Date: Wed, 14 May 2025 19:01:18 +0200 Subject: [PATCH 31/33] chore(docs): more consistent display of opcodes; fix capitalization --- .../checklist_templates/eip_testing_checklist_template.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md index 76f0fa1d4bc..c868d0d5100 100644 --- a/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md +++ b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md @@ -23,7 +23,7 @@ The EIP introduces one or more new opcodes to the EVM. #### Memory expansion -If the opcode execution can expand the memory size, either by writing to memory or reading from an offset that exceeds current memory, or interaction of both parameters (Size of zero should never result in memory expansion, regardless of offset value). +If the opcode execution can expand the memory size, either by writing to memory or reading from an offset that exceeds current memory, or interaction of both parameters (size of zero should never result in memory expansion, regardless of offset value). | ID | Description | Status | Tests | | ------------------------------------------------ | ---------------------------------------------------------------- | ------ | ----- | @@ -52,7 +52,7 @@ If the opcode pushes one or more items to the stack, and the opcode pushes more ##### Stack Underflow -If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. Op.DUPN requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. +If the opcode pops one or more items to the stack, or it has a minimum stack height of one or more (e.g. `DUPN` requires a minimum amount of elements in the stack even when it does not pop any element from it), verify that the opcode execution results in exceptional abort then stack has 1 less item than the minimum stack height expected. | ID | Description | Status | Tests | | --------------------------------- | ---------------- | ------ | ----- | @@ -97,7 +97,7 @@ Verify opcode operation in a subcall frame originated from a `STATICCALL` opcode | ID | Description | Status | Tests | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ----- | | `new_opcode/test/execution_context/staticcall/ban_check` | Verify exceptional abort if the opcode attempts to modify the code, storage or balance of an account. | | | -| `new_opcode/test/execution_context/staticcall/ban_no_modification` | If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. Op.PAY with zero value, or Op.SSTORE to the value it already has in the storage). | | | +| `new_opcode/test/execution_context/staticcall/ban_no_modification` | If the opcode is completely banned from static contexts, verify that even when its inputs would not cause any account modification, the opcode still results in exceptional abort of the execution (e.g. `PAY` with zero value, or `SSTORE` to the value it already has in the storage). | | | | `new_opcode/test/execution_context/staticcall/sub_calls` | Verify sub-calls using other opcodes (e.g. `CALL`, `DELEGATECALL`, etc) also results in the same exceptional abort behavior. | | | ##### `DELEGATECALL` @@ -926,7 +926,7 @@ Verify that the transaction can create new contracts if the transaction type sup | ID | Description | Status | Tests | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----- | | `new_transaction_type/test/sender_account/nonce` | Sender account has its nonce incremented at least by one after the transaction is included in a block (or more if the transaction type introduces a new mechanism that bumps the nonce by more than one). | | | -| `new_transaction_type/test/sender_account/balance` | Sender account has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using Op.BALANCE). | | | +| `new_transaction_type/test/sender_account/balance` | Sender account has its balance reduced by the correct amount (gas consumed and value) at the start of execution (e.g. using `BALANCE`). | | | #### Block Level Interactions From 82aef1d454b42d75c2a7a89977e92c696871e1df Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Wed, 14 May 2025 17:55:31 +0000 Subject: [PATCH 32/33] fix(docs): Add fuzzing snippet --- .../checklist_templates/eip_testing_checklist_template.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md index c868d0d5100..bd63ee657c2 100644 --- a/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md +++ b/docs/writing_tests/checklist_templates/eip_testing_checklist_template.md @@ -15,6 +15,10 @@ Depending on the changes introduced by an EIP, the following template is the min #### Fuzzing +Fuzzing is recommended to be performed on EIPs that introduce new cryptography primitives. + +See [holiman/goevmlab](https://github.com/holiman/goevmlab) for an example of a fuzzing framework for the EVM. + ## New Opcode The EIP introduces one or more new opcodes to the EVM. From 9da1c93cbd8fce18ce0730b599280966a633b80f Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Wed, 14 May 2025 18:00:24 +0000 Subject: [PATCH 33/33] docs: Changelog --- docs/CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ebf85c01bd4..f10ff261707 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -19,6 +19,9 @@ Test fixtures for use by clients are available for each release on the [Github r ### 📋 Misc +- ✨ Added the [EIP checklist template](https://eest.ethereum.org/main/writing_tests/checklist_templates/eip_testing_checklist_template/) that serves as a reference to achieve better coverage when implementing tests for new EIPs ([#1327](https://github.com/ethereum/execution-spec-tests/pull/1327)). +- ✨ Added [Post-Mortems of Missed Test Scenarios](https://eest.ethereum.org/main/writing_tests/post_mortems/) to the documentation that serves as a reference list of all cases that were missed during the test implementation phase of a new EIP, and includes the steps taken in order to prevent similar test cases to be missed in the future ([#1327](https://github.com/ethereum/execution-spec-tests/pull/1327)). + ### 🧪 Test Cases ## [v4.5.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.5.0) - 2025-05-14