[move-vm] Adding negative FV tests for bytecode and runtime verifier #17147

wrwg · 2025-07-23T23:42:18Z

Description

This adds a number of negatiive tests (wrong code) which are run in two ways: one with bytecode verifier enabled where we expect it to be caught in the verifier, one with bytecode verifier disabled where we expect it to be checked by paranoid mode.

One challenge with these tests is that in order to hit paranoid mode, they need to be written such that the issue is reached in an execution path, and not just by loading module code. This is why most tests are triggered via scripts supported by a helper module.

This also changes the way how the verifier is disabled by using a configuration option. The failpoints used before can't be used with parallel test execution.

How Has This Been Tested?

These are tests

Type of Change

Which Components or Systems Does This Change Impact?

wrwg · 2025-07-23T23:42:35Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

...ve-vm/transactional-tests/tests/function_values_safety/closure_assign_local_num.paranoid.exp

vineethk

Minor comments. Great to have these!

third_party/move/move-vm/transactional-tests/tests/tests.rs

third_party/move/move-ir-compiler/src/main.rs

georgemitenkov · 2025-07-25T16:39:21Z

third_party/move/move-binary-format/src/errors.rs


 pub type VMResult<T> = ::std::result::Result<T, VMError>;
 pub type BinaryLoaderResult<T> = ::std::result::Result<T, PartialVMError>;
 pub type PartialVMResult<T> = ::std::result::Result<T, PartialVMError>;

+static STABLE_TEST_DISPLAY: Lazy<Mutex<bool>> = Lazy::new(|| Mutex::new(false));


Why not OnceCell? I think it is designed exactly for this kind of config

This adds a number of tests (wrong code) which are run in two ways: one with bytecode verifier enabled where expect it to be caught in the verifier, one with bytecode verifier disabled where we expect it to be called by paranoid mode. One bug was discovered this far by this approach, where paranoid mode seems to miss a check (#17137). Also changes the way how the verifier is disabled by using a configuration option. The failpoints used before can't be used with parallel test execution.

…ing up.

github-actions · 2025-07-25T21:01:54Z

✅ Forge suite `compat` success on `8e7acd6f8692b1bf870e8232cb95eab97ab5b270` ==> `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`

Compatibility test results for 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a (PR)
1. Check liveness of validators at old version: 8e7acd6f8692b1bf870e8232cb95eab97ab5b270
compatibility::simple-validator-upgrade::liveness-check : committed: 8317.94 txn/s, latency: 4057.56 ms, (p50: 4400 ms, p70: 4600, p90: 4700 ms, p99: 5000 ms), latency samples: 275400
2. Upgrading first Validator to new version: 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2492.55 txn/s, latency: 13543.08 ms, (p50: 14900 ms, p70: 15400, p90: 15900 ms, p99: 16000 ms), latency samples: 93220
3. Upgrading rest of first batch to new version: 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2460.06 txn/s, latency: 13710.92 ms, (p50: 15100 ms, p70: 15600, p90: 16100 ms, p99: 16200 ms), latency samples: 92980
4. upgrading second batch to new version: 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 4309.83 txn/s, latency: 7899.30 ms, (p50: 8700 ms, p70: 9000, p90: 9500 ms, p99: 9800 ms), latency samples: 150840
5. check swarm health
Compatibility test for 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a passed
Test Ok

github-actions · 2025-07-25T21:04:50Z

✅ Forge suite `realistic_env_max_load` success on `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`

two traffics test: inner traffic : committed: 11415.05 txn/s, submitted: 11415.26 txn/s, expired: 0.21 txn/s, latency: 3302.28 ms, (p50: 3300 ms, p70: 3300, p90: 3600 ms, p99: 4500 ms), latency samples: 4340460
two traffics test : committed: 99.98 txn/s, latency: 850.37 ms, (p50: 800 ms, p70: 900, p90: 900 ms, p99: 2400 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.663, avg: 2.573", "ConsensusProposalToOrdered: max: 0.165, avg: 0.163", "ConsensusOrderedToCommit: max: 0.250, avg: 0.210", "ConsensusProposalToCommit: max: 0.413, avg: 0.373"]
Max non-epoch-change gap was: 1 rounds at version 4187010 (avg 0.00) [limit 4], 1.16s no progress at version 4187010 (avg 0.07s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.70s no progress at version 1964263 (avg 0.70s) [limit 16].
Test Ok

github-actions · 2025-07-25T21:11:19Z

✅ Forge suite `framework_upgrade` success on `8e7acd6f8692b1bf870e8232cb95eab97ab5b270` ==> `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`

Compatibility test results for 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a (PR)
Upgrade the nodes to version: 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1557.77 txn/s, submitted: 1561.53 txn/s, failed submission: 3.76 txn/s, expired: 3.76 txn/s, latency: 1904.04 ms, (p50: 1800 ms, p70: 2100, p90: 2400 ms, p99: 3800 ms), latency samples: 140820
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1656.37 txn/s, submitted: 1661.55 txn/s, failed submission: 5.18 txn/s, expired: 5.18 txn/s, latency: 1848.65 ms, (p50: 1800 ms, p70: 1900, p90: 2400 ms, p99: 3800 ms), latency samples: 147200
5. check swarm health
Compatibility test for 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a passed
Upgrade the remaining nodes to version: 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1706.61 txn/s, submitted: 1712.30 txn/s, failed submission: 5.70 txn/s, expired: 5.70 txn/s, latency: 1812.86 ms, (p50: 1800 ms, p70: 2000, p90: 2300 ms, p99: 2700 ms), latency samples: 149740
Test Ok

wrwg requested review from georgemitenkov, junxzm1990, vgao1996, vineethk, GotenJBZ and zi0Black July 23, 2025 23:43

wrwg commented Jul 23, 2025

View reviewed changes

...ve-vm/transactional-tests/tests/function_values_safety/closure_assign_local_num.paranoid.exp Show resolved Hide resolved

vineethk approved these changes Jul 24, 2025

View reviewed changes

third_party/move/move-vm/transactional-tests/tests/tests.rs Show resolved Hide resolved

third_party/move/move-vm/transactional-tests/tests/tests.rs Outdated Show resolved Hide resolved

wrwg mentioned this pull request Jul 25, 2025

[move-vm] Some fixes for runtime checks #17157

Merged

16 tasks

wrwg force-pushed the wrwg/fv-tests branch from 6cf6196 to dbc4907 Compare July 25, 2025 15:44

georgemitenkov approved these changes Jul 25, 2025

View reviewed changes

wrwg and others added 4 commits July 25, 2025 13:31

Fixing false positive bug #17137 (actually a bug in the tests), clean…

cb49950

…ing up.

Apply suggestions from code review

6cea1a3

Addressing reviewer comments

2d532f5

wrwg force-pushed the wrwg/fv-tests branch from 985554e to 2d532f5 Compare July 25, 2025 20:32

wrwg enabled auto-merge (squash) July 25, 2025 20:32

This comment has been minimized.

Sign in to view

wrwg merged commit e1b13f2 into main Jul 25, 2025
92 of 106 checks passed

wrwg deleted the wrwg/fv-tests branch July 25, 2025 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[move-vm] Adding negative FV tests for bytecode and runtime verifier #17147

[move-vm] Adding negative FV tests for bytecode and runtime verifier #17147

wrwg commented Jul 23, 2025 •

edited

Loading

Uh oh!

wrwg commented Jul 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

vineethk left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

georgemitenkov Jul 25, 2025

Uh oh!

wrwg Jul 25, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jul 25, 2025

Uh oh!

github-actions bot commented Jul 25, 2025

Uh oh!

github-actions bot commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

[move-vm] Adding negative FV tests for bytecode and runtime verifier #17147

[move-vm] Adding negative FV tests for bytecode and runtime verifier #17147

Conversation

wrwg commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Type of Change

Which Components or Systems Does This Change Impact?

Uh oh!

wrwg commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vineethk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

georgemitenkov Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

wrwg Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jul 25, 2025

✅ Forge suite compat success on 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a

Uh oh!

github-actions bot commented Jul 25, 2025

✅ Forge suite realistic_env_max_load success on 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a

Uh oh!

github-actions bot commented Jul 25, 2025

✅ Forge suite framework_upgrade success on 8e7acd6f8692b1bf870e8232cb95eab97ab5b270 ==> 2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a

Uh oh!

Uh oh!

Uh oh!

wrwg commented Jul 23, 2025 •

edited

Loading

wrwg commented Jul 23, 2025 •

edited

Loading

✅ Forge suite `compat` success on `8e7acd6f8692b1bf870e8232cb95eab97ab5b270` ==> `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`

✅ Forge suite `realistic_env_max_load` success on `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`

✅ Forge suite `framework_upgrade` success on `8e7acd6f8692b1bf870e8232cb95eab97ab5b270` ==> `2d532f539aaf7b71f1632e5d37b6e37e0ad3a92a`