Skip to content

Testing. #17079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

Testing. #17079

wants to merge 4 commits into from

Conversation

JoshLind
Copy link
Contributor

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Move Compiler
  • Other (specify)

Checklist

  • I have read and followed the CONTRIBUTING doc
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I identified and added all stakeholders and component owners affected by this change as reviewers
  • I tested both happy and unhappy path of the functionality
  • I have made corresponding changes to the documentation

@JoshLind JoshLind added the CICD:run-forge-e2e-perf Run the e2e perf forge only label Jul 16, 2025

This comment has been minimized.

This comment has been minimized.

@JoshLind JoshLind force-pushed the txn_aux_state_sync_4 branch from 3775798 to 912df72 Compare July 16, 2025 01:57

This comment has been minimized.

This comment has been minimized.

@JoshLind JoshLind added CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR and removed CICD:run-forge-e2e-perf Run the e2e perf forge only labels Jul 16, 2025
@JoshLind JoshLind force-pushed the txn_aux_state_sync_4 branch from 912df72 to d3fdfd4 Compare July 16, 2025 13:36

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

❌ Forge suite realistic_env_max_load failure on d3fdfd4d238294067bb662f809e66a1cab216db2

two traffics test: inner traffic : committed: 3934.33 txn/s, latency: 9656.11 ms, (p50: 9300 ms, p70: 9900, p90: 10300 ms, p99: 10600 ms), latency samples: 1496040
Test Failed: test NetworkLoadTest

Caused by:
    The following errors occurred:
    --------------------------------
    Error 1: The following errors occurred:
    --------------------------------
    Error 1: TPS requirement for inner traffic failed. Average TPS 3934.3251294680217, minimum TPS requirement 10000. Full stats: committed: 3934.33 txn/s, latency: 9656.11 ms, (p50: 9300 ms, p70: 9900, p90: 10300 ms, p99: 10600 ms), latency samples: 1496040
    Caused by:
    
    
    Caused by:
    
    

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/result.rs:2014:27
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:347:9
   3: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   5: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:416:18
   6: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:517:10
   7: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:275:14
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   9: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  11: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:324:22
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:323:9
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:302:18
  22: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  23: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
  24: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
  25: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  26: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
  27: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
  28: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
  29: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  30: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  31: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  32: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  33: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  34: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  35: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  36: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  37: main
  38: __libc_start_main
  39: _start
Trailing Log Lines:
  34: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  35: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  36: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  37: main
  38: __libc_start_main
  39: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="2371e020-ae6b-4220-8782-266a1a57cd65">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(network:multi-region-network-emulation(two traffics test)) with ">
            <failure message="test NetworkLoadTest

Caused by:
    The following errors occurred:
    --------------------------------
    Error 1: The following errors occurred:
    --------------------------------
    Error 1: TPS requirement for inner traffic failed. Average TPS 3934.3251294680217, minimum TPS requirement 10000. Full stats: committed: 3934.33 txn/s, latency: 9656.11 ms, (p50: 9300 ms, p70: 9900, p90: 10300 ms, p99: 10600 ms), latency samples: 1496040
    Caused by:
    
    
    Caused by:
    
    

Stack backtrace:
   0: anyhow::error::&lt;impl core::convert::From&lt;E&gt; for anyhow::Error&gt;::from
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: &lt;core::result::Result&lt;T,F&gt; as core::ops::try_trait::FromResidual&lt;core::result::Result&lt;core::convert::Infallible,E&gt;&gt;&gt;::from_residual
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/result.rs:2014:27
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:347:9
   3: &lt;aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest&gt;::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   4: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   5: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:416:18
   6: &lt;dyn aptos_testcases::NetworkLoadTest&gt;::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:517:10
   7: &lt;dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:275:14
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   9: &lt;aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  10: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  11: &lt;aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
  12: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:63
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5
  16: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:31
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  18: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:324:22
  19: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  20: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:323:9
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:302:18
  22: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  23: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
  24: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
  25: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  26: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
  27: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
  28: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
  29: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  30: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  31: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  32: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  33: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  34: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  35: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  36: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  37: main
  38: __libc_start_main
  39: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
[2025-07-16T14:09:24Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-e2e-pr-17079: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-07-16T14:09:24Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-e2e-pr-17079

failures:
    CompositeNetworkTest
Failed to run tests:
Tests Failed

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
   5: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
   9: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  10: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  12: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  13: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  14: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  15: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                         READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge5dd1ba07-0       1/1     Running     0          12m
aptos-node-0-validator-0                     1/1     Running     0          12m
aptos-node-1-fullnode-eforge5dd1ba07-0       1/1     Running     0          12m
aptos-node-1-validator-0                     1/1     Running     0          12m
aptos-node-2-fullnode-eforge5dd1ba07-0       1/1     Running     0          12m
aptos-node-2-validator-0                     1/1     Running     0          12m
aptos-node-3-fullnode-eforge5dd1ba07-0       1/1     Running     0          12m
aptos-node-3-validator-0                     1/1     Running     0          12m
aptos-node-4-fullnode-eforge5dd1ba07-0       1/1     Running     0          12m
aptos-node-4-validator-0                     1/1     Running     0          12m
aptos-node-5-validator-0                     1/1     Running     0          12m
aptos-node-6-validator-0                     1/1     Running     0          12m
forge-testnet-deployer-gdbbx                 0/1     Completed   0          13m
genesis-aptos-genesis-eforge5dd1ba07-456fb   0/1     Completed   0          12m

Copy link
Contributor

❌ Forge suite compat failure on 258fe3e0e3e3e29692d3be7dfb7c0938cdba3b34 ==> d3fdfd4d238294067bb662f809e66a1cab216db2

Forge test runner terminated:
Trailing Log Lines:
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  34: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  35: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  36: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  37: main
  38: __libc_start_main
  39: _start
Debugging output:

@JoshLind JoshLind force-pushed the txn_aux_state_sync_3 branch 2 times, most recently from f0b5ba8 to 36aed6c Compare July 22, 2025 02:12
Base automatically changed from txn_aux_state_sync_3 to main July 22, 2025 02:55
@JoshLind JoshLind closed this Jul 24, 2025
@JoshLind JoshLind deleted the txn_aux_state_sync_4 branch July 24, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant