Results from GH actions on NVIDIA_RTX4090x1

Author: arjunsuresh

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/offline/TEST01/performance/run_1/mlperf_log_detail.txt

@@ -4,7 +4,7 @@ MLPerf Results Summary
 SUT name : Server_3DUNet
 Scenario : Offline
 Mode     : PerformanceOnly
-Samples per second: 4.13119
+Samples per second: 4.13091
 Result is : VALID
   Min duration satisfied : Yes
   Min queries satisfied : Yes
@@ -13,21 +13,21 @@ Result is : VALID
 ================================================
 Additional Stats
 ================================================
-Min latency (ns)                : 298712163
-Max latency (ns)                : 676560238746
-Mean latency (ns)               : 338326669742
-50.00 percentile latency (ns)   : 338139075542
-90.00 percentile latency (ns)   : 609730279416
-95.00 percentile latency (ns)   : 642730489228
-97.00 percentile latency (ns)   : 656637634523
-99.00 percentile latency (ns)   : 670082985816
-99.90 percentile latency (ns)   : 676138334761
+Min latency (ns)                : 298535141
+Max latency (ns)                : 676605939537
+Mean latency (ns)               : 338358768315
+50.00 percentile latency (ns)   : 338208959742
+90.00 percentile latency (ns)   : 609759218730
+95.00 percentile latency (ns)   : 642763934616
+97.00 percentile latency (ns)   : 656673216829
+99.00 percentile latency (ns)   : 670124197203
+99.90 percentile latency (ns)   : 676183478194
 
 ================================================
 Test Parameters Used
 ================================================
-samples_per_query : 2753
-target_qps : 4.17174
+samples_per_query : 2756
+target_qps : 4.17593
 target_latency (ns): 0
 max_async_queries : 1
 min_duration (ms): 600000

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/offline/TEST01/performance/run_1/mlperf_log_summary.txt

@@ -1,4 +1,4 @@
 Verifying performance.
-reference score = 4.13044
-test score = 4.13119
+reference score = 4.13458
+test score = 4.13091
 TEST PASS

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/offline/TEST01/verify_performance.txt

@@ -4,38 +4,38 @@ MLPerf Results Summary
 SUT name : Server_3DUNet
 Scenario : SingleStream
 Mode     : PerformanceOnly
-90th percentile latency (ns) : 438111926
+90th percentile latency (ns) : 438341980
 Result is : VALID
   Min duration satisfied : Yes
   Min queries satisfied : Yes
   Early stopping satisfied: Yes
 Early Stopping Result:
  * Processed at least 64 queries (5934).
  * Would discard 538 highest latency queries.
- * Early stopping 90th percentile estimate: 438304240
- * Early stopping 99th percentile estimate: 504327018
+ * Early stopping 90th percentile estimate: 438520623
+ * Early stopping 99th percentile estimate: 504614656
 
 ================================================
 Additional Stats
 ================================================
 QPS w/ loadgen overhead         : 4.43
 QPS w/o loadgen overhead        : 4.43
 
-Min latency (ns)                : 28882761
-Max latency (ns)                : 514368406
-Mean latency (ns)               : 225482846
-50.00 percentile latency (ns)   : 176110159
-90.00 percentile latency (ns)   : 438111926
-95.00 percentile latency (ns)   : 503573566
-97.00 percentile latency (ns)   : 503841379
-99.00 percentile latency (ns)   : 504231354
-99.90 percentile latency (ns)   : 504874330
+Min latency (ns)                : 28868490
+Max latency (ns)                : 514085641
+Mean latency (ns)               : 225585142
+50.00 percentile latency (ns)   : 176219743
+90.00 percentile latency (ns)   : 438341980
+95.00 percentile latency (ns)   : 503845557
+97.00 percentile latency (ns)   : 504166942
+99.00 percentile latency (ns)   : 504519643
+99.90 percentile latency (ns)   : 505430196
 
 ================================================
 Test Parameters Used
 ================================================
 samples_per_query : 1
-target_qps : 4.93531
+target_qps : 4.93586
 target_latency (ns): 0
 max_async_queries : 1
 min_duration (ms): 600000

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/singlestream/TEST01/performance/run_1/mlperf_log_detail.txt

@@ -1,4 +1,4 @@
 Verifying performance.
-reference score = 437936140
-test score = 438304240
+reference score = 438082736
+test score = 438520623
 TEST PASS

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/singlestream/TEST01/performance/run_1/mlperf_log_summary.txt

@@ -2,7 +2,7 @@
 
 ## Host platform
 
-* OS version: Linux-6.8.0-53-generic-x86_64-with-glibc2.29
+* OS version: Linux-6.8.0-60-generic-x86_64-with-glibc2.29
 * CPU version: x86_64
 * Python version: 3.8.10 (default, Feb  4 2025, 15:02:54) 
 [GCC 9.4.0]
@@ -17,7 +17,7 @@ pip install -U mlcflow
 
 mlc rm cache -f
 
-mlc pull repo mlcommons@mlperf-automations --checkout=06b95fa9f0b3e5cedf5295a7b630442b2f9ffac3
+mlc pull repo mlcommons@mlperf-automations --checkout=cc1d43d1d5eeebee7efa08a1aa0f1cc62fcb1560
 
 
 ```
@@ -41,4 +41,4 @@ Model Precision: int8
 `DICE`: `0.86236`, Required accuracy for closed division `>= 0.86084`
 
 ### Performance Results 
-`Samples per second`: `4.13044`
+`Samples per second`: `4.13458`

closed/GATEOverflow/compliance/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/singlestream/TEST01/verify_performance.txt

@@ -1,8 +1,8 @@
-[2025-05-01 21:47:27,092 main.py:229 INFO] Detected system ID: KnownSystem.Nvidia_05f8e0d586fd
-[2025-05-01 21:47:27,367 harness.py:249 INFO] The harness will load 3 plugins: ['build/plugins/pixelShuffle3DPlugin/libpixelshuffle3dplugin.so', 'build/plugins/conv3D1X1X1K4Plugin/libconv3D1X1X1K4Plugin.so', 'build/plugins/conv3D3X3X3C1K32Plugin/libconv3D3X3X3C1K32Plugin.so']
-[2025-05-01 21:47:27,367 generate_conf_files.py:107 INFO] Generated measurements/ entries for Nvidia_05f8e0d586fd_TRT/3d-unet-99.9/Offline
-[2025-05-01 21:47:27,367 __init__.py:46 INFO] Running command: ./build/bin/harness_3dunet --plugins="build/plugins/pixelShuffle3DPlugin/libpixelshuffle3dplugin.so,build/plugins/conv3D1X1X1K4Plugin/libconv3D1X1X1K4Plugin.so,build/plugins/conv3D3X3X3C1K32Plugin/libconv3D3X3X3C1K32Plugin.so" --logfile_outdir="/mlc-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99.9/offline/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=43 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=8 --map_path="data_maps/kits19/val_map.txt" --mlperf_conf_path="/home/mlcuser/MLC/repos/local/cache/get-git-repo_08fd7192/inference/mlperf.conf" --tensor_path="build/preprocessed_data/KiTS19/inference/int8" --use_graphs=false --user_conf_path="/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/c2cf6864065340ebbfa6602c226164b0.conf" --unet3d_sw_gaussian_patch_path="/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_f962b684/preprocessed_data/KiTS19/etc/gaussian_patches.npy" --gpu_engines="./build/engines/Nvidia_05f8e0d586fd/3d-unet/Offline/3d-unet-Offline-gpu-b8-int8.custom_k_99_9_MaxP.plan" --max_dlas=0 --slice_overlap_patch_kernel_cg_impl=false --scenario Offline --model 3d-unet
-[2025-05-01 21:47:27,367 __init__.py:53 INFO] Overriding Environment
+[2025-06-01 06:45:54,205 main.py:229 INFO] Detected system ID: KnownSystem.Nvidia_0aef8e1065ca
+[2025-06-01 06:45:54,486 harness.py:249 INFO] The harness will load 3 plugins: ['build/plugins/pixelShuffle3DPlugin/libpixelshuffle3dplugin.so', 'build/plugins/conv3D1X1X1K4Plugin/libconv3D1X1X1K4Plugin.so', 'build/plugins/conv3D3X3X3C1K32Plugin/libconv3D3X3X3C1K32Plugin.so']
+[2025-06-01 06:45:54,486 generate_conf_files.py:107 INFO] Generated measurements/ entries for Nvidia_0aef8e1065ca_TRT/3d-unet-99.9/Offline
+[2025-06-01 06:45:54,486 __init__.py:46 INFO] Running command: ./build/bin/harness_3dunet --plugins="build/plugins/pixelShuffle3DPlugin/libpixelshuffle3dplugin.so,build/plugins/conv3D1X1X1K4Plugin/libconv3D1X1X1K4Plugin.so,build/plugins/conv3D3X3X3C1K32Plugin/libconv3D3X3X3C1K32Plugin.so" --logfile_outdir="/mlc-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99.9/offline/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=43 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=8 --map_path="data_maps/kits19/val_map.txt" --mlperf_conf_path="/home/mlcuser/MLC/repos/local/cache/get-git-repo_08fd7192/inference/mlperf.conf" --tensor_path="build/preprocessed_data/KiTS19/inference/int8" --use_graphs=false --user_conf_path="/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/25ed99cceccc4ff6bda93b605eee4f85.conf" --unet3d_sw_gaussian_patch_path="/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_f962b684/preprocessed_data/KiTS19/etc/gaussian_patches.npy" --gpu_engines="./build/engines/Nvidia_0aef8e1065ca/3d-unet/Offline/3d-unet-Offline-gpu-b8-int8.custom_k_99_9_MaxP.plan" --max_dlas=0 --slice_overlap_patch_kernel_cg_impl=false --scenario Offline --model 3d-unet
+[2025-06-01 06:45:54,486 __init__.py:53 INFO] Overriding Environment
 benchmark : Benchmark.UNET3D
 buffer_manager_thread_count : 0
 data_dir : /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_f962b684/data
@@ -11,23 +11,23 @@ gpu_copy_streams : 1
 gpu_inference_streams : 1
 input_dtype : int8
 input_format : linear
-log_dir : /home/mlcuser/MLC/repos/local/cache/get-git-repo_32fceb49/repo/closed/NVIDIA/build/logs/2025.05.01-21.47.26
+log_dir : /home/mlcuser/MLC/repos/local/cache/get-git-repo_32fceb49/repo/closed/NVIDIA/build/logs/2025.06.01-06.45.53
 map_path : data_maps/kits19/val_map.txt
 mlperf_conf_path : /home/mlcuser/MLC/repos/local/cache/get-git-repo_08fd7192/inference/mlperf.conf
 offline_expected_qps : 0.0
 precision : int8
 preprocessed_data_dir : /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_f962b684/preprocessed_data
 scenario : Scenario.Offline
 slice_overlap_patch_kernel_cg_impl : False
-system : SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='13th Gen Intel(R) Core(TM) i9-13900K', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=24, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=131.634496, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=131634496000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA GeForce RTX 4090', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=23.98828125, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=25757220864), max_power_limit=450.0, pci_id='0x268410DE', compute_sm=89): 1})), numa_conf=None, system_id='Nvidia_05f8e0d586fd')
+system : SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='13th Gen Intel(R) Core(TM) i9-13900K', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=24, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=131.63447200000002, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=131634472000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA GeForce RTX 4090', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=23.98828125, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=25757220864), max_power_limit=450.0, pci_id='0x268410DE', compute_sm=89): 1})), numa_conf=None, system_id='Nvidia_0aef8e1065ca')
 tensor_path : build/preprocessed_data/KiTS19/inference/int8
 test_mode : AccuracyOnly
 unet3d_sw_gaussian_patch_path : /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-nvidia-scratch-space_f962b684/preprocessed_data/KiTS19/etc/gaussian_patches.npy
 use_deque_limit : True
 use_graphs : False
-user_conf_path : /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/c2cf6864065340ebbfa6602c226164b0.conf
-system_id : Nvidia_05f8e0d586fd
-config_name : Nvidia_05f8e0d586fd_3d-unet_Offline
+user_conf_path : /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/25ed99cceccc4ff6bda93b605eee4f85.conf
+system_id : Nvidia_0aef8e1065ca
+config_name : Nvidia_0aef8e1065ca_3d-unet_Offline
 workload_setting : WorkloadSetting(HarnessType.Custom, AccuracyTarget.k_99_9, PowerSetting.MaxP)
 optimization_level : plugin-enabled
 num_profiles : 1
@@ -39,23 +39,23 @@ power_limit : None
 cpu_freq : None
 &&&& RUNNING MLPerf_Inference_3DUNet_Harness # ./build/bin/harness_3dunet
 [I] mlperf.conf path: /home/mlcuser/MLC/repos/local/cache/get-git-repo_08fd7192/inference/mlperf.conf
-[I] user.conf path: /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/c2cf6864065340ebbfa6602c226164b0.conf
+[I] user.conf path: /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/25ed99cceccc4ff6bda93b605eee4f85.conf
 Creating QSL.
 Finished Creating QSL.
 Setting up SUT.
 [I] [TRT] Loaded engine size: 31 MiB
-[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 79, GPU 1097 (MiB)
-[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 79, GPU 1107 (MiB)
+[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 80, GPU 1097 (MiB)
+[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 80, GPU 1107 (MiB)
 [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +29, now: CPU 0, GPU 29 (MiB)
-[I] Device:0: ./build/engines/Nvidia_05f8e0d586fd/3d-unet/Offline/3d-unet-Offline-gpu-b8-int8.custom_k_99_9_MaxP.plan has been successfully loaded.
+[I] Device:0: ./build/engines/Nvidia_0aef8e1065ca/3d-unet/Offline/3d-unet-Offline-gpu-b8-int8.custom_k_99_9_MaxP.plan has been successfully loaded.
 [E] [TRT] 3: [runtime.cpp::~Runtime::401] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::401, condition: mEngineCounter.use_count() == 1 Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.)
-[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 48, GPU 1813 (MiB)
-[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 48, GPU 1821 (MiB)
+[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 49, GPU 1813 (MiB)
+[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 49, GPU 1821 (MiB)
 [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +2218, now: CPU 0, GPU 2247 (MiB)
 [I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: true
 Finished setting up SUT.
 Starting warmup. Running for a minimum of 5 seconds.
-Finished warmup. Ran for 5.44798s.
+Finished warmup. Ran for 5.46776s.
 Starting running actual test.
 
 No warnings encountered during test.
@@ -72,8 +72,8 @@ Device Device:0 processed:
   PerSampleCudaMemcpy Calls: 43
   BatchedCudaMemcpy Calls: 0
 &&&& PASSED MLPerf_Inference_3DUNet_Harness # ./build/bin/harness_3dunet
-[2025-05-01 21:47:44,594 run_harness.py:166 INFO] Result: Accuracy run detected.
-[2025-05-01 21:47:44,594 __init__.py:46 INFO] Running command: python3 code/3d-unet/tensorrt/accuracy_kits.py --log_file /mlc-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99.9/offline/accuracy/mlperf_log_accuracy.json
+[2025-06-01 06:46:11,679 run_harness.py:166 INFO] Result: Accuracy run detected.
+[2025-06-01 06:46:11,679 __init__.py:46 INFO] Running command: python3 code/3d-unet/tensorrt/accuracy_kits.py --log_file /mlc-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99.9/offline/accuracy/mlperf_log_accuracy.json
 Loading necessary metadata...
 Loading loadgen accuracy log...
 Running postprocessing...

closed/GATEOverflow/measurements/RTX4090x1-nvidia-gpu-TensorRT-default_config/3d-unet-99.9/offline/README.md

@@ -1,6 +1,6 @@
 {
   "MLC_HOST_CPU_WRITE_PROTECT_SUPPORT": "yes",
-  "MLC_HOST_CPU_MICROCODE": "0x12b",
+  "MLC_HOST_CPU_MICROCODE": "0x12c",
   "MLC_HOST_CPU_FPU_SUPPORT": "yes",
   "MLC_HOST_CPU_FPU_EXCEPTION_SUPPORT": "yes",
   "MLC_HOST_CPU_BUGS": "spectre_v1 spectre_v2 spec_store_bypass swapgs eibrs_pbrsb rfds bhi",