High MAPE

[huydinhtran]

Hi,

So I'm following the "AccelWattch MICRO'21 Artifact Appendix Manual" using a GTX 1050Ti. The simulated power results with SASS and PTX I got were not that accurate compared to the real power results. I got around 60% MAPE on average while in the paper they stated that they got a 11±3.8% MAPE with the Pascal TITAN X. What are some potential reasons for this inaccuracy?

Thanks!

[JRPan]

Can you check how the performance counter (cache, memory, cycle, etc.) looks? Are they correlating well with the hw?

[huydinhtran]

Is that from running the "profile_validation_perf.sh"? I ran it but it gave an error below.

Traceback (most recent call last):
  File "/home/htran197/myStuff/accel-sim-framework/gpu-simulator/../util/hw_stats/run_hw.py", line 123, in <module>
    exec_path = common.file_option_test(os.path.join(edir, exe),"",this_directory)
  File "/home/htran197/myStuff/accel-sim-framework/util/hw_stats/../job_launching/common.py", line 160, in file_option_test
    raise PathMissing("Error - cannot open file {0}".format(name))
common.PathMissing: Error - cannot open file /home/htran197/myStuff/accel-sim-framework/gpu-app-collection/src/..//bin/11.0/release/backprop-rodinia-2.0-ft
Done

[JRPan]

I meant running Accel-Sim only. Without the power model.
Run ./util/job_launching/run_simulations.py -B rodinia_2.0-ft -C QV100-SASS -T ./hw_run/traces/device-<device-num>/<cuda-version>/ -N myTest, then collect HW stats ./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D <gpu-device-num-to-run-on>, and last collect results and correlate
./util/job_launching/get_stats.py -k -K -R -N all-apps | tee all-apps.csv
./util/plotting/plot-correlation.py -c all-apps.csv -H ./hw_run/QUADRO-V100/device-0/9.1/

Or you can check the results manually. The steps are documented here
https://github.com/accel-sim/accel-sim-framework

The power model depends on Accel-Sim's counters. So to get accurate power, you have to get accurate counters first.

[huydinhtran]

I was trying to run ./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D 0 . It gave me an error of GPGPU-Sim ** ERROR: Cannot open config file 'gpgpusim.config'. What are the causes for this?

[JRPan]

Start a new terminal to clear environmental variables. You sourced the setup_envrionment script, so the app is looking for gpgpu-sim instead of the real GPU.

[huydinhtran]

At ./util/job_launching/get_stats.py -k -K -R -N all-apps | tee all-apps.csv I received Could not find logfiles for job with the name "all-apps"

[JRPan]

The name is the what you set when running run_simulationr.py
For example, ./util/job_launching/run_simulations.py -B rodinia_2.0-ft -C QV100-SASS -T ./hw_run/traces/device-<device-num>/<cuda-version>/ -N myTest the name is myTest. You can ignore the -N all-apps option. It will get the latest stat if not specified.

[huydinhtran]

I'm quite confused here since we didn't touch the run_hw.py file before. Do I run ./util/hw_stats/run_hw.py -B rodinia_2.0-ft -C other_stats after the steps that you mentioned?

[huydinhtran]

So I ran ./util/hw_stats/run_hw.py -B rodinia_2.0-ft -C other_stats and this is the output I got which doesn't look similar to yours. What are the potential causes for this?

==32411== Profiling result: "Start","Duration","Grid X","Grid Y","Grid Z","Block X","Block Y","Block Z","Registers Per Thread","Static SMem","Dynamic SMem","Size","Throughput","SrcMemType","DstMemType","Device","Context","Stream","Name","Correlation_ID" us,us,,,,,,,,KB,B,KB,GB/s,,,,,,, 224831.753000,2.816000,,,,,,,,,,16.003906,5.419927,"Pageable","Device","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy HtoD]",114 224885.387000,45.058000,,,,,,,,,,272.066406,5.758417,"Pageable","Device","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy HtoD]",115 224937.806000,19.584000,1,256,1,16,16,1,20,1.062500,0,,,,,"NVIDIA GeForce GTX 1050 Ti (0)","1","7","_Z22bpnn_layerforward_CUDAPfS_S_S_ii",116 224970.927000,2.368000,,,,,,,,,,16.000000,6.443745,"Device","Pageable","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy DtoH]",119 225018.353000,0.864000,,,,,,,,,,0.066406,0.073299,"Pageable","Device","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy HtoD]",122 225051.602000,44.642000,,,,,,,,,,272.066406,5.812077,"Pageable","Device","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy HtoD]",123 225121.973000,44.802000,,,,,,,,,,272.066406,5.791321,"Pageable","Device","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy HtoD]",124 225169.847000,38.817000,1,256,1,16,16,1,20,0.000000,0,,,,,"NVIDIA GeForce GTX 1050 Ti (0)","1","7","_Z24bpnn_adjust_weights_cudaPfiS_iS_S_",125 225209.816000,2.304000,,,,,,,,,,16.003906,6.624355,"Device","Pageable","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy DtoH]",126 225223.993000,37.345000,,,,,,,,,,272.066406,6.947724,"Device","Pageable","NVIDIA GeForce GTX 1050 Ti (0)","1","7","[CUDA memcpy DtoH]",127

[JRPan]

Try./util/hw_stats/run_hw.py -B rodinia_2.0-ft --nsight_profiler --disable_nvprof -C other_stats
The steps are detailed in the readme of Accel-Sim.

[huydinhtran]

I'm not too sure what I did wrong here since I tried to follow the readme and couldn't get what you had. To make it simple, I just changed the gpgpu-sim config file of TITANX to 1050Ti and keep the same name. I also changed the file ./util/plotting/correl_mappings.py to "TITANX" : set("NVIDIA GeForce GTX 1050 Ti")
Here are the steps that I did:

1. ./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D 0
2. ./util/job_launching/run_simulations.py -B rodinia_2.0-ft -C TITANX-SASS -T ./hw_run/traces/device-0/11.0/ -N myTest
3. ./util/hw_stats/run_hw.py -B rodinia_2.0-ft -C other_stats (--nsight_profiler --disable_nvprof failed)

However, the output file in ./hw_run/ is still similar to what I had above. I might have run the steps wrong in some way that I'm not aware of.

[MWHYNOT]

@JRPan Hello, JRPan, I wonder is the command . /util/job_launching/run_simulations.py -B rodinia_2.0-ft -C QV100-SASS -T . /hw_run/traces/device-<device-num>/<cuda-version>/ -N myTest running all the rodinia_2.0-ft benchmark suite using the simulator ?
and what is the purpose of -N myTest?

When I use the command . /util/job_launching/run_simulations.py -B rodinia_2.0-ft -C QV100-SASS -T . /hw_run/traces/device-<device-num>/<cuda-version>/ -N myTest, it always appears error:
Cannot resolve default server host 'torqueserver' - check server_name file. qsub: cannot connect to server torqueserver (errno=15008) Access from host not allowed, or unknown host Job queued (backprop-rodinia-2.0-ft-4096___data_result_4096_txt QV100-SASS)

So I have to run the benchmark using the command ./justrun in the /accel-sim-framework/sim_run_11.2/bfs-rodinia-2.0-ft/__data_graph4096_txt___data_graph4096_result_txt/QV100-SASS/

I would really appreciate it if you could give me some advice

[JRPan]

You already got the simulation results. run_hw_traceis to get the traces, and run simulation with run_simulations. run_hw is not depending on any of those. So you don't need to worry about anything else other than the run_hw.
What is the fail msg? Do you have nsight compute installed? It should come with the CUDA package.

[huydinhtran]

I installed CUDA as how it is in the instruction.
When running with --nsight_profiler --disable_nvprof, I got the error here.

==ERROR== Error: ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM
==PROF== Disconnected from process 3803
==ERROR== An error occurred while trying to profile.
==WARNING== No kernels were profiled.

I'm very confused here. So I followed instructions exactly from the Artifact Manual using the old release-accelwattch branch. After receiving the SASS and PTX simulated results, I just need to run ./util/hw_stats/run_hw.py -B rodinia-3.1_validation,parboil_validation,cuda_samples_11.0_validation -C other_stats (benchmarks are from that manual) right? So we don't run the steps mentioned at the beginning of the discussion of the run_simulations.py and run_hw_trace.py scripts?

[JRPan]

I asked you to run all run_hw, run_simulations and run_hw_trace. I don't remember the exact steps in the artifact. But I asked you to do this because you said you have a high error, and I want you to check if the performance counters match with the real HW.

run_simulations is just launching the Accel-Sim to run simulations. If you have already done that, then no need to re-run the simulator.
run_hw_trace is to collect the HW traces using your physical GPU. You cannot use traces collected on a different GPU generation (for example, you cannot use an Ampere trace collected on a 3070 to simulate a 1050Ti GPU). This is not needed for PTX mode.

Now, assuming your simulation is finished with the correct traces, you have to compare the simulation results with counters from the real GPU. To collect those data, you use run_hw. The error you are having is because you don't have correct permission. To resolve that, please follow the link from the output.

After you have collected HW data, correlate HW with the simulator results. The error would probably be high. Then you need to update your config file, then see if the correlation is better.
When the correlation improves, your MAPE should also improve. Because the power is calculated based on the counters from the Accel-Sim.