Xiangshan running Spec2006 score not matching on Bi-Week report

[Jerry-Tianchen]

Hi I was working on replicating xiangshan V3's spec06 score as shown in the Bi-week report. I have been able to go through the entire process of profiling and checkpointing, and eventually running on emu and speculate the score for the astar and gcc benchmark.

The score is pretty similar to the one in the Bi-Week Report.

I was wondering if Xiangshan could share theinitramfs-spec.txtand run script for the rest of benchmarks in spec2006. I can try to do it one by one on my own, but just to ensure consistency of benchmark such as perlbench, which require many libs. I am not sure if Xiangshan include all the libs for perl in the CPU2006LiteWrapper/cpu2006_run_dir/perlbench/data/all/input/lib

Just for score consistency purpose. Would appreciate it a lot xD.

[Jerry-Tianchen]

Any way, Since I am a bit short of time, I directly proceed and just assume "Every file are needed" and build the initramfs.
I am able to then run profiling/clustering/checkpointing with interval 1M and coverage rate 100%
I use maxK = 30
The average results variance is < 0.03

Then I am able to simulate Xiangshan with all the checkpoints generated, with Warmup instruction counts 100K, and 1M instructions per checkpoint. I am using Default Xiangshan Config (make emu)

The Spec2006 was compiled with according to CPU2006LiteWarpper with jemalloc + RISCVGCB, without vector:
make ARCH=riscv64 CROSS_COMPILE=riscv64-unknown-linux-gnu- OPTIMIZE="-O3 -flto -march=rv64gc_zba_zbb_zbc_zbs" build-int -j8, with gcc15

Strangely, after simulation, all the spec INT score I got is slightly (1-14%, 471.omnetpp by 40%) off than the score in Xiangshan's BI-Week in April(I use commit from April). I double-check the config of Xiangshan on Bi-Weel Report, which do match the default Xiangshan Config

May I ask guys if you have any insight on what might be causing this?

[Jerry-Tianchen]

I just realized we have Deterload Repo. I will compare my checkpointing method with the one in the repo and update perhaps later.

[eastonman]

Sorry for the late response. I can provide some information about how the checkpoint used in the biweekly report is generated.

First, I believe that what is included in initramfs does not affect the score. DeterLoad seems to include all files as well and can produce very similar score results to the checkpoint used in the biweekly report.

Second, regarding your parameter settings: a 1M interval with a 100K warmup is way too small for evaluating the performance of modern out-of-order processors. Typically, we need at least 10–20M instructions to warm up. With insufficient warmup, memory- or branch-intensive applications (such as mcf or omnetpp) tend to have significantly lower estimated performance. In the biweekly report, we use a 20M interval with a 20M warmup. If you want to save simulation time while maintaining accurate estimation, you can try specifying different intervals and warmup periods for different workloads (this is not currently supported by any XiangShan official scipts, you may have to modified them).

I hope this is helpful to you.

[eastonman]

There are some minor differences between the checkpoint we used in biweekly and DeterLoad.

1. For test case with multiple input data, the checkpoint process we used in the past (which is how the checkpoint in biweekly generated) clusters simpoints separately on each input data. While in DeterLoad, the whole test case is clustered together.
2. We turned off timer interrupt in the past, while in DeterLoad, timer interrupt is now turned on as in real world.
3. glibc and maybe some other libraries in DeterLoad has function multi-versioning support. In DeterLoad generated checkpoint, some very little vector instructions will be executed although your SPEC workload isn't compile with vector extension.

The checkpoint we used is generated in 2023 using an old and manual workflow and we keep using it for consistency.

Between DeterLoad generated checkpoints with different init seed, there is a around 1% variation of SPEC INT score and 2-5% maximum difference on single workload. This variation I believe comes from simpoints algorithm. As you know, simpoint is clustering on BBV, which does not represent memory behaviour very will. So on some memory sensitive test cases, the variation is expected to be larger.

From my personal experience, if you increase maxK and the amount of seeds tried for each K on those memory sensitive test cases, score fluctuations will be lower to 2-3% maximum. But the clustering process will run much slower.

I will come back to you for the exact compiler version and options used in biweekly.

[Jerry-Tianchen]

Thank you. For now I will share the result of Deterload with you just for fun LOL:

I am pretty sure simpoint got terribly bad on hammer due to bad luck this time. I might re-cluster on hammer and ran only hammer. While 483 is unstable by nature unless with very high K value so....

May I ask if you have validated the current simpoint Spec2006's score on NanHu? I am a bit unsure which score is closer to the "Real" Performence, as Real Spec06 do multiple input in the same round LOL.

I am doing this both for a Research Purpose and for fun. I modified Boom Core to have same device tree as Xiangshan, and want to evaluate Xiangshan and Boom Core in both Simpoint Spec2006's score LOL, with the same checkpoint for fairness, and perhaps other Open-source RISCV Core.

[eastonman]

FYI the GCC version used in the biweekly checkpoint is 12.2.0. Workload are compiled with rv64gcb.

[eastonman]

Simulation estimation error is around 5-10% in Yanqihu.

Estimation error comes from mulitple misalignments between simulation environment and sillicon. For example, memory controller behaviour and DDR refreshing. 5-10% estimation error is pretty much the best you can achieve with opensource workflow. For more accurate result, you need to run more and longer checkpoints on emulators with commerical memory controllers and NoC, or run full SPEC workload on FPGA with calibrated memory controller.

If you have better idea about lowering the estimation error, you can do some research work on this topic. There are some latest research work on this too: Memory Access Vectors: Improving Sampling Fidelity for CPU Performance Simulations

[eastonman]

mcf, xalancbmk and hmmer is known to have estimation error to us. These three test cases are very sensitive to memory. I suggest increase maxK on these three.