Memory timing might be too tight

[bukatlib]

Describe the bug

I have noticed that a prime sieve might give me an incorrect count of primes mostly shortly after the boot.

I do not think that it is a SW bug since sanitizers report no errors and I observed both:

1. the incorrect count (some ones were zeroed)
2. the correct count but later write of the memory image to file contained some bits cleared (cleared between the counting and write to a binary file)

There is one example for the second case:

$ cmp -l memory_primes_right.img memory_primes_right_differ.img
5985247185 100   0
5985247187  40   0
5985247189   2   0
5985247191  21   0
5985247192  40   0
5985247193 100   0
5985247194 100   0
5985247195  42   0
5985247196   1   0
5985247197   1   0
5985247201  44   0
5985247202   4   0
5985247204   2   0
5985247209  10   0
5985247210 202   0
5985247211  54   0
5985247212  20   0
5985247213 200   0
5985247214 100   0
5985247215   2   0
5985247216 200   0

It seems to me that memory refresh interval is too long which maybe depends on the temperature or other tuning parameters. Note that memory images have 6750 MB (primes to 100 G) and I have not observed these mismatched on x86_64 architecture.

Steps to reproduce the behaviour

I may provide the source code if desired, or other memory tester writing patterns and reading them with a delay (under the full load) may help.

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf146
SDRAM_BANKLOW=1

System

Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 48efb5fc5485fafdc9de8ad481eb5c09e1182656, stage5

vcgencmd bootloader_version
2025/03/27 00:16:44
version 897f94edff080e2b325930669fa4e9caf469e090 (release)
timestamp 1743034604
update-time 1746369352
capabilities 0x0000007f

vcgencmd version
2025/03/27 00:16:44
Copyright (c) 2012 Broadcom
version 897f94ed (release) (embedded)

uname -a
Linux raspberrypi 6.12.25-bpf+ #1 SMP PREEMPT Sun May 4 15:25:58 CEST 2025 aarch64 GNU/Linux

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[stu-spp]

Yes, the source code is desired. I would like to try to reproduce your results.

[bukatlib]

Here is the code for Eratosthenes sieve:
EratosthenesSieve.zip

Steps to build it:

$ mkdir build; cd build
$ cmake ..
$ make -j4

To count primes to 100G and write 6.75 GB memory image to disk (hardcoded to memory_primes.img for testing purposes), run the following command (close memory hungry apps beforehand):

$ ./prime_sieve_bench 100000000000 -
sieve size: 49999999999 bits (6250000000 bytes)
Sieve initialized in 2.25054 secs
Primes sieved in 15.8042 secs.
Found 4118054813 primes in 0.507386 secs (bandwidth 12318 MB/s).
Binary image written in 20.9711 secs.

I expect the following sha256 hash of the memory footprint is (openssl instead of sha256sum since it is much faster):

$ openssl dgst -sha256 memory_primes_right.img
SHA2-256(memory_primes_right.img)= bc773de13417112ba5df5bd234069ea489528d3c0d4cfdc4bf8e35d438adb323

Note that the incorrect results are occasionally observed shortly after the boot.

[timg236]

It seems unlikely that it's memory timing related. Try boosting the core-voltage e.g over_voltage_delta=40000 in-case it's cache voltage margin issue on this board.

[popcornmix]

Does memtester (sudo apt install memtester) also show errors?

[popcornmix]

Also, do you have any cooling on the Pi5?
Your test code does seem like quite a hard stress test, and temperature throttles quickly on an uncooled board.

[bukatlib]

There is one loop of memtester, the results are OK:

memtester version 4.6.0 (64-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 16384
pagesizemask is 0xffffffffffffc000
want 7168MB (7516192768 bytes)
got  7168MB (7516192768 bytes), trying mlock ...locked.
Loop 1/10:
  Stuck Address       : ok         
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok         
  Block Sequential    : ok         
  Checkerboard        : ok         
  Bit Spread          : ok         
  Bit Flip            : ok         
  Walking Ones        : ok         
  Walking Zeroes      : ok         
  8-bit Writes        : ok
  16-bit Writes       : ok

Loop 2/10:
  Stuck Address       : testing   2^C

[bukatlib]

There is a simple script I created for a stress test:

#!/bin/bash

for i in $(seq 10); do
    ./prime_sieve_bench 100000000000 -
    sha256=$(openssl dgst -sha256 memory_primes.img | awk '{print $2}')
    if [ "${sha256}" != "bc773de13417112ba5df5bd234069ea489528d3c0d4cfdc4bf8e35d438adb323" ]; then
        mv memory_primes.img memory_primes_invalid_${i}.img
        echo "Invalid ${sha256} sum of sieve image." 1>&2
    else
        rm memory_primes.img
    fi
done

exit 0

I use an active cooler, there are the sensors values under the load:

CPU frequencies:
2400 2400 2400 2400 

load average:
2,32, 1,60, 1,26

rp1_adc-isa-0000
Adapter: ISA adapter
in1:           1.47 V  
in2:           2.53 V  
in3:           1.37 V  
in4:           1.42 V  
temp1:        +54.3°C  

pwmfan-isa-0000
Adapter: ISA adapter
fan1:        6027 RPM

cpu_thermal-virtual-0
Adapter: Virtual device
temp1:        +62.8°C  

rpi_volt-isa-0000
Adapter: ISA adapter
in0:              N/A  

nvme-pci-10100
Adapter: PCI adapter
Composite:    +42.9°C  (low  = -273.1°C, high = +89.8°C)
                       (crit = +94.8°C)
Sensor 1:     +42.9°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +40.9°C  (low  = -273.1°C, high = +65261.8°C

[pelwell]

Could you run the memtester at the same time as the prime sieve?

[bukatlib]

It is interesting that I reproduced it again in the first iteration (usually observed in the first or second iteration) of 10 after the boot but not the others:

sieve size: 49999999999 bits (6250000000 bytes)
Sieve initialized in 1.09111 secs
Primes sieved in 14.8186 secs.
Found 4118054813 primes in 0.495501 secs (bandwidth 12613.5 MB/s).
Binary image written in 20.6925 secs.
Invalid c5ae25e8b8e2b920c1b91d544828a3e90c8a6a661c5f00919f4522cf0667859f sum of sieve image.
sieve size: 49999999999 bits (6250000000 bytes)
Sieve initialized in 2.22051 secs
Primes sieved in 15.5266 secs.
Found 4118054813 primes in 0.500797 secs (bandwidth 12480.1 MB/s).
Binary image written in 20.2859 secs
...

$ cmp -l memory_primes_invalid_1.img memory_primes_right.img
2889326545   0 200
2889326547   0 300
2889326549   0  40
2889326550   0 101
2889326551   0 244
2889326552   0   4
2889326553   0   2
2889326557   0   4
2889326558   0 100
2889326560   0   2
2889326564   0  10
2889326565   0 100
2889326566   0   4
2889326568   0  20
2889326569   0  60
2889326571   0   1
2889326572   0  40
2889326574   0   2

[bukatlib]

Note that I have not overclocked rpi5 so far. What is the safe over_voltage_delta and is there something else what should be set, e.g., force_turbo? Are there some config options to disable some memory optimizations in EEPROM?

[bukatlib]

Started the sieve stress load and memtester in parallel on a reduce set size (50 G sieve, 3500M memtest) and observed no errors at least in the first memtest loop. Tomorrow I will try again.

[bukatlib]

Note that all the cleared bytes in the provided examples are within 64-byte range:

(gdb) print 2889326574>>6
$1 = 45145727
(gdb) print 2889326545>>6
$2 = 45145727
(gdb) print 5985247216>>6
$3 = 93519487
(gdb) print 5985247185>>6
$4 = 93519487

If I am not wrong, the cache line size is 64 bytes, it might be related.

[timg236]

Note that I have not overclocked rpi5 so far. What is the safe over_voltage_delta and is there something else what should be set, e.g., force_turbo? Are there some config options to disable some memory optimizations in EEPROM?

The firmware and hardware won't allow unsafe voltages but it will get warmer. 40mV should be plenty but you could try up to 100mV.

force_turbo=1 might be useful for consistent clock frequencies but it shouldn't make any difference.

[bukatlib]

Thank you for the information, tried +20 mV, +40 mV, and +50 mV and still able to reproduce. Is someone else able to reproduce it? I start to think that I have a defective piece...

[bukatlib]

Under clocked to 2 GHz and still managed to reproduce it. It is easiest to reproduce when RPI 5 starts cold (after power-off and low CPU temperature), when the system "heats up", it works more reliably. Tested again on x86_64 with primes to 1000 G and no errors observed.