Ramdisk slower than ssd, what is the problem ?

[grobalt]

I am doing a lot of test with Dual socket AMD servers, with high frequency 7F32 and 7F52. Both systems are with 512GB memory, one has 2 sata SSD, the other one has 4 SAS SSD. Memory is octachannel etc.
One is running Centos8, the other ubuntu 20.04 - but behave the same.

i was always wondering what it is that slow (compared to 5900x for example where people have ~ 21 minute plots without ramdisk).
Anyone an idea ?

output for the 2x 16 core 7F52 with Ramdisk, ~ 26 minutes:
./chia_plot -n 16 -r 32 -u 128 -t /media/ramdisk/ -2 /media/ramdisk/ -d /mnt/nfs/chia/ -p *-f *

Multi-threaded pipelined Chia k32 plotter - 8903136
Final Directory: /mnt/nfs/chia/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 810119
Number of Threads: 32
Number of Buckets: 2^7 (128)
Pool Public Key: *
Farmer Public Key: *
Working Directory: /media/ramdisk/
Working Directory 2: /media/ramdisk/
Plot Name: plot-k32-2021-06-17-00-14-41c6481e72a1777b262afff94cafd0c6f8e47889d9e1f0672bf98621f8cd11d5
[P1] Table 1 took 8.56067 sec
[P1] Table 2 took 86.1686 sec, found 4294954996 matches
[P1] Table 3 took 98.2255 sec, found 4294982203 matches
[P1] Table 4 took 100.492 sec, found 4295028759 matches
[P1] Table 5 took 98.8719 sec, found 4295051267 matches
[P1] Table 6 took 101.751 sec, found 4295051924 matches
[P1] Table 7 took 96.4011 sec, found 4295053111 matches
Phase 1 took 590.512 sec
[P2] max_table_size = 4295053111
[P2] Table 7 scan took 15.5913 sec
[P2] Table 7 rewrite took 42.9016 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 22.596 sec
[P2] Table 6 rewrite took 37.0149 sec, dropped 581320433 entries (13.5347 %)
[P2] Table 5 scan took 21.5728 sec
[P2] Table 5 rewrite took 34.8297 sec, dropped 762066309 entries (17.7429 %)
[P2] Table 4 scan took 21.2078 sec
[P2] Table 4 rewrite took 34.0536 sec, dropped 828868394 entries (19.2983 %)
[P2] Table 3 scan took 21.3837 sec
[P2] Table 3 rewrite took 34.2284 sec, dropped 855088090 entries (19.909 %)
[P2] Table 2 scan took 21.5272 sec
[P2] Table 2 rewrite took 33.4577 sec, dropped 865534473 entries (20.1524 %)
Phase 2 took 360.336 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 43.0946 sec, wrote 3429420523 right entries
[P3-2] Table 2 took 38.4596 sec, wrote 3429420523 left entries, 34
[P3-1] Table 3 took 37.6702 sec, wrote 3439894113 right entries
[P3-2] Table 3 took 32.2231 sec, wrote 3439894113 left entries, 3439894113 final
[P3-1] Table 4 took 38.3537 sec, wrote 3466160365 right entries
[P3-2] Table 4 took 34.1066 sec, wrote 3466160365 left entries, 3466160365 final
[P3-1] Table 5 took 38.9104 sec, wrote 3532984958 right entries
[P3-2] Table 5 took 37.9167 sec, wrote 3532984958 left entries, 3532984958 final
[P3-1] Table 6 took 40.0396 sec, wrote 3713731491 right entries
[P3-2] Table 6 took 37.8524 sec, wrote 3713731491 left entries, 3713731491 final
[P3-1] Table 7 took 41.7964 sec, wrote 4295053111 right entries
[P3-2] Table 7 took 43.4927 sec, wrote 4294967296 left entries, 4294967296 final
Phase 3 took 470.277 sec, wrote 21877158746 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 136.029 sec, final plot size is 108835557771 bytes
Total plot creation time was 1557.2 sec (25.9533 min)

and here the same with no ramdisk at all:
./chia_plot -n 1 -r 32 -u 128 -t /home/gts-data/temp/ -2 /home/gts-data/temp/ -d /mnt/nfs/chia/ -p *-f *
Multi-threaded pipelined Chia k32 plotter - 8903136
Final Directory: /mnt/nfs/chia/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 4663
Number of Threads: 32
Number of Buckets: 2^7 (128)
Pool Public Key: *
Farmer Public Key: *
Working Directory: /home/gts-data/temp/
Working Directory 2: /home/gts-data/temp/
Plot Name: plot-k32-2021-06-17-08-51-485ff0b1e304fcedfa5ef34746a4a5af14b2356c1f656dbd0b213691d55b408d
[P1] Table 1 took 7.94515 sec
[P1] Table 2 took 90.0888 sec, found 4294978321 matches
[P1] Table 3 took 94.7127 sec, found 4294954449 matches
[P1] Table 4 took 102.981 sec, found 4294956504 matches
[P1] Table 5 took 101.857 sec, found 4294911529 matches
[P1] Table 6 took 101.898 sec, found 4294709635 matches
[P1] Table 7 took 94.6853 sec, found 4294507834 matches
Phase 1 took 594.221 sec
[P2] max_table_size = 4294978321
[P2] Table 7 scan took 14.276 sec
[P2] Table 7 rewrite took 34.5422 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.073 sec
[P2] Table 6 rewrite took 37.2488 sec, dropped 581307506 entries (13.5354 %)
[P2] Table 5 scan took 22.2402 sec
[P2] Table 5 rewrite took 35.9056 sec, dropped 762034604 entries (17.7427 %)
[P2] Table 4 scan took 21.6304 sec
[P2] Table 4 rewrite took 35.1032 sec, dropped 828879517 entries (19.2989 %)
[P2] Table 3 scan took 23.1427 sec
[P2] Table 3 rewrite took 35.7578 sec, dropped 855098649 entries (19.9094 %)
[P2] Table 2 scan took 21.3356 sec
[P2] Table 2 rewrite took 33.3884 sec, dropped 865577928 entries (20.1533 %)
Phase 2 took 358.8 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 40.4224 sec, wrote 3429400393 right entries
[P3-2] Table 2 took 37.6269 sec, wrote 3429400393 left entries, 3429400393 final
[P3-1] Table 3 took 37.2499 sec, wrote 3439855800 right entries
[P3-2] Table 3 took 34.8328 sec, wrote 3439855800 left entries, 3439855800 final
[P3-1] Table 4 took 38.9809 sec, wrote 3466076987 right entries
[P3-2] Table 4 took 36.3283 sec, wrote 3466076987 left entries, 3466076987 final
[P3-1] Table 5 took 37.0777 sec, wrote 3532876925 right entries
[P3-2] Table 5 took 36.8505 sec, wrote 3532876925 left entries, 3532876925 final
[P3-1] Table 6 took 39.4927 sec, wrote 3713402129 right entries
[P3-2] Table 6 took 39.9895 sec, wrote 3713402129 left entries, 3713402129 final
[P3-1] Table 7 took 41.6374 sec, wrote 4294507834 right entries
[P3-2] Table 7 took 45.1145 sec, wrote 4294507834 left entries, 4294507834 final
Phase 3 took 471.309 sec, wrote 21876120068 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 143.555 sec, final plot size is 108829301090 bytes
Total plot creation time was 1567.93 sec (26.1321 min)

I have absolutely no clue ...

[madMAx43v3r]

I've measured my tmpfs/ramfs speed, and it's only 2 GB/s, so if your SSD is that fast, there is no diff...

[grobalt]

ok, so that could be an idea for the ramdisk question ... now the question, why is the single socket consumer stuff with 12 cores faster :)

[cyperbg]

I've measured my tmpfs/ramfs speed, and it's only 2 GB/s, so if your SSD is that fast, there is no diff...

How do you measure it?

[grobalt]

dd if=/dev/zero of=test.img bs=1M count=65536
to write ~ 64GB to memory