required space for ramdisk plotting only?

[stubmirror]

would i need 330gib ramdisk or is 220gib enough if i wanna test ram only?

[madMAx43v3r]

more like 330, not sure about the exact peak amount..

[msg7086]

I ran with 288GiB of memory with no swap, and plotting ends without issue. You can probably squeeze it further, down to about 265GiB area.

[madMAx43v3r]

ah yeah, that's because max disk usage of tmpdir and tmpdir2 are not at the same time. tmpdir2 has max usage at phase 1 tables 4 and 5, whereas tmpdir has max usage at end of phase 2.

[msg7086]

I was fully ramdisk, no SSD involved. In other words, tmpdir and tmpdir2 point to the same place.

The ramdisk was mounted by

mount -t tmpfs -o size=256G,nr_inodes=10000,mode=0755 tmpfs /root/mnt

And plotting command line is

/root/chia_plot --threads $cpu --tmpdir /root/mnt/chiya/ ...

Hope this clarifies.

[msg7086]

Actually, since I only allocate 256GiB on ramdisk mount, the actual disk usage is at or less than 256GiB. A bit more memory was needed to run the process, so I have to assign a bit more than 256GiB memory to make it work.

[madMAx43v3r]

Makes sense, good info to know.

[msg7086]

Using CPU 64
Final Directory: /root/mnt/chiya/
Number of Plots: 1
Process ID: 2041
Number of Threads: 64
Number of Buckets: 2^7 (128)
Pool Public Key:   x
Farmer Public Key: x
Working Directory:   /root/mnt/chiya/
Working Directory 2: /root/mnt/chiya/
Plot Name: plot-k32-2021-06-10-05-35-x
[P1] Table 1 took 5.79287 sec
[P1] Table 2 took 78.1272 sec, found 4294922472 matches
[P1] Table 3 took 82.0309 sec, found 4294760997 matches
[P1] Table 4 took 95.3012 sec, found 4294395897 matches
[P1] Table 5 took 94.8111 sec, found 4293778805 matches
[P1] Table 6 took 93.2843 sec, found 4292659194 matches
[P1] Table 7 took 79.5587 sec, found 4290224333 matches
Phase 1 took 528.949 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 8.95525 sec
[P2] Table 7 rewrite took 34.6238 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 16.8568 sec
[P2] Table 6 rewrite took 27.4925 sec, dropped 581630155 entries (13.5494 %)
[P2] Table 5 scan took 16.5018 sec
[P2] Table 5 rewrite took 25.6141 sec, dropped 762280741 entries (17.7531 %)
[P2] Table 4 scan took 16.6844 sec
[P2] Table 4 rewrite took 26.983 sec, dropped 829109341 entries (19.3068 %)
[P2] Table 3 scan took 16.4115 sec
[P2] Table 3 rewrite took 25.3536 sec, dropped 855311639 entries (19.9152 %)
[P2] Table 2 scan took 16.7392 sec
[P2] Table 2 rewrite took 25.0913 sec, dropped 865748172 entries (20.1575 %)
Phase 2 took 272.026 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 91.5906 sec, wrote 3429174300 right entries
[P3-2] Table 2 took 25.8004 sec, wrote 3429174300 left entries, 3429174300 final
[P3-1] Table 3 took 65.6307 sec, wrote 3439449358 right entries
[P3-2] Table 3 took 25.5207 sec, wrote 3439449358 left entries, 3439449358 final
[P3-1] Table 4 took 69.503 sec, wrote 3465286556 right entries
[P3-2] Table 4 took 24.5214 sec, wrote 3465286556 left entries, 3465286556 final
[P3-1] Table 5 took 68.5684 sec, wrote 3531498064 right entries
[P3-2] Table 5 took 24.8509 sec, wrote 3531498064 left entries, 3531498064 final
[P3-1] Table 6 took 72.4099 sec, wrote 3711029039 right entries
[P3-2] Table 6 took 26.9011 sec, wrote 3711029039 left entries, 3711029039 final
[P3-1] Table 7 took 117.163 sec, wrote 4290224333 right entries
[P3-2] Table 7 took 28.5977 sec, wrote 4290224333 left entries, 4290224333 final
Phase 3 took 644.629 sec, wrote 21866661650 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 100.953 sec, final plot size is 108771846690 bytes
Total plot creation time was 1546.57 sec

-rw-r--r-- 1 root root 108771846690 Jun 10 06:01 /root/mnt/chiya/plot-k32-2021-06-10-05-35-x.plot

From 64 vCPUs EPYC and 288GiB. Not all CPU resources were utilized during some steps. The sweet spot is probably around 32 threads, 128GiB memory and a 256G SSD for phase 1 and 2.

[msg7086]

Even if you are doing full ramdisk you'll lose some CPU. Not all stages consume all CPU resources. But the loss would be very little. If I'm getting a cloud instance to plot, I would do 16/24/32 cores with 136GiB memory and a SSD as tmpdir1, seems to be best bang for the buck.

[marctanw90]

Sorry, how did you get full ram disk as 288? Won’t it require 220+110 + slippage for r64 for full ram disk? Or are you mounting 220 temp1 and ssd temp2 which will require around 288 for this setup?

[msg7086]

No, it doesn't need 220+110.

tmp1 uses 220GiB max, and tmp2 uses 110GiB max, but they don't max out at the same time. They combined never exceed 256GiB. So if you do pure ramdisk plotting, you only need to specify tmpdir, and tmpdir only needs 256GiB.

288GiB is the full memory capacity, not ramdisk size. That includes 256GiB ramdisk, and 32GiB memory space to run the plotter.

[marctanw90]

Thanks for breaking it down to layman terms for me :) I get it now. I was confused how you got 192 (hybrid) or 384 (full RAM disk) for your bang for buck e5 v1/v2 build. Isnt 288 enough like you mentioned for full RAM disk? Am I missing something here

[maxpla3]

With the 32c TR Pro I can plot with 2 contemporary, and manual staggering, 220G ram disk for both and core affinity to 30t each, raid nvme as temp1, 1700s, that is roughly a plot every 12 minutes

So you have 256GB total RAM? I just ordered another 128GB to get to 256GB. Will try this with a 3970X. Thanks!
What do you mean by core affinity to 30c each? Why and how did you do that?
If one staggers 2 plotters, it is probably possible to reduce the 220GiB ramdisk slightly in size, right? Or do the plotters require the full 110GiB at the same time occasionally?

[maxpla3]

Hi, if I reserve 110 GiB to a ramdisk, I will have 18 Gib left. Will it be enough for r=64?

[stubmirror]

to be safe you want something like threads / 2, ram available.
its not exact, it uses a bit less, but at 64 i saw at least 28gib usage at one point, you will probabaly be better of running 2x32thread or 4x16thread on nvme though

[madMAx43v3r]

Will it be enough for r=64

No, 16 maybe 24

[msg7086]

I just did a test run with 64 threads and 128GiB memory. It went OOM with 16GiB swap file, but works with 32GiB swap file. The most memory critical place is around P1-table4. That means you need at least 160GiB memory to run 64 threads as during P1 it eats both tmpdir1 and tmpdir2, and tmpdir2+40GiB RSS would easily burst 128GiB memory.

[msg7086]

Did a test run with 32 threads and 136GiB memory (plus NVMe SSD). Total plot creation time was 2110.48 sec (35 min). That's only 10 minutes more compared to 64+288 pure ramdisk plotting.

[maxpla3]

Ok, thank you all! I'm considering selling my 3.2TB enterprise SSD and buying additional 128GiB RAM for that money. The crucial question is however, whether this plotter will be able to create pool-plots in the near future. Because if not, I would need my SSD back.

[sc0-0001]

I was expecting to max out at 110GB, but it seems to maybe be CPU/thread dependent?

Is there a straightforward calculation to determine temp2/ramdisk space required? I'm test running on NVMes for temp1 and temp2, and I'm definitely seeing temp2 usage peaking over 150GB (threads -r set to 18).

[sc0-0001]

I'm testing with each of my temp1 and temp2s as 4TB (2x2TB raid0 for each) volumes (and yes, ~3.6TB after filesystem stuff). I'm seeing temp2 usage exceed 110 by a meaningful amount.

What -r did you use for your madmax plot?

[msg7086]

Tested both 32 and 64 threads on 111GiB ramdisk.

[rHilkner]

@msg7086 I have a 5950x (16 cores, 32 threads) with 64gb ram. Do you think I'll have any performance improvement if I go to 128gb? If I make a ramdisk with 110gb will the 18gb remaining be able to run 30-32 threads and keep the OS up?

[sc0-0001]

@rHilkner the answer to your question about 110+18 is in the thread above, in detail.

As far as I can tell, you can't really use less than 110GB for temp2, regardless of how few threads you run. I tried running it on 60GB ramdisk with as few as 8 threads, and even that ran out of memory at 60GB and crashed the program.

[msg7086]

The basic OS uses less than 1G memory, assuming you don't use the GUI. Ramdisk uses 110GiB / 256GiB regardless. If you run 16 threads it should barely fit into the 18GiB remaining but I didn't test it.

If you don't use ramdisk, then you'd just plot like you used to with official plotter, just that this runs faster from start to end. The overall throughput would be similar though. It reduces the needs for many SSDs -- now you only need one SSD that's 256GiB large and you can reach a high throughput. The best bang for the buck plotter would be a Xeon E5v1/v2 server with 192 or 384GiB DDR3 RECC memory and you can do hybrid / full ramdisk plotting at very low cost.

[maxpla3]

This is a bit off-topic, but do you guys know under which conditions quad-channel memory works with Threadripper 3000? I have 4x32GiB installed and I want to upgrade the memory to be able to run tempdir2 on a ramdisk and use r=32 or even 64. Therefore I would need something like 32GiB more RAM. So I could insall additional 4x8GiB RAM-Sticks. Would quad-channel still work with that configuration (4x64GiB and 4x8GiB)?

[sc0-0001]

I tested running temp1 and temp2 on higher end consumer NVMes (WD SN850 / Sabrent Rocket 4), on a 3960x threadripper system with 128GB DDR4-3200MHz, and got madmax plot times of ~1270 seconds.

I then increased RAM to 256GB on the same system, and ran with temp1 as SN850 and temp2 as ramdisk, and am getting plot times of ~1270 seconds.

Does this result make sense? I would expect ramdisk temp2 to be much faster, but it's identical.

[ESP4Ever]

Plotting using only a RAM disk now and according to the PCP memory usage graph it takes 241 Gbyte maximum (20x threads, 256 buckets)

[maxpla3]

You mean one can plot completely on RAM with 256GB total RAM?

[ESP4Ever]

I think so...
241 GB RAM disk + 15 GB for the OS (Ubuntu in my case) and the plotting SW

[seeker003]

I am able to plot only in RAM disk with the following config:

Server

DL380p Gen8
RAM: 256GB
CPU: 2 x E5-2620
OS: Ubuntu 20.04
Plotter: latest code

Mount RAM disk

mount -t tmpfs -o size=246G,nr_inodes=10000,mode=0777 tmpfs /chia/ramdisk

Plotter command

./chia_plot -n 1 -r 22 -u 256 -t /chia/ramdisk/ -2 /chia/ramdisk/ -d /chia/remote-farm/chia-node-sg/ -p KEY -f KEY

Conclusions and results

System doesn't swap. Speed is not better than plotting on 980 Pro 1TB but at least it is not killing the SSD. Probably I can improve the performace if I add more RAM.

[P1] Table 1 took 20.6795 sec
[P1] Table 2 took 202.74 sec, found 4294994869 matches
[P1] Table 3 took 227.826 sec, found 4294977583 matches
[P1] Table 4 took 275.582 sec, found 4295067322 matches
[P1] Table 5 took 275.8 sec, found 4295185562 matches
[P1] Table 6 took 260.55 sec, found 4295169782 matches
[P1] Table 7 took 203.212 sec, found 4295166773 matches
Phase 1 took 1466.42 sec
[P2] max_table_size = 4295185562
[P2] Table 7 scan took 13.5466 sec
[P2] Table 7 rewrite took 48.847 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 46.9441 sec
[P2] Table 6 rewrite took 63.5417 sec, dropped 581413416 entries (13.5364 %)
[P2] Table 5 scan took 43.8217 sec
[P2] Table 5 rewrite took 61.0114 sec, dropped 762143756 entries (17.7441 %)
[P2] Table 4 scan took 42.6915 sec
[P2] Table 4 rewrite took 59.5652 sec, dropped 828930028 entries (19.2996 %)
[P2] Table 3 scan took 42.6143 sec
[P2] Table 3 rewrite took 59.5535 sec, dropped 855066809 entries (19.9085 %)
[P2] Table 2 scan took 41.9976 sec
[P2] Table 2 rewrite took 59.1902 sec, dropped 865587797 entries (20.1534 %)
Phase 2 took 609.877 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 50.8815 sec, wrote 3429407072 right entries
[P3-2] Table 2 took 47.2777 sec, wrote 3429407072 left entries, 3429407072 final
[P3-1] Table 3 took 65.7463 sec, wrote 3439910774 right entries
[P3-2] Table 3 took 46.0395 sec, wrote 3439910774 left entries, 3439910774 final
[P3-1] Table 4 took 66.111 sec, wrote 3466137294 right entries
[P3-2] Table 4 took 46.7877 sec, wrote 3466137294 left entries, 3466137294 final
[P3-1] Table 5 took 67.7182 sec, wrote 3533041806 right entries
[P3-2] Table 5 took 46.8932 sec, wrote 3533041806 left entries, 3533041806 final
[P3-1] Table 6 took 69.5548 sec, wrote 3713756366 right entries
[P3-2] Table 6 took 48.2348 sec, wrote 3713756366 left entries, 3713756366 final
[P3-1] Table 7 took 53.2381 sec, wrote 4295166773 right entries
[P3-2] Table 7 took 56.8802 sec, wrote 4294967296 left entries, 4294967296 final
Phase 3 took 672.233 sec, wrote 21877220608 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 90.5918 sec, final plot size is 108835804299 bytes
Total plot creation time was 2839.21 sec (47.3202 min)

[seeker003]

There is only one problem with this setup, I need fast storage to offload the file. Otherwise process crashes because there isn't enough space to copy and to plot.

[maxpla3]

Do you have a SSD? If yes, set it as your destination. Then run a script in background which copies the plots form this SSD to a HDD. If your server doesn't have USB 3.0, you can easily install a PCIe extension card for that, to be able to copy to external HDDs.

[cvilasis69]

New Single CPU record: 22.32 mins per plot

Server
CPU: AMD Ryzen Threadripper PRO 3955WX
RAM: 512GB
OS: Windows Server 2019 DC running Ubuntu 20.04 with WSL

Mount RAM disk
OSFMount in Windows (RamDisk 360GB)

Plotter command
./chia_plot -n 1 -r 24 -u 256 -t /mnt/r/Temp/ -d /mnt/h/New/ -p x -f x

Code
Latest code from 2021.06.16 and modified with 32 threads for phases 2 & 4.

Results
Multi-threaded pipelined Chia k32 plotter - 8903136
Final Directory: /mnt/h/New/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 20
Number of Threads: 24
Number of Buckets: 2^8 (256)
Pool Public Key: x
Farmer Public Key: x
Working Directory: /mnt/r/Temp/
Working Directory 2: /mnt/r/Temp/
Plot Name: plot-k32-2021-06-16-17-04-b0362fc33f0672da28010b4895e8bc3a0fdc3f44bfdcb333d732d2e2f7cdd122
[P1] Table 1 took 9.8238 sec
[P1] Table 2 took 71.4587 sec, found 4295016197 matches
[P1] Table 3 took 81.447 sec, found 4294960559 matches
[P1] Table 4 took 98.9424 sec, found 4294855076 matches
[P1] Table 5 took 94.2225 sec, found 4294728870 matches
[P1] Table 6 took 90.5682 sec, found 4294549398 matches
[P1] Table 7 took 70.1626 sec, found 4294121720 matches
Phase 1 took 516.734 sec
[P2] max_table_size = 4295016197
[P2] Table 7 scan took 9.37824 sec
[P2] Table 7 rewrite took 46.4228 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 14.2644 sec
[P2] Table 6 rewrite took 31.306 sec, dropped 581355703 entries (13.5371 %)
[P2] Table 5 scan took 13.9156 sec
[P2] Table 5 rewrite took 30.0611 sec, dropped 762010112 entries (17.7429 %)
[P2] Table 4 scan took 14.3659 sec
[P2] Table 4 rewrite took 29.6107 sec, dropped 828891754 entries (19.2996 %)
[P2] Table 3 scan took 14.3708 sec
[P2] Table 3 rewrite took 29.6026 sec, dropped 855120731 entries (19.9099 %)
[P2] Table 2 scan took 14.381 sec
[P2] Table 2 rewrite took 29.4561 sec, dropped 865605022 entries (20.1537 %)
Phase 2 took 303.312 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 31.2223 sec, wrote 3429411175 right entries
[P3-2] Table 2 took 31.9308 sec, wrote 3429411175 left entries, 3429411175 final
[P3-1] Table 3 took 35.2852 sec, wrote 3439839828 right entries
[P3-2] Table 3 took 34.9597 sec, wrote 3439839828 left entries, 3439839828 final
[P3-1] Table 4 took 35.3178 sec, wrote 3465963322 right entries
[P3-2] Table 4 took 35.5227 sec, wrote 3465963322 left entries, 3465963322 final
[P3-1] Table 5 took 31.8518 sec, wrote 3532718758 right entries
[P3-2] Table 5 took 35.698 sec, wrote 3532718758 left entries, 3532718758 final
[P3-1] Table 6 took 36.5274 sec, wrote 3713193695 right entries
[P3-2] Table 6 took 36.8183 sec, wrote 3713193695 left entries, 3713193695 final
[P3-1] Table 7 took 34.6076 sec, wrote 4294121720 right entries
[P3-2] Table 7 took 41.5733 sec, wrote 4294121720 left entries, 4294121720 final
Phase 3 took 429.85 sec, wrote 21875248498 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 89.1021 sec, final plot size is 108824063811 bytes
Total plot creation time was 1339.11 sec (22.3185 min)
Started copy to /mnt/h/New/plot-k32-2021-06-16-17-04-b0362fc33f0672da28010b4895e8bc3a0fdc3f44bfdcb333d732d2e2f7cdd122.plot
Copy to /mnt/h/New/plot-k32-2021-06-16-17-04-b0362fc33f0672da28010b4895e8bc3a0fdc3f44bfdcb333d732d2e2f7cdd122.plot finished, took 93.7153 sec, 1107.43 MB/s avg.

[ESP4Ever]

Some notes on a solely RAM disk plotting:

• slow final file transfer & disabled swap:
  

• even having the final file on a RAM disk, cifs caches a whole file while transmitting. Tried to disable caching and got the problem described above due to a dramatic transfer rate drop
  

[cvilasis69]

Because you need a faster destination to flush your previous plot before [P1] Table5 in phase1 was completed (aprox 50% of phase1).

[ESP4Ever]

I know it but my destination is a network share over a 1Gb link.
This info is for those who running into the app crash problem