DDR3 vs SSD PCIE 5.0

[FNsi]

With almost the same speed, which hints...

[Green-Sky]

yes

[staviq]

SSD has a limited lifespan ( writes budget ), RAM does not.

[Green-Sky]

intel optane ?
also, not everything needs to be written. most of the model is frozen.

[staviq]

I mean I interpreted this as using SSD as swap/pagefile, and in that case you cannot control what ends up being written there. And mmap doesn't explicitly prevent the OS from loading a file ( or parts of it ) into RAM, as you cannot have direct disk to L-cache transfers

Unless I'm misunderstanding the original post

[Green-Sky]

@FNsi is probably half meming anyway...

And mmap doesn't explicitly prevent the OS from loading a file ( or parts of it ) into RAM, as you cannot have direct disk to L-cache transfers

yea, true.

[FNsi]

@FNsi is probably half meming anyway...

its a way isn't it?
😂

That might lead an Architecture change for future LLM base system?

[FNsi]

Considering io_uring instead of mmap?

[FNsi]

found out llm in flash

[ejrydhfs]

On second thought it would be interesting to see this combined with hybrid inference or powerinfer which is optimized for the traditional PC architecture which has no unified memory and instead has separated RAM for the CPU and VRAM for the GPU

The paper is optimized for unified memory architectures it seems

[FNsi]

I see 3FS from deepseek
Kind of make sense

[ejrydhfs]

It would also be really nice to see this, working together with powerinfer and distributed inference vía llama.cpp's RPC or exo, currently there's a PR for powerinfer in llama.cpp

[cmp-nct]

I don't know about DDR3 practical speeds but my DDR4 runs at ~100GB/sec read and write (seq)
My two striped "990 Pro" NVMEs are around 11GB/sec seq and that's mostly limited by the pcie bus I think - a 3rd stripe would unlikely bring a huge boost.
In random read/write my SSDs reach around 4-5GB/sec

DDR3 is slower but not that much. The max performance of DDR4 is currently not being reached by the chipsets (maybe that got better in the past year since I purchased mine).

So RAM is significantly better but still way too slow for fast full inference

[Alumniminium]

You are forgetting about the latency. DDR3 is magnitues lower latency than PCIE 5 NVME

[FNsi]

it's almost clear that pcie5.0 with raid0 is absolutely useful...

[ejrydhfs]

It might seem so with sequential access speeds, but the random access speeds of SSDs are still often lower than those of RAM sticks.

The sequential access speeds of RAM as far as I understand is the same as its random access speeds, but the sequential access speed of SSDs is often lower than that of their random access speeds.

Also, from what I understand from reading the AMD64 Architecture Programmer’s Manuals, is that the processor can't execute code from SSDs directly so it will need to be transferred to RAM first which implies an overhead and latency since access for execution is not direct but not sure how big latency is of an issue

CXL might let you use SSDs as RAM, but then SSDs would likely still have slower random access speeds

With that said it is possible to run LLMs directly off SSDs but since they don't support execution in place nor byte addressing it's not very efficient, I would still expect it to be slower than running them off RAM or VRAM, although LLMs in a flash might make things somewhat faster

Edit thinking about it given that tensors must currently be accessed sequentially, random access speeds are irrelevant for inference. So it might work. But you would need like 10 of these drives to achieve speeds comparable to dual channel ddr5 8800 https://www.pcworld.com/article/2620479/samsung-new-9100-pro-ssd-is-the-fastest-consumer-drive-in-the-world.html which is 140.8 GB/s or 70.4 GB/s per channel DRAM would cost around twice as much as the SSDs per GB but using the SSDs would require some sort of RAID and probably one of those PCIe cards that take several m.2 SSDs

Then there would still be the problem of accessing the data directly from the SSD instead of having to place it in RAM first for access by the processor which is something CXL could solve if flash memory was used as DRAM kind of like Intel optane persistent memory, and Kioxia's XL-Flash persistent memory.

[ejrydhfs]

Latency would not be a problem if we had a way to make inference throughput independent of latency, not sure if buffers or asynchronous data retrieval could make it happen