Where is stored a vaex dataframe if not in RAM?

[yohplala]

Hi,
I could read (I think in different places, but I have mostly a comment from @maartenbreddels in mind) that processing with vaex does not take RAM
But my understanding is that chunk of data are 'necessarily' loaded to be operated, 'somewhere' (if read from pyarrow file for instance).
So, please, what is being actually done when vaex pocess data?

• chunks of data are loaded in RAM, but so small you can basically neglect the size they take? (chunks of 1024 rows, or a multiple of it?)
• chunks of data are loaded, but they are so small, and CPU cache is nowadays so large, that they are actually loaded directly in CPU cache? (so it does really take 0 RAM)
• something else? (hijacking RAM of GPU behind the scene? :) - joking here, but only half joking :))

Thanks for sharing your insights,
Have a good day, bests,

[maartenbreddels]

Good questions.

df.executor.chunk_size_for(len(df)) will tell you the chunk size it will use, which can also be configured (see https://vaex.io/docs/conf.html )
If an hdf5 file, all the data is memory mapped, which is similar to a swapfile. Only when the array gets accessed will the operating system find out if it's not in RAM, and if it's not, it will put it from disk into RAM. For parquet, the story is a bit more complicated, we iterator/scan over the datasets, with a bit of read ahead, so a few chunks are in RAM at a time. Note though, that each thread has it's own chunks, so if you have 64 cores, you have 64 chunks in RAM.
CPU cache can be ignored, it's only a speedup, but it doesn't give you extra RAM, or make you use less RAM.

Does this help?

[yohplala]

Thanks! Yes, it shed light on this topic! :)