Very large lookup table/container

[memsermann]

Hello,
We are scanning a 300 GB Rev 1 Segy file and the resulting container is roughly 18 GB large. We include only the keys ["GroupX","GroupY"], i.e., SegyCon = segy_scan(cloudpath, "segyfile.segy",["GroupX","GroupY"]). I was expecting a much smaller sized container because in the following SLIM/SegyIO demo video: https://www.youtube.com/watch?v=tx530QOPeZo (around 14 minutes into the video), 13.3 TB of data were scanned and the resulting container was only 110 MB. Am I missing parameter that must be set? I followed precisely the the steps in the video. Given the "reduction-factor" in memory in the video, ~ 8E-06, I would have expected a container of size 2.4 MB for a 300 GB segy file.
Thank you! Any guidance and help are appreciated.

[mloubout]

How many shots is there in that file? The size of the resultiing container will depend on the number of unique source position as it is sorted that way currently.

If I remember correctly you can bypass this by doing segy_scan(path, name, keys, blocksize) where blocksize is the number of trace per block in the container. I.e the container will have ntrace/ blocksize blocks.
If not you will have one block per unique SourceX, SourceY poisition and corresponding trace header keys which might get big if you have a lot of sources.