write_raw_bids copies the data

[behinger]

Description of the problem

I'm trying to export a raw file, that I constructed to be exported via mne-bids

mne.export.export_raw("~/sub-016_gap_test.set", raw,overwrite=True)

mne_bids.write_raw_bids(raw, mne_bids.BIDSPath(subject="016",task="test",root="~/bids_test"), overwrite=True, verbose=False,symlink=False, format= "EEGLAB",allow_preload=True)

I'm getting close to my memory limit (why do I need 38GB to safe a 2h EEG File anyway 🤷). What's happening in mne-bids vs. mne.raw?

Steps to reproduce

`mne.export.export_raw("~/sub-016_gap_test.set", raw,overwrite=True)`
vs.
`mne_bids.write_raw_bids(raw, mne_bids.BIDSPath(subject="016",task="test",root="~/bids_test"), overwrite=True, verbose=False,symlink=False, format= "EEGLAB",allow_preload=True)`

---
I could try to share a dataset if absolutely necessary

Expected results

both methods should have the same memory footprint

Actual results

mne-bids takes much more RAM to save

Additional information

Platform             Linux-5.15.0-121-generic-x86_64-with-glibc2.31
Python               3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0]
Executable           /home/recorder/miniconda3/envs/autobids-new/bin/python
CPU                  AMD Ryzen 3 5300U with Radeon Graphics (8 cores)
Memory               38.0 GiB

Core
├☑ mne               1.9.0 (latest release)
├☑ numpy             2.2.4 (unknown linalg bindings (threadpoolctl module not found: No module named 'threadpoolctl'))
├☑ scipy             1.15.2
└☑ matplotlib        3.10.1 (backend=module://matplotlib_inline.backend_inline)

Numerical (optional)
└☐ unavailable       sklearn, numba, nibabel, nilearn, dipy, openmeeg, cupy, pandas, h5io, h5py

Visualization (optional)
└☐ unavailable       pyvista, pyvistaqt, vtk, qtpy, ipympl, pyqtgraph, mne-qt-browser, ipywidgets, trame_client, trame_server, trame_vtk, trame_vuetify

Ecosystem (optional)
├☑ mne-bids          0.16.0
├☑ eeglabio          0.0.3
├☑ edfio             0.4.6
├☑ pybv              0.7.6
└☐ unavailable       mne-nirs, mne-features, mne-connectivity, mne-icalabel, mne-bids-pipeline, neo, mffpy

[sappelhoff]

Thanks for the report @behinger, without having looked too deeply into it, here are my first thoughts:

1. that's a lot of memory usage indeed
2. mne_bids.write_raw_bids does a lot more than raw.export (which is the same as mne.export.export_raw) ... and actually, the last step is, that is calls exactly that function:

   mne-bids/mne_bids/write.py

   Lines 1218 to 1231 in a1328bd

   	def _write_raw_eeglab(raw, bids_fname, overwrite):
   	"""Store data as EEGLAB.
   	Parameters
   	----------
   	raw : mne.io.Raw
   	Raw data to save.
   	bids_fname : str
   	The output filename.
   	overwrite : bool
   	Whether to overwrite an existing file or not.
   	"""
   	assert str(bids_fname).endswith(".set")
   	raw.export(bids_fname, overwrite=overwrite)

3. following 2., I am not surprised that write_raw_bids has a higher memory footprint that a simple export call ... it is likely due to some copies that are being created under the hood

Conclusion (pending a deeper look into the issue):

why do I need 38GB to safe a 2h EEG File anyway 🤷

I don't know, I share your puzzlement 🤔

Furthermore it is possible that write_raw_bids could be making an unneeded copy at some points in the code. Very possible, that this could be optimized.

[behinger]

Thanks!

o.c. write_raw_bids does more, I tried to look through the functions but could spot where it makes any of these copies.
It is a 1000Hz x 129 channels dataset => ~7million x 129 datapoints => 6GB of data. I don't understand why the raw file I have has more than it requires in the first place, but I dont understand MNE in the first place as well :-D

As I understand there is also no way to switch to Float32.

(in earlier versions of my scripts, I tried other converters, e.g. fif/brainvision, but I got worse results. Eeglab was the most memory efficient. But now I got files that are longer than previously.)

[behinger]

ok, I feel stupid for not just ctrl+f this.

mne-bids/mne_bids/write.py

Line 1765 in a1328bd

raw = raw.copy()

This line copies the data, presumably raw is modified downstream and one doesnt want to mutate.

would it be ok to add a PR with a new flag copy::Bool=True that allows to disable this behavior?

[sappelhoff]

would it be ok to add a PR with a new flag copy::Bool=True that allows to disable this behavior?

yes, PR welcome! Please have a look at potential repercussions, too :-)