Back up adata fills up disk space #6
Description
In h5ad_identify_variable_genes.cgi, line 74 adata.filename = dest_datafile_path + ".backed.h5ad" writes adata as a dense matrix. There was this comment: This next command hits a memory issue if not backed. But the previous commands fail for various reasons if backed. So back up now.
The problem is that writing a dense matrix can take up to ~50GB or even more, even after filtering, and thus not only quickly fills diskspace under /tmp, but takes a lot of time to write... In all cases, in our current test setup, this ends up with No space left on device...