Truncated BAM file causing samtools error

[yaaminiv]

@sr320 I'm using the deduplicated and sorted BAM files you generated for the Cv gonad samples (saved here). I need to merge all the files before calling SNPs. Tried doing so in this Jupyter notebook but got the following error:

Is there a reason why I'm getting this error/how would I check if this file is actually truncated? File sizes for reference:

edited to include the correct second screenshot and notebook link.

[kubu4]

A few things:

1. How did you transfer the files to your computer (i.e. via rsync, wget, curl, etc)?
2. If you did not use rsync, did you verify checksums after transfer? Also, just re-download using rsync instead - no need to verify checksums.
3. Does your computer have sufficient space (both HDD and memory)? That's a LOT of BAM files...
4. Your link to your Jupyter notebook actually just links to this GitHub Discussion.

[yaaminiv]

1. Did not transfer the files at all. I'm working directly from the gannet folder (mounted it onto my computer)
2. I'm writing directly to gannet, so there should be enough memory!
3. Whoops. Notebook: https://github.com/epigeneticstoocean/2018_L18-adult-methylation/blob/main/code/04-BS-Snper.ipynb

[kubu4]

Ah! You will not (and should not) be able to work this way. The files are too large to transfer over the internet while processing them. Also, processing files this way is MASSIVELY slower than running things locally, due to the data transfer requirements back and forth, combined with the slower processor of Gannet having to handle that stuff simultaneously.

[kubu4]

In regards to Number 2, memory (RAM) and hard drive (HDD) are different things. The memory to run this is still based off of what amount of RAM your computer has (since this is what's running the commands).