Sporadic inconsistent errors with xtea_long

[yuliamostovoy]

Hi,

I'm trying to run xTea_long on the cloud (Terra) and I'm seeing weird issues where I run it on exactly the same data with the same command and configuration and sometimes it fails and other times it completes successfully. The failures seem to be related to generating the "all_ins_seqs.fa" file, although I'm finding it hard to tell exactly what's going wrong from the logs, and I'm confused as to why it's inconsistent. I tried adding a lot of RAM in case that was the issue, but the errors are still happening. I'm attaching the logs from a few failed runs and two successes on the same data. I'd like to run xTea_long on a number of samples, so I'd really appreciate any help in resolving this. Thanks!
xtea_logs.tar.gz

[simoncchu]

Inside the failed ones, there are earlier errors like:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/xTea/xtea_long/x_reference.py", line 50, in unwrap_gnrt_flank_regions
return XReference.run_gnrt_flank_region_for_chrm(*arg, **kwarg)
File "/xTea/xtea_long/x_reference.py", line 109, in run_gnrt_flank_region_for_chrm
s_left_region = f_fa.fetch(ref_chrm, istart, pos)
File "pysam/libcfaidx.pyx", line 301, in pysam.libcfaidx.FastaFile.fetch
KeyError: "sequence '2' not present"
"""

The code is trying to extract the flanking sequences of a region on chromosome "2" and failed, but before that there is code checking the reference one is "chr2" or "2", so I am not sure why there is an error reported there.

Is it possible to share with me the workspace for a testing?

[yuliamostovoy]

Thanks, yes, I can share the workspace (testing with public data). Should I use your listed gmail account?

[yuliamostovoy]

@simoncchu I shared a couple of workspaces - I think the second has data that you should be able to see directly without any further permission-wrangling. Let me know if there are any issues, and thanks!

[simoncchu]

I found the shared workspaces. I checked the failed jobs, and still cannot find out why the errors were triggered. I will download one of the smaller bam and test locally and get back to you.

[yuliamostovoy]

Thank you, I really appreciate it!

[nogay]

Hi,

I am getting the same "sequence not present" error during different steps of runs. One run I get sequence 4, another retry I get sequence Y, another retry it runs with no errors etc.

I don't know if it's the same with Yulia but in my case it happens when I run it using a singularity container for xtea_long as part of a nextflow pipeline, running on slurm. When I run xtea without nextflow and singularity, just submit to slurm with the conda environment, it works just fine.

wtpoa-cns -i output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.gz -fo output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.fa -t 16
 Running command with output: wtpoa-cns -i output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.gz -fo output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.fa -t 16
 
Running command: mkdir output/tmp/l_asm_tmp/3500/flank_regions

Command error:
 -- date         Thu Mar 27 23:19:51 2025
 --
  -- Error: not a PGZF format at 2 block, in thread_pgz_func -- pgzf.h:369 --
 
 0 contigs 0 edges 0 bases
 ** PROC_STAT(TOTAL) **: real 0.105 sec, user 0.000 sec, sys 0.030 sec, maxrss 22528.0 kB, maxvsize 2066488.0 kB
 ---
 --
 -- total memory     1056461412.0 kB
 -- available         965930468.0 kB
 -- 128 cores
 -- Starting program: wtpoa-cns -i output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.gz -fo output/tmp/l_asm_tmp/3500/chrY~56879824_3500_wtdbg2.ctg.lay.fa -t 16
 -- pid                    528138
 -- date         Thu Mar 27 23:19:52 2025
 --
 
 1 contigs 24 edges 7932 bases
 ** PROC_STAT(TOTAL) **: real 0.205 sec, user 1.050 sec, sys 0.080 sec, maxrss 58368.0 kB, maxvsize 2280736.0 kB
 ---
 multiprocessing.pool.RemoteTraceback: 
 """
 Traceback (most recent call last):
   File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 125, in worker
     result = (True, func(*args, **kwds))
   File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
     return list(map(*args))
   File "/xTea/xtea_long/x_reference.py", line 50, in unwrap_gnrt_flank_regions
     return XReference.run_gnrt_flank_region_for_chrm(*arg, **kwarg)
   File "/xTea/xtea_long/x_reference.py", line 109, in run_gnrt_flank_region_for_chrm
     s_left_region = f_fa.fetch(ref_chrm, istart, pos)
   File "pysam/libcfaidx.pyx", line 301, in pysam.libcfaidx.FastaFile.fetch
 KeyError: "sequence '4' not present"
 """
 
 The above exception was the direct cause of the following exception:
 
 Traceback (most recent call last):
   File "/xTea/xtea_long/l_main.py", line 246, in <module>
     lcaller.call_MEIs_for_sites_with_rep_kmer_filtering(sf_bam_list, sf_merged_sites, sf_ref, l_extd_len_tmp, sf_out_fa,
   File "/xTea/xtea_long/l_MEI_caller.py", line 346, in call_MEIs_for_sites_with_rep_kmer_filtering
     m_tmp, m_refined_pos = self.call_MEIs_from_asm_in_parallel(sf_ref, sf_tmp_to_asm,
   File "/xTea/xtea_long/l_MEI_caller.py", line 730, in call_MEIs_from_asm_in_parallel
     xref.gnrt_flank_region_for_sites(sf_sites, i_extend, self.n_jobs, sf_ref, s_working_folder)
   File "/xTea/xtea_long/x_reference.py", line 183, in gnrt_flank_region_for_sites
     pool.map(unwrap_gnrt_flank_regions, list(zip([self] * len(l_records), l_records)), 1)
   File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 364, in map
     return self._map_async(func, iterable, mapstar, chunksize).get()
   File "/opt/conda/envs/xtea/lib/python3.9/multiprocessing/pool.py", line 771, in get
     raise self._value
 KeyError: "sequence '4' not present"

Thank you!

[nogay]

FYI: I've since assigned 18 cpus to the xtea run that uses 16 cpus and it has become more stable.