Refine3D crashes after adding optics groups #1291
Description
Hi all, my refine3D jobs have been crashing with "A particle image was compared to the reference and resulted in all-zero weights" ever since I added the optics group information to the particle star file (based on EPU's AFIS). It seems ctf refinement jobs all work fine, and the results in log file indicate that beamtilt/trefoil refinement is working. But 3D refinement seems to not be working and I consistently get the aforementioned "error" regardless of whether I use ctf-refined or original (+ optics group) particles.
I used my own custom scripts to extract beamtilt information of EPU's xml files and to cluster micrographs based using k-means algorithm. My script is here, if you want to take a look: https://github.com/BhattaArjun2p/cryoem-pytools/blob/main/epu_afis_extract_optics_group.py. The reason I wrote my own script was because I wanted to add the optic group information to my already cleaned/refined particle set and I noticed that Dustin Morado's script only creates a micrograph star file with optics groups.
Anyway, as I see it, the only difference between the two star files are the entries for optics groups. But while I can get refinement to work just fine with the one without optics groups, the one with optics group crashes.
I would appreciate help with troubleshooting this. I have attached parts of my original and new particle star files, and a plot of optics group clusters.
Environment:
- OS: CentOS Linux release 7.8.2003
- MPI runtime: openmpi-4.0.4
- RELION version: 5.0-beta-3-commit-7d79f3
- Memory: 128 GB
- GPU: RTX 2080 Ti
Dataset:
- Box size: 512 px
- Pixel size: 0.72 A/px
- Number of particles: 332,059
- Description: Approx. 300 KDa complex
Job options:
- Type of job: Refine3D
- Number of MPI processes: 5
- Number of threads: 6
- Full command:
`which relion_refine_mpi` --o Refine3D/job029/run --auto_refine --split_random_halves --i J15_run_data_wOpticsGroup.star --ref Refine3D/job015/run_class001.mrc --firstiter_cc --trust_ref_size --ini_high 25 --blush --dont_combine_weights_via_disc --scratch_dir /scratch --pool 3 --pad 1 --ctf --particle_diameter 220 --flatten_solvent --zero_mask --solvent_mask MaskCreate/job010/mask.mrc --solvent_correct_fsc --oversampling 1 --healpix_order 2 --auto_local_healpix_order 4 --offset_range 5 --offset_step 2 --sym C1 --low_resol_join_halves 40 --norm --scale --j 4 --gpu "" --pipeline_control Refine3D/job029/
WARNING: FSC curve between unmasked maps never drops below 0.8. Using unmasked FSC as FSC_true...
WARNING: This message should go away during the later stages of refinement!
/opt/miniconda/miniconda-default/envs/relion-5.0/lib/python3.10/site-packages/mrcfile/mrcobject.py:554: RuntimeWarning: Data array contains NaN values
warnings.warn("Data array contains NaN values", RuntimeWarning)
/opt/miniconda/miniconda-default/envs/relion-5.0/lib/python3.10/site-packages/mrcfile/mrcobject.py:554: RuntimeWarning: Data array contains NaN values
warnings.warn("Data array contains NaN values", RuntimeWarning)
fn_img= 011109@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 011762@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 005196@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 013167@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 005176@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 011285@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 004576@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 001249@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 012747@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 006665@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 001682@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 005842@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 005700@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 010760@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 002158@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
fn_img= 007148@/scratch/relion_volatile/opticsgroup1_particles.mrcs
adaptive_fraction= 0.999
min_diff2= 3.40282e+38
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
follower 4 encountered error: === Backtrace ===
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x4f) [0x4cfe6f]
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi() [0x4ae793]
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi() [0x714875]
/lib64/libgomp.so.1(+0x16405) [0x7f7affc43405]
/lib64/libpthread.so.0(+0x7ea5) [0x7f7b0066cea5]
/lib64/libc.so.6(clone+0x6d) [0x7f7aff74796d]
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
follower 1 encountered error: === Backtrace ===
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x4f) [0x4cfe6f]
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi() [0x4ae793]
/opt/relion/relion-5.0beta-cuda-11.8/build/bin/relion_refine_mpi() [0x714875]
/lib64/libgomp.so.1(+0x16405) [0x7fd8dbb7d405]
/lib64/libpthread.so.0(+0x7ea5) [0x7fd8dc5a6ea5]
/lib64/libc.so.6(clone+0x6d) [0x7fd8db68196d]
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
Dumped data: error_dump_pdf_orientation, error_dump_pdf_orientation and error_dump_unsorted.
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
2255
ERROR:
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
in: /opt/relion/relion-5.0beta-cuda-11.8/src/acc/acc_ml_optimiser_impl.h, line 2255
ERROR:
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
No orientation was found as better than any other.
A particle image was compared to the reference and resulted in all-zero
weights (for all orientations). This should not happen, unless your data
has very special characteristics. This has historically happened for some
lower-precision calculations, but multiple fallbacks have since been
implemented. Please report this error to the relion developers at
github.com/3dem/relion/issues
slurmstepd: error: *** STEP 83480.0 ON turing006 CANCELLED AT 2025-08-03T21:03:54 ***
srun: error: turing006: task 0: Killed
srun: error: turing006: tasks 1,4: Killed
srun: error: turing006: task 3: Killed
srun: error: turing006: task 2: Killed
micrographs_v5_clusters.pdf
particle_star_original.txt
particle_star_wopticsgroup.txt