Facing error while running dpgen

[chayanroy3]

Please check the error code. No matter how I try to submit one task at a time, it tries to do all at once and giving error.

I want one fp tasks at a time , it is submitting all 20 tasks at once. I have tried group size 0 and paradeg 1 also.

Description

2023-07-05 15:38:17,841 - INFO : info:check_all_finished: False
2023-07-05 15:38:17,859 - INFO : job: f94c6777e65081001fb1767cf242237956515bc9 submit; job_id is 238 09
2023-07-05 15:38:48,945 - INFO : job: f94c6777e65081001fb1767cf242237956515bc9 23809 terminated;fail _cout is 1; resubmitting job
2023-07-05 15:38:48,963 - INFO : job:f94c6777e65081001fb1767cf242237956515bc9 re-submit after termin ated; new job_id is 24246
2023-07-05 15:38:49,246 - INFO : job:f94c6777e65081001fb1767cf242237956515bc9 job_id:24246 after re- submitting; the state now is <JobStatus.running: 3>
(dpgen) chayan@iitb:~/dpgen_testing/dpgen_example/run$ Traceback (most recent call last):
File "/home/chayan/.local/lib/python3.6/site-packages/dpdispatcher/submission.py", line 245, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/home/chayan/.local/lib/python3.6/site-packages/dpdispatcher/submission.py", line 609, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:f94c6777e65081001fb1767cf242237956515bc9 24615 failed 3 times.job_detail:{'f94c6777e65081001fb1767cf242237956515bc9': {'job_task_list': [{'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000019', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000005', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000014', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000004', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000009', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000013', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000015', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000018', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000006', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000012', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000017', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000010', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000001', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000011', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000002', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000016', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000007', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000008', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000000', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -np 2 /home/iitb/software/vasp.6_gpu/bin/vasp_std.gpu', 'task_work_path': 'task.000.000003', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 8, 'gpu_per_node': 2, 'queue_name': '', 'group_size': 0, 'custom_flags': [], 'strategy': {'if_cuda_multi_devices': True, 'ratio_unfinished': 0.0}, 'para_deg': 2, 'module_purge': True, 'module_unload_list': [], 'module_list': ['/opt/nvidia/hpc_sdk/modulefiles/nvhpc/23.1'], 'source_list': [], 'envs': {'LD_LIBRARY_PATH': '$LD_LIBRARY_PATH://opt/amd-fftw-4.0/lib'}, 'wait_time': 30, 'kwargs': {}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': 24615, 'fail_count': 3}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/chayan/miniconda3/envs/dpgen/bin/dpgen", line 8, in
sys.exit(main())
File "/home/chayan/.local/lib/python3.6/site-packages/dpgen/main.py", line 185, in main
args.func(args)
File "/home/chayan/.local/lib/python3.6/site-packages/dpgen/generator/run.py", line 3642, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/home/chayan/.local/lib/python3.6/site-packages/dpgen/generator/run.py", line 3628, in run_iter
run_fp (ii, jdata, mdata)
File "/home/chayan/.local/lib/python3.6/site-packages/dpgen/generator/run.py", line 3019, in run_fp
forward_common_files=forward_common_files)
File "/home/chayan/.local/lib/python3.6/site-packages/dpgen/generator/run.py", line 2997, in run_fp_inner
submission.run_submission()
File "/home/chayan/.local/lib/python3.6/site-packages/dpdispatcher/submission.py", line 212, in run_submission
self.handle_unexpected_submission_state()
File "/home/chayan/.local/lib/python3.6/site-packages/dpdispatcher/submission.py", line 254, in handle_unexpected_submission_state
) from e
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/chayan/llzo/remote_root/1553915fcbecde0d2b24db6a55190211126fa5f1.
Debug information: submission_hash==1553915fcbecde0d2b24db6a55190211126fa5f1.
Please check the dirs and scripts in remote_root. The job information mentioned above may help.

[njzjz]

The log seems correct. It submitted only one job, which contained 20 tasks.

[chayanroy3]

I want a job with one task in it. group is set as 0 and para_dim set as 1 , even then it is shoing this error

[njzjz]

group_size=0 means a job with all tasks. para_deg=1 means executing tasks one by one.

By setting group_size=1, each job contains one task. The jobs will always be submitted simultaneously.