The problem of dpgen /home/haichao/anaconda3/envs/deepmd/lib/python3.10/importlib/__init__.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged. _bootstrap._exec(spec, module)

[zhanghuang103031]

I encountered a problem of running dpgen on the local server, the problem description, the potential function training part is normal, I can train four potential functions at the same time, but the error will be reported after the training is completed
Version of deepmdkit 2.2.9 installed by conda
debug information /bin/sh: line 1: 140014 Aborted (core dumped) dp train input.json

ux-gnu.so2024-10-30 02:06:44,039 - INFO : job: f5dd75978fefc50e8387502acd89bcee87b890fa 155536 terminated; fail_cout is 3; resubmitting job
Traceback (most recent call last):
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 358, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 862, in handle_unexpected_job_state
raise RuntimeError(err_msg)
RuntimeError: job:f5dd75978fefc50e8387502acd89bcee87b890fa 155536 failed 3 times.
Possible remote error message: ==> /home/haichao/Documents/Al_train/dpgen-Al-Sc-3/cde3806730ffe81b4ea9f0255598b2ee3148f843/002/train.log <==
hon.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

DEEPMD INFO average training time: 0.0000 s/batch (exclude first 1000 batches)
DEEPMD INFO finished training
DEEPMD INFO wall time: 0.663 s
/bin/sh: line 1: 155541 Aborted (core dumped) dp train input.json --restart model.ckpt

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/haichao/anaconda3/envs/deepmd/bin/dpgen", line 8, in
sys.exit(main())
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 5394, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 4725, in run_iter
run_train(ii, jdata, mdata)
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 868, in run_train
submission.run_submission()
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 261, in run_submission
self.handle_unexpected_submission_state()
File "/home/haichao/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 362, in handle_unexpected_submission_state
raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/haichao/Documents/Al_train/dpgen-Al-Sc-3/cde3806730ffe81b4ea9f0255598b2ee3148f843.
Debug information: submission_hash==cde3806730ffe81b4ea9f0255598b2ee3148f843.
Please check error messages above and in remote_root. The submission information is saved in /home/haichao/.dpdispatcher/submission/cde3806730ffe81b4ea9f0255598b2ee3148f843.json.
For furthur actions, run the following command with proper flags: dpdisp submission cde3806730ffe81b4ea9f0255598b2ee3148f843