model_devi increase along iteration

[changxiaoju]

How to judge a model can be used? The method I adopt now is to see whether the model deviation will decrease with each iteration, until there is no obvious decreasing trend, and then the training of the model is considered to be OK.
but the thing is, when i received a good result that the model_devi decreased, it will increase in the next iteration.
the two chart below is the change of model deviation between iterations. we can see from the first chart that the model devition from iter06 to iter07 and form iter07 to iter08 decreased, but in the second chart it increased from iter08 to iter 09 and from iter09 to iter10.


i have two questions:
1)Are the methods I use to determine whether the model is usable reasonable?
2)Whether such a change in model devition is normal during the training process?

More information:

1. Important parameter settings in the param.json:
   "model_devi_dt": 0.0002,
   "model_devi_skip": 4000,
   iter.06:
   "model_devi_f_trust_lo": 0.55,
   "model_devi_f_trust_hi": 1.0,
   iter.07:
   "model_devi_f_trust_lo": 0.47,
   "model_devi_f_trust_hi": 0.80,
   iter.08:
   "model_devi_f_trust_lo": 0.40,
   "model_devi_f_trust_hi": 0.65,
   iter.09:
   "model_devi_f_trust_lo": 0.40,#the same with iter08, because the model_deviation did not decrease between iter08 and iter09
   "model_devi_f_trust_hi": 0.65,
   {
   "sys_idx": [2,3,4,5,6,7],
   "temps": [12000,13000,14000,15000],
   "press": [10000000],
   "trj_freq": 10,
   "nsteps": 10000,
   "ensemble": "npt",
   "_idx": "07"
   },

2. Percentage of candidate, accurate and failed:
   

3. RDF comparison with abinitio
   

[amcadmus]

From the attached table I find it is difficult to read out information. Could you please present your result in a more clear way? For example the percentage of candidate, accurate and failed against each iteration.

[changxiaoju]

Thank you very much for your advice, i have added information

[AnguseZhang]

You'are using a more and more strict criterion from iter6 to iter8, so we cannot draw conclusion from results. But it seems the model is getting better, if you directly compare iter9 and iter6: the criterion is more strict, and the accurate ratio is similar (system4 to system7 is more accurate). So I would suggest continue some iterations and compare accurate/candidate ratio using the same criterion. (Meanwhile you can increase MD steps, 0.00002 ps/step * 10000 steps = 2ps seems too short.)

[changxiaoju]

Thank you! I will reset according to your suggestion