Info About New Features Still Need to be Added To This Framework

[AnuragKr]

I am from Computer Science And Data Science Background. I want to contribute to this project. I need ideas on what new feature or improvement regarding Model Training or Improvement enhancement on GPU I can do.

[amcadmus]

Dear Anurag,

Thank you very much for the interest in the DeePMD-kit project. Current we are working on integrating the attention structure, which greatly improves the accuracy and data efficiency of the DP model. We find that the attention is very expansive and becomes the hot spot of the code. You are welcome to benchmark and let us know how to improve the efficiency of the model, especially on GPUs.

The attention code has not yet been PRed, but you may check it via the link:
https://github.com/iProzd/deepmd-kit/tree/attention_at_sel

Best regards,
Han

[AnuragKr]

@denghuilu If you could just share resources that would also work.

[denghuilu]

1. The GPU implementation bases on the TensorFlow framework. And we have tested the performance of DP model(compressed) with several network sizes as showing below:
   
   Please refer to this paper for more details about the construction of the DP model as well as the related test setting: https://arxiv.org/abs/2107.02103.

2. We mainly use the profiling tools provided by NVIDIA: nvprof and tools such as Nsight system or Nsight compute.Sometimes TensorFlow's timeline API is also used to profiling the training process.

[AnuragKr]

Thanks for the answer. I will try to do it myself in case I face difficulty I will get back.

[Zhang-pchao]

Dear Anurag,

Thank you very much for the interest in the DeePMD-kit project. Current we are working on integrating the attention structure, which greatly improves the accuracy and data efficiency of the DP model. We find that the attention is very expansive and becomes the hot spot of the code. You are welcome to benchmark and let us know how to improve the efficiency of the model, especially on GPUs.

The attention code has not yet been PRed, but you may check it via the link: https://github.com/iProzd/deepmd-kit/tree/attention_at_sel

Best regards, Han

Can the graph neural network improve the fitting accuracy of the DP model ?

[wanghan-iapcm]

"graph neural network" is a general concept. We cannot say whether it improves or not the accuracy of the DP.

[AnuragKr]

Hello @wanghan-iapcm,
Like there is one problem mentioned above regarding optimizing the attention mechanism for GPUS, Is there any other open problem like any section of code that still needs to be worked on with respect to any aspect of Data Science and the HPC domain? How about implementing a different matrix multiplication operator and seeing if it improves existing performance? If any control flow of the model architecture need to be fixed?
Please, suggest what will be the best thing to work on. I just want to explore all possible problems and then see which will be a good point to start and which can be solved.
Thanks

[wanghan-iapcm]

Hello @AnuragKr
The most urgent performance issue is the attention model DPA-1 . Compared to the standard DP model, the hot-spot moves from the embedding net to the self-attention layers, so the model compression technique would not help much. We are seeking for ideas of improve the performance of DPA-1, but the progress is slow.

The gemm is the bottleneck of the standard DP model (not compressed), but we find that that the performance is memory bound, not computational bound, at least on the V100 GPU. This means that faster (in the sense of floating point operations) gemm would not help improve the over-all performance. For example using the tensorcores of V100 does not significantly improves the speed of DP. (unless far-beyond-necessarily-large embedding nets are used).

By the model compression technique we have removed the gemm in the embedding net, thus the hot-spot becomes tabulated energy and force evaluation and the gemms in the fitting net. it is definitely welcome to raise new ideas of improving the performance of these parts.

[AnuragKr]

Hello, @wanghan-iapcm I came across one CNN-based model Schnet. Is it a good idea to extend the DeePMD-kit fitting net architecture option apart from the feed-forward neural network, keeping the input data format and loss function the same?

[wanghan-iapcm]

New architectures are welcomed.