feat(usability): Refine model inject helper to support huggingface models

[zigzagcai]

Motivation

Improve usability:

1. Add pack_sample_into_one=True mode for streaming dataloader. (Completed)
   • For the purpose of long sequence training verification
2. Refine model_inject_helper to support more general modeling files (such like huggingface) without even one line of code change. (Completed)
   • For easier adaption for third-party models, to demonstrate InternEvo usability
   • model_inject_helper support pipeline parallel mode
3. Refine and optimize the implementation of MockedDataset. (Completed)
   • added sanity check to ensure global_bsz and data equivalence between loaded/saved data

Related repo:

https://github.com/InternLM/InternEvo-HFModels

Examples:

• Huggingface InternLM1-7B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

• Huggingface InternLM2-7B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

• Huggingface Yi-6B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

• Huggingface LLaMA2-7B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

• Huggingface Baichuan2-7B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

• Huggingface Qwen2-7B

  • pure dp mode without code change
  • pack and isp mode with simple lines of code change (via patch)

Modification

internlm/train/pipeline.py

BC-breaking (Optional)

None

Use cases (Optional)

None

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.