feat(cpu_offload.py): support selective layers' activation cpu offload async

[huangting4201]

本pr依赖 #383 的合入

Motivation

功能支持

性能调优：显存和计算之间的折中

1. 支持用户指定layer层数的中间激活进行异步的CPU offloading，配置样例如下，对前10个layer的中间激活做CPU offload：

cpu_offloading = dict(
    enable=True,  # 设置是否开启CPU offload功能，默认值为False
    num_layers=10,  # 设置进行CPU offload的层数，默认值为0
)

2. 支持CPU offloading和activation ckpt结合使用，并能对ckpt layer开启selective_checkpoint，如下配置，设置前3个layer的激活进行CPU offload，ckpt layer的数目为：
   ckpt_layers=total_layers*model.checkpoint - cpu_offloading.num_layers

cpu_offloading = dict(
    enable=True,
    num_layers=3,
)
selective_checkpoint = True
selective_checkpoint_offload = False
model = dict(
    num_chunks=1,  # if num_chunks > 1, interleaved pipeline scheduler is used.
    checkpoint=0.5,
    ...
)

精度验证

测试在去掉随机性（seed一致且flashattn开启deterministic）情况下的loss曲线，sc，sc+offload与基线能完全对齐，sc+offload3layers与基线能保证前三位小数对齐：

Modification

internlm/core/parallel/comm/cpu_offload.py: cpu offload api.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.