From 03f91a64476cd68f2ab3c0ca53c0bc8289b6dd52 Mon Sep 17 00:00:00 2001 From: sallyjunjun Date: Thu, 29 Aug 2024 12:03:04 +0800 Subject: [PATCH 1/5] fix doc --- .../long_text_example.txt | 0 .../locales/en/LC_MESSAGES/checkpoint.po | 188 +++++---- .../locales/en/LC_MESSAGES/initialize.po | 209 +++++---- .../locales/en/LC_MESSAGES/install.po | 197 ++++++--- doc/code-docs/locales/en/LC_MESSAGES/moe.po | 124 +++--- .../locales/en/LC_MESSAGES/monitor.po | 316 ++++++-------- .../locales/en/LC_MESSAGES/parallel.po | 394 +++++++++++++---- .../locales/en/LC_MESSAGES/profiler.po | 23 +- doc/code-docs/locales/en/LC_MESSAGES/usage.po | 397 +++++++++++++----- doc/code-docs/source/checkpoint.rst | 38 +- doc/code-docs/source/initialize.rst | 126 +++++- doc/code-docs/source/profiler.rst | 4 +- doc/en/usage.md | 62 ++- doc/usage.md | 71 ++-- 14 files changed, 1435 insertions(+), 714 deletions(-) rename doc/{aux_ materials => aux_materials}/long_text_example.txt (100%) diff --git a/doc/aux_ materials/long_text_example.txt b/doc/aux_materials/long_text_example.txt similarity index 100% rename from doc/aux_ materials/long_text_example.txt rename to doc/aux_materials/long_text_example.txt diff --git a/doc/code-docs/locales/en/LC_MESSAGES/checkpoint.po b/doc/code-docs/locales/en/LC_MESSAGES/checkpoint.po index 6d592af6f..b18f9e6d7 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/checkpoint.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/checkpoint.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-27 19:31+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/checkpoint.rst:2 msgid "模型加载与保存" @@ -24,12 +24,13 @@ msgstr "Model Checkpointing" #: ../../source/checkpoint.rst:4 msgid "" -"InternEvo 使用 ``internlm.utils.model_checkpoint.CheckpointManager`` " -"来管理模型保存。其中，可以使用 ``CheckpointManager.try_save_checkpoint(train_state)`` " +"InternEvo 使用 ``internlm.checkpoint.checkpoint_manager.CheckpointManager``" +" 来管理模型保存。其中，可以使用 ``CheckpointManager.try_save_checkpoint(train_state)`` " "来保存指定 step 的模型状态。" msgstr "" -"InternEvo uses ``internlm.utils.model_checkpoint.CheckpointManager`` to " -"manage model checkpointing. In the implementation, we use " +"InternEvo uses " +"``internlm.checkpoint.checkpoint_manager.CheckpointManager`` to manage " +"model checkpointing. In the implementation, we use " "``CheckpointManager.try_save_checkpoint(train_state)`` to checkpoint " "training states at specific steps. " @@ -54,88 +55,99 @@ msgstr "" "file. Currently, the relevant parameters are as follows" #: ../../source/checkpoint.rst:13 -msgid "``enable_save_ckpt``: 是否开启检查点存储功能（不影响检查点加载）。参数类型 ``bool``，必选参数。" +msgid "``enable_save_ckpt`` : 是否开启检查点存储功能（不影响检查点加载）。参数类型 ``bool`` ，必选参数。" msgstr "" "``enable_save_ckpt``: Whether to enable checkpoint storage functionality " "(does not affect checkpoint loading). Parameter type: `bool`, it is a " "required parameter." #: ../../source/checkpoint.rst:15 -msgid "``save_ckpt_folder``: 检查点存储路径，参数类型 ``str``，默认为： ``None``，在开启检查点存储功能时为必选参数。" +msgid "" +"``save_ckpt_folder`` : 检查点存储路径，参数类型 ``str`` ，默认为： ``None`` " +"，在开启检查点存储功能时为必选参数。" msgstr "" "``save_ckpt_folder``: Checkpoint storage path. Parameter type: ``str``. " "This is a required parameter when enabling checkpoint storage " "functionality." #: ../../source/checkpoint.rst:17 -msgid "``checkpoint_every``: 检查点存储频率，参数类型 ``int``，默认为： ``50``。" +msgid "``checkpoint_every`` : 检查点存储频率，参数类型 ``int`` ，默认为： ``50`` 。" msgstr "" -"``checkpoint_every``: Checkpoint storage frequency. Parameter type: " -"``int``." +"``checkpoint_every`` : Checkpoint storage frequency. Parameter type: " +"``int`` . default is: ``50`` . " #: ../../source/checkpoint.rst:19 msgid "" -"``load_ckpt_folder``: 初始化检查点/权重加载路径。参数类型 ``str``，默认为： ``None``，详见 :ref" -":`load-ckpt-folder`。" +"``load_ckpt_info`` : 初始化检查点/权重加载信息。参数类型 ``dict`` ，默认为： ``None`` ，详见 :ref:" +" `load-ckpt-info` 。" msgstr "" -"``load_ckpt_folder``: Initialization checkpoint/weight loading path. " -"Parameter type: ``str``. Default is ``None``. :ref:`load-ckpt-folder`" +"``load_ckpt_info`` : Initialization checkpoint/weight loading path. " +"Parameter type: ``dict`` . Default is ``None`` . :ref: `load-ckpt-info` " #: ../../source/checkpoint.rst:21 -msgid "``async_upload``: 是否开启异步上传，默认值为：``False``，详见 :ref:`asyncupload`。" +msgid "``async_upload`` : 是否开启异步上传，默认值为： ``False`` ，详见 :ref: `asyncupload` 。" msgstr "" -"``async_upload``: Whether to enable asynchronous uploading. See " -"documentation for more details :ref:`asyncupload`" +"``async_upload``: Whether to enable asynchronous uploading. Default is " +"``False`` . See documentation for more details :ref:`asyncupload`" #: ../../source/checkpoint.rst:23 -msgid "``async_upload_tmp_folder``: 异步上传临时存储路径。" +msgid "``async_upload_tmp_folder`` : 异步上传临时存储路径。" msgstr "" "``async_upload_tmp_folder``: Temporary storage path for asynchronous " "uploading." #: ../../source/checkpoint.rst:25 msgid "" -"``oss_snapshot_freq``: 快照存储频率，默认值为：``checkpoint_every``的一半。详见 " -":ref:`snapshot`。" +"``oss_snapshot_freq`` : 快照存储频率，默认值为： ``checkpoint_every`` 的一半。详见 :ref: " +"`snapshot` 。" msgstr "" "``oss_snapshot_freq``: Snapshot storage frequency. See documentation for " "more details :ref:`snapshot`." #: ../../source/checkpoint.rst:27 -msgid "``auto_resume``: 是否开启检查点自动恢复，默认值为：``True``，详见 :ref:`autoresume`。" +msgid "``auto_resume`` : 是否开启检查点自动恢复，默认值为： ``True`` ，详见 :ref: `autoresume` 。" msgstr "" -"``auto_resume``: Whether to enable automatic checkpoint resume. See " -"documentation for more details :ref:`autoresume`." +"``auto_resume``: Whether to enable automatic checkpoint resume. Default " +"is ``True`` . See documentation for more details :ref:`autoresume`." #: ../../source/checkpoint.rst:29 -msgid "``stop_file_path`` : 检查点存储控制文件的路径，默认值为：``None``，详见 :ref:`stopfile`。" +msgid "``stop_file_path`` : 检查点存储控制文件的路径，默认值为： ``None`` ，详见 :ref: `stopfile` 。" msgstr "" -"``stop_file_path``: Path to the checkpoint storage control file. See " -"documentation for more details :ref:`stopfile`." +"``stop_file_path``: Path to the checkpoint storage control file. Default " +"is ``None`` . See documentation for more details :ref:`stopfile`." -#: ../../source/checkpoint.rst:32 +#: ../../source/checkpoint.rst:31 +msgid "" +"``enable_internevo2hf_ckpt`` : " +"是否同时保存huggingface格式的权重。如果开启，会将被并行切分后的权重整合起来保存成huggingface格式，方便后续使用hf方式加载权重，避免因并行方式不同导致的权重转换问题。默认值为：" +" ``False`` 。" +msgstr "" +"``enable_internevo2hf_ckpt`` : " +"Whether to save the weights in the Hugging Face format at the same time. If enabled, the weights that have been parallelly divided will be integrated and saved in the Hugging Face format, which is convenient for loading weights in the Hugging Face way later, to avoid weight conversion issues caused by different parallel methods. The default value is: ``False`` ." + +#: ../../source/checkpoint.rst:34 msgid "下面给出config文件的参数设置例子：" msgstr "Here is an example of parameter settings in the config file." -#: ../../source/checkpoint.rst:53 +#: ../../source/checkpoint.rst:55 msgid "加载与存储格式约定" msgstr "Model loading and saving path format conventions." -#: ../../source/checkpoint.rst:58 +#: ../../source/checkpoint.rst:60 msgid "(1) 路径格式约定" msgstr "(1) Path format conventions." -#: ../../source/checkpoint.rst:60 +#: ../../source/checkpoint.rst:62 msgid "InternEvo对config中出现的所有存储路径都遵循以下的路径格式约定:" msgstr "" "InternEvo follows the following path format conventions for all storage " "paths specified in the config:" -#: ../../source/checkpoint.rst:66 +#: ../../source/checkpoint.rst:68 msgid "对于不同backend的路径，有以下的规则需要注意:" msgstr "For paths of different backends, the following rules should be noted:" -#: ../../source/checkpoint.rst:68 +#: ../../source/checkpoint.rst:70 msgid "" "如果需要使用boto3的路径，需要在运行前提前导入 ``S3_ACCESS_KEY_ID`` 和 " "``S3_SECRET_ACCESS_KEY_ID`` 这两个环境变量。" @@ -144,7 +156,7 @@ msgstr "" "``S3_ACCESS_KEY_ID`` and ``S3_SECRET_ACCESS_KEY_ID`` environment " "variables before running." -#: ../../source/checkpoint.rst:70 +#: ../../source/checkpoint.rst:72 msgid "" "如果需要使用volc的路径，需要在运行前提前导入 ``VOLC_ACCESS_KEY_ID`` 和 " "``VOLC_SECRET_ACCESS_KEY_ID`` 这两个环境变量。" @@ -153,7 +165,7 @@ msgstr "" "``S3_ACCESS_KEY_ID`` and ``S3_SECRET_ACCESS_KEY_ID`` environment " "variables before running." -#: ../../source/checkpoint.rst:72 +#: ../../source/checkpoint.rst:74 msgid "" "如果需要使用oss2的路径，需要在运行前提前导入 ``ALI_ACCESS_KEY_ID`` 和 " "``ALI_SECRET_ACCESS_KEY_ID`` 这两个环境变量。" @@ -162,77 +174,85 @@ msgstr "" "``S3_ACCESS_KEY_ID`` and ``S3_SECRET_ACCESS_KEY_ID`` environment " "variables before running." -#: ../../source/checkpoint.rst:74 +#: ../../source/checkpoint.rst:76 msgid "bucket的endpoint一般分为Inside IP和Outside IP，如果可以尽量使用inside IP，会获得更佳的存储速度。" msgstr "" "The bucket's endpoint is typically divided into Inside IP and Outside IP." " Whenever possible, it's advisable to use the Inside IP to achieve better" " storage speed." -#: ../../source/checkpoint.rst:79 -msgid "(2) 模型加载(load_ckpt_folder)格式约定" -msgstr "(2) Model loading format conventions (load_ckpt_folder)." - #: ../../source/checkpoint.rst:81 -msgid "load_ckpt_folder 由三个字段组成， ``path`` 、 ``content`` 和 ``ckpt_type`` 。" -msgstr "" -"``load_ckpt_folder`` consists of three fields: ``path``, ``content``, and" -" ``ckpt_type``." +msgid "(2) 模型加载(load_ckpt_info)格式约定" +msgstr "(2) Model loading format conventions (load_ckpt_info)." #: ../../source/checkpoint.rst:83 +msgid "load_ckpt_info 由三个字段组成， ``path`` 、 ``content`` 和 ``ckpt_type`` 。" +msgstr "" +"``load_ckpt_info`` consists of three fields: ``path``, ``content``, and " +"``ckpt_type``." + +#: ../../source/checkpoint.rst:85 msgid "``path``：给出了检查点/初始化模型权重的加载路径（path的格式见下小节）" msgstr "" "``path``: Specifies the loading path for the checkpoint/initial model " "weights (the format of the path is described in the following " "subsection)." -#: ../../source/checkpoint.rst:85 +#: ../../source/checkpoint.rst:87 msgid "``content``: 表示需要加载的内容，目前支持的字段包括：" msgstr "" "``content``: Indicates the content to be loaded, currently supported " "fields include:" -#: ../../source/checkpoint.rst:87 +#: ../../source/checkpoint.rst:89 msgid "``model``：加载模型权重。" msgstr "``model``: Load model weights." -#: ../../source/checkpoint.rst:88 +#: ../../source/checkpoint.rst:90 msgid "``sampler``：加载sampler状态。" msgstr "``sampler``: Load sampler state." -#: ../../source/checkpoint.rst:89 +#: ../../source/checkpoint.rst:91 msgid "``scheduler``：加载lr_scheduler状态。" msgstr "``scheduler``: Load lr_scheduler state." -#: ../../source/checkpoint.rst:90 +#: ../../source/checkpoint.rst:92 msgid "``optimzier``：加载optimizer状态。" msgstr "``optimizer``: Load optimizer state." -#: ../../source/checkpoint.rst:91 +#: ../../source/checkpoint.rst:93 msgid "``all``：表示所有状态均加载，一般在resume训练使用。" msgstr "" "``all``: Indicates that all states should be loaded, typically used for " "resuming training." -#: ../../source/checkpoint.rst:93 +#: ../../source/checkpoint.rst:95 msgid "``ckpt_type``：表示加载的模型权重类型，目前支持的字段包括：" msgstr "" "``ckpt_type``: Represents the type of model weight to be loaded, " "currently supported fields include:" -#: ../../source/checkpoint.rst:95 -msgid "``internlm``：internevo约定的checkpoint存储格式。" -msgstr "``internlm``: Checkpoint storage format as per InternEvo conventions." - #: ../../source/checkpoint.rst:97 +msgid "``internevo``：internevo约定的checkpoint存储格式。" +msgstr "``internevo``: Checkpoint storage format specified by internevo." + +#: ../../source/checkpoint.rst:98 +msgid "``llama``：huggingface llama约定的checkpoint存储格式。" +msgstr "``llama``: Checkpoint storage format specified by huggingface llama." + +#: ../../source/checkpoint.rst:99 +msgid "``hf``：huggingface 模型约定的checkpoint存储格式。" +msgstr "``hf``: Checkpoint storage format specified by huggingface model." + +#: ../../source/checkpoint.rst:101 msgid "下面给出两个例子：" msgstr "Here are two examples:" -#: ../../source/checkpoint.rst:111 +#: ../../source/checkpoint.rst:114 msgid "异步上传" msgstr "Asynchronous upload." -#: ../../source/checkpoint.rst:113 +#: ../../source/checkpoint.rst:116 msgid "" "异步上传会先同步的将模型存储到 ``async_upload_tmp_folder`` " "中，再异步的写入远端存储（OSS/NFS）中。从而避免存储ckpt阻塞训练过长时间。" @@ -242,18 +262,18 @@ msgstr "" "storage (OSS/NFS). This helps prevent blocking training for extended " "periods while storing checkpoints." -#: ../../source/checkpoint.rst:115 ../../source/checkpoint.rst:133 -#: ../../source/checkpoint.rst:149 ../../source/checkpoint.rst:164 +#: ../../source/checkpoint.rst:118 ../../source/checkpoint.rst:136 +#: ../../source/checkpoint.rst:152 ../../source/checkpoint.rst:167 msgid "config.ckpt 中相关的参数：" msgstr "The parameters related to ``config.ckpt`` are:" -#: ../../source/checkpoint.rst:117 +#: ../../source/checkpoint.rst:120 msgid "``async_upload``: 是否开启异步上传。参数类型 ``bool/None``，默认为 ``False``。" msgstr "" "``async_upload``: Whether to enable asynchronous upload. Parameter type: " "``bool/None``. Default is ``False``." -#: ../../source/checkpoint.rst:119 +#: ../../source/checkpoint.rst:122 msgid "" "``async_upload_tmp_folder``: 异步上传临时存储路径。参数类型 ``str/None``, 默认值为 " "``/dev/shm/{JOB_NAME}_tmp_ckpt/``。" @@ -262,14 +282,14 @@ msgstr "" "upload. Parameter type: `str/None`. Default value is " "``/dev/shm/{JOB_NAME}_tmp_ckpt/``." -#: ../../source/checkpoint.rst:121 +#: ../../source/checkpoint.rst:124 msgid "需要注意的是，异步上传功能仅在backend为非local时才会有效果，bcakend为local时只支持同步存储。" msgstr "" "It's important to note that asynchronous upload functionality is only " "effective when the backend is set to \"boto3.\" When the backend is set " "to \"local,\" only synchronous storage is supported." -#: ../../source/checkpoint.rst:123 +#: ../../source/checkpoint.rst:126 msgid "" "``async_upload_tmp_folder`` " "设置的的原则为尽量设置为计算节点的local目录，这样才可以获得最佳的异步上传速度，一般来说建议为 ``/dev/shm`` 或 " @@ -281,11 +301,11 @@ msgstr "" "or ``/nvme``. If If you use synchronous upload, this path does not need " "to be given." -#: ../../source/checkpoint.rst:129 +#: ../../source/checkpoint.rst:132 msgid "快照检查点" msgstr "Snapshot Checkpoint" -#: ../../source/checkpoint.rst:131 +#: ../../source/checkpoint.rst:134 msgid "" "快照检查点是一种特殊的检查点，其是为了减少模型因为训练崩溃（ECC error, NCCL error, " ".etc）等问题导致训练任务崩溃而损失的训练进度。其采用交替覆盖写的策略，所占用的存储大小为两个step的检查点所需的空间。配合上异步的检查点写入，在不影响训练速度和存储容量的条件下极大的增大了检查点的存储频率。" @@ -299,13 +319,13 @@ msgstr "" "frequency of checkpoints without affecting training speed and storage " "capacity." -#: ../../source/checkpoint.rst:135 +#: ../../source/checkpoint.rst:138 msgid "``oss_snapshot_freq``: 快照存储频率。参数类型 ``int/None``，默认为 ``50``。" msgstr "" "``oss_snapshot_freq``: Snapshot storage frequency. Parameter type " "``int/None``, default is ``50``" -#: ../../source/checkpoint.rst:137 +#: ../../source/checkpoint.rst:140 msgid "" "``oss_snapshot_freq`` 可以根据模型每step时间酌情设置，一般快照频率在1小时以下，半小时以上为怡/不给（默认值是 " "``checkpoint_every`` 的二分之一）。" @@ -315,11 +335,11 @@ msgstr "" "is Yi/Non for more than half an hour (the default value is one-half of " "``checkpoint_every``)" -#: ../../source/checkpoint.rst:143 +#: ../../source/checkpoint.rst:146 msgid "检查点自动恢复" msgstr "Checkpoint automatic recovery" -#: ../../source/checkpoint.rst:145 +#: ../../source/checkpoint.rst:148 msgid "" "检查点自动加载功能的目的是在resume训练时，自动加载 ``save_ckpt_folder`` " "路径下最新的检查点（包括snapshot检查点）。配合上自动重启机制，可以实现无人干预的任务自动恢复。" @@ -330,36 +350,37 @@ msgstr "" "automatic restart mechanism, tasks can be automatically restored without " "human intervention." -#: ../../source/checkpoint.rst:147 +#: ../../source/checkpoint.rst:150 msgid "" -"该功能默认开启，所以要注意如果需要加载 ``load_ckpt_folder`` 路径下的模型权重，要将 ``auto_resume`` 设置为 " +"该功能默认开启，所以要注意如果需要加载 ``load_ckpt_info`` 路径下的模型权重，要将 ``auto_resume`` 设置为 " "False，否则可能会产生预期外的行为。" msgstr "" "This function is enabled by default, so please note that if you need to " -"load the model weights under the ``load_ckpt_folder`` path, you must set " +"load the model weights under the ``load_ckpt_info`` path, you must set " "``auto_resume`` to ``False``, otherwise unexpected behavior may occur." -#: ../../source/checkpoint.rst:151 +#: ../../source/checkpoint.rst:154 msgid "``auto_resume``: 是否开启检查点自动恢复。参数类型 ``bool``，默认为 ``True``。" msgstr "" "``auto_resume``: Whether to enable automatic checkpoint recovery. " "Parameter type ``bool``, default is ``True``" -#: ../../source/checkpoint.rst:153 +#: ../../source/checkpoint.rst:156 msgid "" "``auto_resume`` 如果为True，则尝试从 ``save_ckpt_folder`` " -"路径中自动加载最新的ckpt，如果找不到，则从step 0开始训练。如果为False，则尝试从 ``load_ckpt_folder`` " +"路径中自动加载最新的ckpt，如果找不到，则从step 0开始训练。如果为False，则尝试从 ``load_ckpt_info`` " "中加载模型参数。" msgstr "" -"``auto_resume`` If True, attempts to save_ckpt_folder`Automatically load " -"the latest ckpt in the path. If not found, training will start from step " -"0. If False, try to load model parameters from ``load_ckpt_folder``" +"``auto_resume`` If True, attempts to automatically load the latest ckpt " +"in the path specified in ``save_ckpt_folder`` . If not found, training " +"will start from step 0. If False, try to load model parameters from " +"``load_ckpt_info`` " -#: ../../source/checkpoint.rst:159 +#: ../../source/checkpoint.rst:162 msgid "手动控制检查点存储" msgstr "Manual control of checkpoint storage" -#: ../../source/checkpoint.rst:161 +#: ../../source/checkpoint.rst:164 msgid "" "在模型距离下一次检查点存储还有很长时间，这时如果希望立刻停止一个任务，又不希望丢失目前训练进度时可以使用手动控制检查点存储功能。通过向一个位于NFS上的" " ``stop_file_path`` 文件中写入希望任务停止的step步数，Global Rank " @@ -375,14 +396,14 @@ msgstr "" "all training processes, and it is agreed that each process will store a " "checkpoint when training reaches this step, and choose whether to exit." -#: ../../source/checkpoint.rst:166 +#: ../../source/checkpoint.rst:169 msgid "``stop_file_path``：检查点存储控制文件的路径，参数类型 ``str/None``，默认为 ``None``，表示关闭该功能。" msgstr "" "``stop_file_path``: The path of the checkpoint storage control file, " "parameter type ``str/None``, the default is ``None``, indicating to turn " "off this function" -#: ../../source/checkpoint.rst:168 +#: ../../source/checkpoint.rst:171 msgid "下面给出一个写入 ``stop_file_path`` 的例子：" msgstr "An example of writing to ``stop_file_path`` is given below:" @@ -422,3 +443,6 @@ msgstr "An example of writing to ``stop_file_path`` is given below:" #~ msgid "Save checkpoint to the given folder path." #~ msgstr "" +#~ msgid "``hf_model``：适用于加载huggingface所有模型的checkpoint存储格式。" +#~ msgstr "" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/initialize.po b/doc/code-docs/locales/en/LC_MESSAGES/initialize.po index 350e5fc1e..99edfa825 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/initialize.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/initialize.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-27 10:37+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: zh_CN\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/initialize.rst:2 msgid "训练构建" @@ -87,115 +87,108 @@ msgstr "" "default parser with some builtin arguments, users can add custom " "parameters to this parser." -#: internlm.initialize.launch.get_default_parser:1 of -msgid "" -"Reads user command line and uses an argument parser to parse the input " -"arguments. Input arguments include configuration, host, port, world size," -" local rank, backend for torch.distributed." -msgstr "" - -#: internlm.initialize.initialize_trainer.initialize_trainer -#: internlm.initialize.launch.get_default_parser of -msgid "返回" -msgstr "" - -#: internlm.initialize.launch.get_default_parser:4 of -msgid "" -"Returns the parser with the default arguments, the user may add " -"customized arguments into this parser." -msgstr "" - -#: internlm.initialize.initialize_trainer.initialize_trainer -#: internlm.initialize.launch.get_default_parser of -msgid "返回类型" -msgstr "" - -#: ../../source/initialize.rst:45 +#: ../../source/initialize.rst:44 msgid "模型初始化" msgstr "Model Initialization" -#: ../../source/initialize.rst:49 +#: ../../source/initialize.rst:48 msgid "InternEvo 在配置文件中使用字段 ``model_type`` 和 ``model`` 来控制模型初始化过程。示例模型初始化配置定义如下：" msgstr "" "InternEvo uses the field ``model_type`` and ``model`` in the config file " "to control model initialization process. An example model initialization " "configuratio" -#: ../../source/initialize.rst:77 +#: ../../source/initialize.rst:76 msgid "字段 ``model_type`` 指明了要初始化的模型类型" msgstr "" "The field ``model_type`` specifics the model type has been registered and" " to be initialized." -#: ../../source/initialize.rst:78 +#: ../../source/initialize.rst:77 msgid "字段 ``model`` 中的参数指定了在模型初始化过程中的参数设置" msgstr "" "The parameters in field ``model`` specific the configuration settings " "during model initialization." -#: ../../source/initialize.rst:80 -msgid "" -"值得注意的是，用户可以定义新的模型类型，并使用装饰器 ``@MODEL_INITIALIZER.register_module`` " -"注册模型的初始化函数，其中 ``MODEL_INITIALIZER`` 是类 " -"``internlm.util.registry.Registry`` 的一个实例化对象，示例如下所示：" -msgstr "" -"It is worth noting that, users can define new model type, and register " -"model's initialization function by decorater " -"``@MODEL_INITIALIZER.register_module``, which ``MODEL_INITIALIZER`` is an" -" instantiated object of class ``internlm.util.registry.Registry``, the " -"example is shown as follows." +#: ../../source/initialize.rst:79 +msgid "值得注意的是，用户可以定义新的模型类型，并通过 ``register_module`` 注册模型的初始化函数，示例如下所示：" +msgstr "It is worth noting that users can define new model types and register the model initialization functions via register_module. An example is shown as follows:" -#: ../../source/initialize.rst:92 +#: ../../source/initialize.rst:90 +msgid "其中，\"INTERNLM\"为新的模型类型，InternLM1为新模型的入口函数。" +msgstr "In this context, "INTERNLM" is the new model type, and InternLM1 is the entry function for the new model." + +#: ../../source/initialize.rst:93 msgid "优化器初始化" msgstr "Optimizer Initialization" -#: ../../source/initialize.rst:99 -msgid "数据加载器初始化" -msgstr "Dataloader Initialization" - -#: ../../source/initialize.rst:106 -msgid "Trainer 初始化" -msgstr "Trainer Initialization" - -#: internlm.initialize.initialize_trainer.initialize_trainer:1 of +#: ../../source/initialize.rst:97 msgid "" -"Core function to wrap the essential training components with our " -"functionality based on the config which is loaded into gpc.config." +"InternEvo 在配置文件中使用字段 ``grad_scaler`` 、 ``hybrid_zero_optimizer`` 、 " +"``adam`` 、 ``lr_scheduler`` 和 ``beta2_scheduler`` " +"来控制优化器初始化过程。示例优化器初始化配置定义如下：" msgstr "" +"InternEvo utilizes the fields grad_scaler, hybrid_zero_optimizer, adam, lr_scheduler, and beta2_scheduler in the configuration file to control the initialization process of the optimizer. An example configuration for initializing the optimizer is defined as follows:" -#: internlm.initialize.initialize_trainer.initialize_trainer of -msgid "参数" +#: ../../source/initialize.rst:158 +msgid "" +"用户通过 ``initialize_optimizer`` 函数初始化优化器，并传入 ``isp_communicator`` 参数，以便处理 " +"ISP 并行模式下的通信。" msgstr "" +"Users initialize the optimizer through the function initialize_optimizer and pass in the isp_communicator parameter to handle communication in the ISP parallel mode." -#: internlm.initialize.initialize_trainer.initialize_trainer:4 of -msgid "Your model instance or a function to build the model." -msgstr "" +#: ../../source/initialize.rst:163 +msgid "数据加载器初始化" +msgstr "Dataloader Initialization" -#: internlm.initialize.initialize_trainer.initialize_trainer:6 of -msgid "Your optimizer for training." +#: ../../source/initialize.rst:167 +msgid "InternEvo 在配置文件中使用字段 ``data`` 来控制数据加载器初始化过程。示例数据加载器初始化配置定义如下：" msgstr "" +"InternEvo uses the field data in the configuration file to control the initialization process of the data loader. The example configuration for initializing the data loader is defined as follows:" -#: internlm.initialize.initialize_trainer.initialize_trainer:8 of -msgid "Your criterion instance." -msgstr "" +#: ../../source/initialize.rst:204 +msgid "这里支持三种数据集的初始化，包括模拟数据集、已分词数据集和流式数据集。" +msgstr "This section supports the initialization of three types of datasets, including dummy datasets, tokenized datasets, and streaming datasets." -#: internlm.initialize.initialize_trainer.initialize_trainer:10 of -msgid "Dataloader for training." -msgstr "" +#: ../../source/initialize.rst:206 +msgid "模拟数据集" +msgstr "dummy dataset" -#: internlm.initialize.initialize_trainer.initialize_trainer:12 of -msgid "Dataloader for testing." -msgstr "" +#: ../../source/initialize.rst:208 +msgid "如果设置TRAIN_FOLDER为None，则随机生成模拟数据集，如果设置的随机种子一样，生成的数据集保持一致。" +msgstr "If the TRAIN_FOLDER is set to None, a dummy dataset will be randomly generated, and if the random seed is the same, the generated dataset will remain consistent." -#: internlm.initialize.initialize_trainer.initialize_trainer:14 of -msgid "Your lr scheduler instance, optional." -msgstr "" +#: ../../source/initialize.rst:210 +msgid "已分词数据集" +msgstr "tokenized dataset" + +#: ../../source/initialize.rst:212 +msgid "如果设置TRAIN_FOLDER为本地指定路径，路径中保存经过分词之后的.bin和.meta文件，则加载已分词数据集。" +msgstr "If the TRAIN_FOLDER is set to a local path where the .bin and .meta files after tokenization are stored, then a tokenized dataset will be loaded." + +#: ../../source/initialize.rst:214 +msgid "流式数据集" +msgstr "streaming dataset" + +#: ../../source/initialize.rst:216 +msgid "如果设置TRAIN_FOLDER为本地指定路径，路径中保存从huggingface下载的数据集，同时在data配置中，新增如下type和tokenizer_path字段，则加载流式数据集。" +msgstr "If the TRAIN_FOLDER is set to a specified local path that contains the dataset downloaded from HuggingFace, and new fields type and tokenizer_path are added in the data configuration, then a streaming dataset will be loaded." -#: internlm.initialize.initialize_trainer.initialize_trainer:17 of +#: ../../source/initialize.rst:225 msgid "" -"A tuple of ``(trainer, train_dataloader, test_dataloader, lr_scheduler)``" -" where only ``trainer`` could not be None." +"已分词数据集和流式数据集格式的详细说明，参见 `使用教程 " +"`_" msgstr "" +"For detailed instructions on the formats for tokenized datasets and streaming datasets, please refer to the `User Guide " +"`_" + +#: ../../source/initialize.rst:228 +msgid "Trainer 初始化" +msgstr "Trainer Initialization" + +#: ../../source/initialize.rst:232 +msgid "通过initialize_trainer函数，初始化训练过程，需要输入创建好的模型、初始化的优化器以及调度器等参数。" +msgstr "The initialize_trainer function is used to initiate the training process, requiring parameters such as the created model, initialized optimizer, scheduler, and other related parameters." #~ msgid "Initialize model with Automatic Mixed Precision." #~ msgstr "" @@ -230,3 +223,71 @@ msgstr "" #~ msgid "A tuple of (train_dl, dataset_types)." #~ msgstr "" +#~ msgid "" +#~ "Reads user command line and uses " +#~ "an argument parser to parse the " +#~ "input arguments. Input arguments include " +#~ "configuration, host, port, world size, " +#~ "local rank, backend for torch.distributed." +#~ msgstr "" + +#~ msgid "返回" +#~ msgstr "" + +#~ msgid "" +#~ "Returns the parser with the default " +#~ "arguments, the user may add customized" +#~ " arguments into this parser." +#~ msgstr "" + +#~ msgid "返回类型" +#~ msgstr "" + +#~ msgid "" +#~ "值得注意的是，用户可以定义新的模型类型，并使用装饰器 " +#~ "``@MODEL_INITIALIZER.register_module`` 注册模型的初始化函数，其中 " +#~ "``MODEL_INITIALIZER`` 是类 " +#~ "``internlm.util.registry.Registry`` 的一个实例化对象，示例如下所示：" +#~ msgstr "" +#~ "It is worth noting that, users can" +#~ " define new model type, and register" +#~ " model's initialization function by " +#~ "decorater ``@MODEL_INITIALIZER.register_module``, which" +#~ " ``MODEL_INITIALIZER`` is an instantiated " +#~ "object of class ``internlm.util.registry.Registry``," +#~ " the example is shown as follows." + +#~ msgid "" +#~ "Core function to wrap the essential " +#~ "training components with our functionality " +#~ "based on the config which is " +#~ "loaded into gpc.config." +#~ msgstr "" + +#~ msgid "参数" +#~ msgstr "" + +#~ msgid "Your model instance or a function to build the model." +#~ msgstr "" + +#~ msgid "Your optimizer for training." +#~ msgstr "" + +#~ msgid "Your criterion instance." +#~ msgstr "" + +#~ msgid "Dataloader for training." +#~ msgstr "" + +#~ msgid "Dataloader for testing." +#~ msgstr "" + +#~ msgid "Your lr scheduler instance, optional." +#~ msgstr "" + +#~ msgid "" +#~ "A tuple of ``(trainer, train_dataloader, " +#~ "test_dataloader, lr_scheduler)`` where only " +#~ "``trainer`` could not be None." +#~ msgstr "" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/install.po b/doc/code-docs/locales/en/LC_MESSAGES/install.po index 9211b4165..2c0393902 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/install.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/install.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-23 14:49+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,104 +16,133 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" -#: ../../../install.md:2 ../../../install.md:28 +#: ../../../install.md:2 msgid "环境安装" msgstr "Installation" -#: ../../../install.md:4 +#: ../../../install.md:3 msgid "环境准备" msgstr "Environment Preparation" -#: ../../../install.md:5 -msgid "首先，需要安装的依赖包及对应版本列表如下：" -msgstr "The required packages and corresponding version are shown as follows:" - -#: ../../../install.md:6 +#: ../../../install.md:4 msgid "Python == 3.10" msgstr "" -#: ../../../install.md:7 -msgid "GCC == 10.2.0" +#: ../../../install.md:5 +msgid "Ampere或者Hopper架构的GPU (例如H100, A100)" +msgstr "GPU with Ampere or Hopper architecture (such as H100, A100)" + +#: ../../../install.md:6 ../../../install.md:132 +msgid "Linux OS" msgstr "" #: ../../../install.md:8 -msgid "MPFR == 4.1.0" -msgstr "" +msgid "pip方式安装" +msgstr "Installation through pip" #: ../../../install.md:9 -msgid "CUDA >= 11.7" +msgid "推荐使用 conda 构建一个 Python-3.10 的虚拟环境，命令如下：" msgstr "" +"It is recommended to build a Python-3.10 virtual environment using conda, command is as follows: " -#: ../../../install.md:10 -msgid "Pytorch >= 1.13.1" -msgstr "" +#: ../../../install.md:15 +msgid "首先，安装指定版本的torch, torchvision, torchaudio以及torch-scatter:" +msgstr "Firstly, install the specified versions of torch, torchvision, torchaudio, and torch-scatter:" -#: ../../../install.md:11 -msgid "Transformers >= 4.28.0" +#: ../../../install.md:21 +msgid "安装InternEvo:" +msgstr "Install InternEvo:" + +#: ../../../install.md:26 ../../../install.md:69 +msgid "安装 flash-attention (version v2.2.1)：" +msgstr "Install flash-attention (version v2.2.1):" + +#: ../../../install.md:31 +msgid "安装 Apex (version 23.05)： apex为非必须安装包，如果安装，参考下述源码方式安装。" +msgstr "Install Apex (version 23.05): apex is an optional installation package; if you choose to install it, refer to the following source code method for installation." + +#: ../../../install.md:34 +msgid "源码方式安装" +msgstr "Installation through Source Code" + +#: ../../../install.md:35 +msgid "依赖包" +msgstr "Dependent Package" + +#: ../../../install.md:36 +msgid "首先，需要安装的依赖包及对应版本列表如下：" +msgstr "The required packages and corresponding version are shown as follows:" + +#: ../../../install.md:37 +msgid "GCC == 10.2.0" msgstr "" -#: ../../../install.md:12 -msgid "Flash-Attention >= v1.0.5" +#: ../../../install.md:38 +msgid "MPFR == 4.1.0" msgstr "" -#: ../../../install.md:13 -msgid "Apex == 23.05" +#: ../../../install.md:39 +msgid "CUDA >= 11.8" msgstr "" -#: ../../../install.md:14 -msgid "Ampere或者Hopper架构的GPU (例如H100, A100)" -msgstr "GPU with Ampere or Hopper architecture (such as H100, A100)" +#: ../../../install.md:40 +msgid "Pytorch >= 2.1.0" +msgstr "" -#: ../../../install.md:15 -msgid "Linux OS" +#: ../../../install.md:41 +msgid "Transformers >= 4.28.0" msgstr "" -#: ../../../install.md:17 +#: ../../../install.md:43 msgid "以上依赖包安装完成后，需要更新配置系统环境变量：" msgstr "" "After installing the above dependencies, some system environment " "variables need to be updated:" -#: ../../../install.md:29 -msgid "将项目`internlm`及其依赖子模块，从 github 仓库中 clone 下来，命令如下：" +#: ../../../install.md:54 +msgid "安装过程" +msgstr "Installation Procedure" + +#: ../../../install.md:55 +msgid "将项目`InternEvo`及其依赖子模块，从 github 仓库中 clone 下来，命令如下：" msgstr "" -"Clone the project `internlm` and its dependent submodules from the github" +"Clone the project `InternEvo` and its dependent submodules from the github" " repository, as follows:" -#: ../../../install.md:34 +#: ../../../install.md:60 msgid "推荐使用 conda 构建一个 Python-3.10 的虚拟环境， 并基于`requirements/`文件安装项目所需的依赖包：" msgstr "" "It is recommended to build a Python-3.10 virtual environment using conda " "and install the required dependencies based on the `requirements/` files:" -#: ../../../install.md:43 -msgid "安装 flash-attention (version v1.0.5)：" -msgstr "Install flash-attention (version v1.0.5):" - -#: ../../../install.md:55 +#: ../../../install.md:79 msgid "安装 Apex (version 23.05)：" msgstr "Install Apex (version 23.05):" -#: ../../../install.md:62 +#: ../../../install.md:89 +msgid "额外安装" +msgstr "Additional Install" + +#: ../../../install.md:94 msgid "环境镜像" msgstr "Environment Image" -#: ../../../install.md:63 +#: ../../../install.md:95 msgid "" "用户可以使用提供的 dockerfile 结合 docker.Makefile 来构建自己的镜像，或者也可以从 " -"https://hub.docker.com/r/internlm/internlm 获取安装了 InternEvo 运行环境的镜像。" +"https://hub.docker.com/r/internlm/internevo/tags 获取安装了 InternEvo 运行环境的镜像。" msgstr "" "Users can use the provided dockerfile combined with docker.Makefile to " "build their own images, or obtain images with InternEvo runtime " -"environment installed from https://hub.docker.com/r/internlm/internlm." +"environment installed from https://hub.docker.com/r/internlm/internevo/tags." -#: ../../../install.md:65 +#: ../../../install.md:97 msgid "镜像配置及构造" msgstr "Image Configuration and Build" -#: ../../../install.md:66 +#: ../../../install.md:98 msgid "" "dockerfile 的配置以及构造均通过 docker.Makefile 文件实现，在 InternEvo 根目录下执行如下命令即可 build" " 镜像：" @@ -122,39 +151,97 @@ msgstr "" " docker.Makefile. To build the image, execute the following command in " "the root directory of InternEvo:" -#: ../../../install.md:70 +#: ../../../install.md:102 msgid "" -"在 docker.Makefile 中可自定义基础镜像，环境版本等内容，对应参数可直接通过命令行传递。对于 BASE_OS 分别支持 " -"ubuntu20.04 和 centos7。" +"在 docker.Makefile 中可自定义基础镜像，环境版本等内容，对应参数可直接通过命令行传递，默认为推荐的环境版本。对于 BASE_OS " +"分别支持 ubuntu20.04 和 centos7。" msgstr "" "In docker.Makefile, you can customize the basic image, environment " "version, etc., and the corresponding parameters can be passed directly " "through the command line. For BASE_OS, ubuntu20.04 and centos7 are " "respectively supported." -#: ../../../install.md:72 +#: ../../../install.md:104 msgid "镜像拉取" msgstr "Pull Standard Image" -#: ../../../install.md:73 +#: ../../../install.md:105 msgid "基于 ubuntu 和 centos 的标准镜像已经 build 完成也可直接拉取使用：" msgstr "" "The standard image based on ubuntu and centos has been built and can be " "directly pulled:" -#: ../../../install.md:82 +#: ../../../install.md:114 msgid "容器启动" msgstr "Run Container" -#: ../../../install.md:83 +#: ../../../install.md:115 msgid "对于使用 dockerfile 构建或拉取的本地标准镜像，使用如下命令启动并进入容器：" msgstr "" "For the local standard image built with dockerfile or pulled, use the " "following command to run and enter the container:" -#: ../../../install.md:87 -msgid "容器内默认目录即 `/InternLM`，根据[使用文档](./usage.md)即可启动训练。" +#: ../../../install.md:120 +msgid "训练启动" +msgstr "Training start" + +#: ../../../install.md:121 +msgid "容器内默认目录即 `/InternEvo`，参考[使用文档](./usage.md)可获取具体使用方法。默认7B模型启动单机8卡训练命令样例：" +msgstr "" +"The default directory in the container is `/InternEvo`, please start " +"training according to the [Usage](./usage.md). Default command example to launch the training of a 7B model on a single machine with 8 GPUs:" + +#: ../../../install.md:126 +msgid "环境安装（NPU）" +msgstr "Installation (NPU)" + +#: ../../../install.md:127 +msgid "" +"在搭载NPU的机器上安装环境的版本可参考GPU，在NPU上使用昇腾torch_npu代替torch，同时Flash-" +"Attention和Apex不再支持安装，相应功能已由InternEvo代码内部实现。以下教程仅为torch_npu安装。" +msgstr "" +"The version for environment installation on machines with NPUs can refer to that of GPUs, using Ascend's torch_npu instead of torch on the NPU. Flash-Attention and Apex are no longer supported for installation, as the corresponding functionalities have been implemented internally by the InternEvo code. The following tutorial is only for the installation of torch_npu." + +#: ../../../install.md:129 +msgid "torch_npu官方文档：https://gitee.com/ascend/pytorch" +msgstr "torch_npu offical documentation：https://gitee.com/ascend/pytorch" + +#: ../../../install.md:131 +msgid "环境安装样例" +msgstr "Installation Example" + +#: ../../../install.md:133 +msgid "torch_npu: v2.1.0-6.0.rc1" msgstr "" -"The default directory in the container is `/InternLM`, please start " -"training according to the [Usage](./usage.md)." + +#: ../../../install.md:134 +msgid "NPU显卡：910B" +msgstr "NPU: 910B" + +#: ../../../install.md:137 +msgid "安装torch_run" +msgstr "Install torch_run" + +#: ../../../install.md:139 +msgid "参考文档：https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/" +msgstr "Reference documentation: https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/" + +#: ../../../install.md:141 +msgid "" +"安装时可尝试根据文档内方式安装，或者从 https://gitee.com/ascend/pytorch/releases " +"下载指定版本torch_npu进行安装，如下所示：" +msgstr "" +"When installing, you may try to install according to the method described in the documentation, or download a specific version of torch_npu for installation from Ascend's PyTorch Releases on Gitee https://gitee.com/ascend/pytorch/releases, as follows:" + +#~ msgid "CUDA >= 11.7" +#~ msgstr "" + +#~ msgid "Pytorch >= 1.13.1" +#~ msgstr "" + +#~ msgid "Flash-Attention >= v1.0.5" +#~ msgstr "" + +#~ msgid "Apex == 23.05" +#~ msgstr "" diff --git a/doc/code-docs/locales/en/LC_MESSAGES/moe.po b/doc/code-docs/locales/en/LC_MESSAGES/moe.po index f2fe3d0aa..937c03c63 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/moe.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/moe.po @@ -3,12 +3,11 @@ # This file is distributed under the same license as the InternLM package. # FIRST AUTHOR , 2023. # -#, fuzzy msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-23 14:49+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -17,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/moe.rst:2 msgid "混合专家模型" @@ -72,8 +71,8 @@ msgstr "Model related settings" #: ../../source/moe.rst:32 msgid "num_experts：专家网络个数。在InternEvo中，每个专家有着相同的网络结构但维护着不同的训练参数。" msgstr "" -"num_experts: The number of expert networks. In InternEvo, each expert has " -"the same network structure but maintains different training parameters." +"num_experts: The number of expert networks. In InternEvo, each expert has" +" the same network structure but maintains different training parameters." #: ../../source/moe.rst:33 msgid "" @@ -146,83 +145,72 @@ msgstr "" "distributed training can start as the normal training process. please " "refer to :doc:`/usage`: Start Training" -#: internlm.model.moe.MoE:1 of -msgid "Initialize an MoE layer." +#: ../../source/moe.rst:65 +msgid "注意：InternEvo支持用户定义自己的MoE结构。internlm.model.moe.MoE是定义MoE网络的接口，目前使用SwiGLU网络实现了专家模型并支持top1gating和top2gating两种门控策略。用户可以在MoE接口中对专家网络和门控策略进行扩展。" msgstr "" +"Note: InternEvo supports users to define their own MoE structure. " +"internlm.model.moe.MoE is the interface that defines the MoE network. " +"Currently, the SwiGLU network is used to implement the experts and " +"supports two gating strategies: top1gating and top2gating. Users can " +"extend the expert network and gating strategy in the MoE interface as " +"needed." -#: internlm.model.moe.MoE of -msgid "参数" -msgstr "parameter" +#~ msgid "Initialize an MoE layer." +#~ msgstr "" -#: internlm.model.moe.MoE:3 of -msgid "" -"the hidden dimension of the model, importantly this is also the input and" -" output dimension." -msgstr "" +#~ msgid "参数" +#~ msgstr "parameter" -#: internlm.model.moe.MoE:5 of -msgid "default=1, the total number of experts per layer." -msgstr "" +#~ msgid "" +#~ "the hidden dimension of the model, " +#~ "importantly this is also the input " +#~ "and output dimension." +#~ msgstr "" -#: internlm.model.moe.MoE:7 of -msgid "default=1, number of ranks in the expert parallel world or group." -msgstr "" +#~ msgid "default=1, the total number of experts per layer." +#~ msgstr "" -#: internlm.model.moe.MoE:9 of -msgid "default=1, top-k gating value, only supports k=1 or k=2." -msgstr "" +#~ msgid "default=1, number of ranks in the expert parallel world or group." +#~ msgstr "" -#: internlm.model.moe.MoE:11 of -msgid "default=1.0, the capacity of the expert at training time." -msgstr "" +#~ msgid "default=1, top-k gating value, only supports k=1 or k=2." +#~ msgstr "" -#: internlm.model.moe.MoE:13 of -msgid "default=1.0, the capacity of the expert at eval time." -msgstr "" +#~ msgid "default=1.0, the capacity of the expert at training time." +#~ msgstr "" -#: internlm.model.moe.MoE:15 of -msgid "" -"default=4, the minimum capacity per expert regardless of the " -"capacity_factor." -msgstr "" +#~ msgid "default=1.0, the capacity of the expert at eval time." +#~ msgstr "" -#: internlm.model.moe.MoE:17 of -msgid "" -"default=None, noisy gate policy, valid options are 'Jitter', 'RSample' or" -" 'None'." -msgstr "" +#~ msgid "" +#~ "default=4, the minimum capacity per " +#~ "expert regardless of the capacity_factor." +#~ msgstr "" -#: internlm.model.moe.MoE:20 of -msgid "default=True, whether to use the default MoE layer." -msgstr "" +#~ msgid "" +#~ "default=None, noisy gate policy, valid " +#~ "options are 'Jitter', 'RSample' or " +#~ "'None'." +#~ msgstr "" -#: internlm.model.moe.MoE:22 of -msgid "" -"default=True, whether to drop tokens - (setting to False is equivalent to" -" infinite capacity)." -msgstr "" +#~ msgid "default=True, whether to use the default MoE layer." +#~ msgstr "" -#: internlm.model.moe.MoE:25 of -msgid "default=True, whether to use Random Token Selection." -msgstr "" +#~ msgid "" +#~ "default=True, whether to drop tokens -" +#~ " (setting to False is equivalent to" +#~ " infinite capacity)." +#~ msgstr "" -#: internlm.model.moe.MoE:27 of -msgid "" -"default=False, make this MoE layer a Residual MoE " -"(https://arxiv.org/abs/2201.05596) layer." -msgstr "" +#~ msgid "default=True, whether to use Random Token Selection." +#~ msgstr "" -#: internlm.model.moe.MoE:30 of -msgid "default=None, the torch module that defines the residual MLP." -msgstr "" +#~ msgid "" +#~ "default=False, make this MoE layer a " +#~ "Residual MoE (https://arxiv.org/abs/2201.05596) " +#~ "layer." +#~ msgstr "" -#: ../../source/moe.rst:65 -msgid "注意：InternEvo支持用户定义自己的MoE结构。internlm.model.moe.MoE是定义MoE网络的接口，目前使用SwiGLU网络实现了专家模型并支持top1gating和top2gating两种门控策略。用户可以在MoE接口中对专家网络和门控策略进行扩展。" -msgstr "" -"Note: InternEvo supports users to define their own MoE structure. " -"internlm.model.moe.MoE is the interface that defines the MoE network. " -"Currently, the SwiGLU network is used to implement the experts and " -"supports two gating strategies: top1gating and top2gating. Users can " -"extend the expert network and gating strategy in the MoE interface as " -"needed." +#~ msgid "default=None, the torch module that defines the residual MLP." +#~ msgstr "" diff --git a/doc/code-docs/locales/en/LC_MESSAGES/monitor.po b/doc/code-docs/locales/en/LC_MESSAGES/monitor.po index 0e0667305..55f0b6d16 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/monitor.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/monitor.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-23 20:19+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/monitor.rst:2 msgid "监控和告警" @@ -38,93 +38,6 @@ msgstr "" " and track training status with " "``internlm.monitor.monitor.MonitorTracker``." -#: internlm.monitor.monitor.initialize_monitor_manager:1 of -msgid "" -"Initialize monitor manager for monitoring training lifetime and alerting " -"exception info to Feishu." -msgstr "" - -#: internlm.monitor.alert.initialize_light_monitor -#: internlm.monitor.alert.send_feishu_msg_with_webhook -#: internlm.monitor.alert.send_heartbeat -#: internlm.monitor.monitor.MonitorManager.start_monitor -#: internlm.monitor.monitor.MonitorTracker -#: internlm.monitor.monitor.initialize_monitor_manager of -msgid "参数" -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.start_monitor:3 -#: internlm.monitor.monitor.initialize_monitor_manager:3 of -msgid "The training job name." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.start_monitor:5 -#: internlm.monitor.monitor.initialize_monitor_manager:5 of -msgid "The Feishu webhook address for sending alert messages." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager:1 of -msgid "" -"Monitor Manager for managing monitor thread and monitoring training " -"status." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.monitor_loss_spike:1 of -msgid "Check loss value, if loss spike occurs, send alert message to Feishu." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.monitor_exception:1 of -msgid "Catch and format exception information, send alert message to Feishu." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.handle_sigterm:1 of -msgid "Catch SIGTERM signal, and send alert message to Feishu." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.start_monitor:1 of -msgid "" -"Initialize and start monitor thread for checking training job status, " -"loss spike and so on." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.start_monitor:7 of -msgid "The time of monitor interval in seconds, defaults to 300." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.start_monitor:9 of -msgid "" -"The limit multiple of current loss to previous loss value, which means " -"loss spike may be occurs, defaults to 1.5." -msgstr "" - -#: internlm.monitor.monitor.MonitorManager.stop_monitor:1 of -msgid "Stop the monitor and alert thread." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker:1 of -msgid "Track job status and alert to Feishu during job training." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker:3 of -msgid "The Feishu webhook address for sending alerting messages." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker:5 of -msgid "The interval in seconds for monitoring checks. Defaults to 300." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker:7 of -msgid "The threshold for detecting loss value spikes. Defaults to 1.5." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker.run:1 of -msgid "start the monitor tracker." -msgstr "" - -#: internlm.monitor.monitor.MonitorTracker.stop:1 of -msgid "Stop the monitor tracker." -msgstr "" - #: ../../source/monitor.rst:18 msgid "告警" msgstr "Alerting" @@ -142,39 +55,6 @@ msgstr "" "Feishu webhook address by calling " "``internlm.monitor.alert.send_feishu_msg_with_webhook()``." -#: internlm.monitor.alert.send_feishu_msg_with_webhook:1 of -msgid "Use Feishu robot to send messages with the given webhook." -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook:3 of -msgid "The webhook to be used to send message." -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook:5 of -msgid "The message title." -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook:7 of -msgid "The message body." -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook of -msgid "返回" -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook:10 of -msgid "The response from the request. Or catch the exception and return None." -msgstr "" - -#: internlm.monitor.alert.initialize_light_monitor -#: internlm.monitor.alert.send_feishu_msg_with_webhook of -msgid "抛出" -msgstr "" - -#: internlm.monitor.alert.send_feishu_msg_with_webhook:12 of -msgid "An exception rasied by the HTTP post request." -msgstr "" - #: ../../source/monitor.rst:25 msgid "轻量监控" msgstr "Light Monitoring" @@ -185,10 +65,10 @@ msgid "" " `grafana dashboard `_ " "直观地呈现这些指标信息，以便用户进行更加全面和深入的训练分析。" msgstr "" -"The InternEvo light monitoring tool employs a heartbeat mechanism to real-" -"time monitor various metrics during the training process, such as loss, " -"grad_norm, and training phase duration. Additionally, InternEvo can " -"present these metric details through a `grafana dashboard " +"The InternEvo light monitoring tool employs a heartbeat mechanism to " +"real-time monitor various metrics during the training process, such as " +"loss, grad_norm, and training phase duration. Additionally, InternEvo can" +" present these metric details through a `grafana dashboard " "`_, allowing users to conduct " "more comprehensive and in-depth training analysis in an intuitive manner." @@ -201,8 +81,8 @@ msgstr "" "The configuration for light monitoring is specified by the ``monitor`` " "field in the configuration file. Users can modify monitoring settings by " "editing the configuration file `config file " -"`_. " -"Here is an example of a monitoring configuration:" +"`_." +" Here is an example of a monitoring configuration:" #: ../../source/monitor.rst:41 msgid "enable_feishu_alert (bool)：是否启用飞书告警。默认值：False。" @@ -235,67 +115,147 @@ msgstr "" " monitoring server uses these heartbeat messages to detect if the " "training encounters any abnormalities and sends alert messages as needed." -#: internlm.monitor.alert.initialize_light_monitor:1 of -msgid "Initialize the lightweight monitoring module." -msgstr "" +#~ msgid "" +#~ "InternLM轻量监控基于心跳机制来监控训练过程中是否出现 " +#~ "loss、grad_norm异常、训练各阶段时间超时等异常，并通过dashboard展示训练指标信息等。" +#~ msgstr "" -#: internlm.monitor.alert.initialize_light_monitor:3 of -msgid "The address of the monitor. Defaults to 'MONITOR_SERVER' environment." -msgstr "" +#~ msgid "" +#~ "Initialize monitor manager for monitoring " +#~ "training lifetime and alerting exception " +#~ "info to Feishu." +#~ msgstr "" -#: internlm.monitor.alert.initialize_light_monitor:6 of -msgid "" -"If any exceptions occur during initialization, they will be caught and " -"logged as warnings." -msgstr "" +#~ msgid "参数" +#~ msgstr "" -#: internlm.monitor.alert.initialize_light_monitor:9 -#: internlm.monitor.alert.send_heartbeat:9 of -msgid "示例" -msgstr "Example" +#~ msgid "The training job name." +#~ msgstr "" -#: internlm.monitor.alert.initialize_light_monitor:10 of -msgid "" -"Initialize the monitoring module with the default address " -"``initialize_light_monitor()``" -msgstr "" +#~ msgid "The Feishu webhook address for sending alert messages." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:1 of -msgid "Send a heartbeat message to a monitoring server." -msgstr "" +#~ msgid "" +#~ "Monitor Manager for managing monitor " +#~ "thread and monitoring training status." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:3 of -msgid "" -"The type of heartbeat message, e.g., \"train_metrics\", \"init_time\", " -"\"stage_time\"." -msgstr "" +#~ msgid "Check loss value, if loss spike occurs, send alert message to Feishu." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:5 of -msgid "A dictionary containing message data to be included in the heartbeat." -msgstr "" +#~ msgid "Catch and format exception information, send alert message to Feishu." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:10 of -msgid "" -"Sending a heartbeat message for training metrics " -"``send_heartbeat(\"train_metrics\", {\"loss\": 0.1, \"accuracy\": " -"0.95})``" -msgstr "" +#~ msgid "Catch SIGTERM signal, and send alert message to Feishu." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:13 of -msgid "" -"Sending a heartbeat message for initialization time " -"``send_heartbeat(\"init_time\", {\"import_time\": 0.25})``" -msgstr "" +#~ msgid "" +#~ "Initialize and start monitor thread for" +#~ " checking training job status, loss " +#~ "spike and so on." +#~ msgstr "" -#: internlm.monitor.alert.send_heartbeat:16 of -msgid "" -"Sending a heartbeat message for stage time " -"``send_heartbeat(\"stage_time\", {\"fwd_time\": 2.3, \"bwd_time\": " -"6.2})``" -msgstr "" +#~ msgid "The time of monitor interval in seconds, defaults to 300." +#~ msgstr "" #~ msgid "" -#~ "InternLM轻量监控基于心跳机制来监控训练过程中是否出现 " -#~ "loss、grad_norm异常、训练各阶段时间超时等异常，并通过dashboard展示训练指标信息等。" +#~ "The limit multiple of current loss " +#~ "to previous loss value, which means " +#~ "loss spike may be occurs, defaults " +#~ "to 1.5." +#~ msgstr "" + +#~ msgid "Stop the monitor and alert thread." +#~ msgstr "" + +#~ msgid "Track job status and alert to Feishu during job training." +#~ msgstr "" + +#~ msgid "The Feishu webhook address for sending alerting messages." +#~ msgstr "" + +#~ msgid "The interval in seconds for monitoring checks. Defaults to 300." +#~ msgstr "" + +#~ msgid "The threshold for detecting loss value spikes. Defaults to 1.5." +#~ msgstr "" + +#~ msgid "start the monitor tracker." +#~ msgstr "" + +#~ msgid "Stop the monitor tracker." +#~ msgstr "" + +#~ msgid "Use Feishu robot to send messages with the given webhook." +#~ msgstr "" + +#~ msgid "The webhook to be used to send message." +#~ msgstr "" + +#~ msgid "The message title." +#~ msgstr "" + +#~ msgid "The message body." +#~ msgstr "" + +#~ msgid "返回" +#~ msgstr "" + +#~ msgid "The response from the request. Or catch the exception and return None." +#~ msgstr "" + +#~ msgid "抛出" +#~ msgstr "" + +#~ msgid "An exception rasied by the HTTP post request." +#~ msgstr "" + +#~ msgid "Initialize the lightweight monitoring module." +#~ msgstr "" + +#~ msgid "The address of the monitor. Defaults to 'MONITOR_SERVER' environment." +#~ msgstr "" + +#~ msgid "" +#~ "If any exceptions occur during " +#~ "initialization, they will be caught and" +#~ " logged as warnings." +#~ msgstr "" + +#~ msgid "示例" +#~ msgstr "Example" + +#~ msgid "" +#~ "Initialize the monitoring module with " +#~ "the default address ``initialize_light_monitor()``" +#~ msgstr "" + +#~ msgid "Send a heartbeat message to a monitoring server." +#~ msgstr "" + +#~ msgid "" +#~ "The type of heartbeat message, e.g., " +#~ "\"train_metrics\", \"init_time\", \"stage_time\"." +#~ msgstr "" + +#~ msgid "A dictionary containing message data to be included in the heartbeat." +#~ msgstr "" + +#~ msgid "" +#~ "Sending a heartbeat message for training" +#~ " metrics ``send_heartbeat(\"train_metrics\", {\"loss\":" +#~ " 0.1, \"accuracy\": 0.95})``" +#~ msgstr "" + +#~ msgid "" +#~ "Sending a heartbeat message for " +#~ "initialization time ``send_heartbeat(\"init_time\", " +#~ "{\"import_time\": 0.25})``" +#~ msgstr "" + +#~ msgid "" +#~ "Sending a heartbeat message for stage" +#~ " time ``send_heartbeat(\"stage_time\", {\"fwd_time\":" +#~ " 2.3, \"bwd_time\": 6.2})``" #~ msgstr "" diff --git a/doc/code-docs/locales/en/LC_MESSAGES/parallel.po b/doc/code-docs/locales/en/LC_MESSAGES/parallel.po index 326945f60..c32bc4a6e 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/parallel.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/parallel.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-02-04 15:50+0800\n" +"POT-Creation-Date: 2024-08-23 14:49+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/parallel.rst:2 msgid "并行训练" @@ -64,11 +64,10 @@ msgstr "" "retain the complete optimizer state parameters." #: ../../source/parallel.rst:23 -msgid "" -"当 ``size > 1`` 且 ``size <= data_parallel_size``，则 zero1 进程组是数据并行进程组的子集" +msgid "当 ``size > 1`` 且 ``zero1 <= data_parallel_size``，则 zero1 进程组是数据并行进程组的子集" msgstr "" -"When ``size > 1`` and ``size <= data_parallel_size``, the zero1 " -"process group is a subset of the data parallel process group." +"When ``size > 1`` and ``size <= data_parallel_size``, the zero1 process " +"group is a subset of the data parallel process group." #: ../../source/parallel.rst:25 msgid "tensor：张量并行策略" @@ -77,8 +76,8 @@ msgstr "tensor: tensor parallel strategy" #: ../../source/parallel.rst:27 msgid "size：张量并行大小，通常是每个节点的 GPU 数量，默认值为 1" msgstr "" -"size: int, tensor parallel size, usually the number of GPUs per node, the " -"default value is 1" +"size: int, tensor parallel size, usually the number of GPUs per node, the" +" default value is 1" #: ../../source/parallel.rst:28 msgid "mode：张量并行模式，支持['mtp', 'msp', 'fsp', 'isp']，其中，" @@ -86,19 +85,27 @@ msgstr "mode: the tensor parallel mode, should be in ['mtp', 'msp', 'fsp', 'isp' #: ../../source/parallel.rst:30 msgid "mtp：表示使用 Megatron-LM 的张量并行实现方案，不包含序列并行，mtp 为默认模式" -msgstr "mtp: defaults to 'mtp', means the pure megatron tensor parallel without sequence parallel" +msgstr "" +"mtp: defaults to 'mtp', means the pure megatron tensor parallel without " +"sequence parallel" #: ../../source/parallel.rst:31 msgid "msp：表示使用 Megatron-LM 的序列化并行实现方案，序列并行大小 = 张量并行大小" -msgstr "msp: megatron tensor parallel with sequence parallel, sequence parallel size = tensor parallel size" +msgstr "" +"msp: megatron tensor parallel with sequence parallel, sequence parallel " +"size = tensor parallel size" #: ../../source/parallel.rst:32 msgid "fsp：表示使用 flash-attn 实现模式的张量并行与序列并行，序列并行大小 = 张量并行大小" -msgstr "fsp: tensor parallel by flash-attn with sequence parallel, sequence parallel size = tensor parallel size" +msgstr "" +"fsp: tensor parallel by flash-attn with sequence parallel, sequence " +"parallel size = tensor parallel size" #: ../../source/parallel.rst:33 msgid "isp：InternEvo 系统自研的序列化并行方案，可以与权重并行结合使用，序列并行大小与权重并行大小互相独立" -msgstr "isp: customed intern sequence parallel without tensor parallel, can be used with weight parallel" +msgstr "" +"isp: customed intern sequence parallel without tensor parallel, can be " +"used with weight parallel" #: ../../source/parallel.rst:35 msgid "pipeline：流水线并行策略" @@ -116,7 +123,9 @@ msgstr "" #: ../../source/parallel.rst:40 msgid "weight：权重并行策略，只能与 isp 张量并行模式结合使用" -msgstr "weight: weight parallel strategy, only can be used with 'isp' tensor parallel mode" +msgstr "" +"weight: weight parallel strategy, only can be used with 'isp' tensor " +"parallel mode" #: ../../source/parallel.rst:42 msgid "size：权重并行大小，默认值为 1" @@ -124,7 +133,9 @@ msgstr "size: weight parallel size, the default value is 1" #: ../../source/parallel.rst:43 msgid "overlap：是否开启计算与通信的 overlap，默认值为 False" -msgstr "overlap: bool, enable/disable all_gather/reduce_scatter communication overlap, defaults to False" +msgstr "" +"overlap: bool, enable/disable all_gather/reduce_scatter communication " +"overlap, defaults to False" #: ../../source/parallel.rst:44 msgid "memory_pool：是否开启权重显存池，默认值为 False" @@ -146,12 +157,14 @@ msgid "" "'fsp', 'isp']，前三种模式为基于 Megatron-LM 的张量并行和序列化并行的策略实现，最后一种模式为 InternEvo " "系统自研，可与权重并行（Weight Parallel）结合使用的一种新的序列化并行方式。接下来详细介绍这几种张量并行模式的区别。" msgstr "" -"The InternEvo system version ``v0.3.0`` has significant updates in the tensor parallelism strategy. " -"The current tensor parallelism supports four modes: ['mtp', 'msp', 'fsp', 'isp']. The first three modes " -"are based on the Megatron-LM's tensor parallelism and sequence parallelism strategy. The last mode is a " -"self-developed strategy by the InternEvo system, which is a new sequence parallelism method that can be used " -"in conjunction with weight parallelism. The following provides a detailed explanation of the differences among " -"these tensor parallelism modes." +"The InternEvo system version ``v0.3.0`` has significant updates in the " +"tensor parallelism strategy. The current tensor parallelism supports four" +" modes: ['mtp', 'msp', 'fsp', 'isp']. The first three modes are based on " +"the Megatron-LM's tensor parallelism and sequence parallelism strategy. " +"The last mode is a self-developed strategy by the InternEvo system, which" +" is a new sequence parallelism method that can be used in conjunction " +"with weight parallelism. The following provides a detailed explanation of" +" the differences among these tensor parallelism modes." #: ../../source/parallel.rst:53 msgid "MTP" @@ -163,10 +176,11 @@ msgid "" "Parallel `_ 并行方案，如下图所示，带有张量并行的 " "``Transformer`` 层：" msgstr "" -"MTP (Megatron-LM Tensor Parallel) is the default tensor parallelism model, inspired by the Megatron-LM Tensor Parallel parallel scheme, " -"as referenced in the paper `Megatron-LM Tensor Parallel `_. " -"The following diagram illustrates the ``Transformer`` layer with tensor parallelism:" - +"MTP (Megatron-LM Tensor Parallel) is the default tensor parallelism " +"model, inspired by the Megatron-LM Tensor Parallel parallel scheme, as " +"referenced in the paper `Megatron-LM Tensor Parallel " +"`_. The following diagram illustrates " +"the ``Transformer`` layer with tensor parallelism:" #: ../../source/parallel.rst:61 msgid "Transformer layer with tensor parallelism." @@ -181,18 +195,23 @@ msgid "" " 这两个模块进行张量并行操作。假设张量并行大小为 ``tp``，输入数据的序列长度为 ``seqlen``，隐藏层大小为 ``hidden " "size``，则张量并行期间产生的激活值 ``shape`` 为 ``[seqlen, hidden_size/tp]``。" msgstr "" -"MTP primarily applies tensor parallelism operations to the attention and the linear module. " -"Assuming the tensor parallelism size is tp, the sequence length of input data is seqlen, and the hidden layer size is hidden size, " -"then the shape of the activation values generated during tensor parallelism is ``[seqlen, hidden_size/tp]``." +"MTP primarily applies tensor parallelism operations to the attention and " +"the linear module. Assuming the tensor parallelism size is tp, the " +"sequence length of input data is seqlen, and the hidden layer size is " +"hidden size, then the shape of the activation values generated during " +"tensor parallelism is ``[seqlen, hidden_size/tp]``." #: ../../source/parallel.rst:65 msgid "" "MTP 张量并行引入的通信如上图所示，其中 ``f`` 和 ``f̄`` 是共轭的。在前向传递中 ``f`` 是无操作，而在反向传递中进行 " "``all-reduce`` 操作。而 ``f̄`` 在前向传递中进行 ``all-reduce`` 操作，在反向传递中是无操作。" msgstr "" -"The communication introduced by MTP is illustrated in the above diagram, where ``f`` and ``f̄`` are conjugates. " -"In the forward pass, ``f`` corresponds to a no-operation, while in the backward pass, it involves an ``all-reduce`` operation. " -"On the other hand, ``f̄`` performs an ``all-reduce`` operation in the forward pass and is a no-operation in the backward pass." +"The communication introduced by MTP is illustrated in the above diagram, " +"where ``f`` and ``f̄`` are conjugates. In the forward pass, ``f`` " +"corresponds to a no-operation, while in the backward pass, it involves an" +" ``all-reduce`` operation. On the other hand, ``f̄`` performs an ``all-" +"reduce`` operation in the forward pass and is a no-operation in the " +"backward pass." #: ../../source/parallel.rst:67 msgid "MSP" @@ -204,8 +223,9 @@ msgid "" "`_ 并行方案，如下图所示，带有张量并行和序列化并行的 " "``Transformer`` 层：" msgstr "" -"MSP (Megatron-LM Sequence Parallel) is adopted from the Megatron-LM Sequence Parallel parallel scheme. " -"The following diagram illustrates the Transformer layer with both tensor parallelism and sequence parallelism:" +"MSP (Megatron-LM Sequence Parallel) is adopted from the Megatron-LM " +"Sequence Parallel parallel scheme. The following diagram illustrates the " +"Transformer layer with both tensor parallelism and sequence parallelism:" #: ../../source/parallel.rst:75 msgid "Transformer layer with tensor and sequence parallelism." @@ -218,11 +238,15 @@ msgid "" "``seqlen``，隐藏层大小为 ``hidden size``，则序列化并行期间产生的激活值形状为 ``[seqlen/tp, " "hidden_size]``，张量并行期间产生的激活值形状为 ``[seqlen, hidden_size/tp]``。" msgstr "" -"Compared to MTP, it is evident that MSP primarily focuses on modules without tensor parallelism, such as ``LayerNorm`` " -"and ``Dropout``, and performs sequence parallelism operations. It is important to note that the size of sequence parallelism " -"is equal to the size of tensor parallelism, and they share the same communication group. Assuming the tensor parallelism size is tp, " -"the input data has a sequence length of seqlen, and the hidden layer size is hidden size, the shape of activation values during " -"sequence parallelism is ``[seqlen/tp, hidden_size]``, while during tensor parallelism, it is ``[seqlen, hidden_size/tp]``." +"Compared to MTP, it is evident that MSP primarily focuses on modules " +"without tensor parallelism, such as ``LayerNorm`` and ``Dropout``, and " +"performs sequence parallelism operations. It is important to note that " +"the size of sequence parallelism is equal to the size of tensor " +"parallelism, and they share the same communication group. Assuming the " +"tensor parallelism size is tp, the input data has a sequence length of " +"seqlen, and the hidden layer size is hidden size, the shape of activation" +" values during sequence parallelism is ``[seqlen/tp, hidden_size]``, " +"while during tensor parallelism, it is ``[seqlen, hidden_size/tp]``." #: ../../source/parallel.rst:79 msgid "" @@ -230,10 +254,13 @@ msgid "" "gather`` 操作，而在反向传递中进行 ``reduce-scatter`` 操作。而 ``ḡ`` 在前向传递中进行 ``reduce-" "scatter`` 操作，在反向传递中进行 ``all-gather`` 操作。" msgstr "" -"In comparison to MTP, there are variations in the communication primitives in MSP, as illustrated in the diagram above, " -"where ``g`` and ``ḡ`` are conjugates. In the forward pass, ``g`` performs an ``all-gather`` operation, while in the backward pass, " -"it undergoes a ``reduce-scatter`` operation. On the other hand, ``ḡ`` conducts a ``reduce-scatter`` operation in the forward pass and " -"an ``all-gather`` operation in the backward pass." +"In comparison to MTP, there are variations in the communication " +"primitives in MSP, as illustrated in the diagram above, where ``g`` and " +"``ḡ`` are conjugates. In the forward pass, ``g`` performs an ``all-" +"gather`` operation, while in the backward pass, it undergoes a ``reduce-" +"scatter`` operation. On the other hand, ``ḡ`` conducts a ``reduce-" +"scatter`` operation in the forward pass and an ``all-gather`` operation " +"in the backward pass." #: ../../source/parallel.rst:81 msgid "" @@ -243,13 +270,19 @@ msgid "" "scatter``，才能完成 ``seqlen`` 维度的切分，激活值形状变成 ``[seqlen/tp, " "hidden_size]``，从而正常进入序列化并行的阶段。而反向传递时，则是同样的道理。" msgstr "" -"In the forward pass, the communication of ``g`` occurs at the junction of sequence parallelism and tensor parallelism, " -"performing an ``all-gather`` operation along the seqlen dimension of activation values. After this communication is completed, " -"the shape of the activation values becomes the full ``[seqlen, hidden_size]``, and then it enters the scope of the tensor parallelism module. " -"The communication of ``ḡ`` is situated at the junction of tensor parallelism and sequence parallelism, requiring the transformation of the " -"``all-reduce`` communication operation from MTP into a ``reduce-scatter`` operation to achieve the split along the seqlen dimension. " -"This results in the activation values having a shape of ``[seqlen/tp, hidden_size]``, enabling a smooth transition into the sequence parallelism phase. " -"The same principles apply during the backward pass." +"In the forward pass, the communication of ``g`` occurs at the junction of" +" sequence parallelism and tensor parallelism, performing an ``all-" +"gather`` operation along the seqlen dimension of activation values. After" +" this communication is completed, the shape of the activation values " +"becomes the full ``[seqlen, hidden_size]``, and then it enters the scope " +"of the tensor parallelism module. The communication of ``ḡ`` is situated " +"at the junction of tensor parallelism and sequence parallelism, requiring" +" the transformation of the ``all-reduce`` communication operation from " +"MTP into a ``reduce-scatter`` operation to achieve the split along the " +"seqlen dimension. This results in the activation values having a shape of" +" ``[seqlen/tp, hidden_size]``, enabling a smooth transition into the " +"sequence parallelism phase. The same principles apply during the backward" +" pass." #: ../../source/parallel.rst:83 msgid "FSP" @@ -263,17 +296,23 @@ msgid "" "FSP 则只存储 ``seqlen`` 切分后的输入数据，因此在进行 backward 计算时，需要再额外 ``all-gather`` " "一次输入。" msgstr "" -"FSP (Flash-Attn Sequence Parallel) is a sequence parallelism implementation inspired by the flash attention scheme, as referenced in " -"`flash attention `_. The only difference between this implementation and MSP is that, " -"after the ``g`` performs ``all-gather`` communication, MSP stores a complete copy of the input data for backward computation, while FSP only " -"retains the input data split into seqlen segments. Therefore, during backward computation, an additional ``all-gather`` operation " -"is needed to retrieve the complete input data." +"FSP (Flash-Attn Sequence Parallel) is a sequence parallelism " +"implementation inspired by the flash attention scheme, as referenced in " +"`flash attention `_. The " +"only difference between this implementation and MSP is that, after the " +"``g`` performs ``all-gather`` communication, MSP stores a complete copy " +"of the input data for backward computation, while FSP only retains the " +"input data split into seqlen segments. Therefore, during backward " +"computation, an additional ``all-gather`` operation is needed to retrieve" +" the complete input data." #: ../../source/parallel.rst:87 msgid "因此，FSP 与 MSP 性能对比的话，会有更小的显存占用，但是由于引入额外的 ``all-gather`` 通信，会导致训练速度 TGS 降低。" msgstr "" -"Therefore, in terms of performance comparison between FSP and MSP, FSP tends to have a smaller memory footprint. However, " -"the introduction of additional ``all-gather`` communication can lead to a reduction in the training speed, denoted as TGS." +"Therefore, in terms of performance comparison between FSP and MSP, FSP " +"tends to have a smaller memory footprint. However, the introduction of " +"additional ``all-gather`` communication can lead to a reduction in the " +"training speed, denoted as TGS." #: ../../source/parallel.rst:90 msgid "ISP" @@ -284,9 +323,13 @@ msgid "" "ISP(Intern Sequence Parallel)，InternEvo " "系统自研的灵活可扩展序列化并行方案，支持张量并行与序列化并行解耦，通过计算和通信的overlap提高训练性能，并基于显存池管理降低显存碎片化的可能性，提高显存利用率。" msgstr "" -"ISP (Intern Sequence Parallel) is a flexible and scalable sequence parallelism solution developed in-house by the InternEvo system. " -"It supports the decoupling of tensor parallelism and sequence parallelism, enhancing training performance through the overlap of computation and communication. " -"Additionally, it incorporates memory pool management to reduce the likelihood of memory fragmentation, thereby improving memory utilization." +"ISP (Intern Sequence Parallel) is a flexible and scalable sequence " +"parallelism solution developed in-house by the InternEvo system. It " +"supports the decoupling of tensor parallelism and sequence parallelism, " +"enhancing training performance through the overlap of computation and " +"communication. Additionally, it incorporates memory pool management to " +"reduce the likelihood of memory fragmentation, thereby improving memory " +"utilization." #: ../../source/parallel.rst:94 msgid "" @@ -298,24 +341,31 @@ msgid "" "``overlap``，可提高训练性能。将 ``weight.memory_pool`` 字段设置为 ``True`` " "即为开启显存池管理功能，可一定程度降低 GPU 显存碎片的可能性，提高显存利用率。" msgstr "" -"Taking the configuration file `configs/7B_isp_sft.py `_ as an example, " -"set the tensor.mode field to isp, where the tensor.size field represents the size of data split along the seqlen dimension. " -"The ISP algorithm can be combined with weight parallel, where the weight.size field represents the model weight split size. " -"Setting weight.overlap to True enables computation and communication overlap, enhancing training performance. " -"Setting weight.memory_pool to True activates the memory pool management feature, which helps to some extent " -"in reducing the likelihood of GPU memory fragmentation and improving memory utilization." +"Taking the configuration file `configs/7B_isp_sft.py " +"`_" +" as an example, set the tensor.mode field to isp, where the tensor.size " +"field represents the size of data split along the seqlen dimension. The " +"ISP algorithm can be combined with weight parallel, where the weight.size" +" field represents the model weight split size. Setting weight.overlap to " +"True enables computation and communication overlap, enhancing training " +"performance. Setting weight.memory_pool to True activates the memory pool" +" management feature, which helps to some extent in reducing the " +"likelihood of GPU memory fragmentation and improving memory utilization." #: ../../source/parallel.rst:105 msgid "如下图所示，带有序列化并行和权重并行的 ``Transformer`` 层：" -msgstr "As illustrated in the diagram below, there is a Transformer layer with both sequence parallelism and weight parallelism:" +msgstr "" +"As illustrated in the diagram below, there is a Transformer layer with " +"both sequence parallelism and weight parallelism:" #: ../../source/parallel.rst:111 msgid "" "如图所示，ISP 的序列化并行范围覆盖整个 ``Transformer`` 模型层，模型权重并行主要针对 ``Attention`` 和 " "``MLP Block`` 的 ``Linear module``。" msgstr "" -"As shown in the figure, the sequence parallelism scope of ISP covers the entire Transformer model layer, " -"while the weight parallelism primarily targets the Linear module within the Attention and MLP Block." +"As shown in the figure, the sequence parallelism scope of ISP covers the " +"entire Transformer model layer, while the weight parallelism primarily " +"targets the Linear module within the Attention and MLP Block." #: ../../source/parallel.rst:113 msgid "" @@ -323,10 +373,12 @@ msgid "" "``Linear`` 在进行后向计算前需要进行模型权重的 ``all-gather`` 通信，在后向计算后，需要进行模型权重的梯度的 " "``reduce-scatter`` 通信操作。" msgstr "" -"The changes in communication primitives are as follows: during the forward pass, each Linear module requires ``all-gather`` communication " -"for model weight. In the backward pass, before performing the backward computation, each Linear module requires ``all-gather`` communication for model weight. " -"After the backward computation, there is a ``reduce-scatter`` communication operation for the gradients of model weights." - +"The changes in communication primitives are as follows: during the " +"forward pass, each Linear module requires ``all-gather`` communication " +"for model weight. In the backward pass, before performing the backward " +"computation, each Linear module requires ``all-gather`` communication for" +" model weight. After the backward computation, there is a ``reduce-" +"scatter`` communication operation for the gradients of model weights." #: ../../source/parallel.rst:115 msgid "" @@ -334,9 +386,12 @@ msgid "" "``Self-Atten`` 前后各增加了一个 ``all-to-all`` 通信操作，用于完成激活值形状的转置，目的是在进行 " "``attention score`` 计算时能保持原有的张量并行的模式。" msgstr "" -"It is important to note that, in comparison to MSP and FSP, there are some changes in communication primitives for attention score calculation in ISP. " -"For instance, before and after ``Self-Atten``, an additional ``all-to-all`` communication operation is introduced to transpose the shape of activation values. " -"The purpose is to maintain the original tensor parallelism pattern during the attention score calculation." +"It is important to note that, in comparison to MSP and FSP, there are " +"some changes in communication primitives for attention score calculation " +"in ISP. For instance, before and after ``Self-Atten``, an additional " +"``all-to-all`` communication operation is introduced to transpose the " +"shape of activation values. The purpose is to maintain the original " +"tensor parallelism pattern during the attention score calculation." #: ../../source/parallel.rst:117 msgid "" @@ -344,8 +399,9 @@ msgid "" "Language Model Training via Hybrid Parallelism and Redundant Sharding " "`_。" msgstr "" -"For more design details and performance evaluation of the ISP algorithm, please refer to the paper " -"`InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding " +"For more design details and performance evaluation of the ISP algorithm, " +"please refer to the paper `InternEvo: Efficient Long-sequence Large " +"Language Model Training via Hybrid Parallelism and Redundant Sharding " "`_." #: ../../source/parallel.rst:121 @@ -385,6 +441,134 @@ msgstr "" "To use non-interleaved pipeline scheduler, users need to set " "``model.num_chunks = 1`` in the config file." +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:1 of +msgid "" +"A helper schedule class for pipeline parallelism running environment. It " +"uses non-interleaved 1F1B strategy. Other properties are similar as " +":class:`NonPipelineSchedule`." +msgstr "" + +#: ../../source/parallel.rst +msgid "参数" +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:5 of +msgid "The number of microbatches." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:7 of +msgid "Type of data. torch.float by default." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:9 of +msgid "" +"The post processing function which receives a micro batch of data, and it" +" will be executed in `load_micro_batch`." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:12 of +msgid "Specified shape in pipeline communication." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:14 of +msgid "" +"If set to `True`, communication will be reduced over pipeline when using " +"1D tensor parallelization." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:16 of +msgid "List of scheduler hooks." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.pre_processing:1 +#: of +msgid "To perform actions before running the schedule." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.pre_processing:3 +#: of +msgid "InternLM engine for training and inference." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:1 +#: of +msgid "" +"Runs non-interleaved 1F1B schedule, with communication between pipeline " +"stages. Returns a tuple with losses if the last stage, an empty tuple " +"otherwise." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:4 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:4 +#: of +msgid "Colossalai engine for training and inference." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:6 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:6 +#: of +msgid "" +"Dataloader as the form of an iterator, obtained by calling " +"iter(dataloader)." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:8 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:8 +#: of +msgid "" +"Whether run forward step only. Default is false. If true, no backward " +"will be run." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:10 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:10 +#: of +msgid "Whether returns the loss value. Default is true." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:12 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:12 +#: of +msgid "If False, the output and label won't be returned." +msgstr "" + +#: ../../source/parallel.rst +msgid "返回" +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:15 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:15 +#: of +msgid "" +"A tuple of (output, label, loss, moe_loss), loss and label could be None." +" The loss would be returned only in the last stage. And the moe_loss " +"is accumulated from all stages." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:17 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:17 +#: of +msgid "A tuple of (output, label, loss, moe_loss), loss and label could be None." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:18 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:18 +#: of +msgid "" +"The loss would be returned only in the last stage. And the moe_loss is " +"accumulated from all stages." +msgstr "" + +#: ../../source/parallel.rst +msgid "返回类型" +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:19 +#: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:19 +#: of +msgid "Tuple[:class:`torch.Tensor`]" +msgstr "" + #: ../../source/parallel.rst:142 msgid "交错式流水线调度" msgstr "scheduler for interleaved 1F1B strategy" @@ -395,6 +579,17 @@ msgstr "" "To use interleaved pipeline scheduler, users need to set " "``model.num_chunks > 1`` in the config file." +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler:1 of +msgid "Interleaved Pipeline Scheduler." +msgstr "" + +#: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:1 +#: of +msgid "" +"Run interleaved 1F1B schedule (model split into model chunks), with " +"communication between pipeline stages as needed." +msgstr "" + #: ../../source/parallel.rst:148 msgid "值得注意的是，在使用交错式流水线调度器时可启用通信优化功能，即在 1F1B 阶段启用异步通信，以充分利用上行/下行带宽并实现通信与计算重叠。" msgstr "" @@ -503,6 +698,53 @@ msgstr "" "These optimizations can speed up the training process and improve " "training efficiency." +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer:1 of +msgid "Hybrid Zero Optimizer." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.belongs_to_current_rank:1 +#: of +msgid "" +"Check whether a parameter is supposed to be updated by the process of the" +" current rank" +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.belongs_to_current_rank:3 +#: of +msgid "A :class:`torch.Tensor` object" +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.belongs_to_current_rank:6 +#: of +msgid "" +"True if the parameter should be updated by the current rank. Otherwise " +"false." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.zero_grad:1 +#: of +msgid "" +"Set parameter gradients to zero. If set_to_none = True, gradient will be " +"set to None to save memory." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.zero_grad:4 +#: of +msgid "Whether set the gradient to None. Default value is True." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.step:1 of +msgid "Performs a single optimization step." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.step:3 of +msgid "A closure that reevaluates the model and returns the loss." +msgstr "" + +#: internlm.solver.optimizer.hybrid_zero_optim.HybridZeroOptimizer.step:7 of +msgid "Whether the gradient is success updated, and the gradient." +msgstr "" + #~ msgid "A tuple of (output, label, loss), loss and label could be None." #~ msgstr "" diff --git a/doc/code-docs/locales/en/LC_MESSAGES/profiler.po b/doc/code-docs/locales/en/LC_MESSAGES/profiler.po index 636b0937f..a6582200a 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/profiler.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/profiler.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-27 10:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/profiler.rst:2 msgid "性能分析" @@ -26,14 +26,14 @@ msgstr "Profiler" msgid "Torch Profiler" msgstr "" -#: ../../source/profiler.rst:9 +#: ../../source/profiler.rst:9 msgid "" -"InternEvo 使用 ``internlm.train.initialize_llm_profile()`` " +"InternEvo 使用 ``internlm.train.pipeline.initialize_llm_profile()`` " "来收集和分析模型训练或推理期间的性能数据，如 CPU/CUDA/memory 等性能数据。这个实现基于 `torch.profiler " "`_ ，输出的性能分析 trace 文件可以使用 " "`tensorboard `_ 进行可视化。" msgstr "" -"InternLM uses ``internlm.train.initialize_llm_profile()`` to profile " +"InternLM uses ``internlm.train.pipeline.initialize_llm_profile()`` to profile " "performance data, execution time duration and breakdown analysis of step " "time. The implementation is based on `torch.profiler " "`_ and output tracing " @@ -77,10 +77,6 @@ msgstr "" "`_" -#: internlm.train.training_internlm.initialize_llm_profile:1 of -msgid "Initialize and return the profiler context manager instance." -msgstr "" - #: ../../source/profiler.rst:38 msgid "Memory Profiler" msgstr "" @@ -126,8 +122,7 @@ msgstr "An example of model parameters sunburst chart is as follows:" msgid "A memory profiler for a llm model." msgstr "" -#: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler -#: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler.point of +#: ../../source/profiler.rst msgid "参数" msgstr "" @@ -159,8 +154,7 @@ msgstr "" msgid "Whether to create a new memory record file. Defaults to False." msgstr "" -#: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler.point -#: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler.step of +#: ../../source/profiler.rst msgid "返回" msgstr "" @@ -173,3 +167,6 @@ msgstr "" msgid "Update the memory state of the optimizer state." msgstr "" +#~ msgid "Initialize and return the profiler context manager instance." +#~ msgstr "" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/usage.po b/doc/code-docs/locales/en/LC_MESSAGES/usage.po index e32dae4a4..5d9632eac 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/usage.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/usage.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-27 11:16+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,7 +16,7 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../../usage.md:2 msgid "使用教程" @@ -46,18 +46,72 @@ msgstr "" msgid "数据准备 （预训练）" msgstr "Dataset Preparation (Pre-training)" -#: ../../../usage.md:11 -msgid "InternLM训练任务的数据集包括一系列的`bin`和`meta`文件。使用`tokenizer`从原始文本文件生成训练用数据集。通过在`tools/tokenizer.py`中指定模型参数路径的方式来导入tokenizer模型。目前提供`tokenizer_internlm.model`来生成tokens。若想使用不同的模型，可直接修改`tokernizer.py`中的模型参数路径。" +#: ../../../usage.md:10 +msgid "使用huggingface格式数据集" +msgstr "Using Hugging Face format dataset" + +#: ../../../usage.md:12 +msgid "如果使用huggingface数据集，需要先将数据集和需要使用的tokenizer下载到本地。" +msgstr "" +"If you use a Hugging Face dataset, you need to first download the dataset" +" and the required tokenizer to your local machine." + +#: ../../../usage.md:14 +msgid "以`roneneldan/TinyStories`这个数据为例，数据准备阶段需要通过如下命令将数据集下载到本地：" +msgstr "" +"Using the roneneldan/TinyStories dataset as an example, the data " +"preparation phase requires downloading the dataset to your local system " +"with the following command:" + +#: ../../../usage.md:18 +msgid "其中，\"/mnt/petrelfs/hf-TinyStories\" 为需要将数据集保存的本地路径。" +msgstr "" +"In this case, \"/mnt/petrelfs/hf-TinyStories\" is the local path where " +"the dataset needs to be saved." + +#: ../../../usage.md:20 +msgid "" +"然后将tokenizer下载到本地，例如，使用internlm2的tokenizer，则将`https://huggingface.co/internlm/internlm2-7b/tree/main`中的`special_tokens_map.json`、`tokenizer.model`、`tokenizer_config.json`、`tokenization_internlm2.py`和`tokenization_internlm2_fast.py`文件下载到本地路径，如\"/mnt/petrelfs" +"/hf-internlm2-tokenizer\"中。" +msgstr "" +"Next, download the tokenizer to your local system. For instance, if you " +"are using the internlm2 tokenizer, download the files " +"special_tokens_map.json, tokenizer.model, tokenizer_config.json, " +"tokenization_internlm2.py, and tokenization_internlm2_fast.py from the " +"repository at \"https://huggingface.co/internlm/internlm2-7b/tree/main\" " +"to a local directory, such as \"/mnt/petrelfs/hf-internlm2-tokenizer\"." + +#: ../../../usage.md:22 +msgid "将配置文件做如下改动：" +msgstr "Make the following changes to the configuration file:" + +#: ../../../usage.md:30 +msgid "" +"type默认为\"tokenized\"，这里需要改为\"streaming\"类型。同时需要指定`tokenizer_path`, " +"如果使用下述tokenized之后的数据集，则不需要设置该字段。`TRAIN_FOLDER`指定本地数据集路径。" +msgstr "" +"The type is set to \"tokenized\" by default, but here it needs to be " +"changed to the \"streaming\" type. Also, you need to specify the " +"tokenizer_path. If you are using the dataset that has been tokenized as " +"described below, you do not need to set this field. The TRAIN_FOLDER " +"specifies the path to the local dataset." + +#: ../../../usage.md:32 +msgid "使用tokenized之后数据集" +msgstr "Use the dataset after tokenization" + +#: ../../../usage.md:34 +msgid "InternEvo训练任务的数据集包括一系列的`bin`和`meta`文件。使用`tokenizer`从原始文本文件生成训练用数据集。通过在`tools/tokenizer.py`中指定模型参数路径的方式来导入tokenizer模型。目前提供`V7_sft.model`来生成tokens。若想使用不同的模型，可直接修改`tokernizer.py`中的模型参数路径。" msgstr "" -"The dataset for the InternEvo training task includes a series of `bin` and" -" `meta` files. A `tokenizer` is used to generate the training dataset " +"The dataset for the InternEvo training task includes a series of `bin` " +"and `meta` files. A `tokenizer` is used to generate the training dataset " "from the original text files. The tokenizer model is imported by " "specifying the model parameter path in `tools/tokenizer.py`. Currently, " -"`tokenizer_internlm.model` is provided to generate tokens. If you want to use a " -"different model, you can directly modify the model parameter path in " -"`tokenizer.py`." +"`tokenizer_internlm.model` is provided to generate tokens. If you want to" +" use a different model, you can directly modify the model parameter path " +"in `tokenizer.py`." -#: ../../../usage.md:13 +#: ../../../usage.md:37 msgid "可以运行以下命令生成原始数据对应的`bin`和`meta`文件，其中参数`text_input_path`表示原始文本数据路径，目前支持`txt`、`json`和`jsonl`三种输入格式，`bin_output_path`表示生成的`bin`文件的保存路径。" msgstr "" "You can run the following command to generate `bin` and `meta` files " @@ -66,30 +120,30 @@ msgstr "" "`txt`, `json`, and `jsonl` formats, while `bin_output_path` represents " "the save path of the generated `bin` files." -#: ../../../usage.md:18 +#: ../../../usage.md:42 msgid "下面是一个数据处理的例子：" msgstr "Here is an example of data processing:" -#: ../../../usage.md:20 +#: ../../../usage.md:44 msgid "给定一个包含原始数据集的文件`raw_data.txt`，原始数据集如下所示：" msgstr "" "Given a file `raw_data.txt` containing the raw dataset, the raw dataset " "is shown below:" -#: ../../../usage.md:27 +#: ../../../usage.md:51 msgid "可以通过运行以下命令来生成`bin`和`meta`文件：" msgstr "" "You can generate the `bin` and `meta` files by running the following " "command:" -#: ../../../usage.md:32 +#: ../../../usage.md:56 msgid "需要注意的是，生成的`bin`文件需要保存在`cn`或者`en`或者`code`或者`ja`或者`ar`或者`kaoshi`这六个目录下，以区分数据集的类型。" msgstr "" "It should be noted that the generated `bin` files need to be saved in one" " of the following directories: `cn`, `en`, `code`, `ja`, `ar`, or " "`kaoshi`, depending on the type of dataset." -#: ../../../usage.md:34 +#: ../../../usage.md:58 msgid "其中，`cn`表示中文数据集；`en`表示英文数据集；`code`表示代码数据集；`ja`表示日语数据集；`ar`表示阿拉伯语数据集；`kaoshi`表示考试数据集。" msgstr "" "Here, `cn` represents the Chinese dataset, `en` represents the English " @@ -97,22 +151,22 @@ msgstr "" " dataset, `ar` represents the Arabic dataset, and `kaoshi` represents the" " exam dataset." -#: ../../../usage.md:36 +#: ../../../usage.md:60 msgid "生成的bin文件的格式如下：" msgstr "The format of the generated `bin` files is as follows:" -#: ../../../usage.md:42 +#: ../../../usage.md:66 msgid "`bin`文件中的每一行均对应原始数据集中的每一个句子，表示每个句子的`token`（下文将用sequence指定）。" msgstr "" "Each line in the `bin` file corresponds to each sentence in the original " "dataset, representing the tokens of each sentence (referred to as " "sequence below)." -#: ../../../usage.md:44 +#: ../../../usage.md:68 msgid "生成的`meta`文件的格式如下：" msgstr "The format of the generated `meta` file is as follows:" -#: ../../../usage.md:48 +#: ../../../usage.md:72 msgid "" "在`meta`文件中，每个元组对应着`bin`文件中每一个`sequence`的元信息。其中，元组的第一个元素表示每个`sequence`在所有`sequence`中的`starting" " index`，第二个元素表示每个`sequence`中有多少个`tokens`。" @@ -122,7 +176,7 @@ msgstr "" "index` of each `sequence` among all `sequences`, and the second element " "indicates the number of `tokens` for each `sequence`." -#: ../../../usage.md:50 +#: ../../../usage.md:74 msgid "" "例如，对于第一个`sequence`，`starting index`为 0，有 11 " "个`tokens`；对于第二个`sequence`，由于第一个`sequence`转换为`string`后的长度为`89`，因此它的`starting" @@ -131,17 +185,17 @@ msgstr "" "For example, the first `sequence` starts at index 0 and has 16 `tokens`. " "The second `sequence` starts at index 110 and has 24 `tokens`." -#: ../../../usage.md:52 +#: ../../../usage.md:76 msgid "`json`和`jsonl`类型的文件的`bin`和`meta`文件格式和`txt`一致，此处不再赘叙。" msgstr "" "The `bin` and `meta` file formats for `json` and `jsonl` type files are " "the same as for `txt`, so we won't go over them here." -#: ../../../usage.md:54 +#: ../../../usage.md:78 msgid "数据准备 （微调）" msgstr "Data Preparation (Fine-tuning)" -#: ../../../usage.md:56 +#: ../../../usage.md:80 msgid "" "微调任务的数据集格式与预训练任务保持一致，生成的数据格式为一系列的`bin`和`meta`文件。以下以 Alpaca " "数据集为例，介绍微调的数据准备流程。" @@ -151,7 +205,7 @@ msgstr "" "the Alpaca dataset as an example to explain the data preparation process " "for fine-tuning." -#: ../../../usage.md:58 +#: ../../../usage.md:82 msgid "" "下载 [Alpaca 数据集](https://github.com/tatsu-" "lab/stanford_alpaca/blob/main/alpaca_data.json)" @@ -159,87 +213,108 @@ msgstr "" "Download the [Alpaca dataset](https://github.com/tatsu-" "lab/stanford_alpaca/blob/main/alpaca_data.json)." -#: ../../../usage.md:60 +#: ../../../usage.md:84 msgid "对 Alpaca 数据进行 tokenize，使用以下命令" msgstr "Tokenize the Alpaca dataset using the following command:" -#: ../../../usage.md:66 +#: ../../../usage.md:90 msgid "建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize" msgstr "" "It is recommended that users refer to alpaca_tokenizer.py to write new " "scripts to tokenize their own datasets" -#: ../../../usage.md:68 +#: ../../../usage.md:92 msgid "训练配置" msgstr "Training Configuration" -#: ../../../usage.md:70 +#: ../../../usage.md:94 msgid "以 7B Demo 的配置文件`configs/7B_sft.py`为例：" msgstr "" "Taking the configuration file `configs/7B_sft.py` for the 7B demo as an " "example," -#: ../../../usage.md:237 +#: ../../../usage.md:307 msgid "接下来将详细介绍启动一个模型训练所需要进行的数据、模型、并行和监控等相关的配置。" msgstr "" "let's discuss the data, model, parallel and monitoring configurations " "required to start a model training." -#: ../../../usage.md:239 +#: ../../../usage.md:309 msgid "数据配置" msgstr "Data Configuration" -#: ../../../usage.md:240 +#: ../../../usage.md:310 msgid "数据相关的关键参数配置及释义如下所示：" msgstr "Here are the key parameters and their explanations for data configuration:" -#: ../../../usage.md:255 +#: ../../../usage.md:325 msgid "![pack_into_one](./imgs/pack_into_one.png)" msgstr "" -#: ../../../usage.md:255 +#: ../../../usage.md:325 msgid "pack_into_one" msgstr "" -#: ../../../usage.md:258 +#: ../../../usage.md:328 msgid "目前支持传入数据集文件路径`train_folder`，且要求文件格式如下：" msgstr "" "Currently, it supports passing the dataset file path `train_folder`, and " "the file format is required to be as follows:" -#: ../../../usage.md:265 +#: ../../../usage.md:335 msgid "数据集的详细内容可参考``数据准备``模块相关的介绍。" msgstr "" "For detailed information about the dataset, please refer to the \"Data " "Preparation\" section." -#: ../../../usage.md:267 +#: ../../../usage.md:337 +msgid "同时，也支持huggingface格式的数据集处理。" +msgstr "" +"Additionally, it supports processing of datasets in the Hugging Face " +"format." + +#: ../../../usage.md:339 +msgid "train_folder设置为从huggingface上下载的本地数据集路径，如：\"/mnt/petrelfs/hf-TinyStories\"" +msgstr "" +"Set the train_folder to the local path of the dataset downloaded from " +"Hugging Face, for example: \"/mnt/petrelfs/hf-TinyStories\"." + +#: ../../../usage.md:341 +msgid "在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如：" +msgstr "" +"In the data section, you need to add new fields for type and " +"tokenizer_path to indicate that the dataset is in Hugging Face format and" +" to specify the path of the tokenizer, for example:" + +#: ../../../usage.md:359 msgid "模型配置" msgstr "Model Configuration" -#: ../../../usage.md:269 +#: ../../../usage.md:361 msgid "如果在启动训练时要加载模型 `checkpoint`，可进行如下相关配置：" msgstr "" "If you want to load a model checkpoint when starting the training, you " "can configure it as follows:" -#: ../../../usage.md:282 +#: ../../../usage.md:385 msgid "注意：" msgstr "Note:" -#: ../../../usage.md:283 -msgid "路径若以 `local:` 为前缀，则存储在本地文件系统；若以 `boto3:` 为前缀，则存储在远程 oss 上" +#: ../../../usage.md:386 +msgid "" +"路径若以 `local:` 为前缀，则存储在本地文件系统；若以 `boto3:` 为前缀，则存储在远程 oss " +"上；若无前缀，为huggingface上可以直接下载的模型路径。" msgstr "" "If the path starts with `local:`, it means the file is stored in the " "local file system. If it starts with `boto3:`, it means the file is " "stored in the remote OSS." -#: ../../../usage.md:285 +#: ../../../usage.md:388 msgid "模型相关关键参数配置如下所示：" msgstr "The configuration for the model is as follows:" -#: ../../../usage.md:309 -msgid "注意：用户可自定义模型类型名和模型结构，并配置相对应的模型参数。通过`utils/registry.py`下的`MODEL_INITIALIZER`对象进行模型初始化函数接口注册，在训练主函数`train.py`中初始化模型时，可通过`model_type`配置获取指定的模型初始化接口函数。" +#: ../../../usage.md:412 +msgid "注意：用户可自定义模型类型名和模型结构，并配置相对应的模型参数。通过`internlm/model/registry.py`下的`model_initializer`对象进行模型初始化函数接口注册，在训练主函数`train.py`中初始化模型时，可通过`model_type`配置获取指定的模型初始化接口函数。" msgstr "" "Note: Users can customize the model type name and model structure, and " "configure the corresponding model parameters. The model initialization " @@ -249,7 +324,7 @@ msgstr "" "interface function can be obtained through the `model_type` " "configuration." -#: ../../../usage.md:311 +#: ../../../usage.md:414 msgid "" "*如果基于 InternLM 7B继续训练，可以参考 " "[ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo) 中 " @@ -259,76 +334,130 @@ msgstr "" "OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-" "zoo) to download weights*." -#: ../../../usage.md:313 +#: ../../../usage.md:416 msgid "并行配置" msgstr "Parallel Configuration" -#: ../../../usage.md:315 +#: ../../../usage.md:418 msgid "训练并行配置样例如下：" msgstr "Training parallel configuration example:" -#: ../../../usage.md:324 -msgid "zero1：zero 并行策略，分如下三种情况，默认值为 -1" -msgstr "" -"zero1: zero parallel strategy, divided into the following three cases, " -"default value is -1" +#: ../../../usage.md:427 +msgid "zero1（字典）：" +msgstr "zero1 (dict): " -#: ../../../usage.md:325 +#: ../../../usage.md:428 +msgid "size: 整数" +msgstr "size: int " + +#: ../../../usage.md:429 msgid "当`zero1 <= 0`，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" msgstr "" -"When `zero1 <= 0`, the size of the zero1 process group is equal to the " +"When `zero1 <= 0` , the size of the zero1 process group is equal to the " "size of the data parallel process group, so the optimizer state " "parameters will be split within the data parallel range." -#: ../../../usage.md:326 +#: ../../../usage.md:430 msgid "当`zero1 == 1`，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" msgstr "" "When `zero1 == 1`, zero1 is not used, and all data parallel groups retain" " the complete optimizer state parameters." -#: ../../../usage.md:327 +#: ../../../usage.md:431 msgid "当`zero1 > 1`且`zero1 <= data_parallel_world_size`，则 zero1 进程组是数据并行进程组的子集" msgstr "" "When `zero1 > 1` and `zero1 <= data_parallel_world_size`, the zero1 " "process group is a subset of the data parallel process group." -#: ../../../usage.md:328 -msgid "tensor：张量并行大小，通常是每个节点的 GPU 数量，默认值为 1" +#: ../../../usage.md:432 +msgid "fsdp: 布尔值，启用/禁用torch的完全分片数据并行，默认为False。" +msgstr "" +"fsdp: A boolean value that enables or disables fully sharded data " +"parallelism in torch, with the default being False." + +#: ../../../usage.md:433 +msgid "tensor（字典）：" +msgstr "tensor (dict): " + +#: ../../../usage.md:434 +msgid "size: 整数，张量并行的大小。" +msgstr "size: int, size of tensor parallem" + +#: ../../../usage.md:435 +msgid "mode: 字符串，张量并行模式，应该是 ['mtp', 'msp', 'fsp', 'isp'] 中的一个，" msgstr "" -"tensor: tensor parallel size, usually the number of GPUs per node, " -"default is 1" +"mode: string, tensor parallel mode, should be one of ['mtp', 'msp', " +"'fsp', 'isp'] " -#: ../../../usage.md:329 -msgid "pipeline：流水线并行策略" +#: ../../../usage.md:436 +msgid "默认为 'mtp'，意味着没有序列并行的纯Megatron张量并行。" +msgstr "" +"Default is 'mtp', which means there is no sequence parallelism, just pure" +" tensor parallelism for Megatron." + +#: ../../../usage.md:437 +msgid "msp: 带序列并行的Megatron张量并行，序列并行大小 = 张量并行大小。" +msgstr "" +"msp: Megatron Tensor Parallelism with Sequence Parallelism, where the " +"size of sequence parallelism is equal to the size of tensor parallelism." + +#: ../../../usage.md:438 +msgid "fsp: 通过flash-attn带序列并行的张量并行，序列并行大小 = 张量并行大小。" +msgstr "" +"fsp: Tensor Parallelism with Sequence Parallelism facilitated by flash-" +"attn, where the size of sequence parallelism is equal to the size of " +"tensor parallelism." + +#: ../../../usage.md:439 +msgid "isp: 定制的内部序列并行，不带张量并行，可以与权重并行一起使用。" +msgstr "" +"isp: Custom internal sequence parallelism, without tensor parallelism, " +"which can be used in conjunction with weight parallelism." + +#: ../../../usage.md:440 +msgid "pipeline（字典）：" msgstr "pipeline: pipeline parallel strategy" -#: ../../../usage.md:330 -msgid "size：流水线并行大小，默认值为 1" -msgstr "size: pipeline parallel size, the default value is 1" +#: ../../../usage.md:441 +msgid "size: 整数，流水线并行的大小。" +msgstr "size: int, size of pipeline parallel" -#: ../../../usage.md:331 -msgid "interleaved_overlap：bool 类型，交错式调度时，开启或关闭通信优化，默认值为关闭" +#: ../../../usage.md:442 +msgid "interleaved_overlap: 布尔值，启用/禁用在使用交错流水线调度器时的通信重叠，默认为False。" msgstr "" -"interleaved_overlap: bool type, when interleaved scheduling, enable or " -"disable communication optimization, the default value is False" +"interleaved_overlap: A boolean value that enables or disables " +"communication overlapping when using an interleaved pipeline scheduler, " +"with the default being False." + +#: ../../../usage.md:443 +msgid "weight（字典）：" +msgstr "weight (dict):" -#: ../../../usage.md:332 -msgid "sequence_parallel：是否开启序列化并行，默认值为 False" +#: ../../../usage.md:444 +msgid "size: 整数，权重并行的大小。" +msgstr "size: int, size of weight parallel" + +#: ../../../usage.md:445 +msgid "overlap: 布尔值，启用/禁用all_gather/reduce_scatter通信重叠，默认为False。" msgstr "" -"sequence_parallel: Whether to enable sequence parallelism, the default " -"value is False" +"overlap: bool, enable/disable all_gather/reduce_scatter communication " +"overlap, default is False" + +#: ../../../usage.md:446 +msgid "memory_pool: 布尔值，启用/禁用内存池，默认为False。" +msgstr "memory_pool: bool, enable/disable memory pool, default is False" -#: ../../../usage.md:334 +#: ../../../usage.md:448 msgid "注意：`数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小`" msgstr "" "Note: `Data parallel size = Total number of GPUs / Pipeline parallel size" " / Tensor parallel size`" -#: ../../../usage.md:336 +#: ../../../usage.md:450 msgid "启动训练" msgstr "Start Training" -#: ../../../usage.md:338 +#: ../../../usage.md:452 msgid "完成了以上数据集准备和相关训练配置后，可启动 Demo 训练。接下来分别以 slurm 和 torch 环境为例，介绍训练启动方式。" msgstr "" "After completing the data preparation and relevant training " @@ -336,91 +465,153 @@ msgstr "" "following examples demonstrate how to start the training in both slurm " "and torch environments." -#: ../../../usage.md:340 +#: ../../../usage.md:454 ../../../usage.md:489 msgid "若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on slurm with 16 GPUs across " "multiple nodes, use the following command:" -#: ../../../usage.md:345 +#: ../../../usage.md:459 msgid "若在 torch 上启动分布式运行环境，单节点 8 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on torch with 8 GPUs on a " "single node, use the following command:" -#: ../../../usage.md:350 +#: ../../../usage.md:464 msgid "运行结果" msgstr "Training Results" -#: ../../../usage.md:352 +#: ../../../usage.md:466 msgid "以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下：" msgstr "" "Taking the configuration of the demo training on a single machine with 8 " "GPUs on slurm as an example, the training result log is shown below:" -#: ../../../usage.md:373 +#: ../../../usage.md:487 +msgid "加载训练的checkpoint并生成" +msgstr "Load the training checkpoint and generate." + +#: ../../../usage.md:494 +msgid "在配置文件中添加`generation`配置" +msgstr "Add generation configuration to the configuration file." + +#: ../../../usage.md:512 msgid "长文本生成" msgstr "Long Text Generation" -#: ../../../usage.md:375 +#: ../../../usage.md:514 msgid "" "在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K " "的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK " "RoPE。" msgstr "" -"In the inference phase, we can use Dynamic NTK RoPE instead of the original RoPE, allowing the model to handle" -" long-text input and output, achieving extrapolation effects up to 16K. Currently, InternLM supports the use of" -" Dynamic NTK RoPE in models formatted in both Hugging Face format and InternLM's native format." +"In the inference phase, we can use Dynamic NTK RoPE instead of the " +"original RoPE, allowing the model to handle long-text input and output, " +"achieving extrapolation effects up to 16K. Currently, InternLM supports " +"the use of Dynamic NTK RoPE in models formatted in both Hugging Face " +"format and InternLM's native format." -#: ../../../usage.md:378 +#: ../../../usage.md:517 msgid "" "对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 " "`config.json` 中的 `rotary.type` 修改为 `origin`；" msgstr "" -"For models in Hugging Face format, Dynamic NTK RoPE is currently enabled by default. " -"If users wish to disable this behavior, they can modify `rotary.type` in the `config.json` file to `origin`." +"For models in Hugging Face format, Dynamic NTK RoPE is currently enabled " +"by default. If users wish to disable this behavior, they can modify " +"`rotary.type` in the `config.json` file to `origin`." -#: ../../../usage.md:379 +#: ../../../usage.md:518 msgid "" "对于 InternLM " "本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。" msgstr "" -"For models in InternLM's native format, during inference, you can enable this behavior by adding " -"use_dynamic_ntk_rope=True to the configuration dictionary when initializing the model." +"For models in InternLM's native format, during inference, you can enable " +"this behavior by adding use_dynamic_ntk_rope=True to the configuration " +"dictionary when initializing the model." -#: ../../../usage.md:381 +#: ../../../usage.md:520 msgid "" "用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE " -"是如何生效的。例如文件[长文本示例](./aux_%20materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用" +"是如何生效的。例如文件[长文本示例](../../aux_materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用" " Dynamic NTK， 模型是完全无法回答该文本对应的问题。而使用 Dynamic NTK RoPE 后 InternLM Chat 7B " "v1.1 模型的回答如下所示：" msgstr "" -"Users can visually compare and observe how Dynamic NTK RoPE takes effect directly through the web_demo. " -"For example, in the file Long Text Example, there is a text with token length exceeding 2200. " -"Without using Dynamic NTK, the model is unable to answer questions related to this text. " -"However, after applying Dynamic NTK RoPE, the response from the InternLM Chat 7B v1.1 model is as follows:" +"Users can visually compare and observe how Dynamic NTK RoPE takes effect " +"directly through the web_demo. For example, in the file Long Text " +"Example, there is a text with token length exceeding 2200. Without using " +"Dynamic NTK, the model is unable to answer questions related to this " +"text. However, after applying Dynamic NTK RoPE, the response from the " +"InternLM Chat 7B v1.1 model is as follows:" -#: ../../../usage.md:384 +#: ../../../usage.md:523 msgid "![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png)" msgstr "" -#: ../../../usage.md:384 +#: ../../../usage.md:523 msgid "dynamic_ntk_answer" msgstr "" -#: ../../../usage.md:386 +#: ../../../usage.md:525 msgid "关于 Dyanmic NTK 的原理，详细请参考" msgstr "Regarding the principle of Dyanmic NTK, please refer to" -#: ../../../usage.md:387 -msgid "https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases" +#: ../../../usage.md:526 +msgid "[dynamically_scaled_rope_further_increases](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases)" msgstr "" -#: ../../../usage.md:388 -msgid "https://kexue.fm/archives/9675" +#: ../../../usage.md:527 +msgid "[https://kexue.fm/archives/9675](https://kexue.fm/archives/9675)" msgstr "" #~ msgid "`load_model_only_folder`与`load_ckpt_folder`不能同时设置" #~ msgstr "" #~ "`load_model_only_folder` and `load_ckpt_folder` " #~ "cannot be set at the same time." + +#~ msgid "zero1：zero 并行策略，分如下三种情况，默认值为 -1" +#~ msgstr "" +#~ "zero1: zero parallel strategy, divided " +#~ "into the following three cases, default" +#~ " value is -1" + +#~ msgid "tensor：张量并行大小，通常是每个节点的 GPU 数量，默认值为 1" +#~ msgstr "" +#~ "tensor: tensor parallel size, usually " +#~ "the number of GPUs per node, " +#~ "default is 1" + +#~ msgid "size：流水线并行大小，默认值为 1" +#~ msgstr "size: pipeline parallel size, the default value is 1" + +#~ msgid "interleaved_overlap：bool 类型，交错式调度时，开启或关闭通信优化，默认值为关闭" +#~ msgstr "" +#~ "interleaved_overlap: bool type, when " +#~ "interleaved scheduling, enable or disable " +#~ "communication optimization, the default value" +#~ " is False" + +#~ msgid "sequence_parallel：是否开启序列化并行，默认值为 False" +#~ msgstr "" +#~ "sequence_parallel: Whether to enable sequence" +#~ " parallelism, the default value is " +#~ "False" + +#~ msgid "https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases" +#~ msgstr "" + +#~ msgid "https://kexue.fm/archives/9675" +#~ msgstr "" + +#~ msgid "数据准备 （使用huggingface数据集）" +#~ msgstr "" + +#~ msgid "如果使用huggingface数据集进行在线加载并且在线tokenize的话，那么以`roneneldan/TinyStories`这个数据为例，数据准备阶段只需要将配置文件做如下改动：" +#~ msgstr "" + +#~ msgid "" +#~ "同时，也支持huggingface格式的数据集处理。 " +#~ "train_folder设置为huggingface上可以通过load_dataset直接下载的数据集路径，如：\"roneneldan/TinyStories\"" +#~ " " +#~ "在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如：" +#~ msgstr "" + diff --git a/doc/code-docs/source/checkpoint.rst b/doc/code-docs/source/checkpoint.rst index c01c69504..4170d420c 100644 --- a/doc/code-docs/source/checkpoint.rst +++ b/doc/code-docs/source/checkpoint.rst @@ -1,7 +1,7 @@ 模型加载与保存 =================== -InternEvo 使用 ``internlm.utils.model_checkpoint.CheckpointManager`` 来管理模型保存。其中，可以使用 ``CheckpointManager.try_save_checkpoint(train_state)`` 来保存指定 step 的模型状态。 +InternEvo 使用 ``internlm.checkpoint.checkpoint_manager.CheckpointManager`` 来管理模型保存。其中，可以使用 ``CheckpointManager.try_save_checkpoint(train_state)`` 来保存指定 step 的模型状态。 InternEvo支持启动时自动加载最新的模型备份，并在接收信号退出训练时自动进行模型备份。 @@ -10,23 +10,25 @@ CheckpointManager ``CheckpointManager`` 是InternEvo负责进行模型加载和保存的工具类，其会使用config文件中的ckpt字段的初始化参数字典初始化自身的参数，目前相关的参数有： -- ``enable_save_ckpt``: 是否开启检查点存储功能（不影响检查点加载）。参数类型 ``bool``，必选参数。 +- ``enable_save_ckpt`` : 是否开启检查点存储功能（不影响检查点加载）。参数类型 ``bool`` ，必选参数。 -- ``save_ckpt_folder``: 检查点存储路径，参数类型 ``str``，默认为： ``None``，在开启检查点存储功能时为必选参数。 +- ``save_ckpt_folder`` : 检查点存储路径，参数类型 ``str`` ，默认为： ``None`` ，在开启检查点存储功能时为必选参数。 -- ``checkpoint_every``: 检查点存储频率，参数类型 ``int``，默认为： ``50``。 +- ``checkpoint_every`` : 检查点存储频率，参数类型 ``int`` ，默认为： ``50`` 。 -- ``load_ckpt_info``: 初始化检查点/权重加载信息。参数类型 ``dict``，默认为： ``None``，详见 :ref:`load-ckpt-info`。 +- ``load_ckpt_info`` : 初始化检查点/权重加载信息。参数类型 ``dict`` ，默认为： ``None`` ，详见 :ref: `load-ckpt-info` 。 -- ``async_upload``: 是否开启异步上传，默认值为：``False``，详见 :ref:`asyncupload`。 +- ``async_upload`` : 是否开启异步上传，默认值为： ``False`` ，详见 :ref: `asyncupload` 。 -- ``async_upload_tmp_folder``: 异步上传临时存储路径。 +- ``async_upload_tmp_folder`` : 异步上传临时存储路径。 -- ``oss_snapshot_freq``: 快照存储频率，默认值为：``checkpoint_every``的一半。详见 :ref:`snapshot`。 +- ``oss_snapshot_freq`` : 快照存储频率，默认值为： ``checkpoint_every`` 的一半。详见 :ref: `snapshot` 。 -- ``auto_resume``: 是否开启检查点自动恢复，默认值为：``True``，详见 :ref:`autoresume`。 +- ``auto_resume`` : 是否开启检查点自动恢复，默认值为： ``True`` ，详见 :ref: `autoresume` 。 -- ``stop_file_path`` : 检查点存储控制文件的路径，默认值为：``None``，详见 :ref:`stopfile`。 +- ``stop_file_path`` : 检查点存储控制文件的路径，默认值为： ``None`` ，详见 :ref: `stopfile` 。 + +- ``enable_internevo2hf_ckpt`` : 是否同时保存huggingface格式的权重。如果开启，会将被并行切分后的权重整合起来保存成huggingface格式，方便后续使用hf方式加载权重，避免因并行方式不同导致的权重转换问题。默认值为： ``False`` 。 下面给出config文件的参数设置例子： @@ -36,16 +38,16 @@ CheckpointManager ckpt = dict( enable_save_ckpt=False, # enable ckpt save. save_ckpt_folder=SAVE_CKPT_FOLDER, # Path to save training ckpt. - load_ckpt_info=dict(path="local:/mnt/mfs/ckpt", content=["all",], ckpt_type="internlm"), + load_ckpt_info=dict(path="local:/mnt/mfs/ckpt", content=("all",), ckpt_type="internevo"), auto_resume=False, # disable auto-resume, internlm will load model checkpoint from the path of 'load_ckpt_info'. checkpoint_every=CHECKPOINT_EVERY, async_upload=True, # async ckpt upload. (only work for boto3, volc and oss2 ckpt) async_upload_tmp_folder="/dev/shm/internlm_tmp_ckpt/", # path for temporarily files during asynchronous upload. oss_snapshot_freq=int(CHECKPOINT_EVERY / 2), # snapshot ckpt save frequency. + enable_internevo2hf_ckpt=True, # save huggingface format ckpt at the same time. ) - -.. autoclass:: internlm.utils.model_checkpoint.CheckpointManager +.. autoclass:: internlm.checkpoint.checkpoint_manager.CheckpointManager :members: @@ -93,9 +95,8 @@ load_ckpt_info 由三个字段组成， ``path`` 、 ``content`` 和 ``ckpt_type - ``ckpt_type``：表示加载的模型权重类型，目前支持的字段包括： - ``internevo``：internevo约定的checkpoint存储格式。 - - ``llama``：llama约定的checkpoint存储格式。 - - ``hf_llama``：huggingface llama约定的checkpoint存储格式。 - - ``hf_model``：适用于加载huggingface所有模型的checkpoint存储格式。 + - ``llama``：huggingface llama约定的checkpoint存储格式。 + - ``hf``：huggingface 模型约定的checkpoint存储格式。 下面给出两个例子： @@ -107,10 +108,6 @@ load_ckpt_info 由三个字段组成， ``path`` 、 ``content`` 和 ``ckpt_type # 从文件存储相对路径 ckpt_model 中加载所有的状态，适合断点续训的场景 load_ckpt_info = dict(path="local:ckpt_model", content=("all",), ckpt_type="internevo") - # 从 huggingface 下载指定模型，加载checkpoint - load_ckpt_info = dict(path="internlm/internlm-7b", content=("model",), ckpt_type="hf_model") - - .. _asyncupload: 异步上传 @@ -179,4 +176,3 @@ config.ckpt 中相关的参数： # 如果存入的step>0，则任务会在存储ckpt后自动退出 # 如果存入的step<0，则任务会在存储ckpt后会继续训练 echo "999" > ./llm_alter/1006_pr.log - diff --git a/doc/code-docs/source/initialize.rst b/doc/code-docs/source/initialize.rst index ff3985ee8..311ad5265 100644 --- a/doc/code-docs/source/initialize.rst +++ b/doc/code-docs/source/initialize.rst @@ -38,7 +38,6 @@ InternEvo 使用 `argparse `_ .. autofunction:: internlm.initialize.get_default_parser - .. _InternLM-model-init: 模型初始化 @@ -95,16 +94,139 @@ InternEvo 在配置文件中使用字段 ``model_type`` 和 ``model`` 来控制 .. autofunction:: internlm.train.initialize_optimizer +InternEvo 在配置文件中使用字段 ``grad_scaler`` 、 ``hybrid_zero_optimizer`` 、 ``adam`` 、 ``lr_scheduler`` 和 ``beta2_scheduler`` 来控制优化器初始化过程。示例优化器初始化配置定义如下： + +.. code-block:: python + + grad_scaler = dict( + fp16=dict( + # the initial loss scale, defaults to 2**16 + initial_scale=2**16, + # the minimum loss scale, defaults to None + min_scale=1, + # the number of steps to increase loss scale when no overflow occurs + growth_interval=1000, + ), + # the multiplication factor for increasing loss scale, defaults to 2 + growth_factor=2, + # the multiplication factor for decreasing loss scale, defaults to 0.5 + backoff_factor=0.5, + # the maximum loss scale, defaults to None + max_scale=2**24, + # the number of overflows before decreasing loss scale, defaults to 2 + hysteresis=2, + ) + + hybrid_zero_optimizer = dict( + # Enable low_level_optimzer overlap_communication + overlap_sync_grad=True, + overlap_sync_param=False, + # bucket size for nccl communication params + reduce_bucket_size=512 * 1024 * 1024, + # grad clipping + clip_grad_norm=1.0, + # whether use new optm + use_split_tensor_optim=False, + # when use split tensor optm + # Perform all gather with a set of parameters of all_gather_size + all_gather_size=512 * 1024 * 1024, + ) + + adam = dict( + lr=1e-4, + adam_beta1=0.9, + adam_beta2=0.95, + adam_beta2_c=0, + adam_eps=1e-8, + weight_decay=0.01, + ) + + lr_scheduler = dict( + total_steps=data["total_steps"], + init_steps=0, # optimizer_warmup_step + warmup_ratio=0.01, + eta_min=1e-5, + last_epoch=-1, + ) + + beta2_scheduler = dict( + init_beta2=adam["adam_beta2"], + c=adam["adam_beta2_c"], + cur_iter=-1, + ) + +用户通过 ``initialize_optimizer`` 函数初始化优化器，并传入 ``isp_communicator`` 参数，以便处理 ISP 并行模式下的通信。 + .. _InternLM-dl-init: 数据加载器初始化 ------------------------- -.. autofunction:: internlm.train.get_train_data_loader +.. autofunction:: internlm.train.build_train_loader_with_data_type + +InternEvo 在配置文件中使用字段 ``data`` 来控制数据加载器初始化过程。示例数据加载器初始化配置定义如下： + +.. code-block:: python + + TRAIN_FOLDER = None # "/path/to/dataset" + VALID_FOLDER = None # "/path/to/dataset" + data = dict( + seq_len=SEQ_LEN, + # micro_num means the number of micro_batch contained in one gradient update + micro_num=4, + # packed_length = micro_bsz * SEQ_LEN + micro_bsz=2, + # defaults to the value of micro_num + valid_micro_num=4, + # defaults to 0, means disable evaluate + valid_every=50, + pack_sample_into_one=False, + total_steps=50000, + skip_batches="", + # rampup_batch_size (str): A string with three space-separated integers representing the + # starting batch size, the increment, and the number of steps between + # each increment. For example, "192 24 8" means that the batch size (micro_num) + # starts at 192 and increases by 24 every 8 steps. Defaults to None. + # (IMPORTANT): The interval step size is 'micro_bsz'. + rampup_batch_size="", + # Datasets with less than 50 rows will be discarded + min_length=50, + train_folder=TRAIN_FOLDER, + valid_folder=VALID_FOLDER, + empty_cache_and_diag_interval=200, + diag_outlier_ratio=1.1, + # whether use shared memory to load meta files + use_shm=False, + # when use shm, the default shm_path is "/dev/shm/metacache" + # shm_path="/dev/shm/metacache" + ) + +这里支持三种数据集的初始化，包括模拟数据集、已分词数据集和流式数据集。 + +- 模拟数据集 + + 如果设置TRAIN_FOLDER为None，则随机生成模拟数据集，如果设置的随机种子一样，生成的数据集保持一致。 + +- 已分词数据集 + + 如果设置TRAIN_FOLDER为本地指定路径，路径中保存经过分词之后的.bin和.meta文件，则加载已分词数据集。 + +- 流式数据集 + + 如果设置TRAIN_FOLDER为本地指定路径，路径中保存从huggingface下载的数据集，同时在data配置中，新增如下type和tokenizer_path字段，则加载流式数据集。 + +.. code-block:: python + + type="streaming", + tokenizer_path="/path/to/tokenizer", .. _InternLM-trainer-init: +已分词数据集和流式数据集格式的详细说明，参见 `使用教程 `_ + Trainer 初始化 ------------------------- .. autofunction:: internlm.initialize.initialize_trainer + +通过initialize_trainer函数，初始化训练过程，需要输入创建好的模型、初始化的优化器以及调度器等参数。 diff --git a/doc/code-docs/source/profiler.rst b/doc/code-docs/source/profiler.rst index df8b22f34..babd9646c 100644 --- a/doc/code-docs/source/profiler.rst +++ b/doc/code-docs/source/profiler.rst @@ -6,7 +6,7 @@ Torch Profiler ----------------- -InternEvo 使用 ``internlm.train.initialize_llm_profile()`` 来收集和分析模型训练或推理期间的性能数据，如 CPU/CUDA/memory 等性能数据。这个实现基于 `torch.profiler `_ ，输出的性能分析 trace 文件可以使用 `tensorboard `_ 进行可视化。 +InternEvo 使用 ``internlm.train.pipeline.initialize_llm_profile()`` 来收集和分析模型训练或推理期间的性能数据，如 CPU/CUDA/memory 等性能数据。这个实现基于 `torch.profiler `_ ，输出的性能分析 trace 文件可以使用 `tensorboard `_ 进行可视化。 用户如果想使用这个 torch 性能分析工具，需要在启动训练时传递 ``--profiling`` 参数以启用性能分析。完成 torch 性能分析后，用户可以在 ``{JOB_NAME}/{start_time}/traces/rank{}_dp{}_tp{}_pp{}`` 文件夹中看到性能分析结果。 @@ -32,7 +32,7 @@ InternEvo 使用 ``internlm.train.initialize_llm_profile()`` 来收集和分析 :scale: 45% :class: with-border -.. autofunction:: internlm.train.initialize_llm_profile +.. autofunction:: internlm.train.pipeline.initialize_llm_profile Memory Profiler ----------------- diff --git a/doc/en/usage.md b/doc/en/usage.md index f1d0f65e7..b701a5bbc 100644 --- a/doc/en/usage.md +++ b/doc/en/usage.md @@ -6,17 +6,35 @@ To start a demo model training, you need to prepare three things: **installation Please refer to the [installation guide](./install.md) for instructions on how to install the necessary dependencies. -### Dataset Preparation (HuggingFace Datasets) -If you are using the HuggingFace datasets for on-the-fly streaming load and tokenize, taking the `roneneldan/TinyStories`` dataset as an example, the data preparation stage only requires the following adjustments in the configuration file: -```python -TRAIN_FOLDER = "roneneldan/TinyStories" +### Dataset Preparation (Pre-training) + +#### Using Hugging Face Format Dataset + +If you are using a Hugging Face dataset, you will first need to download the dataset and the required tokenizer to your local machine. + +Take the dataset `roneneldan/TinyStories` as an example. During the data preparation phase, you need to download the dataset to your local machine using the following command: + +```bash +huggingface-cli download --repo-type dataset --resume-download "roneneldan/TinyStories" --local-dir "/mnt/petrelfs/hf-TinyStories" +``` + +here, "/mnt/petrelfs/hf-TinyStories" is the local path where you want to save the dataset. + +Then, download the tokenizer to your local machine. For example, if you are using the internlm2 tokenizer, download the files special_tokens_map.json, tokenizer.model, tokenizer_config.json, tokenization_internlm2.py, and tokenization_internlm2_fast.py from the URL "https://huggingface.co/internlm/internlm2-7b/tree/main" to a local path such as "/mnt/petrelfs/hf-internlm2-tokenizer". + +Make the following changes to the configuration file: + +```bash +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" data = dict( - type="hf", - tokenizer_path="internlm/internlm-7b", + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", ) ``` -### Dataset Preparation (Pre-training) +The type defaults to "tokenized", but here it needs to be changed to "streaming". Also, you need to specify the `tokenizer_path`. If you are using the dataset after tokenization as described below, you do not need to set this field. `TRAIN_FOLDER` specifies the local path of the dataset. + +#### Using a Dataset After Tokenization The dataset for the InternEvo training task includes a series of `bin` and `meta` files. A `tokenizer` is used to generate the training dataset from the original text files. The tokenizer model is imported by specifying the model parameter path in `tools/tokenizer.py`. Currently, `tokenizer_internlm.model` is provided to generate tokens. If you want to use a different model, you can directly modify the model parameter path in `tokenizer.py`. @@ -117,7 +135,7 @@ ckpt = dict( # 1. the 'path' indicate ckpt path, # 2. the 'content‘ means what states will be loaded, support: "model", "sampler", "optimizer", "scheduler", "all" # 3. the ’ckpt_type‘ means the type of checkpoint to be loaded, now only 'normal' type is supported. - load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internlm"), + load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internevo"), checkpoint_every=CHECKPOINT_EVERY, async_upload=True, # async ckpt upload. (only work for boto3 ckpt) async_upload_tmp_folder="/dev/shm/internlm_tmp_ckpt/", # path for temporarily files during asynchronous upload. @@ -282,6 +300,28 @@ Currently, it supports passing the dataset file path `train_folder`, and the fil For detailed information about the dataset, please refer to the "Data Preparation" section. +Additionally, it also supports processing datasets in the Hugging Face format. + +Set train_folder to the local path of the dataset downloaded from Hugging Face, such as: "/mnt/petrelfs/hf-TinyStories". + +In the data dictionary, you need to add new fields type and tokenizer_path to indicate that the dataset is in the Hugging Face format and to specify the path of the tokenizer, for example: + +```python +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" +SEQ_LEN = 2048 +data = dict( + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", + seq_len=SEQ_LEN, # Length of the data samples, default value is 2048 + micro_num=1, # Number of micro_batches processed in one model parameter update, default value is 1 + micro_bsz=1, # Packed_length = micro_bsz * SEQ_LEN, the size of data processed in one micro_batch, default value is 1 + total_steps=50000, # Total number of steps to be executed, default value is 50000 + min_length=50, # If the number of lines in the dataset file is less than 50, it will be discarded + train_folder=TRAIN_FOLDER, # Dataset file path, default value is None; if train_folder is empty, training will be done using randomly generated datasets + pack_sample_into_one=False, # Logic for data arrangement, determines whether to calculate attention based on the seq_len dimension or the actual length of the sequence +) +``` + #### Model Configuration If you want to load a model checkpoint when starting the training, you can configure it as follows: @@ -295,8 +335,8 @@ ckpt = dict( # When resuming training from a breakpoint,: # (1) 'path' is the path of the loaded checkpoint. # (2) 'content' indicates which state will be loaded, support: "model", "sampler", "optimizer", "scheduler", "all" - # (3) 'ckpt_type' indicates which type ckpt will be loaded, currently supported: "internlm" - load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internlm"), + # (3) 'ckpt_type' indicates which type ckpt will be loaded, currently supported: "internevo" + load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internevo"), ) ``` @@ -427,4 +467,4 @@ model = dict( Regarding the principle of Dyanmic NTK, please refer to 1. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases -2. https://kexue.fm/archives/9675 \ No newline at end of file +2. https://kexue.fm/archives/9675 diff --git a/doc/usage.md b/doc/usage.md index e05cc6938..98852100d 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1,22 +1,34 @@ -## 使用教程 +# 使用教程 启动一个 Demo 模型训练，需要进行三项准备，**安装**，**数据集准备**和**模型训练配置**。接下来，首先会介绍数据准备相关的操作，再简要描述模型训练配置相关的内容。 -### 安装 +## 安装 请参考[安装文档](./install.md)进行安装。 -### 数据准备 （使用huggingface数据集） +## 数据准备 （预训练） +### 使用huggingface格式数据集 -如果使用huggingface数据集进行在线加载并且在线tokenize的话，那么以`roneneldan/TinyStories`这个数据为例，数据准备阶段只需要将配置文件做如下改动： -```python -TRAIN_FOLDER = "roneneldan/TinyStories" +如果使用huggingface数据集，需要先将数据集和需要使用的tokenizer下载到本地。 + +以`roneneldan/TinyStories`这个数据为例，数据准备阶段需要通过如下命令将数据集下载到本地： +```bash +huggingface-cli download --repo-type dataset --resume-download "roneneldan/TinyStories" --local-dir "/mnt/petrelfs/hf-TinyStories" +``` +其中，"/mnt/petrelfs/hf-TinyStories" 为需要将数据集保存的本地路径。 + +然后将tokenizer下载到本地，例如，使用internlm2的tokenizer，则将`https://huggingface.co/internlm/internlm2-7b/tree/main`中的`special_tokens_map.json`、`tokenizer.model`、`tokenizer_config.json`、`tokenization_internlm2.py`和`tokenization_internlm2_fast.py`文件下载到本地路径，如"/mnt/petrelfs/hf-internlm2-tokenizer"中。 + +将配置文件做如下改动： +```bash +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" data = dict( - type="hf", - tokenizer_path="internlm/internlm-7b", + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", ) ``` +type默认为"tokenized"，这里需要改为"streaming"类型。同时需要指定`tokenizer_path`, 如果使用下述tokenized之后的数据集，则不需要设置该字段。`TRAIN_FOLDER`指定本地数据集路径。 -### 数据准备 （预训练） +### 使用tokenized之后数据集 InternEvo训练任务的数据集包括一系列的`bin`和`meta`文件。使用`tokenizer`从原始文本文件生成训练用数据集。通过在`tools/tokenizer.py`中指定模型参数路径的方式来导入tokenizer模型。目前提供`V7_sft.model`来生成tokens。若想使用不同的模型，可直接修改`tokernizer.py`中的模型参数路径。 @@ -62,7 +74,7 @@ $ python tools/tokenizer.py --text_input_path raw_data.txt --bin_output_path cn/ `json`和`jsonl`类型的文件的`bin`和`meta`文件格式和`txt`一致，此处不再赘叙。 -### 数据准备 （微调） +## 数据准备 （微调） 微调任务的数据集格式与预训练任务保持一致，生成的数据格式为一系列的`bin`和`meta`文件。以下以 Alpaca 数据集为例，介绍微调的数据准备流程。 @@ -76,7 +88,7 @@ python tools/alpaca_tokenizer.py /path/to/alpaca_dataset /path/to/output_dataset 建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize -### 训练配置 +## 训练配置 以 7B Demo 的配置文件`configs/7B_sft.py`为例： ```python @@ -106,7 +118,8 @@ ckpt = dict( # 'load_ckpt_info' setting guide: # 1. the 'path' indicate ckpt path, # 2. the 'content‘ means what states will be loaded, support: "model", "sampler", "optimizer", "scheduler", "all" - # 3. the ’ckpt_type‘ means the type of checkpoint to be loaded, support: "internevo", "llama", "hf_llama", "hf_model". + # 3. the ’ckpt_type‘ means the type of checkpoint to be loaded, support: "internevo", "hf", or other custom-defined + # load function such as "llama" load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internevo"), # 'auto_resume' is designed to automatically load the latest checkpoint from 'save_ckpt_folder' when encountering # training interruptions/hangs caused by hardware failures, using a scheduling system (such as k8s/slurm) @@ -292,7 +305,7 @@ monitor = dict( ``` 接下来将详细介绍启动一个模型训练所需要进行的数据、模型、并行和监控等相关的配置。 -#### 数据配置 +### 数据配置 数据相关的关键参数配置及释义如下所示： ```python TRAIN_FOLDER = "/path/to/dataset" @@ -321,14 +334,16 @@ data = dict( 数据集的详细内容可参考``数据准备``模块相关的介绍。 同时，也支持huggingface格式的数据集处理。 -train_folder设置为huggingface上可以通过load_dataset直接下载的数据集路径，如："roneneldan/TinyStories" + +train_folder设置为从huggingface上下载的本地数据集路径，如："/mnt/petrelfs/hf-TinyStories" + 在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如： ```python -TRAIN_FOLDER = "roneneldan/TinyStories" +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" SEQ_LEN = 2048 data = dict( - type="hf", - tokenizer_path="internlm/internlm-7b", + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", seq_len=SEQ_LEN, # 数据样本长度，默认值为 2048 micro_num=1, # micro_num 是指在一次模型参数更新中会处理的 micro_batch 的数目，默认值为 1 micro_bsz=1, # packed_length = micro_bsz * SEQ_LEN，为一次处理的 micro_batch 的数据大小，默认值为 1 @@ -340,7 +355,7 @@ data = dict( ) ``` -#### 模型配置 +### 模型配置 如果在启动训练时要加载模型 `checkpoint`，可进行如下相关配置： ```python @@ -353,10 +368,8 @@ ckpt = dict( checkpoint_every=float("inf"), # 每多少个 step 存储一次 checkpoint，默认值为 inf # 断点续训时，加载模型和优化器等权重的路径，将从指定的 step 恢复训练 # content 表示哪些状态会被加载，支持： "model", "sampler", "optimizer", "scheduler", "all" - # ckpt_type 表示加载的模型类型，目前支持: "internevo", "llama", "hf_llama", "hf_model" - # 其中，"hf_model"类型表示从huggingface上下载模型加载ckpt，MODEL_ONLY_FOLDER需要设置为可以 - # 通过AutoModel直接加载的模型路径，如："internlm/internlm-7b" - load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internlm"), + # ckpt_type 表示加载的模型类型，目前支持: "internevo", "llama", "hf" + load_ckpt_info=dict(path=MODEL_ONLY_FOLDER, content=("model",), ckpt_type="internevo"), # 'auto_resume' 旨在在遇到由硬件故障引起的训练中断/挂起时，自动从 'save_ckpt_folder' 加载最新的检查点， # 使用调度系统（例如 k8s/slurm）在训练重启时自动重启机制。 # 请注意，如果未设置 auto_resume（其默认值为 True），它将不会默认加载 load_ckpt_info 中指定的检查点路径。 @@ -399,7 +412,7 @@ model = dict( *如果基于 InternLM 7B继续训练，可以参考 [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo) 中 OpenXLab 链接下载权重* -#### 并行配置 +### 并行配置 训练并行配置样例如下： ```python @@ -433,7 +446,7 @@ parallel = dict( 注意：`数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小` -### 启动训练 +## 启动训练 完成了以上数据集准备和相关训练配置后，可启动 Demo 训练。接下来分别以 slurm 和 torch 环境为例，介绍训练启动方式。 @@ -447,7 +460,7 @@ $ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python trai $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" ``` -### 运行结果 +## 运行结果 以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下： ```bash @@ -470,7 +483,7 @@ $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py - 2023-07-07 12:29:16,994 INFO train.py:323 in record_current_batch_training_metrics -- tflops=189.3109313713174,step=5,loss=9.822169303894043,tgs (tokens/gpu/second)=4262.67,lr=1.4000000000000001e-06,loss_scale=65536.0,grad_norm=47.10386835560855,micro_num=4,num_consumed_tokens=786432,inf_nan_skip_batches=0,num_samples_in_batch=17,largest_length=2048,largest_batch=6,smallest_batch=3,adam_beta2=0.95,fwd_bwd_time=3.69 ``` -### 加载训练的checkpoint并生成 +## 加载训练的checkpoint并生成 若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示： ```bash @@ -495,7 +508,7 @@ generation = dict( ) ``` -### 长文本生成 +## 长文本生成 在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K 的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK RoPE。 @@ -503,11 +516,11 @@ generation = dict( 1. 对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 `config.json` 中的 `rotary.type` 修改为 `origin`； 2. 对于 InternLM 本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。 -用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE 是如何生效的。例如文件[长文本示例](./aux_%20materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用 Dynamic NTK， +用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE 是如何生效的。例如文件[长文本示例](../../aux_materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用 Dynamic NTK， 模型是完全无法回答该文本对应的问题。而使用 Dynamic NTK RoPE 后 InternLM Chat 7B v1.1 模型的回答如下所示： ![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png) 关于 Dyanmic NTK 的原理，详细请参考 1. [dynamically_scaled_rope_further_increases](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases) -2. [https://kexue.fm/archives/9675](https://kexue.fm/archives/9675) \ No newline at end of file +2. [https://kexue.fm/archives/9675](https://kexue.fm/archives/9675) From e0cf8b1dfadc564d992b5c7c75b3763889f826df Mon Sep 17 00:00:00 2001 From: sallyjunjun Date: Thu, 29 Aug 2024 15:34:54 +0800 Subject: [PATCH 2/5] fix InternEvo logo --- README-ja-JP.md | 22 ++++++---------------- README-zh-Hans.md | 26 ++++++++------------------ README.md | 22 ++++++---------------- doc/imgs/InternEvo_logo.png | Bin 0 -> 66734 bytes doc/imgs/logo.svg | 23 ----------------------- 5 files changed, 20 insertions(+), 73 deletions(-) create mode 100644 doc/imgs/InternEvo_logo.png delete mode 100644 doc/imgs/logo.svg diff --git a/README-ja-JP.md b/README-ja-JP.md index 1bd8df3c9..679e2a25f 100644 --- a/README-ja-JP.md +++ b/README-ja-JP.md @@ -2,17 +2,7 @@
- -
 
-
- InternEvo - - - HOT - - -
 
-
+ [![Documentation Status](https://readthedocs.org/projects/internevo/badge/?version=latest)](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) [![license](./doc/imgs/license.svg)](./LICENSE) @@ -28,13 +18,13 @@
-

- 👋 Discord と WeChat で私たちに参加してください -

- ### 最新ニュース 🔥 +- 2024/08/29: InternEvoは、huggingface形式のストリーミングデータセットをサポートしています。データフローの詳細な手順を追加しました。 + +- 2024/04/17: InternEvoは、NPU-910Bクラスターでモデルのトレーニングをサポートしています。 + - 2024/01/17: InternLMシリーズのモデルについてさらに詳しく知りたい方は、当社の組織内の[InternLM](https://github.com/InternLM/InternLM)をご覧ください。 @@ -49,7 +39,7 @@ InternEvoトレーニングフレームワークを基に、当社はInternLM-7B InternEvoのインストール、データ処理、事前トレーニング、およびファインチューニングを開始するためには、[使用チュートリアル](./doc/en/usage.md) を参照してください。 -詳細については、以下をご確認ください: [internevo.readthedocs.io](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) +詳細については、以下をご確認ください: [internevo.readthedocs.io](https://internevo.readthedocs.io/en/latest/?badge=latest) ## システムアーキテクチャ diff --git a/README-zh-Hans.md b/README-zh-Hans.md index 237a50e28..fc9e78d4d 100644 --- a/README-zh-Hans.md +++ b/README-zh-Hans.md @@ -2,17 +2,7 @@
- -
 
-
- InternEvo - - - HOT - - -
 
-
+ [![使用文档](https://readthedocs.org/projects/internevo/badge/?version=latest)](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) [![license](./doc/imgs/license.svg)](./LICENSE) @@ -28,13 +18,13 @@
-

- 👋 加入我们的 Discord 和 微信群 -

- ### 新闻 🔥 +- 2024/08/29: InternEvo支持流式加载huggingface格式的数据集。新增详细数据流程说明的指导文档。 + +- 2024/04/17: InternEvo支持在NPU-910B集群上训练模型。 + - 2024/01/17: 更多关于InternLM系列模型的内容，请查看组织内的 [InternLM](https://github.com/InternLM/InternLM) @@ -46,13 +36,13 @@ InternEvo是一个开源的轻量级训练框架，旨在支持无需大量依 ## 快速开始 -请查看 [Usage Tutorial](./doc/en/usage.md) 来开始InternEvo的安装、数据处理、预训练与微调。 +请查看 [用户说明](./doc/usage.md) 来开始InternEvo的安装、数据处理、预训练与微调。 更多细节请查看文档 [internevo.readthedocs.io](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) ## 系统架构 -Please refer to the [System Architecture document](./doc/en/structure.md) for architecture details. +系统架构细节请参考：[系统架构文档](./doc/structure.md) ## 框架性能 @@ -63,7 +53,7 @@ InternEvo深度集成了Flash-Attention、Apex等高性能计算库，以提高 | TGS | 4078 | 3939 | 3919 | 3944 | 3928 | 3920 | 3835 | 3625 | | TFLOPS | 193 | 191 | 188 | 188 | 187 | 185 | 186 | 184 | -TGS表示每张GPU每秒可处理的平均Tokens数量。更多模型性能测试数据细节请查看 [Training Performance document](./doc/en/train_performance.md) +TGS表示每张GPU每秒可处理的平均Tokens数量。更多模型性能测试数据细节请查看 [训练性能文档](./doc/train_performance.md) ## 贡献 diff --git a/README.md b/README.md index f05d4ae78..4fd668408 100644 --- a/README.md +++ b/README.md @@ -2,17 +2,7 @@
- -
 
-
- InternEvo - - - HOT - - -
 
-
+ [![Documentation Status](https://readthedocs.org/projects/internevo/badge/?version=latest)](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) [![license](./doc/imgs/license.svg)](./LICENSE) @@ -28,13 +18,13 @@
-

- 👋 join us on Discord and WeChat -

- ### Latest News 🔥 +- 2024/08/29: InternEvo supports streaming dataset of huggingface format. Add detailed instructions of data flow. + +- 2024/04/17: InternEvo supports training model on NPU-910B cluster. + - 2024/01/17: To delve deeper into the InternLM series of models, please check [InternLM](https://github.com/InternLM/InternLM) in our organization. @@ -49,7 +39,7 @@ Based on the InternEvo training framework, we are continually releasing a variet Please refer to [Usage Tutorial](./doc/en/usage.md) to start InternEvo installation, data processing, pre-training and fine-tuning. -For more details, please check [internevo.readthedocs.io](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) +For more details, please check [internevo.readthedocs.io](https://internevo.readthedocs.io/en/latest/?badge=latest) ## System Architecture diff --git a/doc/imgs/InternEvo_logo.png b/doc/imgs/InternEvo_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..771637fb0abe6737f1ecdbd904167282dafa11cf GIT binary patch literal 66734 zcmeFYWm8;T*DV}^G}^c~9)df8MuWRUf_tz)paYG&yCpzycPEhG5S-u|2=4A4^xe7N z>$;y)b^gHl@KiNL?fqr1HP@6e#+)l$O+^+9oecfivu9ZHa#9-4o*_Y=Jws4OMFHN- zShV2){}7xsWF?+ejzD&S2P7+TW$|avYGN_&O_70TG?<*O)3awp^nd>l3PISE8p_I8i_O}~Z{E1>j$|-5dEdoU&%^hL4v5~oUCKq*3HGR1)_xmt z-1GHjc(M6{6Ht2c~tw{8Y8U&D~As; zei}6!pFL?0hoLyy;M?)iH?}qpx5{bkbw6PH=D6$wk&110H|3xves;blUY0d15dL03- z8KW?5YNyGndgsZ^Tqj+1`PZi9$kFcYoHY*2tl$vREb_siknqb;Y4phKlM}g!u&Bb~ z9VnB6Sd>n^DgNPNebz`uvsk11ANaauFyvDXEwzr{zm_We(-D9;X(!%<0$s;kBUp2B~~_5o+sjYHQs~qrMRIXp|td< z=H|!0wx|t21{MCM2~`mGP^v%HQdhA@pb+wj0KP6Y&K0H?mRJ~saJVC+jDd2gOM#F> z>GTx6-fEuaR`I&;DlcDvj(O^O@7p6}kWu`FpkR`jS<=9|(8n`^%}rvpH*XT5kFu2b z?{04e{)rS$B~HLMZ1{edR#sNVyOYE~1Y`hD`izUPC5K!~VqyUn|Iy|7At!!d&$G%{ zNckZ2Cl1XjiRJ2zs6tCDuMzn&8pg~WM=;1D#@neSwy?NZN3YhBf&QO1*2-ld!BNzM z>@fVN*_wEA9$rbFB~7|@D{>|NGP~OryO53wPJl!!@T+`LtZ4MbKnfpc0YfC}ztPW2 z2n2d2`lUec4OkL|^8>vjx834*%@HzO${dJe?dM8k+gbvoAanMoHU6#59xI(tgIF8) zC$`iil0vD_rsd|;bD=)b#uQ0AoI)ps1{pCa#Z?*_Q{=;?)-T*VJkd17vASOWy3AHd zC*U&g#3ZDKIMv^}B*sv^mS@qfNOUKSQ&JcC9lBnThnpam&9;C;xS$+~6>|++Fzt19 z!2o9>Av?73Q^oQ^ih^x$J848o)G2W&&CI_Hl<<$f%q%Jc;&b@nF5UXmKM5?k=s5;> zNE0-U2}DpqK!C*U@ffb&H%B&U*&ZQ^_Yd!kQ}G}R8O|(D#N_2lLZ@$+UnMnSV6iXn zuJgt8oGuLyJxA(JNuDjJb6G?Ybh!^oZ$+$i6ZW~EW&HRr=t*Ab3P3!c@*9I~Ur{tU z(x+V=3W*Kky=Bfp2q)Uqil6i66PoEf4QUkE8Xl*n6*C!ACyKGg_?Uh)*x2IAU(-Sa zsoU{`V25G)pZIO0hJq<_?(Zd zCP^vOh^rFrrZ0BbDAKiDT7Zh=N0kdd{G|^teQlT-|4u9UXRh z)#2gF*{^JWP3B03GV${pZ}o{~{`X7CCV#`(dgZ4eWHCKmSkUW+vEtLDUazLLPC}Hm z-)uI-;xfUQ4D!AU0_ID1GG&epDpl7xjH)5^Q5#?R1! zPn-6j`Kh>(4*cf^SiH?4S(B$oIbNzY;#k4w<8dAw&0GT+WPaK zloiGBcbm_1$(UIj)aN$gbB(qfY|1)J!N_d9(C+8hZ?l{#LaB>QQE!xV5=?u!{kPzp zPfLs8{2bdiOBU@JW{6U8f`eq1j|;}q+Crya1z!2vyXIeK3afi51 z6)off-@DA2_g##)gp^+lI_LNGwyZe=h?^}hNUmp~dTb0u4?Y_7_#~C&u zB8R+$7;dKPr@4#`7eaiCW^qUf8J-l1=5?~**{3Nq(1yyc-J50-AKUv^Pe!A({-Znmzg`h`Y#i0$^mYI5Y}@bRKDhw;pbkP-%y z5_F5b{i701xECICtUn%FE+c_F+bNkO?MW*aFD)&z3|Qy~mrP%`W&-DZbWHf?tW_a9 z#R|;|{wW=*bhmPSUxsV>BMcl8*WVcMbB(@^hu1=ROp-LIk zS8<{GN-%d2%IXwZZXnzxn&E-77D}D*@znErw9eH1kLPMasPqa?Dt5A!zLBa3yeqds%(Pt+8G z@MJKRfP!ocq)76K0ym2yPM+BHXjWRk-YyS|obTuIOIBl*8pSUfN(K<=zFt%inpcXV&WvcE%Y|&^j*_Eg{5BTdCST-Jbf({O!`l(`fZ<7GDH=pr_*Qt z3zClkG;oSBWCnY}+^3qEuPD!y@HtJ^Vslf= zeV^#fn>T0-&8hF-zYkjcG7cG_>}Psm3V;~x!Rh|xg;v*C`?qh^Y}`pLjDF_UZjVg( zOMZK>&c>g=zc_8bF=qmPI&I z`w`!Z?%lLfqVC zl(mR4=cTB6^Sg*NxG07 z$d5({RuIh>nWElFN$k^Rff3ezOdy?>*eL=X8GerfQ;9H#%20n=8PC%zt+-@+zgiN8E%TxqakYE7yhYw-*QU<94;&E_ zPG|wKNoZttJ-pxl3g@sA8fJy|-;SRQ;E`MwU{Rd;E}J~bdnN*tzUHB?Y|5t-)||+c{bqw3Jo7lpe3bAFS!~)PYD><>xSaDJ80-dc}3Hwwf2pjb>r0j++5>kN8!t%Q6Cj z_|kPsqGOMoT}|(iJFYTAp`+pHeVM%v?&V5J3r) z%xrUp7)>8jgvCaE`t(!T38tBwy&DiS!7MwkiE}TrG6#JaZE_s@&g7o%O++@ta4*v4+ zM%RQ2zCXY+9H`+6>%4hf*ZaCXzxw)UXxw&nY9|M5}zSlj95yr-$fT_IO2Y=Sd$3 zn9}7#XQ|VbZRVI>Ej5RMI)~Q$ySQG7epV@)KU4T2=7Vxc-q>1=8WC=&K+gXmr~7=n zq^?{EOjqr>(OHW!Lh*^eFu!no#L&~)CLmK;YSRsdDvG<-LiQcW0_>EIUo^YI%bMq7 zMFdIM$?ir;8AYhF@QEgw`|2!Y*!W5E`q+F9#tHFDTt!Z9RD?{pqg$}KAo3dIC-?FdG=Y?wn~S1O4(TU z5DrcEEp6VKbuEn`_46)y!;YMs^?yUR&>hImbRC??FTxgDoYP=r!JnC&p8aqS%Cm8i zGgOX5nTwCci7C{ziyL7}>iEhj2^(wwOmla}L3x9sQ!4tRF?Odnj?;F zT8mxburnFn@Q+5FSmPZwdYE7FAW`QPqgmBWxq-Vak>_MgNsCP%7&bQv z$MR&It*Tbe2mkT4Q9d9ObDL$CAC$tutE{C#fjEeJ=wS9We>be0jst7Gl*tal{fT|Q z;IdVGIlczdE}16S>T|<(pRNf3i~H5^D7}*M^8CSF(zE;CUV6rHgl(0~&rp6RnQWaD z0g#k;K%nrk8M*rSDG;gb%&e>~8%zHqyy>B^LhE2Fccu$Zm5ZJQ=R3(~8j;UCAtNu- zwM;ETCk{-0Svezt1Ptgtt*w2T$ZYjw;&`n(BHVEk^9?a}v<_Fq?L;j?p-&|}p;k9Y zHus)JxcEGG1V2}EC?4D^(C5yBh81(1yGIN$DW)Sas&U$;Ou6{ngYpmGv=OLY84K_~ z)^^|LJ6v=XgNAm7Pl1c&VVtqYp*kj;TB*ty7;l$N$zfu+zx_4jrmiuAY;LiUn%zNAu!>)uZw3BpdO3BM$b1%_gGFTt!@YB+^N`-@=b zmdF``Kpn5rVH+VF`5bDB+qs7iIZ45Tkl|6}T6haNZLx}Y-Sp4m|I73<-vImeUqN&A z@F;{mp2GE^aac;;yTfYeA0`9{_dmshC;X=aBnjB`YXBrFpv=$03_(LmDFPv(4!w*2 zlq6$2Q&kjZjSAW`pgRs26nM24@U#WQ=waHBr_<#|zWD}Q@D#K=#`R|8uF*(RFn6O=hoMii`+A4lW(Y`& z2fM}8t(74Wuah{J!&zx>&&z_PMTugE>!qn8eTP%MWVgdQQTyXXtmQHbiKfe3+`!Pu zJ;E|6*{kn0^o2};P47dwFemRD+}B^SMH?yv-eDIS8G5G7Ioa+{DyQskA6+A!d$QC% zUDJd#a*)%u9WOP4yrCbtDHeR=j`(56mLwtLK<<(7)BKX2AOWaoO%Yg^Qc!D2WYEKf9vT>?l?Pz$LoGeM_e$=htOt(YNc}#wd^~>%XiEuZ>q!w#XdlJ0tVRO@ zYbzUkw(a#`VeRYT&?D}s#_T=(x#doKOm!J%O?*)B>9g5-Mnj&czH1KIRu+b zrR+COQQy?zgZ)KEH}+q^-wP-})(9r$ zhCV(tA7&d;I=L3a$$}8NXw9(VJ6@4eOQG(Hk(pnA+gbS?-+enZxlaKgu|Lyg(Q%MZ z$e`v9{9h3R3VYJc5#@L{VM-*$1 zxJYePxKx)(3m!;-w(?A-cMPv~)VH&i=MzUcu z{S+T1r&6}|-U%RuW-7qcRD;TeJ}i-8fQLi(_Q>X%EXbot1V1%JfCX|jTWTonuO6E- zb|Xi(#uCd}fVC^}biIN7@cz;C!J)GEuORr?8PLTO)|lsUyd)y%`$)^C-!?$Q2bBgD zr67lLnjXcN8FtyB4ERe~ds&l&s7p<{tilbOUDivSlWN|Su{C>bO;@_>efFJg{@x2@ zHU`zqq%M|scxvemA0I2aky&c>h>qV#MXNt`dvMjQ4W&pmd0uDMTy6zx!gg4KpO=lK zR`mu*p69GtB3I~qmjL^tb&v1d8u6#z{rLi?|BMWrfjvONI5Ew!Fur+wFfu2wggwBJU)``#a3EzI(Y+t$pnCZMdd8N9S175 zP%yx3X`epfq2KXGpN&30r5ea)g z)-krbWVjqG92SFaBGaix7N|x@+3IwsnQB-3OhuRn5F}Y>ZMu4uxfOk#$n-h_BnP(^ z8l8B}=P*eZ|0>^gO8=1@YC@1ht>T(pk0(krt=`d!ZVshny=X?qnrp!)@8;GCT_vg1 zYYmy0 z?yPh?F$%skioo)~Y=xt<0o&pa2qfZ&d%?{v>K~H@6|c8nwwK8;>w4AmnCePOD|zW% zT2Y)2>M795$W(S!%_u4r0qq<$4a50g`rALW1yy=~OO6nI_;gk{nw$P}a(4ivnZN4% zIHX#<5$O(p*zbYw&-m`xPiZtYThhHboKtra9%QQNyWPBeaB$DX z&Z&7#PthC7rN0hVpitLHKI4#&D=G6gL$M|I5tsutYB;)D;WPKV#uIqmFmb`Gr`d0rjm+0LZ%pz%J$a9yb$ z?Bkop2AxNwz1#fYVW1&{J;76Qd%{(UW0CZ9ehmNfSpaT2C8cn{6E@;}DE(#WP{zig zMRm8TMKXn>O(0szJ2ZMq~Jxd00hJS4U4ho$3+0s>{9+37*AKehQJEc&$jws0UwA$rPjuMHOj1I{^c6 zz;2CUp0{q@@0T-`r>HvUv|I7lM#IE0|MeLpxU$1yoB4~2$iy{2JhY%`I;eF`$LFzU zpGUHzlEMoA5_li+oYSes6u+3eb=eCkXl1Ozz|Ldvi>7)Xe8Yz$*#WX^1p3Pz$uCPj zS)YtOLp6*%GA0CDBuYLAjcxK#Fy_4hHJwR-2E21bHZfS-+C&INmq&Wef;UBHVjR1d zSfgC;zClGFtSLFsD7SE$Y`~9%Z%4VMh+^ zpoPH1+;Hyn+SB|UdbpK;kcx|j+RiwPRW9DWdk`0?Efr#a9{9L$6KSJI$N9!Kx%i7| zeBBGy0EvQXLfOAfey44Yu~yGqJK@}cbsIp88tizOV_{{@f9-o0iE>FH)%qOH4Fp#5 zmZBm7glw+{;Ol_2mdbRLp)SRrUsy0G70+Cf=L8MLgXA)5rE0>)OiwI9;{zK|?^V=F zgKr@d?8b$6*J%x<(YfN?&mcGNKT4K2NXI%k?k;6|C!9aT!d*5<1Q$!l>08fGEWUZC zoMk$wB9DG6$lab-KRv&hPl_^%8)q|p{P@NJ`4C64k0A__{mdZ^gf zRK@0}evWdPkHeJDwHy_g+4LI~(9!gCU{f?UVSb$5D5z%YxhgW!>UNzPb8Sux8Bp&} zU!7{DADWc0DTT9TV=4JrLL-qwlN1#k(Y_O>L$gIVKqz}^Qb=n1Mg!xgMhT(PMAjh9ecq-Ujqi48nQV7m@a$0aa22*B; z$inwyQeSnE*smgWF<;aXt3H)4;DJx$&HET%`jkN0@;=F;f6h%R4K21MVfL3x@eT!# zZ|S5%$UvaI4;0)3LWo3uU=>@wr^DW2 zo+`!pMI+lU{C&O(;}&30LdYAsD=7$H!TIYhZhx`nv+3ON<%XYO_Nak!D+J`iXke^J ztTj*JXmDiqw|zb}d!x2=$H_;PYAkRq0eMiZM}2Ch$=<*itZA(1b%yt@Z^GQF#lZXd z_%l7U?I+~b4RUo11y0VT$m1znZS|b|M5mpgKTe^fStUCldcGY-)bp=;c4JkEv(Ex= za*)C#P7QtbW8ejciU|vkgf+)i6F)~5i{FjAUTl~#i*yjA#Zs1-zX5&tS}1O#EAV0Q zdj@STt~$Q_s=7J>lTlx9&tyn++V0SAC&%P&ppo?sjB~zJRy|bc#M;&gq?ir)Lwv!H z2T0MRTuB~qFstc{sJ~ddZI&GhKb970y2@UalpQnny=Tj~HfkGe2V9q zzCNBbuNfUJ#qS9S^vxE{o{Tu_n&htIk;C&it~6bHB??SxO>v0usAwYGeRsN3A>(?N zPXK|9+>1Q5208n$octv#EX}`-S!SQ+`o5jDtlsC8mrCm&LffJBYpufVYYXO$D`9h^ zL=7mg)~Tr}=-?!00==kq*97IO{jc30Me(i$r!rFCPu8T)A#dD>>&o@jqe0$t(n6Y? zk5tTMWAi@fe^&=4v(}qFy&Pan4(XR9^|n6EsMPmrNosbo~-jN|tic|M>l3c-^dEq!ISm-dx@ay-U$x8m%A8gJxZE~*Qk+^QF(*a*x{|{U2{Gu_G{JbJ^0qkCosTG4cy(L+$xD z1Y}Y?y&qYLFj@}P14s(DDvFEC^_P}Xp&Z2=8c3YK4lrgxQ;lE0ayP+G@jfgyq)0rW z74$5#7a}_mj{(}-Zd$PMWbE>jT+UO-MCu#co%X;`CLtF6rFL@WKA;{9oPA$#<-j`h zna*I|y!S*qI@rr<80mgoee`#1@;MZ=$hR0H1o;`;**UoexijNQNAJ5B=ApL8%a;zS7qiEh=s=NdhDRE@Qp011S0PE=~HuOu15sP$4!5VxSvzHajK{wid&y(88Hg==pHn=Xt&b8=lBuNe+2yYH#b}NsLVg!6 zGBg0JC7^sTo|un{3O1c>YIkffNkDm@20jz=*eV#Z=kh)EE&7LTHC zvT9i2MKw1|4T9yj^4|;fn{qNXj|K;>re)xS`CEDg_g&?`=o|g%CiJK%e;j|jQBrrx zl`OM&IGcXSFEwyQjRk@L^=rSsiQG{eBBY4;!)rVz+;!pkkn*SXtGmuNbCDqG+wV0RI&+QR4fALX5ap2doyKOtbL*)cVROR zJys7kYRmqeB~%i0q0#xfnpR`&zE6q()l2}BWDX%2Y)54&Zp)TqP*{YaxeK^!0ATQp zmtxj8?z{bx0*iKeesLJ%?fG^44$JC>GDIl;HsawGgZ6nJHS}Ur#(kKQ{RmeaM7z zM=URNC~%6<0dAq6a8DPP_c_{HO#tm>nzA+k9F0Q z@ui-jP?P9t)z5Bj49q)?1z(@rRVzWC3+N)g_mAFZ*8+8a*k^*>4(Eu9uZWMgNsp(u zF?zOjP>!dq&bv?79uvLko5eql#8_g@`^jnQX((8!W)#fs=;$$4o};ZT37LgiXMr#i zFP+647Ci3;U39;Vc1m=A`=wvU-HptF3<}*bq2aZbJPK*JK@#-dD&LV?;`#myy22D{_pU;a z^#;{6h@WNmD1KdwTJTPO(L-IKPkSQ@Ke%G0u0C-s06xpr0(7cjvxE67z^IE(`0AKq z(UOGS*cLmU+t*tpP}!y1Ldg8~EPz8i5OO7D6>Oi-4c)3REJkjlO0OvL=k^Jcasyc< zFEtVhP6N=${uYoCjNWIeBs@mt%?y{)Mb9bRpnh1%aJ*m|aoyLe)Q^{a#(;)yS^k@+ zA!ZzxiIp+G3xqv68tGdozy4C5p)M9C>IvUv3Z>0%i)E@6zsAMK?nK5uK^A2N+(UAhN{JF$#9aC~dwOz8O07HHi)7Jo5* z5xarU7ba?vJF;ryZ645cH@TLTZykW-;XFuXu{?jXf1B|=?udrhX--K&N$H{~lh$ly z?T_LFf~-i8re9O+KC;D&L8YX!vXW8tBaveA^f&5MC%ms!;@8Z~b|+Ou3)I&PFppmA9DTNvm0CR8A`AS<*JGV^ zDdN_>63=@XXv6Jsxbldo6=zIMS3{=1W1b3Ph4N~SsTsC=xg58S}{Eo__SW6UPXq z`xK33m*2ZvOvLm1`ceowvs5~t>Q;cj@4AU6&KYZZ$tZ4b& zUiHi4HBmwIT^#c-``o_Zf?T^eH9QLof|PUmNmO3q4g}Sxz9@*~qM)2GpQvj zxiAr)m)O_d+c$TQsz8L;vku^doB4v4A*6Se*UPsg&-q*Ji``{^PEqKTt)KnM?_wGtv51+$aXUdG@8XEd zujRu3m6OTw=u=aU4gHQ2>FCf8gM98)%?$*B2g1^Cf8hH{Id}K;PJ-}XCs26aQ*N&6 zshBN>G|J2*iykkfS*qO0eOJBoFEMPt_o_5ldjAAm17MjHO_%A5q5M1&^GY>{=S(N0 z*PL z0fCBt$PQ&xRM>|JFVS-Y8Y@@G&b9?5nwJYSwQrBojyIicwgtZF5zmo?`MiJnDB5kD zPMpUEj`yv&o*sY=1xhc5#e^lP!;eUM-g013@KlPRM#&wa(X1ejA7N5(N=-4;N#c;C z-`Ty{ZsEFrTR+;cg>pqI1|*NfaY&lL+nDi-TrRvTDR?ra(a)Uy@sA0jUcB_}XQ}X( zO&s)O_z@O^A#{uTYey$FUjk`>WJ&yd2kWzjR3HcJk7m539mXNW%t6R1IleqLKn8t_ z;)xK)!1O>36Gib>I=kt1K-d44h)eO4qgCb$C_2B)v_|W5ill1XOb*dzfkK>(aH?id z*R1;bJBDBH)0Rjek)u|=D!X4M4Kv>*3lY~lEazZsvrlBTKgi?vDkZ3@jQ>WP!gc1K zKHX)?;C`Vm-0&0!51A$?TUx)}ctpz(_K^rF$P2Xnw%Ge>ilJ7QEf14I41UkqyDZev z!H-iBVCq7GcYYcbYcrd@`ZX$j`}?YQ&~&Q|HS;uKl8MgS*SeCk#|iS9jNGiBG&D6V z9C=(1;1bUu>VwO7Kl2&MsB^-MCJ(l?Dye>$ISY3%(fR8Y0N)Gv$`DFg6LJtoxio zno_56!6lmcSkurO(!Z~bA@Hkj{k_|O;Wp3d{L`elGZAd;hL>$@|zo^&z3s zH+SSu!xJ z!69Z*WPEyCdcSL&fw7-0d^`Xv*^HDdM z0+V4ArfyAPGE^#v5VG-~-f=D|3QtB7HV-I2OW13JCk?HkM=G*L0$WG4fnE8tV=AdG zI~f6i&Cu*+{)@)u>1$75;AHrH&TwNp2}*8AfHl{1cLuEq#ml>vw-h^$&Du(ORjyU-m^cBhs8~)O@~J)LtIQ!Dk$7QfZuXof#$LQynRrlTp<*?)nnM53PHl zrUQc^t8f;ccE|C)sR|<99irI(Q6p@5Z8bOlfGE+i5YPEfMTm59U z3LCWe18_+vEpql54+eN(ifh%4Mr`H!kja?!^W2q%0ug^_XVznOv%HO#exMvEA%|VT zekV{_5f5@QAHMz)tET_^dy2N?-=`&2cH@nm*RWYI#f@qN zjSmewu>&&5ID^}kM{o^Xt5-;)nXzhu3Jxm_oQzJvE>upD@s_w95X>G4Zok|AP1fc! zg@YXQWPX2=nrK@B;kiaaQw<25EWa9`hAH7J(@M~7Xd1gRJG`BUVH2`B>t^DX`b}i@ z#?EOYP91)5^3j|nGd`dTf(m~8W;}5Ud~Lg^rrV?ZgO~It%*7$YVOGGyGi~O4Q zV@tym9oj{CO4gXi$3tQQf+i^`ZZ>S7JwQWJ)M9&W#Kb3Nq80!svb-;#sAVkB;(nA_ zpglR>>UkN$cuAH4ZcP9&=c6%jl3Szmj(+b zCXpi`L`0>T`}a5=cxbz}J$pluDVOwg{drtKmwXHihY^=4I_bVuYU^nwm%Wdu;hL2Vu6Nf-Ym2EuRKo;ggcX0ySKaAWE7gO} zJcuno<6BwnkBMvekwBjnI8ah+I|V7aT3*v&O_R6~IZzn3f?aiLh6Dy!53VN3n6Tov z6H~NUBaL4 zO7~t&S$bFZo!WD$#nDJ(>kp?X-+tbMN6CFRtt`iL?BSB`Ea#&2uUcL3DK@ymoKXjf zk0<9^q~tRsi%m|CwY&vBXJ%_&){RPCv|gCyD^K z^mOJCn*Vh-mK6ti(6-nCg5&mVBoJ!uGoiD+jIj?Ky>`*}u8jC=KFrv5YU3&$)i*HsoQ>NtGl9LJ$lYDnKV!GTG)?bF?w&$PslqhWuJK zty6@>U8nGbbGy@ID&-0zVzGcTOY$v}ndnS+xnt2V9tw`)(967G{O9fxjU~Sgx4ig5 z9i1Etx)4GklPEr1cNnD7Cfdz3%H*XQO6RO%S@RDb-~9}xKJ%Ov~c7pJYCU$3itO7`V3 zz%UcsimyJy9!>c9UgmN?px3qTjvjBXA5OKFg;Qsb{k>5psM;TJVPEB>PzeZ$2D%&R zq)D@t&)SrR4>i`wf=rbH1-H)|HTt*YLX$wiSXLBppWpg?>jMELrJ|`R4n-Z1E>Tn^ znM_R86|DV$d-4KEASwRt-X2YV>+a`oj1Zq=#la%pW1#Q$R{4$Uj+>f%=JmC(QnIxuwbh{Yu*cpC=; zf_=4IJG;UTO_mNNP`CWjt-Lw+Nv@QQi8!+mT2y#`t2KJcqi{~Hfjc`X1I2OkxIM>i z9|Nre)cAWr}Mj`#nBnsIU@*3eNJn2q)NuvC+1;ow`O^)W!Mk1JX)R zEe43;?ZR~{jarAekV&y`|MJ;kuES2PZWbxH@O(DE%i17KbyULoqIw)|vMj#7a~OyW zvgM3ol(~5QfrVIIHhzEO`1NDq{!GBGzg2vs+FetPnR^ zO&i83wE8wnV0a^22nnG!Mf%xOxm@c1$I@AbMfHAP|Er|L0MasmfJisOkkTNX(j`(- zLzjT$&><+@-Q78K*U+8PUDD6-_x#`aip%SqIrqKyUTb~!m51pQDqwZ`$jd?p3k_~} zAazdBk%;c2eew{b*yL^c9*v-Sz6Hev$-vT(cBMMsc&OT=akxoQaltckUxDW8+l1~1 za+J>>)s9aak|P~N&|JHk6r^6ceZnLHM+Y!7U4%jdd!*EevfaO$(h(nKR8So%+5}OM3B)~cskg=e>Hv5gbBOltE&X*@1bTknGK6-E#LjZ2NahSRmM*x4B8ofNM`td@5g*3U@kJT`{N$wMJB6kIUgVY zb}2a|eG;gUgLVvKQAR#TU!Sb7;rAKuzSxh~@HB2Sdua4_%7OoqKjFW0c?BBWQx?|~ z3%DXGPxqGapI81}B-j`EhOUP)*xCh89d3N!60hx>M8;4RIf8%F?=3vb!F2HC?V5BA z=jK0O#Y?p7Rb(%dT9X96z)pVJb zZS|*Q3(9~i>^(U{w!5SR-0*2{&m{Yl*n3b zd1x*0wE@!MJTE5NpRp8Fj?U$Zl;-FREG~0;#LJi0gn6q1!hIaC9G{m4fqHbWp+tfC zq6K1bwg@zo47puaPWE#UkDpG7yH7M4Gp`8*F+na9uTjtr)Ajq_FpKB=?Ec7fIv#dE zVQSJUvqwQ&w#RB=M#2LhFBN7fknv2QZx%LB-z5h+Yidv29Iu$mO&P5D`E$nYHNW7`5zxzax{(hL%tg`alhUW6FPUH$24+3aT!;8oBk!1eYevlOAOHd2Wq#c#{-x(vBGCCF<2iKx>szy?-w|cc3&VXU^XiMj;h$M`O+6`K8=cg57I*U6 z*P}Lf_35}^DQt3vvGjHk0=dRG`Rt%5&V{#>b*tUO^KF+|R9UP>MMeU!4i1HXRpBdT zDUK9s9+yP<*M;Ue%3u7cAQC+~7@){?g3e(-|47>gD96CndOfYEgF@wc(*5PzZBE)q z67hKcHrd-!e&x#b3c1V|e|`b6s*9?>#o4G!fiHK=b*z)EK9xPVpJ{$S){-JYP6nUdbQasG4AMD<3^Gks z5caz0EJR~4dFpKVyEh--|F}uGd0zeaq!DPuZ5+K2w`At~x4|B!Zj?S^RvYg6r+n?l z{m^aqMdznf`WgJc-h%pSqsL2QJ4K{aFI#NpsUR5?)bsk3>v1%R*$9r=QGcOr8%5#U zon5srsX1Xc&-d`Wa$MBMzn+Qv3a#8v`2EEG#Bs`|&hIRgF z>x>-Q{1DHtytFxd`*G9W{0GUkg$I^{-3;n+tEZPW|4EiAsOjj*rhA_^s@VQH;x$+6 zxqKs(K+ zoC^MS;*$f&)!*T>Mk1=Xy#mzKWWY$SV-)SvV|V3=jtSUH%!tS8T0Q;ym2g@@!iEx*B_5Ah`2)0IBpi`&ic6e+WN+Z?>V; zO2bw_W43pC*bVt1wmgC?4i!l~pqx|+1yPRu4Hmw&_}yhTrXKV)N72#S@TRpn@0poG*&?-NEeQ zI6>u39`|025eE8~*Cp50>jyPTQEfFAZLZZl7RpUc0eI6@g2pJ$M^g$IzkK!6uM+Ri zEUR^9C}MBSD~~=Vs{QO#O2vpCP4HONoj|=kEaKFV_@3KiOH6Mwrx42%;-*TJ;17O!P>Nxy+#bOc-IvI0A#phJfmf^bHTP;df@0vKMxzZ@xkJLu8IG**h%r4u7D)?nWqf2tUsj zTJ$SN(!j#AF5{d|dac4Ffk~4n(ml3=`|Eu07erI;#T@J|_@Jmh>VE6nU||8GF;)qV z%~hwd8V!PcUJi`q@|c3`S*)ICjb?CvE3&fan@Q#DUNVF&R2M21%fFi|X1h8;qz7SB zNa6P?MqlBc++QC{w~9;jRl|tLOJ>yC6A$9OBQDxbha}ZMZ@k(5aorW~na1lU?zMyR z`NmU&b@0vTrt++y4~%X@Myk0{(a;n4W4qbi}zR&K=wt%j}r;4jtDR*v_gT`8_ZBJbDKW%=@ zc=WuH%j^J^>9@+ZKc5RYyVng!BLTE8`KjhdVf^SM^JwMI0=9I_4iNt76hErlzJ7uvlUB zTQWUlv{h_8b6g3B$3t@yjNzVDw%@Zg(|2S%hs_H1u_x%flOTr&Ec(XJe`Ngx(zKtgYuk&`ALd zG#$uTr)b@4f?=iGTCKIVaPIY~AyX`AUH+|&R)?8i>jJPn(?G&kOxDwpfLXiln_S}2 zg390!`QCz4a$#b%C&UU4R)v?|34(L9*7}K}Esg6!5)D2ym?= zeQ+dFNaxpYqvZKYa^=rOmx8nBl;CX_kG(4t{dBu&Sl(r8hu-(XZ)PAie|IXRDd_OJ z+2t^Ap38iQ;rft&XLn60Sum9+tnPMd05PeUj_bp6of#S8C1k(xF7=~Nnyj3hL|>_~ zN~MmTInW2@c(<;}qj_UFH2lZEw~^ulv9Bs)^oayzU(MEmT2cLz7^d%e!jeQt}txU0M!`BB+-#xe6d{@WqBa)Y*00o}GT z44Y{5=!Y1?<9Y_;Sg#pc2f16BhUX4@=oS(At>^QX1tMJBNj{yI2 z2^4g!GHQRPsgaYw?RrZkZJPxZhvrD9z3JKA&ihmb_x`to_m?J3w|!&kt=~vBDCZQW zL!=A&udp=CQ}XwR!_>WQ!5W5wqSMkhs#kd*)CubX%2;(I5>5Ihd&-vS_af>UayOY& zfArfe`i3qvNb$+Qk<40ti*w}^ZbCm`!r|->pW~w%Tfn9?Gx2P7F*C=sP26tC%O4#h z)6_pIXZgPxs~z`dXBG>=yveDy2mFE3UQvl|~c2Gtz z-c z6*O(o{9gW{r$GVB@bAru&-RoZ=eg*EFs?LWaIM6P=fKe;e2;@n#Px0qO0JS$g0)Kc zhj_9+tnC*$ZS$zRX85NkYgSXeQRPpI`}1f@L90_U8<2zY8;F!(V}RoV^lM1``@t8V z_;<#%Md%_^;}q>4XVVW{E${&(A-#?sq6-#|-k`_v?hhjBWA;#C{3AvFa}6@u*ZRI| zTn#SUDYwozKfJ7w#Tsw?qxo9R_iKx_Uc5G1)uM5l++NQ|uG}`~>VTc)omcCrRr*4! z5hkb!2wvh%v%7WRCUMxjmz0#4sLbx8rxSny56|?aw3g+kLM&RtRtM~nlqb9^LTA~K znCoM*YcxCvZqs4^Cg6DNDljID;g$j6qt}FsNhHtzG8=mcm{dcU-s7PM|J0({2cOew z*^LQwT`g8SD!t){(wFMj&S*8;W!M-nFOVO~WgJCnd_^+DynR}v+|}FQD=I0#^-4hd zZG&2diH>MkE!C+DmBOafFaK|&e4WHx@iF+d65XID<;5W|A_IQz9?N0m*do5WapeZp zX<{+-oScfG=-46H>`Hhb=)&uO>U5@*W~ni(Fw`RhY>kjni=u{!{Fc8T0U1gJ>iW}I zCZHgeYC{0#2*Ovaxn*c0BFjNyjfWdn7p@Y9%}P!#sQLa)UTz}Ktv?YNuxtCYgNBne zjB>F%bN4~r>zG@pzgeu-lvaX}H@GMOq3IaE3m3|gp6>h>kk9b6LY%&gzq;HM(##;l z%A9fkDSr2O)JFaE;!6{|2Xd~HNBpFKOI#NV2FG^(5{~g*0+yq~S4-0}F@GLcIDD{T zMX++D`?*zt@8gqHRM=qmQ`p6cSNjx^a{f9lApwE1tSns>0~Bus*ID_S_gQ}RfHdhzqyHU zSI;P6*n3@sDo)M0Or4aPO;jm)5MN2*WaS!n!gTpz2DAI;N!wi{uI-y$zFDdpT<{<7 zCt6c;Q?$pXxPgf}YkX&-)QOHE64M_RQUCnFJr~3*JH7RcW;cpuMC898@Qd=sF*6*) znjkJykVNFNqQUm-OnYRs{7kTMfhNiR%QM#UYYU>;|xPCZD3D!o<)VHJ!C2##kv z-Xk7O&Br8WCGk8VhZBJ>IWoQF(*;vMGwX_KlWL#4eSJa>Yl`&v4-L1o%hk{a&;)7u zwd}fx9ns%@@!FJ*ia(<6ZV~}HA!De)&EWz~w$33`uQDoO&uF$z|3!LN7BK91N$+Ti3#1LEQr`pKh1zZ5mlJ>IVod%GYVDUHC$MWRbZk;c+|%uWp*zk+?(jGcU&TpA zL%gEHaDuU>r>_+ro&v5C-nNkxj;nDJFHkF!max$azEA{Aw7DCoN4Q{iwZP;CBhxfM z0E*FVH;zrY&9cYqqqpS&8tTHs*^j%E0;vp`x7YzRC8?&aFl~jK0)tHf24?hp?KfbX z7T)2W7x6=DcDj}c3$MkHgu4vUu{Cw-v*WJG_a=4iRPIhQ)$h7ooy^BriyD7scKb`q zh-U1)5hXseb{e* z&JZ!J2A+Sob0f{DzCO|nhIT}UcJPmSq*M@qh3|iseKn|DQ6dxFJv6Tt=DJ=E;$Lj# z1#BC!qbdAyH&!M14Ik8pceOSs(~9()pdrHdTU?3I$yFP^?=cPfY<(hO2e2JzMIf*QZls?VcPs(_;b2KLH~T|i@g zZq7s>mpRHIdJ|a+7;Aa1!aJMg6Ph;k6ogfra4`>7SyfN1U#P1OnrCHy$9$QtEEj_y z24(?>ZhQPdF7l4X1IEZthxn0JD@(T&f!B8;1n7}{&UebDzH!9fVaDA%u=dEuvRSHbLn8l)gm<$-pUKI3F>iC%F4me`*(L&k*o-4xzuS`B>7X$SpF?g*) zvafq0^BJ`GdzzSgDkg3<6eLw@QNrkYl6z^KP2ajK*LlXvw-~fpl4^29F90?(cHo1p z{ig=1pC@Jei4bM9&&;X)E1&dtIKiD7=aw`8aCBkN;($fmSAE3mZ;tY$-RBuYUwrW; zcw=-dJ$)J@JDw<}<#;5XwJ&g)Q}ClG@=03K=RuZ%MoM|%c?F28k%wCV-)WqzS|3A{ z1~~h+Vfar~O7>FC+<4?*F5l3opue(iLc%X658`KgdJ}Y4d>#+MZkMKjZN;gl^Z#*~oJ2u$BEbFqJW&(%)fUO8 zZ#bSh{Ia=cV=k4Pw0RI;5iFK}lq3uYy`ssl!GBQop+DGR4Eu+nwjKP@r}mEvS57Bo zl&|zyk~AQ>O!4Jezz;IyPNRO)#`M^v&={)9v2AW(n`UUbpylUZwi8RIma0tle& z_!3{WavJhb=Qe6w4yIE7jOmLI@-l_9`~}^Cg8L8fC*wU(wc4S_94TLV&Ix-Lypa5c zYZwrwlb?#O?rpNXQiPp@lwO%G!>@cjCo5 zir?)u7#st>gGAgkm0TLq^3;5gZ<&dq`l2m#fJXf`6o|#z802iV`R_&+=qlmveQ_^= zBRQWO&P#-bbMDs?;GDk63qXc7{D-)j|BDr(HtoRJ`Qm`jVN0t+3G$Fj3#h7{YvJny z-GD{<>rY+7iCUaU6n-zabJN;V5$o>gn7YwM9Qb;22092H1?_P58$P}kzCsVHN^HZDWcvC6e7Ye$ zL9>>5UU}>gDx(Cd0sZh-TMH6=G_JDWmsHy+hj&|HMqYeJRq|LRL45NFv;Kgt<@KB+G1rj}L6couC??N*){M4p?VpDX!vtE^vg@J`Q!d6Dih0LtF8{Kn*p_%qqhHcPszMl{^@maq*!> z?Th$5C~m5lZu>isSLVB;oHk%0UMb%V*f)M{>n2EncEz^-@cALiM$lWPww|%zu9J`P z`QC1j1QI-H_+9dlEN}EsCihFaTu&RngJSfK?cd3o-(}_b}|we&PWI&*zQ|JoIn? z+Y?$BntYimuwrgjAN!jTVnlf#Y7#W}-wv+QD^9F-RSZ6ki4>_waWPwKyY zR|8;SK52e7gw;#OQ*e6^yDe0D!3^EOet;EmnV z5oTju+AXCOFH@R<6ht5Hz~Q_!Nq3c@_d`#6KN!G7ol*>YpY|NGZMh6PD;nSk&xlIA z3I z#2YH<*RN|Brvi6Q9^ZKVoGXBkZX0PP`!eLI?6f5a!Bl^$Lfp5w01Al$34|k_&3rO+ z+iEJodGEVB0MVv;*?Go2_l~FIQ&;R7dGj^;>PN^e8S_1k#dv_w;7Imlx_4%rEB^%H zFf~>BZnN3LZ#<IUA^CuA0HYt-Q{$c2XAR|pyh+I+#u+YpoTo<6G{ zs}$M2{H>sCeb7x0ho|MlEC5>^M7BiCJo3K(G>5JI0MYSR<uVPG=Ats!E`~bt5NX5SW z@u7GX&pIElFAmUMxnImq%pQ$k*ehM4VdW=$F!`+&?{)K0LRO^a@xI(Ad>56HRS+lR zqy74Wo;0Zz? zjFVEijxqou(s)R0zCVeFEii7x*KtfgesB#a3r`hEJTlA^yTJ9^`9*Vh{MJfM(ZMs` z0ziRQib7UPuMBA+1R*DwJ?8Vkgh941GLFX$S6@j>HtO5~um1ftrmUM$L>Vw2k)2q7 zA;ij(nXFr{+&eW8_t1L~x2$+5Ac2O?8MKxmiKZ`g=L`@Iy%Fu=PRN6-x+(r~v#Ys3 zPkb_^Q6%>T3tO!SzDcCc`K(13BxjAz(D8IsgtP0HkD$VhKUp9H3n8R5%f8-|vyz~* zXhTVazGK-eR1DP7Rg*MFVJ!$*d;ffu@7%S3Es6^+No%_|kKo98d8_7PyUIY$yIrDt z5>_7+c}Us*WUIRsOgdE-O6PP61(+lyxM2O2Z#{L7q-2fJ%1geSkuuM>Gh9H5EHo-4 z-s{Dmc|d4?KRl95*eUuqDl0n%Zukl`3HKx2@N4GlMveVPB+jy!KmQe%mi7t9pQvbS z2ADLz<5$6v8!AwT1X^a{|D*#izh;2Yqs>;=;V2_a|Gd|c;ga`-qN!qJPO+_UAdI@~! z6=p=UOEh5ctp8u^ypiybi2REkwz(v%X&piwm?c3__Ug_6zVqzsDsqrTDh}}kX5>l8NB)MJ#NnM`2eZc%>)DS zydN$jrtRUoECJ?PGY2Ih3XU_zgZ= zm3l=eIAK`*t(3t2$jYi)IHU@W+jO8gJ^tRuk-8y%uT=L}4YP6L?#dIJS^Fk3eC)sT@&+vq-GQ;Kuc1QsiN?`L_|nGI1rQ?wx=DV@4SPUFA5w+LD6IIK;D=j3K1`o zao3DiKk@i9ZlBTb_8`D$Uk{AoEW+4z1IgY0V~;bQ>oDH%ZXDEVzA>6Xl}C_Tw081I zf04pkJ>L;lqrMaOc|OCrItrBNcD~8w{mF>o_}n6_X7f2XeB+F_J&^A0PYMAC`S%93 z2NdZ3B7!#;Y?94!z3$*NV5Nga=i9=S{SODZ${FNLN5R03jS47D6Z|lm54$}o?`Y9g z+0>LKh#`|$!#>QmOh=kn{7y|LIZ_=t=08w5G%j`2UO!gRUdYe?HCeYq>6s=UqKA!s zB2x=82S^_ifvziAKhud3*)QKoasL!w*j3>@t-T3WJdM@;<>a~JB1_8mQ`PW$Acmy& z9s^fcMAhW`A!HvFP#)*xYB8r12FQ$~SH*Ut*>7iYvM2sYx^TnHlNW&8js`!SJ_WxN zdcsg4)+{q%*{@N9vP<(r_FD?t0hPp?Ek4FilE7xM&Yv+&jD!Lcbo~kxtUOz)LK1en@3K;%J}&T#sSpwKgxqF}B~&9&p~?h1MN_S5Z}3$#34a<**w% z&ncXuXg-#5#S||g>Ec_)Ih*?Ad=e?P7yDtGbsf~4yfvRLO5Vu7hvbnb^aOKJ$ z&cT;f60A+^Be@6WSs^dC*=cCP0JN#NoLEfYDzqg4s3t0am@4G!KcebZ|3t6XOJ)z1 zC7RqwJZrI@i0W+Rb_~4(RfQh| zD9kXV@7gSCa^^CSVL{~ELGh<(e{p|%BMTGo7SQpQA&#yNcE8jg$4!@ILqLEG{4i}w zJ0$Vo92ZiUGQ3>?0LV-^6Z2S^vJbm=`W<7I@{nk8$VD|gukkdNu{%|C{g0N2ob4A! z^^0tWc2TD^Xx|2qW%)lSPT2|NP;)XW&NzAr0W{DMn%UCTQ}x%Z-jDu04!Ncg@Jl032G zhBfRGNTsXVHMTYN9Y?HX#7_)S5l^B|w0Ef|`>ySI2P6hmjNzt@+~dgzyo~wUnE;el z_qMt}8#xsKUR@IX^z;X(^)Ihk2+6w$Wn>S-&JzT&j@B51?BOG8@5G@{$Y`&XjeF^$ibpfZsY#Q~!|e|@ zYVUyHYm8|;@kMPPN@^h3{RYUZcYYLUifzRrzTE^XMPp*VP4c+G?laFx!*@{nyKwnR%{o&M-F{ZRoB7wGJ<&TwX;*^wZxXB@+_=_8n%4nQZtYnwps2OJK#C zP02EEBqg6>Y_Gz31Em=iJa*6Y(xCy=7eOJNuN2Y-{m<=NdMHNNF3irG-Hhzj)QS;+bt^jnbSsA<8 z7{!)r{$-u2fHd{b{rxW**+yO*=I>@q=j5tPZZX6yotZLnN%BGh(6ob!N-kZ`Jyr5X zb7GDP^t9$NZF=~X4Hm*3@P|Yy`iX~fT)+Q1Z%>f^5XqNBh6WLd|EWsP=uw!w_M;%p zLM2~g8J3ZE(B_#MKN~~^dH0T)IQ1`!N~_%wzKP2(cCY&datNUN(QWAi8t}-B>jFUDCHzF(?0+uFN4%{ViyypWjOp zCZ+g4oH=3`pw0IStdL=G5KPxs>3|HFt3@WpbYFuS{23w;4s%WObf15Qm@Bwme-tyH%p<`--P5=t`s`I4m?f6yrHd zG9eGJ(<{W0ZS>aKJyV5-YVW&+9#*(e3UvRoMLw)*A$ z$GCGdK)os}b%Nr}E6)^VL)}uV8Dw^5amh-695+I-+9@SaJd*B;UEbe)uP1(QDuLtj z9i8q-i_vBT$8N^qY4?F5p$4$ofX>h-?0lg}o2jP2hzKj?4*>=62U7ch)3P%B_NHg7 zHaq-zHfXX>F{{Xtc-!5I`K`C<&X%h6g9ETyh*nnPvosWnzwq@5P0Q_FPY+A382{5ut+M?5p$|QkAfkihi#@%&ohToM%ae+`qjY7RFA4WnHmvE> z7xzAaG>tYxJn@C^0f%TtYqGF=6BU1f$xS{**2?;K@3TGPmD=<#c#vBfSmdQ!TgyBS zGXijN14xhZ))Wuo`_Hb=<1i9Vs7!~r!0`LP=mEsnCw!)ttz#LZG#@3}S9b5`Xdy1hOQ@AS&j-FbkBX^5)#)f=1D%4BNl%>aa{V9y6W{Wc5`dkK!34G)GzH-&Ik(WSg> z>JMHTGzh@Hs%$a+=iUZ} z0J{MrBeS!mqj+u8UIQ~b;Eu9EXkyl_{qO+t9}3H>2kXJmQ(6Wc#}7hv_q#sQ$x;DG zm>})!~dq22HXSH_IPe&A9*N^2uArUA!a3Dt6+r z&*af)15MpP$F^jdD*=P>2KnZ~v<=pexFkDjz?psn&rZu7Tl`^uwrD{JNI^TSn?(k&z*58#<8Am5<2=G27q*sSEG zu3_=0W{{w$c3 zMT{s02B$}6>318RUt{pwRwD4%`K%T^Y$}5t;o0A+0?zqg)^|TL+qrqje!r7L($WrQ4q_nXWM5wMRa!()WJoMcH4HSGI!-7yUagqY_@a zDf?)`sg(UUpC-bi_6(+lOysT|?yqfh8o%cr`HAni3Lt>EJ5|1-X&N@r1^8ufFw-+< z0mCkge9X-`AR|pgT9X)vq7-y?T512vX``EAy^CiO8xy+;`qS2CPKMayD(|w6Jcs@* zx#`~S3?bQIOnL_{c$iCLiVjkcNzlJM{+n~LCfm`jO>#OWMn`66U_lWnEI5$&!2#t+OjV)HA3L3 z_y-Ca5xAsouNXxa+Tfn;=l#6xI?ph>SYx%CTt#KMC5;gN9fC;vJSkIMxwb~h@)=5U zL;xOVok5V9d7+^M^-?8B^1ldf31Ss z7$vnGI`k+FJAG+0v`NAN*^hWPnMkrmOBYyrsyNC(g9*F^K zxT49D?o=1fCcxSdOwi}W%V|Fr$Xln|F~jG$(TwTqbd4_j{&`9-SpZh{Q|YxAu-|Pp z0$`@ofO`j^o)FhD#$7a{It)EB=W?zS6tpQrNXtcA0^5muTE`)~r|U^}B~9-PpXQ3d zFx8aYgNw+?3j$(tI-33mlrC*3rG}q>Djo=U&f(hXj$@?rraP_Mz`B4Mv*|bcbPLkW zWpWI_#gyp%eC68o{sH~ZJYW%ikIvt18P5gGr!;Di{eU_HZU}#o1qf-y^dCNq?Bol0 z)4ZvX&Js6hGs4sNK2jL;U3;6Vm%pZ*zPqO2?X&&=v&F!nM8uX@%kClK%W4ip0_mmgTZ+KHELGssz=Z3kRMCyB6ZALbPn$vH zw5%4DvH=@@)xo(>PCL7=hBYk-;;cd%+p1vzquAov_nDrPe?`*zR1(@zfq|8ej)|F% zeZZ^i&i6C>4VeWpfL?{eza_T;tqDw*u-;-|HIs9A=G*>iPpvah%I6lF zrlm*H_;d}L7H=I&{&C!9<(Xm=F7Oa`pCsUSI&J?!u{$kXmRp)!A9^2m5FIK~E?slQ zl|PcR_{;Z(s-%PcX#3TW4Y{iCMO5PVr@KwICht_3x*!u!@gf%%HnQboNT_%I9H(E0 zqOd|rbXtE`Rh7-n%-iBl#1sz45Bh@!ELM zxq*ek&g%E%G8Q=z>o+Y-CenJATb>qc!4BUY9#MaJos-f!yM;JJkn(L+hB;Ia*~t6( z(@O*3hcaj`kAl-q(Ngb2BP}$ll_r9ayF&I&9M-_!zHq0-#d~o{wk{qc|AcM zC}`h_iA1et%d)ewz7Y3O&jCDTKmTbSuUcSQQy}dYI=6w0rc#Ev!;g=;f5>DY|9c0K(wyOvS$YT|a({Ich;ufqf_3%KUfJ~LZh zT4=m?`}aT_UCWbzEq^j+a+(E&FR{_E6o32pi33YF>eJ58FSel(M(dNJMA#>-aszQ3 z*6&Q~yJ{kXWQ-sKGm%xChRA4FZ7ink<4L0~$6`3I@k1z#AtIZ6$)Y>gHbo)9Ik9Yy z{}_wh0c57frm%N0xTNeQV%ViXPVGKA^h+>SsX!7k3>3%8EDlj})hxw##Y@$js+Hcn z8>J{Jw%VQAlL!rQ#sK}n4kM;R~^;e_42?+6M z$GkEQ=WAg>An-S;hQD^W;5=`L#!9*03u@d>@h(Zexw#v-m*J9{av2F~zaZJaCk@A6 zMF3g{P(3q1cIn!D(Lo2iwmBO=zZBI%yKPS9zBqU8mGiqNse0aI>#^`8Fu#SvYbaWJ zp`j%+HhUW#@XLlAiBR$}0$^g0?GM}SPd-ugXeO2_SCBJJSgz1j0|22a)IH z1mJG-MD_`ecAEV~d}%Wy%@Fuis*q8_*ezy!&e~fP9uuuVfd7gPI>rO(Kt=Qo8^2+j zn(y@QwY&k+mpWx@LxmU4gq@YkzSUaSxq zCb%wR?gX3yd5ZZu^AJ*A!rNyq5d~}^S%vIuAu;;W$Du4NJrtP3W;sG$oN>!)(~Aim zCur?0)&w>!F2&aBbsUTE(D!1l<-b>K0O#DyTJWzR?x=^VLz2pBsvlBb|D^&HcNHc% z30xfvTwFsueUtxCvx$CPt|RgQ+t?M@3krG$dk`6CK7*z`>FEV zE#n)0qXf;J0YqCyBwJE_a203OzJ&T6!rO!>)iO4J-edKEYJV&N&tz}>Nxb*-A&Oeq zX!1pF1CARjogwz@*6wwd2`mg;*M=a*cP75K4toVN9^t+bEhJo9n%L5CZ zDbq5F7wL4z^qgJAwu6XW9PQmp1Y(S<`O@=zPT$sWWsgod2|unXj&5Y;<2sM|W@f+9 zlaGtxKGTV>B=V8Pf>S9yjNGjXs9%pnT~<`g&AFYr|2@Ua%`=_C1A|pSLE~%G{3Nsf z)oDxV2gtd?0p1_OF5ZJ48P59Dx-U*Jq>ge&r1^ z3|QdaAN(Hd9}$Q1UhU*~&QRc*Pa^ak;0ekTzsFIn#f|}zs{gA)^3eFdZ$Tvcn4R&! zTXW7`^L~kvfp=JbT{B0Edpahjo?Z_LzyL?JXA)V~{l;q$7K@ob6LUJgg)YY>Fcie4 zHx`YNpIw5CMGI+BG`%1ik2P1omRvM&OsCl1nwY2RVkqrm0#=jho3TF{#FyTIx66an zAs-!oXI6pK|AVjR@)o+S%wA^T33TEF;z$2ZljN`;()g*)3xDCf0#qr)D_=pdQyE|d zLCQ}x)i-uhL_nL#;v~~UEdlAkt2O_a`+t04zz#>x%2-#Mc1?Jo!|p0Py;S^LO_AqA z*KN?aQ5q#>`INv8lTyY1st@4)Y+ut6@;a}9HmTqs6^1qVfbxG0eSTx*;&9d}dH0>? zocFKcs{Kha#O(FaN@ZW2CuJ52piosefHdw8m7XwMyFA{8KU`m&e51MIbrtsYoVx9H zj50$sR`0yIg5&-F@>>1#N|rw?64$+t2|rt2T6JK5{7<2MkC+Oh?N#ouXs%kY#<{~;RDyKMaRX-fA}!!p6?HI#Vp*N z#uGQdW$!4QET>JHD2flZ)oT_7CS(;oRH{T`l;PP7EX0G(^-!O0=(zdSYNXTg(hA?TSXUjtZQAwq9-tyt@{=&1OEDGoNsA+Lr|ypQLjK z2Que6j`t1!t3ZGT{~q)p9h0@Lz40Exj7LPY{q(io`;3a2`e)sdg&ur;qSi|NVaG7; z{7-Wn8$J;RU45;CH+dIBP9uN+!l!E~DE$ZO_YU26zt-5(za5~1uwChs1xZxxX_FPN zDdh`~-a>*+V5Fv|th_g4Q#)D1y0v(wGg1#4J1?`MgmJdBuqyRJw9H)=HR+$T#|qvv z^#SLH?^<-sE4XoDC=?yvc?lP+9*49K)rX7fk_4a{uj2t!yq|tuv_#;2;8)oS^y@tz zzgH(%ldDb?k^we;uEaGat?H7F%72rq?Z)Z_v{jY>DJ%*pz>cwzX74;Vfx@9UnZ`VO!Em#-<%SO%)( z%>VBNY(92vja85!%=sHU)6I4h9n`>}uBoX8fV*4VQHD668IBMvkpQG1b*)#JMGs&3 z$jY7F2gh>9$AboSiZ9-xplyB^fr@N=Vt~}Rhn$;z7n@GhJY?mGDnN%Z14}nDXyvHM zfw1EH$ha*3qMOe|M6cj41~l;2aV1~9Vxr%Bjf_U2Y8-@u=81wfO!|z#`G^LIYA%L< zN(Qn;sd9qmi{4Eaz!Yks*^15iAfRwZ%zEwycYm=MXCz%9%GQK`S?U|_VdnZh&}9DK zS?x!JbZL0`D%yv3To0AL{nMs!MmJYTB;B@Yu4!i?Htn$A+TvCCqcUpdmnC|8#k%8) zlN6GXaR5lX>wiYM;6VPZ%gY0x0oRyqzp2F|>uBs8Z)c!X#5+QzmUX(-I`OX$sv_t6 zifp6Ryi)$IT^11f;tWknZkx`@8?|a~yo-!=4@2TIV_g@Qf^Q3y1szS{*%bzMlOheKG!`om@6Y z^!_5?uQzwd-R(LJFw-_cr02as(j^SwQrlxfiXu{Yy*Db=c-3IPTl6B2X)W;x%YdjK z!#c5{T>}w=X1ybM@M`^!lV@JalPX0>2s$c;haK549w7lWcnL7U02C9_Xvf0NlDp`4 z)$tse>853_xrjV*LyU-|@FhX0g|#iey;z*7X6h1cE_!4CK5gzR6_w;X77edAE*0@P zdq14W)|&;#a35{&zo(l(4@ox!-+i7|7woVzO?u3|{Gv1xpe1N2-A2IXwc36c-W`zH zXPscZoPbkkSmrBYFj$ltu$sjOC~_I` zZf|;2OUlswd|;l~f0{+!+_c8d5EqNVQa(wHDT*rpI+8567nKIohy;zumbWTm*Jm`J zC0NBf4o_-Jvva#jb#ZX_o_&!HOCEnA`j_^c!~-|WxU9wFODPT$*Y2(&U;?2&GDO3; zdC?{ij%R^fKI$t|*=!Cv$tmhByQc9S85*Qz-I;Al2!>FLJG9=Rx14q*w=ZNk0$I{W z6b!R53FQw%&t2^f&2CLIrudbcohDGXbeOY=y2uWwCM6$uy6qE&_FzXx;e>U|Trq|>d z*?x!R7TBg^ra(}V1%`aeQ!u&SyEbA%>IwXJNd`3o1J=LeETXQ@NL-Tw^cyzEjC0RWaTRou zolbiHl&HraR3*O(Fcv36*5Wg%E>;pmS1)TRg~tt(%130atPNkvto~qjEL&*uO0URm zbf1j^(>t29>rTC`aViPgM-IHzbjGW7hJMc{42j3yr&@ded!B%a6}H1EBLCUdC4-$Y7og^i!nf9=RRQ)eokjMZfHtb7Bz>xFgLYHVW9WB8g8) z+CQox4&pkHUzW;kr{&~czcxoA@c=CZ02io+SQUL%vVIY0wfUx@r$3Ue*76O_QKia% z)O@V`P-Q+z#OLx814rgL51X8#-TL#7b7B6|+~EH~5uce0MvShcDK_}KC!mHLdpW)< zvh;?k6GQ^6swLq+7{&z5{mjgO;7do}as@_y+HW?;!N0|XXqZ4P4rryb6AR2ZvGiSf zj;WJ#L_Q2Oipfz49iks>C_V{zoo<+aU;1ziFIc3N3spMUrfvreCGfi!yk&)`smnJb zih;!lmji0ZP`T_(y)jXILIR~OCQnVx$?BP1V~Yt%Zp&y8b|A_Q(755??**@|5qaOd z@Li{ehYlK(89YEf95wmMiJ7+a;n0r^&h5Es<#}G>wlQB9)N%}IVc3ZT^A0ScXG0Cn z>$do1U#yVE8<=u0@CswrSo$!{t)D>k`bOpf)>T{_e^>mH1VABx9+XxWvlkwPKr|A% zdeFk3{9>_&AjJk%m&=yQLC|xfx8VFll)VHfB`N`FpfB$cL$Ib8N5(6OqGbmlCboTD~mwVnTS-<#*#m&pjW@!I%dTrF(epmZ*l zKX)LRc3xb}CKxWOl(eLR#jQDeJdRKV-iwM%QR!A=ca)>B|CLINDd8P5Q7YB-jvir% zpa*k&;8W%jh1ux zYp?B+I&08PRePP=3l_JxNbKNi2bi)GK%I^^`^!~0KsYB9i~#Zr(CsSOIt`P$AGQbF z>1i(<@3I`n*A<+6-`^B7j;(MEa`w|Z>wltLo2c>V(QMnBIW&w=JW#I^P*yLMzZ=(a zl<2SBeO)W?Ph)@s*_(VZaOroAKiY%LF0iz`hSI`nP(1N7{O)fO$3TxnWz&~KW z%RfNhKj57yAvO&OwF2pYehlx_Q$so5t*`W3xb0g3NqpB=r`y%5`;23CGKScPRR&oc zQ4}I!{0?*C(^^g&*A9M|h?!pmSC=7%KN}!sJ-WBAUd>lJexGuyc$s`fG~?-tp9#Y8 zLhr&h$`+38$hZbv?d{X7nx3dC$(RN3Z|{28z%-}+(D;6y6#SnEk!bZSRmf%M0_*ap zfI3Wh4?KFpx>6EZ5>{5IB?c|he8df6ErKw2dU)cCaTm*T1G&w))FWCL=DMcTvoR8? zxN#C_wKWn+2ks%5xt_}*+yhEEZk}2Bqdcy+o2!q#I@^1*-;7tW?6WmiSXJ$|=n;($ zfyzGDD$SX#bHGqQ$U?biKnx<(d^LJ$Nf`gg=R3eKdijnc6quqfVOb*l=oB+l0igukmif5U;^(;=nN9?*LGC{-UWJ&2$JDLi3LF{e?pBJtS^QL|I~0D zdj90P!K?gbNA4GdN5dm7?bRjOd9hP{^+>Yw9`qRjSCGe1b*|HVwse1Hs9xRQopPos zHNOH@Vkb6a6E0x}I(*09WtF=ev^SUF`UI|Hw7IEkESyKG7Ev41STJgQ=e;XEV z_zVq0c)7-@w)(|Wcwm^~VsHF+76yhY$F!@gipQxDMGP`5xiXk*?F~Ug8P1$!St$MU zM_fhMlA54(J+SN_G6~_V#>ntjbFXTv)}i&6kY)&G)Sr z*9+HULOBArr|ao1E7Ami@xDs19&M&5-^vIW^#!C@@GS((rYIpr8|lfu#(A-?$B?4YQXEH1=F9L|*`P zLc>4@u%VLIYIlDq=%di_xOkj(r;kWGDrBKVGJV~cGnw5o?D+kn{i~xq0lDfIRjc_e z*J{9#yd0}j8uYD%nCS2ey9SUM1A*j){r_s9ALSxDHqjg}3!2KQ^@GUwDQ6qQox_*4 z!MGTIzG)kt95rudZZ;gjqVx11-@d4J<5;ORWBiXnkrTs(R=3K&1yBWZq_mM$FB4OQ zR@+GFhHUqgiUjd!P{aH}Z?tc2O_HMNL^%y1P15#Rx**EO+&tH|jIh&x8um5JGjbE2 z>H`(ZQ|EtsPH^QpJgKhU^NT+HUkJEAE!+J~(3@&taf;1(PM~VfAOC7Z2Q-z6+Ws3> zhK+G!vb~vBY=O?s0*Au6r!?|{2rC7D6a!M6b+Yz=V-_EMJJc=WR zuJV;X%Y1lhcvn+!JOf~3dHlkua>uHAM}zoAEmI(#?jwNC35f@x1)cOow+#VbgrAp{ zlgJ^Z7%ryEtjln#?Z0+BDHP7>5+8bg4P^F5$WLF+H~28IUtSwto#q>EZp4}z2t7Ri zzCW6Wm)v=WA2+LN(IN_Xb5CK|?(|o`WYU$jtzfrq8=@>@w0}ygtJSd?3EsuVQ$v!8 z73KQOrrW1qafB;u3xt_oOGWKdiMYc~NhaUoSL_)a73MpAKD-2UKHL4`%KxTUZGhkq z)yAxA_ujBy?1TPwkmTbDLBvod1~cJ-65R%z}*ZkH410LA2{h{Ux;3 z*`y>p--ZEw*`EZCFDgWC0p89Cm(zDPh42@Vr4i&Z)4K*a?V3-((@q*TIf508!j@w0 zCeuLt$DeQA63CJS2gp=o?|LcK?QH^4G_y1VA+{D4&my6MVeppOPYZr!ysp7Rv{3p> zEi#dGxCg7bnZ?w-8R!oU9s(1!Y~Y#*4R!j#MiTmt`Gq6~JFH+(S+N&H=hpbfvZ>Wk4h!G)$~2!LOg8cJ4lobi==k)- z+!AjXwIgsel@=c1F1LoK^e!EU)co_}Du1Fv8ZJY=BVl1}nF_f#=WO2`@eVOX<&}sfG-S_=wURP0r z*z(F@@w>wV$K;JcS_-jR$A_Vet_tHcdT$c3&?ea=$t6ETknyO7{S1&wT3HC#H=n+I zS)5xcIGaR}q-OfhFElW1-k*Ea1pv@kM+eIca!P|aYtJ=@9}6GGuZ(TOb-)CFi3NB$ zoSK-dY#zjB^YFb$gw3f8TM764%FK!u&FoLOm$`hl{sc(Whmj4%<(krWIF20ZtFLugZRevnYP@1UcPr)ywO=wtC{?q`STtZA=$LNc;uKr$Itvl3u?0hYi9TW|fuG!1D?cbqh4DmWq(cs1DR&bxlA3W#vVVv^r2!~R7* z=Ej9b#T6Hf-!*j*o0TmjZlOXm=YV;-J#Gm7u491nq;&hefSC~u>vmS^^1y$e1ynVJ z`&BS}MB5)lR^fFnx%3{|^(^x~?&KXvQ`nrFc0Mwg(f)yT|9-fxlB#-!vk7-C3?5#q z`Hk3(Q0mFtQ5Ql|Qh7;aCefoVu`A@?_`lNAa(#8AXxs*XjIvJ(iKH7TE6=xR}VjEb(dQ-Z{~PM}ffL5D0#S zWXZ5(a#5h(P~oF(OEF0clZGQXN`%YxGW>$=^@K z{cQNLn)PQXlj*)81rH*+rQ;;ag%W4Vsa&k+#x@j7!;F+u_3X4Y8EdKyqDRMskE<*O zK6fbWc&dUr8`RnWc6;xispWTjC;$d3_hY>zlU?l|;Tv5wl(jYN3lErt59R42X?Vq? zDYSQWo%t~bxZ9PGe7*fu{}^5bd=^0<#LoWq88%)BZx9{H7E76*f53$v%w#r5%}rvC z;5TZ?@=s5Pg`C+p393$$dvViw*?`|KFVyNPg~rIQjRhf7G`=f&0_A9X-29d`MtD7B zpHwZ{{-yZ19XZD)iz22kEk@IIgT~=#A>@GK^=pO6d{tau zugtnRfR3J4kmW6;1L=>{U?;mNGUG|sEK z{wn-`3{)OR)A>r*5T|xpm}D2$dQs+(lQ|%&VQWOk{W|IG?e%BkNksA&?TwVodu^Ju ziMcfFzEK|o+*21iIP@X!*s1kso=O6T ze&O-890KsmxkxsLf<%%|KkX1tW21(o2!|{!fvh`aelk_#D0?dg+$c2`Im8}kX6!6+|v5t zussBF2N+%{&w!CV2WIW&?$iqr^5cG`WS&&uurT{s=|6@B0PSxcvVYGv=nldmwf=h& zElH(u+V`%_-yF68(zb>RzvNTA+@U$AB(sRQxQ=ei{j!kgUtSWdM|`wEf7T|0>BG0n z0C(OhzV#SPe1-*ned4DrvsLF0q@=q6hAYgY2-7?TD9mL}UDOh-FXmR%byBp?_!V>iw)rt7oa>;*cX z)na0OAqhQ#iQHu4dBA8zqF@XU>y@Ff!!n{w zuO|BvcqK0}&dKJCk;~IhK$^ChNosdARX+KsNLHFjx>x%lUOjc9UsQ)X6-iq1f)WtAEc7 zPU#ZYC$x}!+X4n8Ayn8fdIKcgH>L*T4|lK@pO=&tp8YT!e%MC!bdSn7lkkZ8icdvi z@n_A|rLfA~t(liY&7i{X7_!HHDh{45yRQMEDH9lDp}+j}aG*`hOnyclTzvZCqE$>uwGj*-eyFDVrUs;upC-cOh#-RWgtWxK z?jc95*iaV}qGQY41=q1&2ch@mWvz;ntydEcNU;R8JAr%XmW`w&>gmpc8YCH*O`@S| zPwtILwLR&cY*4e@n*T1qez|TqdSU!;i(Vt@=aw}*<{FFR5(na zF8?B#bHQ-$0(;GeJKj(?(`z>tO`Do zWLkipk)3#QSlI*k=~q6i9r+9RO0#rciVx9O_Dr;XFXreIct zZmXcaAn=dr*WOKj4WHV5Vbs18*N_2r1JkeJ|6*`TtG)u>21DI5s@??+(N z@{s|WR+|~cj1)U@r4U05m5jX|)=Ct_>tLlV4f70Z=D=HhKkOgSm~dfGci7~Z-90+Y z&`pucIZTk)x!2x+^G|rBVmRDYxKg=ro@Z9TTW`-zIdT+8B7%Qu`c3rT?HN*$Pf#;0 zjg3ky^AGUe!ug_@{FpkX&-gseC(vSQ3_D`hs#+Dyn*2>ww%91tCJp!Ei$L|}|v>F?pro8KG zPFEZRCE^r9)op*lMsm3x3nERW={m#4X9ZFx+=;gYeJNkt1g(VMFh$ODe^@Gaz8=Vb z26NvkNU~-F)2hmIDa6KnO>{U2>|#&!M~S+qK#T!)$YxVn8*kUN$zlsB_6`rslSc6d z_?yq}xA1;I;YzIqQOP`Yy&#Im%yPCAP6Y%9H2tR!cn!Q^cM@=?4b1s6Yucj0-HYE- zbg=paLp5UHgT+0WxZjq*ED@Kx&3$1~Z|?)H7qJgj&Jvd<7Ucc-Yhbx$mWC z0Jd@zoZ*_&0M-m0Ha7O=x#4|GM^`!iLiP1P54xRZpF989TC|0?{AAf8*O~Z|Pdp}e zeTY8H?_(?Rq^;;Q$EOwSCC}4w@0PQ2;nKHrZ-%rA#6|U&wog0x8~|Qnk|?wdET!~Y zECRUIf~JR$7NCmBh#rvFHWZAPy!rPxwkb~J2Ur@#FxKF`JHNg!2nC-l@}WZh(9k}R zL0CNpv?w7k^A$FlTgng%z~x2~&l~aF3gV7hM3KLce*bj5VeH8RRYXsIm4wX5+F4mo zCHVGe`Ul+gy9x>hKJuW1`b$u$W{9>*NC{dp}cXD#Ix!K@7;gTTDPWLmE1;7q5yzIE6) zbHQ%55?FcNzk+8sQg1h|+zCtt@76o?-589c941XEh57+eALF{4Un6jQ8nji=Kp-H6 z09Tv8J)7`nh4GEVV9)3AkClG&09Lirts?r!WgrBrHeb=C5OD`D5w5ZPzS_J`Z3iNS zKp;k_eNG@jqy73FAu*!t^&5rk{npDU^iwK58Ndnq9;B@_F?9Zsa%S6LOzq?&U>wFn z_C!}dS0N|2!>t#gRwH zg&7j-@^(;_PCg9th_m<(6QgQlseC=QEt5*$b5ld^&Wv9pH|NJJ(VC^o2eEqOA9Mp9 zRq2WfEc;=+;08>{K;usHlQCs%YeJPk7Vh)Cc_la8(`N1pFxrQ#xwH{BQNR^_8c$s> zakne<5mWmSi1O^Jhe+eG@?oc5j%&0K``|Cz)73*6zu%GDrJ-UZMCIVo!gm2FZRGX2 z_g+QWncJDNoU#FK?)XMA+&$`2cWp3@q2$+rXx`rUHb9oYM-H3 zH1O(V>&-p?NOhtGm)cK2Zd?=b@WY#MNK0`ZGsS%#pu9*dMnm-LV?uXX#&AbhsE_9E z=jPy-&@a@Rj$@tID;>^2YJAA%H5x8C|o%2w0efWXqW-ZeTN5TsIydsSp1>p?(FR^||6=Glu< z@Swu|M`qf@JEc7?=wTac@f${Q^T&29>zDj4n=R*suXgqWMpRu}DL%=x0~5N}`ONZ{ zThCxUNvx8o0WTAI04keJnZ#-c%k+`B8crGfJ9POI=j0;uCDsD4U})a#Y5LYc zrF@zJw13vaH_yKEIYj}dzZh&7`@8#7UoRWj1VD2hxqcIthJ|`%mYJ+paln3fzBET> z(u$^oL~}Qa&}mkGy{-aX&;)Q;=38OA7e#Lj(ulYuihR4{KHdA|oKrU4I4PYm89vQ6 zE4I@5Vf)?B7(y_`6P#}(xv&+JKmLaj$ZL%2!+qefCV#$;K3@A*|b7)Kcy7vk{O0!`~`CA+O;en;L;7`1_KK>F{ z_Y373h#!aQ;XL02+^Ajmv5m#n9Aeu2>G)5$u<%|sp2qn#x$ZqnNPn?y+UU2Kx9OX+ z6!RuOBnlx3v-)Dr(3_#I_^}Y+3OI@tNLw&Vw^F(ArtGyE|E3y(o7ofsp;Ie_8%Cl& zmZlaypph~u2&97<>Buv}3ks;8gNBN_dWwLY1dPUGiD}Yb{mNR|QNWnAk6S8lidtHF zmzP{g8hnb%Yn|hu_@(AG(ip}RS^Y?9f>Xa<5&!{NBJf{Hvm(>em!rYEw2rG0*1Vm! zD-k7=C|j3nV}1W}1b;Oy$c)i>=;29bp?0`|IC*@o|ED-<#eDlNG;C~4i z2gnX7@^hDIB_TZ5MnhS@E006YuCK`=xs18Qq(n@6hbv}HQ$wycb(#H$-RwJ%;mXaZ zFIU-ow`-M<^!J}Qlh`FOLnkjUSIr**qlE<`$!%oS5%&WZdT^_Y=eDDNOPppDgbtZu z87dyC;m9i&=ifSVPpC#C)@M~S5y8tR;t1x7#JiE+e}n=NGQf7^CT${Vt0X}R*0g3=bsbz;0b6Zwv!AFkO3c0>-LUE##Iw!V%572n0j-)FF1<(M{> z!FWRT=T*r+zbM-wE&x%-`}lX<1vtqS0aKu=xx6y;e3R7!m1^UbTi`9BsLx)vS(Zr@ zxHA>W2zyW+Zr*85%&^5n&o=bBo5+mh+hRGx`uEJLki$v>o31GGH((-C^9Xp!R?Rq%0@6ia`8xE9fW9;! z?TF%Ly{xNm*=Wx+P57g*An)4Uv<7~Eunpy@$v-BN+qr@1cLkbFfjK`~sb#;r8?6(= z?%runms#uc+pf815+A7H2r~ARx)=~$C>~D|PaQ%sJIulR``@6C%St%L!Fcydff$1iZ{@ym}Qn%^Y5}0S}GOkR6JU7i-<&{)DTs!3lD$8 z%x|A0VKn4V*8fgNo&{cyi8phQ%&b%A+X9Q|f%-SbJ=MQ7xDo&Q#kI0jt?Q*;hN^Bo<7ahQJPaV^H)y zXE?hn)W#_>Taiw0swT?A0U_&mv&_WG!OCBIVL?F#pkNK74o@U* zhsFOoq7qisHgaW|%G-Ll6<8JFXiM-dJKMz8=G+HN7Sr!;uMH{KXZ(YErwu9TXDX+w zEPxASLxsk8+SG8~RL}yc+d7qx7p7QJQWDP}iQUitcP26L1uJtl9SU!C-*|h;?x9@!mqAstVp4*b&aeTzM1uiUg!?I8VMo9oL8&wWo#^BmwcOaDBhio48TsJE-eaN+CEP>TU=rQN5b3e6_lwWIboK?D8q<`o~>>Ip3qSv=!@| z>xxr?vYKB6))gtlq*st91FM)6yiUa_UiXvfe%Ze%>w1nlYJ%w;>i7KdoZ44I&(8g= zm{1yOe+O^x%T2`Re|G4?4pn{IE-2_XFntI6U1vIBM;1j0iB3JmS8^bXq&pwBvf17% za|m9vG&h_6e2j97uIHA(q35!fFr{>E$1mh8gsy+5^gN3|g*PZD$Yoe>;Ep2dx3_nf z*O;LsHlH-*3VYg~ep(h9VpnX3gM}k=MMhD1L)4hhrpwl&?S~J>Bgyo%P{n{(JI&qo zi*B^k3|VL$y}`I-^bCE^lv3pX@u2~|N;+vXi6?8>I`mTh_@DQe;#!HW%S8`&Skxyi zB_H;*8_!CvZ~_-bk)K1Q^Im#+`B-o(=1zq_XPcvOJU)$x_lCLN%LFUdLQtcpTya_c zPy=JC!wf207F8>xSYoPw06JCq=BR}%QVa(Nbdm9589(;BBFSay9K1;LKCyev*^#YTJZyWv7;Nwcw4QH=gE1PbP z&S==NY;mu5FUPkchyTQUKd{g$RS zttI|h^u3FG;@$AULNS>o+H@rF^}q_H&?gbH^M@b_R>~qhUDQqF_GpEZn39Sq|67p6 zN2YI+WrhhI?Fw{WWPJRCW(meevi{LpoKKl?$T);3l5N8sJ3c*`Grcx1H|Lik zU0UUGA#d6i!wywy^IVD^>to36c;)w_!pFY%)muJ3!cP>T^id%Ez2baoJp~}l1l_UU zKC?!1f?=4k^ryHtEJrMF?cbN~5SYT1Bv&$cui0F$N~|b$rPUmo%KheAc_OHmS_4Af zO?LWVe0oN=VPPJ2tyomQA1d;kp7hj*X~iknHeu~${tfGVQ8Ob$Bw3dhKzIIR826B* z4#j_;82LOTyOxoJis6uko{WYeDeyxq9KiS0)hAsV1Rj-ZS#R96lb!61U94_7-W zNm+MSxK)p#F>S`DGc_)!Gvi%^gJgMxKhzjtvyn5>0`wf+q#Nepi^sY5xE)N1#y6b_ z81x$wbbr?0w(N5${j0+s0(oe*-Ryffl#d<8hV+720>}Vn`m9@?a)OX0rPk*ff22bi zzl9_+34vS2sV}$gb&}I(CseQ+%eDDR!q$qm9i+Rsuemlh9z54Y#Rz=rcR%k265Qd# ztHse4DhjTyMp5Mvh{`A9$dBo|y+%Ur675#e6DT;vMItsh3BX;ByW08OzVSNlNPw$P)mKEEY8=h z+ezgq5gX!`j+0P*tbe!T+<0+h6UUTd9I|onDDCGY2^4CA^7Z4AQSWk0K3WLcM;ypT z{rca(BL%Sxg}Pe(;wpe7!;JoY(3Ltw3b%|C);w`y(@J4X+#4dA< zgp6D|R~mIrPS`0nMoa9MTchI3%VSck4o_hz0gyCqMTlpD!e+n8)62=WO7IFS0sK7- z$CFp{2jP(q(rigcNJhWaP%^X0!>pGpdR2I#yb(5TuDyU8aiA+qx6tC1ju@k8P-*`H zk09*h%s5)S%cyc_c|&d|q^j~?Ib&Q&wskNDNS&2QUz@M6J67L*Gn} zRJd9l2LTmeXcJHZGZ(H<2myQR9k-wR)<1+GmjzIZ^}P_@h~ujI0Sc`48o^At;GZn?ZjGFS(E6e)jn@W{VH8Z-g&q8_ht6&DuTaH7msQZj>V~pa8HND9x1L+o z#J_Mm98M5nJq2WN_XAPSDb%VHUipl2-T^M$4!#!`3qLP_Q}%CE`Dl-S^wv-MMNmHc zs3ac1ta(B;b`EcUXuOM3+^ zw}0gSrxOwT`^|5$1U zwqIYTJ~@}qAQRO$t*Pz;MwfNutWeh}E)`MpBQ>_@HdQNS^kw>I{AnY+Je|u22<-pP zY_Rn^02a%CI5yN)m+#XmJ`+a_Zi^YT3i zrynb^!QYz64Y~RL6WuO2W=KKXt?D#~x*Xmf@EGbDX8wfJfUQ>>cJ)Y$4x{XUOMi`c z1;JnhPMt9pp%7}+kgW9?W$Eo&m$S&p%`U?>A`MNxcUiJN#TVhzn$S}Ls0y}Y7K9Zk z2KioIFv^pKbn?_GEsFV^0=u(M%EvoQqvg* z^Q^ip+0OP)6D9$CEHD>k{|@I}vk}>lCB%BE^;&CeY^)ejmqN@h>L&U|};Q%JIDhUmP z&E^RuSB?n?V}#EC0EheEp8_%XC=!C;_q{o^urSS0(?($ukg}$aM2`2bEXFT=VEPRN zAJrZ>FTJPSN3q-4HA)LF+RnIDPG5Q^Cdyihd#cHWBrt;+UKXW}Yfcs^9J5GlkL%jvz-NYDu;BSw2W8C2N=S07+J?NS{N*gDvfAA-A2KKvpO^G{VubL?e*}d zm8?g`%t7Pj5TOc=8vr+r21n{fc&_t}NwK6gjB8Z=^0dLk>phrdcBJ~LYB51lRHM)L z-{yl)4RA@dlKxS~%EFRu-lz4tTS0^+xawJduSx(rH*VQ@<<@59C@ri&4}yl4s8e2g z{C85#!|=xy9UL?Nnu7$3-OTW>b0M|gy(E(7d;D9< z*u3~OB6$)0wi3*|lP^0WJa-J-YQWBG>}#aze=ASfTTtsz|9?l{6WsOwf}%!{KDgtI z2|4=2-T(Xb=-K_O5Pn-hCA;0d5U;>Vi%(mFkLBO98w>mA;5T?6)h`$_F~M&`2`@_R zPejADd*@smfEB^~NREJ*3xhxm_4M{^W<5ntoh0Zhw(COoVr5vb>$~*w^s2L&Ol*wJ zpSz+6%)zvl7Pb&++2GCq4EJUew9314!}_Di`yr{P?3)DIu;%FJe)mlF3Mu{x8oy~& z>BQGYa^Gys*3m#!h7r{yNZQI*9534!5!lcD&0wL3P*tjX>Upn>@*y8-k=1CHb}`o} zCB|nA=VkWGMp8j$f*YA6eiPI9ljEm1qOBp$_vf7I-U2k7)N?2l3nph_SH5YVdtVaE z&U_o+%b}vMbBw(ynaOjQt!WqNKvJN)Y#b1lozn#ph-1kXNfvNusZmVX_E{AdpL8zh?hC`7OS;D6xj(;;xlgzc zq5QmQ1j%K-&IVfhDy!A;HowqhB15~cVMRr(wlmdOsqpvWF0)p!mO!xxh^Z#_Y2D%u zItdc!?6Zbz?Q~r>h8_ZKh8syVE;&0CJ2vTQJJM7(=nbskA^^7}rXO96-}{12&O%-y zK>p*~dwKR*H?{SQ%*=;crNjlb@qgbrNaJDDAuuO=8HIxahTogRIQ;W@se%}BM@V~A9zRHCqX zSek-(Ul4PCoeRR8|AG+jacNd;WQclbK};q&J=JQ(XueW+s*qd18-x%nGbVoOtx~vk z5BjfnT-ySPlYQ#MH=8e0?#u4G;vG$u-8LU7GdXqKKdSJZ6}Y4nRkm|oYz-)~`=oae zr}q`1y_bLM8B(p~aaTE}p-msHWp|g)uB&{`dddYh_20T3@UTNPdqx}KP$6Gc*NKQs zg2+LFrl0`AwDZdZh%^%yiS_2T*#-1U|$s zbP0-!NLq5pW1iynVSa2M&(p4kD$CO(zk9MbQ#OL(A6S@wZ02yh6TG1!xj(aHVB5^d z(qm6<7riH+zQ24}8U+oW`O>ZDO6iY37D*`LdYM{*63z@5y{RR zYxm)P+>qJ2eWSxHMk|`d;gooBvjAO~)u)W0fS%w7rP45#WzvGf=1 zn7;ep89uKJqWX0XU)Z+a_1SSV%aXH+}e$xndIOZ*vStIi5gXQw#V;<+uQ61qH z_lLRZgW|PY2kUoh4$QY-#+2=xnedU7`7 zjO`cNl|M9^kkMubVakC468dniy)TXcSj6&S4!|v}SN+f*IG_WYyAd$u$4-g|tG&w# zQ8Bw;L0QG$i!QotIy@ToIbB<=^HPInTBj8adedD$o+$o3+?n-zH5uk!M+HA!*t#{Gm{_t_>SL$V;C-%_P;JKDd^Fb@bUGdjRKB`Z8NdO64>36rh9EnsiV z%~=}Cl*lBFJ|Tvvb}TP0NHm0P-``PV-(8TJ-LLztYz@&B%rxPEC;5eBWcU7At)E2d zM>WllR6=J3XVs6xpHPzHfydmO-#v``lx&=u{nrxtS8sx4+G)JbNo}^z|9iP&CLRPy zJkM)vL{HWTc{`|?2{+?e%N4C4U}jrhA`kY~F72K-Y>vy`mTFeZsbZtrXw(^7U;i`F zj@I*9wtcP5=fh`kf+<0i4CTDLszEDi-Z7&+P*5TJ)W}(U{@#sazdc$@>1flQ9 zDcDSFxmH$FGi@dMzvIgYj&E2j?=QgwXV-5HF8z`)GOC85j>mXcjVb0&JTgrR8(U4+ zIpVbDejcwo_{wz}SuRIjs|$ODU1dF*(`|TsIqylk9&j(c?ByphoDSY7u-NH97IzhL zonQ%leX1y+2)VmOudk#8?{{X9M9(xHHR01fv33d_Ztp2X%1dWy(h}Rp?9ftr-U?oh zwtKg@n`O@Wqx?4d>-75oe-#E5SEKZ#=_PhbROeqUFLKB%j8(WnLb!ePcSdRVKJ@RY zjph;=JLp*CiWP`qp5nL3!E?>cbtBYR|1Gf$AM9iP)h{C3=|-x%a2bA1Nf?_BE3cs} zdx*Jq!@Czz>99||Mcsd!THfT$7Q?@t4j$Y{v+Uhh6E>h=ERFcx5?%V;Q99ipiZW9P z5OT7PTjFJg=o=V(48nPmsTP7N0ZUB6TWA+6?aPf;A2KG6jnJHAES^nOz{J2}p`K>s z#PDG%t4U<+v=KH&oeS5tU?@V_ZsIfE<2$o}z=0)|3IOv67Zr~^l55>;y087O zrid|8hE=N;M|>dNI#N~SP}M5dOoeU!Xh0Kp?nqrW!o>IldU86>d+e>Pw9Rubw?uvU z%LTksci;0T)VuSM-x;Kr`a|x+`|odPh$B)7RPjRA@$hP z;_&vYfLfX)J8n~87S44zidFwPy^L4GpT0eFiPiUwV%I0X8v%}j^{@Sj!Qc9c=mw<^ z`D4Um0Yh6_x=Wg!@`+Bn7-~S{NlJV>uXl)QXv_h)%{}Uslmmu6@67`fOFsTz0pSk{ zh+KDwSPb{^?E9UcpVh}djBeIh|G~zId2!;kdu>vRJq3iM|jMgN~)Z&r3{< z{I3|p!(w?zTxR+)yCWS|vx~qomvZ{8?Andk5h_uK#VAlz9e+VvdP}txj#E)*>2~uu zE^wq_F@s~BURe!Ke=)0&^9KEZ>Jfuoc5^dS$Ze-rdqu1?23BDShbY@qV!9*kbVyPG^hdt}8y_raWrHbM=?fs4;`b3xp&5 zzM}7;&$Wv9>t!oVy=P)spx(238I5?Fve#dBf~(d!HX13x;q0Lld!PS%I+y?aFp$7R zjT!p?wfEL7tFLO9OScvw((&$UIC*FoO1O=UY9V+z zw5G;=>c&|j(=1zpQM))@$Z?IM7q%WdbcbYa)H54R+MOE#XPL9KlLGvh>Ya9(ED=zC ze*PvG(JNb#rVfX}IjzB(PQPQZ2G^DB`B*g1SNp_<-1)?J8!VWK>DPpybw9S2>59QG z;kz*{GI#%H_#D4$L1)PTKU6`N5yz$Z0(r9SYS>{)fS}Km3UNrg6Mm3LAYoE2n3lI= z*bqnsW`ezrBwWR)@;n+{GOQ1_rBXC@&tEy5eRwT+|;mxkT40*N_ zK*M^SGnVe*q;p)a{Q5#{6aXR57!!Uk9U%6!(n4Ie_@~*%`3aYXF~P{EDIA>+P!Mt; z$``Uj6(XzVQ##KYI6eag*LjRQ&!J4&#igQRX*4#Yx#z04NMYg$0xJbNY)kZ!^WC@=c}mI5fApc zJCZ*8ll2Ta^9rTI9Ybo_-3d;lNuyF2Qk;R9VNTaKPoxSsYXFf7uVj+0b~tyDDy|Td zbyMv#TpO`ydt&5w@6ojDsLnS$N3QW{qNG(#F|-x|e)GO2)DUpab<);HF+IK~MOY{L zZ+P--j1}u)bwp*k0u?D7ZK93Os;L?TfFnqR4OnXSAdx+o&4(e!{X}XZLmK7)*ss{X zt)(su_3TQM5X9K;na&{YtfYeVGLvM3h_+*@m@wBxPezV+!@ixa<*=z`2&vGxPqIC6 z5C{B>R0Q5LSkcxmM3`2$t~_wdZgYcxQf=7U+8Xt8Xz&}r^Iyf}Dwo9V={fg4r%8RE zddwB03WNUvQm5{jnvx0_(&{?RMLvk%Po@wliyq(U0&rS zhanA}oRWrXXnY!07!|pHcl$~OEflTPs8TdwLI+lI7$*lSIY)A;>Q)0KnneKA!S9_| z{#|wT{AJ*Qs#Ne5?ME>edG{^bspbSRI}|6KR8MhtmL48N=bUw3`GM$PcauM|NX&Vx zUiA_1Y8&2ZRQRo>YXWFvBv>1ju;MM|bP*l5Ol|UYu{iT5BC9^}I;$2;zqOJ~V7?qY z!~1jD>9+iHp_A9Vm2uv)8DE{hHI<)sMc}(RwW+sUQvNz!SlC^L*bsA{=2_q7Z!^K! z4Kvw6*Q3mK6Mb_JfpehsgXjdq-f8_i`mxf7*16w=oNbRF3SwxlrQ*6glv+@eT4DRa zQlo5M-T5I|AO-t5TIu^*U(6PIWUvQ~={b!h9slank46%N)W~&r8sgDf0W$&>dhr8c z+JyXvn3#A|Tx+HH9;4o%(1+Ts;ZpDVr;T@1#eAQF90m0e0X26~UQ4=Aw2f4x4&rhi z*d#8?ZNe{D0Ysd)Tt}@F&2_kWXJDd&VzVc<| zMds8*+38d~T8Gg2Yxtj{Q6?4R+mv9k*9XxvhgT9$D_6b%oWp7bul>C3+$rqKhq1#s zp7-Ox*;qRLMW~$W>aZzk9II_j(m&(vLTI4jLnF|wl-eFRspldd+d52llbUmyStui zCd;jdMC%`@(SOY_ia3cSZp!!=Q|QI-NoNC3Zh1GWY5`kEt`{#}AWvWXLojt- zS>nG~cISbk_=fmHE#G>r@`QH{60^zvdax%H;_s(cQteQ^OFY?zjyZ1%AA%v4> z`v6tQ{76=(I{xvG)|J)(n%_0f-@Z=ZGU~zpl`)(CzSu_t+aWwE);)Le)nP_*r@*dc$Bljc=GG=7dj66XXOhoZoJhiZ48bHwe$@0(y%=s z5{Xzbz=?|@y(<4vV{OrdB*Tr1JsB~^OkfD9*%;bfQowmUjEh!?NGe`~^{eLN&DIXO z*Un#VFYz**lntq!94wG%@ZVRHBNdxtVXb5UV_3+_^Wg{P(r(+Mp;goyfo4CRD7^5t zEnBhub2d&L*iA_shYR;@syj0vlq=dE(y@w!~4 zefZhpDu6yyMd%~@6Rz3XmzDdFh-lAb{o)nx@vyJZ;N381+wbpq4GytQKY^kScLXur zJ{259(gVP3Jkeewmk}h4@jwmJ=Fg(5Tl+}_KWwt5d|2vTIKW1;Ief*sS#xMEu)h$2 zwS(E8&|cMHxlBCq_-h1hZBEb$Q!>u4%)#UOQg9arVW+$HMb49&4B}YSg@aeu8GF)1 zgH+>y4k5Pnpkrdt_IpXaFY+d|L-wt6Lx$8`hqL4Z^^CBr#)HuX!B&>OdY`K{czGmB zNTEEkX9~7O*MY+%%E5&&6Zp`;A635zl46`96?ea0obR={YHOR1mD${qV3Mu6VTmZ& zXt4ioEERMy*~WErX#8;VQS$X4FE1iLEq+w9#Yi6-lRMcEEEGWmi3UJ&R^x>!CvNik z_SE}Jtuzf6$h?gw7}74ZfAGy8I7Iq_A+_>tHdXQl>95$@o5Zc6L$q(zFhX@ew|`Oa z**UYM_hR6TPvlm=Ma4>o7h$c#cZF78I(e≈bVndU{ob52CXsi-Q*GGVZE9aa2*M z^?vSp-|r?3xXZ{FCcDxKovNbY##Z>Cn>{q%Qt(g_ zwR4~7Tm13TjhJh~V47gA>p@493kAx}rhnaYn$AGEpm>7?Uh(5ruFr_fg(Z73Nu5-Z zSk2+Uy8wOElU=i?!v{hOoX$|{+lJ1#42HZk50+)n{w zKvY~jJc}vlaEivFIW^oM(9ZPRqib_7ZW3mgQ|X-vT`o=5YF%=j^O;lxhh%&(DEj*cb>DroVrb|7Mc;sN?h&rjYT)wpL82xM|d zvaB4>7t_EVdSPO zNZ8ERstv#gtutV51S>`0Y$@;VU-&Rd-mnhu-o&h|1iA@X_+``CbD9MWRUNu$@~0+A zV)cL~VSD>ileCJ7)xFnvk+EKM2o1IVM_x``j{E=2z>mG>`c>@16gesrlII#_`0q%! zob`U2e8$>HOgGMsE0uC>dl6G(%I+xfRUH4s`IDY(S2ikA$bWKRJ9Uv5kY*lWnqp33 zh4uB#Xt|f`xn|Wo(zH4KHbo2ceUy>-xzxCVWwrKB$xk8*M-e9)@g5yjTJ{w1D}eyN zk4mQYz9^Uk^ZAdr4=M4SB|fNlJ@MT;yN-`tALrlSU#RkP$6Wb&3CSAHI3;SdsftEH z66F)zFJIz=j+-w|d+2l&#zUV+-nqU!?QzW7puZr;V?Hv`FG3OULpW*`8}FnOOyDh1 zPdfiuNF*B%4s^(?njd$sPm%CmgEAS&>aXnX1vhT4l7msZJdpcEcMjhU^U#im;N365 zVP8&o7B&?F4SzADL|BM7cJW?MBzOX}=RH6U#@|!451Q!}y5+iCOHJP`_Xr}Ro9S3A zNQ(FMg49vu;G-XqMgC_nm9OXcen>h?QtgRUBmDP{ufzN#c6+Rx3qTh}|H!MjP;1Y7 zAs$W*eko50K1wBfN(({!8X$b%5U#f z4;v1>7c--AbEo)5jwe_{!P6LqO4VeemFfvGsg|#y9`EbN~9b6=o zIR07JPwUOD$~l`ncE0d@&Zt}Ng($)su4_8f5f#5DY1Lbm5x5^j8R=@JHWxsD? zzR;g8?#8x_$-mlwk@gQwW(x<8e``y{Dl`^!l-ckaeI-{IFHezD|{f2_|qx_7|4&6i;p%qGSq?SVpJ-7?$ z03}kMi*q-r=L~5Ihv%drS(Zf&posg)ln?ewVX%7!1WprUWEh}lH)Sl-PhpKe!Q@== znNMzH?o5i6WmPzm5s`r%*&r9E${%(DRr^t9RblX%VY7NCM~#9cv&3{%gX>(xrzL-D z?3!H*i=&WjW{P!6CPnLZcNEEiY+c`(c}F8&`m(gY&DyUw{&)q84_4ZX4+LLZ0o{E4 z=~OdG>0Vo>IU8`mfmudJJC`jc+xz8~-Ck%ocr@}*lt^C_Zd21(* zK4YJK>V=()Dh)ukg`=eonr5J`?JonY4<0Tz1=?XgGXb5{PTLY|sXs8MdM0A1+Wk)^ z71PT$Ny_@PLqrx32`&V}v`fY8YWfR@3 z()>$6vWW3(D)Kq!<)(Pt)|pX7mb8cdh=8c`k*GBP^Q!Lpalhrn4!O$4M|#Q{tF!+k zWya75$W`vQCnkNeQv10`bmV?~Qf65L9_;86=qq5S1B?VKHq4-s+8Q7tU8vPZMf4Sy z>TwApS-*x~xW&1iQ-U3aKZ&ehmRKZ1fi59dS6u6+nO?caF8ClgxG7Ta{rG1U%;$X% z7$!=;5-T|o&Iy^kq&ql;u~t%*$~2Ip!8^II74 z&nS_F=H&=ef|q^qBP+LXb;jegWMCDU?ylPXIMeehlO3rOU0fj%NfWDX*?eB63K#NQ z9uMtfC(1{f_B-oLM&=eFKOe*Da8cp10tB=Du{lYOw4<)K#`p9^bqFJgJol1wIs)#8 z!*AtT77US!c#dlAE;;924Yq8!UMJ^#uu)E!+QZhsBB}_}JOo}gi=VAmLj%s7YGY)h zIZSI`#8qrmxv0~Z1Qrz5U&_=zmd9AadnxTL+QQzI&R8dDBv{xHRgd=YwMizEs}n{( zg9hxk{oXl_VFmFdkm{>C6g+0Q|I8;K9{lKzaW+7|)LEO@7-^bg#S zQ0d#22O-EO0xk-1W{ir7>B~_hoKia9W4?+)a%(d>rT+nlSy!7K-CZw8y6jMtF`@+O zf9m{7nbsygoY7H&T`(+BP}0!RPEFZZh}i9|;fx6oRmWg>clQ4#)&V-?5f_uXLC!68 zG1br|-Xq74f6R1eq%;4^62QPcrHo`wphwMfeV6Q+{ZJj&gSN^|@?%pa1Q{`f-%|uy z)#0`H?8RRPkZdmmx)B%SRMl(GN zTPQIsIoZ|7Q6sQ1w5ZeTZW8gP2S;lc8kxA|7xr$0H1&`Q9N`I+^U}d8Z}4a9JqApe z@3`7Q(It0G>o}cX`UDCWa$-!{oJSC>X;%|Z>ne$whv#Xev zif~USsWCt->4v3+!?SC#aub_vR@p7693ls(5D7U zW+R9%;y%GGfM^8m#U-Wpyb|v`E5@wyKvLg3_kIih$*5r!{tgJ}N6f0-AG-@|wSa?7 zCPE3qQFxKn=G}6sUG+KELmz~7Y3NfN%K|5mJP>*~W;KozGB2@^3pZ5`dL_W>&={MD z+wYXD!@U9oan^p?9EdI;xQ_LxlVM){lPVJ@r1(Z@w%jR{s;PtDBH#I6FE;qK%A^Pm zx&uUpi&rZe&U}eT(h_C^o73p27jvFk3*|=s>3-Hk#lZa+!5;>6QmR90tu$xWGQ#R{ zVZ$Xo+pgCsh^L70#_Pi+67xavWa0RXI6F>V=(j&E=+CVFi0nFEi*Hj6X3-5`h!CEg zo!w^n*%JpY&z;xI9oO2-cSJ#@(g0(P{LTe&ufv2xPTNp`tmAo*{mZcKu1zUz5Rc?k zgq%Im;ahz5aG6~G^|O_C$k8-GH%>w3tat{JLR*&Y-TT<90DDndqdWTgD582KNuf|) zsBVPScIPvP6{1FDlw0^Gd)<}&;EE0={r}aumFiN7$%kn{GK@F)PoGv4SKfFNyfUH# zPbU(r(1TTV4E*r#<643<2uf#v98vMQUG?H$`%@q3Hl!{AmedBrf_6vCg!~f}aH=@+ z_7l`^H_tEOAK)L{khbawy}E?G&l7L+YoS5&o|zH5f`8KzlarGV3T>UseZ8Hek|g$8 zy|58bWhq#{m%_lW_-&T43RKJP=d`S_D2A`p=^`H9Upx;G58fjaz0_;qw)4_ohn@l* zk5!ZBwV{TFbl~xd`)7DtmHyj1(kM|d@CN>-WZ|CK+>}>4RjpJQkeS^BcnPA`iB1Mx8L0ZdbEzW+?m3SfiHF(xvm1az^0(WG{8K zo3gp8^J1VDFe-p0|G_=}6~PM0gF8~jNd^4bn?Z8xX0HDB*uwG%#qr(3W~gqh=tAFj ze|dZN6DC$WAtqj2;QK<3SJ75_HYe(ozQPGFYsOz4WBKOZb)y&7!$R*ZT-no8biUCw z0nh?3W?W?SP-!zNBLZmxO9O41e;w)c*EysUDv{|NZuD1REjih>~7=N(O%KGIhMaJ z4u@~KAfBg1{b|2%snsdhVgs#Oc@os_bG{t)?D7kfS-{$P?!XY&e1aulje?B#H%#+9 zvC*N|pq=7Jxc z-42S0yZgVJc!IQHZ^x5P_^6u=f4rWne(OlGA*KrDvuGyR5@_?oSUOw&(0O=MfiSC> zxX780a!Ta8H6@W2d!MnSmWlCBBvEbRE{LCSa?hVOoNRggNggWF_wX$tj9tFsQfh!G ziBjju_=kVkwP?fnqzn8{dc za|s3_1AIW=-rIR%PXJk)%$Vy%va}HL9h}?D`E=cWU;K~(+zE(KWovf zKuY3i@={S;-m}Z9-jQgaSwsQ6U@Y~(>6-AK-_>xI4|Z?q{J1k@14jBcEB8cnWGYl8 ze5coSxy8(j{+Y6NFKRBmuDev1r_m9=8dAMY9zzqpMA(efWd z>id+L>lHtW?OZ5>*?;&mG?&NGIgd4zE4(ED%_|kFQt5BL^4wO0Y_n3sj=K`^E37C4 zFslsx>mRJG>%9-4nSy2t*YJU5xC{9YVS4HwPcyH=uUyea# z$7}`oeIES$fsbVb5>G6zdr81^;SlYT*zlLcve@EbLL>4LsAS%y0n20Sb(Y>}Y$v_~ zn)+dDBgSQbPQ`?gca*gZ28lqx?c62;*<5!d4oArXw{RGpuI#aisLp(ntz6Ve3c=5C z1H$hCyNcYv)iSlW(D?H-&4Rtz!+_7MO^?|m3R!i|M}>Xi$5DFlX_}i|nACeYI1}nE zPZ;Uiaix@nmtx1tb_h_RkR|JNKyocTp^oOB0imbc0-$xd-V4Ci8Hp5u{|nIp3O$d+?mk!T1_TzVab0fc(Ogs!1Gmll#}OP3YU7%Zx5Qu(9rXLo0AebtacDocCoZ#Iy8kT#g7gaJ4z!d(4rlVj*iApX2ZX9cP;Z;Rh=rw-|M(oK#{im~I>$6T3^gXIB&!l$Fs)fZ$0 z-?^KNqe=}b*tXU<|E4i^cb!hthu2?6A8!p~cIRrL9-Fbz6VRXsgz1GMN$Ky3E3*1s zJ8vQmLVn44?MOPxkup$hQG<&z7(txw6D4&xnVXuq3cUUWe(TwdQAD$lwawMG5FcL zgq7zzWl0#Vig45lAYtj6L=4~}0lkEAZ}0u(+(>ItQHB0VFG{ew0~8N{LUcu(#BGx< z$Qhso+jNQ~pZtl?o23GdkyLn3Ae|0|2ND&d)Gc>tq%@u?U^X;1Ug^+i4h^`-ziMRJ$`9+~_&{W&56j{Uz+M(5c#(wsJ!}5dV#)C};QFK#;!G;G zXFFNq+Xp=7D@|@@G*(VC746h_ho8G*Rz2^hfZabX3MM?h3tvx!5rH$Osyqh3Q>>-^ z$23Pq^ldUT9;beu+npQZO%)`RsJ(-g!|GSnv9BkQi4E05=wLEA&IyP8#dZeHnaW)A zpe#gaH>)UyWwA4a8Nk6RSVwXU-E5jjUz^^@Tzj$d5CBHW;Uj z&@8@b+xz|X97KBhO8tP#nEX2#_%)@ad5!NIrUHq8ey2c*OV#>z`p=Rpf4yLsFworoO+l-5?rE6uGL%e(GGp&u8FlHL@JU&BO0}#8O%BRl^ODrvbm#3uVp$(*pwH zu|O_q$5F&CHM)=zObM3pwSxv8^M9I}J1?^olBWWXe|j4IbhPsPmZ#=^s73r$iDH}C z^9p2Qkg2(gjJ;ZBDANl+FWe#{kPUyL)S@|tFbX4Uqvgkp)~gas7naX(*&!wt+dXIq zgAHH(&Fhd1q{AFbc7#Z#C(;%&fmWzMWMQHTlK9C2SIhH`h6D`ybK!JRJFe(N!lcIw zjV?2nigd4e!lBF;ys=CeoK>DY=E-d~NyjAco4D42u0`^vg*in8;Bu*qfW~s>PY$Vfa4g2)i?bQtZSaOm-hpjg9g;Fm@V%bHz%#m2_wJ>n$G z8m1l8w0^wnQ#TgiZj8c~*aYJ2Z7{1U076sD-f8?E2he&j;?n0}p&%0wb+I{8AOq*h zlY?JpC{hWFxT6uhN`2TpeWeS36Y99#keWcQ(E4Jr00_SmJ)px}oDQlz`xc z2^I@5jJLssZmA2DBy5n>O8>2H%rkHAXYbz4w%GBu6nf5hQ7eEEV~ED*RRAuG1*jP| zJ|C@gG@q%5{yKq-=6HC@IMy{yCVYO!;IeRi2%drh;6~Qgc_n7C`2O%H6u`5;{~=b zn98XJcAmHuNWaWF*9x*hJOM>I?Uq&3qLoX`b>wVfgPa#_z>d{D+};BSlL5Pp*?m9? zP#H)O3fWK6cH*+Am`g7Q5*QNHKZ>PSlBWM!khxrb;BdqVf3QqfdF}-|+nZ@z@LO73Aw(GolSIjFMh(;381K5cvw?m-8g(nfS7z8MA_x&v%2km(7F4St$xYmcoox(o- zYK^ON0xWV08hS2|?Ixp}*zn2uD zP3%;q`jtJ|J<8hV=49HB83K8Dc)EO6f+LwjerXqK8Fywfg1IX!#CJ0IOj^`@Qa^SZ zKXCuTa-1}BZ7;C-J+m-ou;VM;Gh;$FYx|K;&UKvBz2n6TK$GvuVsr|ZT7TG2iVv}x z13}SL#el%l8|plE>40gcGLu#?3A32A;B{?_1GMVqR)J<#d~7UHX1^22Ww9v~Xnlth zDIO<#>qr2XYcP2<9Jq1x?$=IfrP~Qw+wATPb&v1|EI2E?qy&q* z>K6@9=x}gybdgb^o|D~;<3Kr}ugU*}p(OJ(P+?MH#xLZ3-+)dV3BI}mH60Q_Q*!1< z*U-Yk_P5JlA%Y`BAh=k1jn&UbW#%anE5Wi60PjNK%2!juP>`R0D$})s<31$GXi~{W~XCn>NME!gT&~84qsW-XVQE3?)ig#4A3hbKtagd zd>=+=53&tm-7Cn+VdEIfNh>wpvQ1Ypd8uBHG6&Hy0kDMZJ``W=rjv)lI3kE>&Rp8} z@QaKW(Ii{)8S5jxh!0}MTUX)6p!^5#D3?x5`@b74C%txx%v*lt)MG2lbMK3aO`bD4 z(b`C9ros%at(_!OQL_U}M2B~WhQ&H*EDo0-zll)j{7FjM_dOpRNQog4x;V4Tn_ zLqXVZj}3Sd1Nm{HgQbvHDTOsv*g}5pj%irIO;YVoES7HjLq-nldMwHfP5EA_dZ53i z3$IYu<(P{?9v=?fK191C6(sZ&tR|z#pnhu#wG)#o4|IC6e;M;o3z)v1oP0N1V<%Z* z#*2>?@ldv5{$)ZTzx)`p2>$xgtyKw+6#(Dg@w2m4HodF^Q;ECQ|LT5vWR#D(x^(Td z*-o^IiceO%lXS~X=`I;q=~oJVcGzhkk6N+7cOp zxR9Vp>ZO+RFF-puEhDV-b!xquNq0?orK?s}luBdjr)AQ7UYUjjN-v*Lq zmwWh1pHHK446Lps6|P}Tz-862I=s|sScPPTd2fdmHGrjg!$ta6PSW&$QGl|;Znm&= zyg$5o2U%g$c&FxUOiBq%xrDCs1#I5D?1o$KK~!O3H0LnRw2qm$&b^jpw*a^?g;V93 zQgBGaKftPzQp~X^Y%0l#dHebK(Mmm!N}5}hATAh2B4a>>A{sMJzEds4WA?vVM@Rwj z4AvgrAJXe&3^V4ck!0*jY`nY_r>7Zg3IC$($?1UcQqWZ3*F*+rH2TS3T;yDWaG!kJ z&Pl22)KBq04Q$la)$WeA4@DhPqgT7-?0dB;sTdvbKwem_5I|`Yw9w%EQZ(=NzvIap zi%^1lIcmaDTH4y7fXQZ3dGp3U3e{+PNX$OiMM_ecy0=OvfS(V$70g0 zSRci*iUXE8lIo~2Oe)yrCresf&v9@PF<6-h0P+6Zps~oQs(Pi!_8-HB3^?S~kH>!s zK782SxDp(rkx5RA3<{1SyRV!<$dju&Hs$sHKis`R4rahR3M z?dyw~`ybFZ^S_kO7tUDO~o!Br18A@>2d#NDf7>jr&#Vnb2)>9fH z%qlCR8fG;aEK?(=ibrRSc}R|ZQrN3uSnKk*q@-lX$0|wj-#r%vRQy;JtiID~U+2*; zG`Pa8t#9Xn;p`Yv#wSo;^!@bPBHzg3e*N+dyu1!v4Dr=mmh0?rQ^7v@Z`%iQL-2mC z=3f{*d-evv9x_FK2^CaWmO z>9O=hvDV^$$CImJWRhx1n5%VQml0t4Y0nR|;8BoyU(-dUza??H9e#&3m{c3NHOv@z^py(;Z>DPXdLhL64>}EmJK_R&Q%3n zicL;WzaM$<-NxWd)MTNB#Xlcu}53Mcf0g@5$z%?Pb}N! zaAs*A-23dFRDiz2vg&^i-TS&bG|^s1R~OaYEl){Bg~k2%VR@iF_5{t#a!PMl`yY8q z{x - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file From 1b3a30f2fbc655392fe823a4bedaed038bcc5cc4 Mon Sep 17 00:00:00 2001 From: sallyjunjun Date: Thu, 29 Aug 2024 16:33:33 +0800 Subject: [PATCH 3/5] fix usage and install --- .../locales/en/LC_MESSAGES/install.po | 155 ++++++++------ doc/code-docs/locales/en/LC_MESSAGES/usage.po | 189 +++++++++--------- doc/en/install.md | 22 +- doc/en/usage.md | 10 +- doc/install.md | 20 +- doc/usage.md | 31 +-- 6 files changed, 247 insertions(+), 180 deletions(-) diff --git a/doc/code-docs/locales/en/LC_MESSAGES/install.po b/doc/code-docs/locales/en/LC_MESSAGES/install.po index 2c0393902..690b54c61 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/install.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/install.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-23 14:49+0800\n" +"POT-Creation-Date: 2024-08-29 16:25+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -20,7 +20,7 @@ msgstr "" #: ../../../install.md:2 msgid "环境安装" -msgstr "Installation" +msgstr "Environment Installation" #: ../../../install.md:3 msgid "环境准备" @@ -34,7 +34,7 @@ msgstr "" msgid "Ampere或者Hopper架构的GPU (例如H100, A100)" msgstr "GPU with Ampere or Hopper architecture (such as H100, A100)" -#: ../../../install.md:6 ../../../install.md:132 +#: ../../../install.md:6 ../../../install.md:142 msgid "Linux OS" msgstr "" @@ -45,104 +45,119 @@ msgstr "Installation through pip" #: ../../../install.md:9 msgid "推荐使用 conda 构建一个 Python-3.10 的虚拟环境，命令如下：" msgstr "" -"It is recommended to build a Python-3.10 virtual environment using conda, command is as follows: " +"It is recommended to build a Python-3.10 virtual environment using conda," +" command is as follows: " #: ../../../install.md:15 msgid "首先，安装指定版本的torch, torchvision, torchaudio以及torch-scatter:" -msgstr "Firstly, install the specified versions of torch, torchvision, torchaudio, and torch-scatter:" +msgstr "" +"Firstly, install the specified versions of torch, torchvision, " +"torchaudio, and torch-scatter:" #: ../../../install.md:21 msgid "安装InternEvo:" msgstr "Install InternEvo:" -#: ../../../install.md:26 ../../../install.md:69 +#: ../../../install.md:26 ../../../install.md:79 msgid "安装 flash-attention (version v2.2.1)：" msgstr "Install flash-attention (version v2.2.1):" -#: ../../../install.md:31 -msgid "安装 Apex (version 23.05)： apex为非必须安装包，如果安装，参考下述源码方式安装。" -msgstr "Install Apex (version 23.05): apex is an optional installation package; if you choose to install it, refer to the following source code method for installation." +#: ../../../install.md:28 +msgid "如果需要使用flash-attention加速训练，且环境中支持，安装方式如下：" +msgstr "If you need to use flash-attention to accelerate training, and it is supported in your environment, install as follows:" + +#: ../../../install.md:33 ../../../install.md:89 +msgid "安装 Apex (version 23.05)：" +msgstr "Install Apex (version 23.05):" + +#: ../../../install.md:35 +msgid "apex为非必须安装包，如果安装，参考下述源码方式安装。" +msgstr "" +"apex is an optional installation package; " +"if you choose to install it, refer to the following source code method " +"for installation." + +#: ../../../install.md:37 ../../../install.md:99 +msgid "安装megablocks（version 0.3.2）:" +msgstr "Install megablocks（version 0.3.2）" + +#: ../../../install.md:39 +msgid "如果是MoE相关模型，需要安装。" +msgstr "If the model you are training is MoE releated, install as follows:" -#: ../../../install.md:34 +#: ../../../install.md:44 msgid "源码方式安装" msgstr "Installation through Source Code" -#: ../../../install.md:35 +#: ../../../install.md:45 msgid "依赖包" msgstr "Dependent Package" -#: ../../../install.md:36 +#: ../../../install.md:46 msgid "首先，需要安装的依赖包及对应版本列表如下：" msgstr "The required packages and corresponding version are shown as follows:" -#: ../../../install.md:37 +#: ../../../install.md:47 msgid "GCC == 10.2.0" msgstr "" -#: ../../../install.md:38 +#: ../../../install.md:48 msgid "MPFR == 4.1.0" msgstr "" -#: ../../../install.md:39 +#: ../../../install.md:49 msgid "CUDA >= 11.8" msgstr "" -#: ../../../install.md:40 +#: ../../../install.md:50 msgid "Pytorch >= 2.1.0" msgstr "" -#: ../../../install.md:41 +#: ../../../install.md:51 msgid "Transformers >= 4.28.0" msgstr "" -#: ../../../install.md:43 +#: ../../../install.md:53 msgid "以上依赖包安装完成后，需要更新配置系统环境变量：" msgstr "" "After installing the above dependencies, some system environment " "variables need to be updated:" -#: ../../../install.md:54 +#: ../../../install.md:64 msgid "安装过程" msgstr "Installation Procedure" -#: ../../../install.md:55 +#: ../../../install.md:65 msgid "将项目`InternEvo`及其依赖子模块，从 github 仓库中 clone 下来，命令如下：" msgstr "" -"Clone the project `InternEvo` and its dependent submodules from the github" -" repository, as follows:" +"Clone the project `InternEvo` and its dependent submodules from the " +"github repository, as follows:" -#: ../../../install.md:60 +#: ../../../install.md:70 msgid "推荐使用 conda 构建一个 Python-3.10 的虚拟环境， 并基于`requirements/`文件安装项目所需的依赖包：" msgstr "" "It is recommended to build a Python-3.10 virtual environment using conda " "and install the required dependencies based on the `requirements/` files:" -#: ../../../install.md:79 -msgid "安装 Apex (version 23.05)：" -msgstr "Install Apex (version 23.05):" - -#: ../../../install.md:89 -msgid "额外安装" -msgstr "Additional Install" - -#: ../../../install.md:94 +#: ../../../install.md:104 msgid "环境镜像" msgstr "Environment Image" -#: ../../../install.md:95 +#: ../../../install.md:105 msgid "" "用户可以使用提供的 dockerfile 结合 docker.Makefile 来构建自己的镜像，或者也可以从 " "https://hub.docker.com/r/internlm/internevo/tags 获取安装了 InternEvo 运行环境的镜像。" msgstr "" "Users can use the provided dockerfile combined with docker.Makefile to " "build their own images, or obtain images with InternEvo runtime " -"environment installed from https://hub.docker.com/r/internlm/internevo/tags." +"environment installed from " +"https://hub.docker.com/r/internlm/internevo/tags." -#: ../../../install.md:97 +#: ../../../install.md:107 msgid "镜像配置及构造" msgstr "Image Configuration and Build" -#: ../../../install.md:98 +#: ../../../install.md:108 msgid "" "dockerfile 的配置以及构造均通过 docker.Makefile 文件实现，在 InternEvo 根目录下执行如下命令即可 build" " 镜像：" @@ -151,7 +166,7 @@ msgstr "" " docker.Makefile. To build the image, execute the following command in " "the root directory of InternEvo:" -#: ../../../install.md:102 +#: ../../../install.md:112 msgid "" "在 docker.Makefile 中可自定义基础镜像，环境版本等内容，对应参数可直接通过命令行传递，默认为推荐的环境版本。对于 BASE_OS " "分别支持 ubuntu20.04 和 centos7。" @@ -161,77 +176,88 @@ msgstr "" "through the command line. For BASE_OS, ubuntu20.04 and centos7 are " "respectively supported." -#: ../../../install.md:104 +#: ../../../install.md:114 msgid "镜像拉取" msgstr "Pull Standard Image" -#: ../../../install.md:105 +#: ../../../install.md:115 msgid "基于 ubuntu 和 centos 的标准镜像已经 build 完成也可直接拉取使用：" msgstr "" "The standard image based on ubuntu and centos has been built and can be " "directly pulled:" -#: ../../../install.md:114 +#: ../../../install.md:124 msgid "容器启动" msgstr "Run Container" -#: ../../../install.md:115 +#: ../../../install.md:125 msgid "对于使用 dockerfile 构建或拉取的本地标准镜像，使用如下命令启动并进入容器：" msgstr "" "For the local standard image built with dockerfile or pulled, use the " "following command to run and enter the container:" -#: ../../../install.md:120 +#: ../../../install.md:130 msgid "训练启动" msgstr "Training start" -#: ../../../install.md:121 +#: ../../../install.md:131 msgid "容器内默认目录即 `/InternEvo`，参考[使用文档](./usage.md)可获取具体使用方法。默认7B模型启动单机8卡训练命令样例：" msgstr "" "The default directory in the container is `/InternEvo`, please start " -"training according to the [Usage](./usage.md). Default command example to launch the training of a 7B model on a single machine with 8 GPUs:" +"training according to the [Usage](./usage.md). Default command example to" +" launch the training of a 7B model on a single machine with 8 GPUs:" -#: ../../../install.md:126 -msgid "环境安装（NPU）" -msgstr "Installation (NPU)" +#: ../../../install.md:136 +msgid "NPU环境安装" +msgstr "NPU Environment Installation" -#: ../../../install.md:127 +#: ../../../install.md:137 msgid "" "在搭载NPU的机器上安装环境的版本可参考GPU，在NPU上使用昇腾torch_npu代替torch，同时Flash-" "Attention和Apex不再支持安装，相应功能已由InternEvo代码内部实现。以下教程仅为torch_npu安装。" msgstr "" -"The version for environment installation on machines with NPUs can refer to that of GPUs, using Ascend's torch_npu instead of torch on the NPU. Flash-Attention and Apex are no longer supported for installation, as the corresponding functionalities have been implemented internally by the InternEvo code. The following tutorial is only for the installation of torch_npu." +"The version for environment installation on machines with NPUs can refer " +"to that of GPUs, using Ascend's torch_npu instead of torch on the NPU. " +"Flash-Attention and Apex are no longer supported for installation, as the" +" corresponding functionalities have been implemented internally by the " +"InternEvo code. The following tutorial is only for the installation of " +"torch_npu." -#: ../../../install.md:129 +#: ../../../install.md:139 msgid "torch_npu官方文档：https://gitee.com/ascend/pytorch" msgstr "torch_npu offical documentation：https://gitee.com/ascend/pytorch" -#: ../../../install.md:131 -msgid "环境安装样例" -msgstr "Installation Example" +#: ../../../install.md:141 +msgid "NPU环境准备" +msgstr "NPU Environment Preparation" -#: ../../../install.md:133 +#: ../../../install.md:143 msgid "torch_npu: v2.1.0-6.0.rc1" msgstr "" -#: ../../../install.md:134 +#: ../../../install.md:144 msgid "NPU显卡：910B" msgstr "NPU: 910B" -#: ../../../install.md:137 +#: ../../../install.md:147 msgid "安装torch_run" msgstr "Install torch_run" -#: ../../../install.md:139 +#: ../../../install.md:149 msgid "参考文档：https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/" -msgstr "Reference documentation: https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/" +msgstr "" +"Reference documentation: " +"https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/" -#: ../../../install.md:141 +#: ../../../install.md:151 msgid "" "安装时可尝试根据文档内方式安装，或者从 https://gitee.com/ascend/pytorch/releases " "下载指定版本torch_npu进行安装，如下所示：" msgstr "" -"When installing, you may try to install according to the method described in the documentation, or download a specific version of torch_npu for installation from Ascend's PyTorch Releases on Gitee https://gitee.com/ascend/pytorch/releases, as follows:" +"When installing, you may try to install according to the method described" +" in the documentation, or download a specific version of torch_npu for " +"installation from Ascend's PyTorch Releases on Gitee " +"https://gitee.com/ascend/pytorch/releases, as follows:" #~ msgid "CUDA >= 11.7" #~ msgstr "" @@ -245,3 +271,12 @@ msgstr "" #~ msgid "Apex == 23.05" #~ msgstr "" +#~ msgid "额外安装" +#~ msgstr "Additional Install" + +#~ msgid "环境安装（NPU）" +#~ msgstr "Installation (NPU)" + +#~ msgid "环境安装样例" +#~ msgstr "Installation Example" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/usage.po b/doc/code-docs/locales/en/LC_MESSAGES/usage.po index 5d9632eac..a5fd94407 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/usage.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/usage.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-27 11:16+0800\n" +"POT-Creation-Date: 2024-08-29 16:25+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -43,33 +43,37 @@ msgstr "" "on how to install the necessary dependencies." #: ../../../usage.md:9 -msgid "数据准备 （预训练）" -msgstr "Dataset Preparation (Pre-training)" +msgid "数据准备" +msgstr "Data Preparation" -#: ../../../usage.md:10 +#: ../../../usage.md:11 +msgid "预训练" +msgstr "Pre-training" + +#: ../../../usage.md:13 msgid "使用huggingface格式数据集" msgstr "Using Hugging Face format dataset" -#: ../../../usage.md:12 +#: ../../../usage.md:15 msgid "如果使用huggingface数据集，需要先将数据集和需要使用的tokenizer下载到本地。" msgstr "" "If you use a Hugging Face dataset, you need to first download the dataset" " and the required tokenizer to your local machine." -#: ../../../usage.md:14 +#: ../../../usage.md:17 msgid "以`roneneldan/TinyStories`这个数据为例，数据准备阶段需要通过如下命令将数据集下载到本地：" msgstr "" "Using the roneneldan/TinyStories dataset as an example, the data " "preparation phase requires downloading the dataset to your local system " "with the following command:" -#: ../../../usage.md:18 +#: ../../../usage.md:21 msgid "其中，\"/mnt/petrelfs/hf-TinyStories\" 为需要将数据集保存的本地路径。" msgstr "" "In this case, \"/mnt/petrelfs/hf-TinyStories\" is the local path where " "the dataset needs to be saved." -#: ../../../usage.md:20 +#: ../../../usage.md:23 msgid "" "然后将tokenizer下载到本地，例如，使用internlm2的tokenizer，则将`https://huggingface.co/internlm/internlm2-7b/tree/main`中的`special_tokens_map.json`、`tokenizer.model`、`tokenizer_config.json`、`tokenization_internlm2.py`和`tokenization_internlm2_fast.py`文件下载到本地路径，如\"/mnt/petrelfs" "/hf-internlm2-tokenizer\"中。" @@ -81,11 +85,11 @@ msgstr "" "repository at \"https://huggingface.co/internlm/internlm2-7b/tree/main\" " "to a local directory, such as \"/mnt/petrelfs/hf-internlm2-tokenizer\"." -#: ../../../usage.md:22 +#: ../../../usage.md:25 msgid "将配置文件做如下改动：" msgstr "Make the following changes to the configuration file:" -#: ../../../usage.md:30 +#: ../../../usage.md:33 msgid "" "type默认为\"tokenized\"，这里需要改为\"streaming\"类型。同时需要指定`tokenizer_path`, " "如果使用下述tokenized之后的数据集，则不需要设置该字段。`TRAIN_FOLDER`指定本地数据集路径。" @@ -96,11 +100,11 @@ msgstr "" "described below, you do not need to set this field. The TRAIN_FOLDER " "specifies the path to the local dataset." -#: ../../../usage.md:32 +#: ../../../usage.md:35 msgid "使用tokenized之后数据集" msgstr "Use the dataset after tokenization" -#: ../../../usage.md:34 +#: ../../../usage.md:37 msgid "InternEvo训练任务的数据集包括一系列的`bin`和`meta`文件。使用`tokenizer`从原始文本文件生成训练用数据集。通过在`tools/tokenizer.py`中指定模型参数路径的方式来导入tokenizer模型。目前提供`V7_sft.model`来生成tokens。若想使用不同的模型，可直接修改`tokernizer.py`中的模型参数路径。" msgstr "" "The dataset for the InternEvo training task includes a series of `bin` " @@ -111,7 +115,7 @@ msgstr "" " use a different model, you can directly modify the model parameter path " "in `tokenizer.py`." -#: ../../../usage.md:37 +#: ../../../usage.md:40 msgid "可以运行以下命令生成原始数据对应的`bin`和`meta`文件，其中参数`text_input_path`表示原始文本数据路径，目前支持`txt`、`json`和`jsonl`三种输入格式，`bin_output_path`表示生成的`bin`文件的保存路径。" msgstr "" "You can run the following command to generate `bin` and `meta` files " @@ -120,30 +124,30 @@ msgstr "" "`txt`, `json`, and `jsonl` formats, while `bin_output_path` represents " "the save path of the generated `bin` files." -#: ../../../usage.md:42 +#: ../../../usage.md:45 msgid "下面是一个数据处理的例子：" msgstr "Here is an example of data processing:" -#: ../../../usage.md:44 +#: ../../../usage.md:47 msgid "给定一个包含原始数据集的文件`raw_data.txt`，原始数据集如下所示：" msgstr "" "Given a file `raw_data.txt` containing the raw dataset, the raw dataset " "is shown below:" -#: ../../../usage.md:51 +#: ../../../usage.md:54 msgid "可以通过运行以下命令来生成`bin`和`meta`文件：" msgstr "" "You can generate the `bin` and `meta` files by running the following " "command:" -#: ../../../usage.md:56 +#: ../../../usage.md:59 msgid "需要注意的是，生成的`bin`文件需要保存在`cn`或者`en`或者`code`或者`ja`或者`ar`或者`kaoshi`这六个目录下，以区分数据集的类型。" msgstr "" "It should be noted that the generated `bin` files need to be saved in one" " of the following directories: `cn`, `en`, `code`, `ja`, `ar`, or " "`kaoshi`, depending on the type of dataset." -#: ../../../usage.md:58 +#: ../../../usage.md:61 msgid "其中，`cn`表示中文数据集；`en`表示英文数据集；`code`表示代码数据集；`ja`表示日语数据集；`ar`表示阿拉伯语数据集；`kaoshi`表示考试数据集。" msgstr "" "Here, `cn` represents the Chinese dataset, `en` represents the English " @@ -151,22 +155,22 @@ msgstr "" " dataset, `ar` represents the Arabic dataset, and `kaoshi` represents the" " exam dataset." -#: ../../../usage.md:60 +#: ../../../usage.md:63 msgid "生成的bin文件的格式如下：" msgstr "The format of the generated `bin` files is as follows:" -#: ../../../usage.md:66 +#: ../../../usage.md:69 msgid "`bin`文件中的每一行均对应原始数据集中的每一个句子，表示每个句子的`token`（下文将用sequence指定）。" msgstr "" "Each line in the `bin` file corresponds to each sentence in the original " "dataset, representing the tokens of each sentence (referred to as " "sequence below)." -#: ../../../usage.md:68 +#: ../../../usage.md:71 msgid "生成的`meta`文件的格式如下：" msgstr "The format of the generated `meta` file is as follows:" -#: ../../../usage.md:72 +#: ../../../usage.md:75 msgid "" "在`meta`文件中，每个元组对应着`bin`文件中每一个`sequence`的元信息。其中，元组的第一个元素表示每个`sequence`在所有`sequence`中的`starting" " index`，第二个元素表示每个`sequence`中有多少个`tokens`。" @@ -176,7 +180,7 @@ msgstr "" "index` of each `sequence` among all `sequences`, and the second element " "indicates the number of `tokens` for each `sequence`." -#: ../../../usage.md:74 +#: ../../../usage.md:77 msgid "" "例如，对于第一个`sequence`，`starting index`为 0，有 11 " "个`tokens`；对于第二个`sequence`，由于第一个`sequence`转换为`string`后的长度为`89`，因此它的`starting" @@ -185,17 +189,17 @@ msgstr "" "For example, the first `sequence` starts at index 0 and has 16 `tokens`. " "The second `sequence` starts at index 110 and has 24 `tokens`." -#: ../../../usage.md:76 +#: ../../../usage.md:79 msgid "`json`和`jsonl`类型的文件的`bin`和`meta`文件格式和`txt`一致，此处不再赘叙。" msgstr "" "The `bin` and `meta` file formats for `json` and `jsonl` type files are " "the same as for `txt`, so we won't go over them here." -#: ../../../usage.md:78 -msgid "数据准备 （微调）" -msgstr "Data Preparation (Fine-tuning)" +#: ../../../usage.md:81 +msgid "微调" +msgstr "Fine-tuning" -#: ../../../usage.md:80 +#: ../../../usage.md:83 msgid "" "微调任务的数据集格式与预训练任务保持一致，生成的数据格式为一系列的`bin`和`meta`文件。以下以 Alpaca " "数据集为例，介绍微调的数据准备流程。" @@ -205,7 +209,7 @@ msgstr "" "the Alpaca dataset as an example to explain the data preparation process " "for fine-tuning." -#: ../../../usage.md:82 +#: ../../../usage.md:85 msgid "" "下载 [Alpaca 数据集](https://github.com/tatsu-" "lab/stanford_alpaca/blob/main/alpaca_data.json)" @@ -213,94 +217,94 @@ msgstr "" "Download the [Alpaca dataset](https://github.com/tatsu-" "lab/stanford_alpaca/blob/main/alpaca_data.json)." -#: ../../../usage.md:84 +#: ../../../usage.md:87 msgid "对 Alpaca 数据进行 tokenize，使用以下命令" msgstr "Tokenize the Alpaca dataset using the following command:" -#: ../../../usage.md:90 +#: ../../../usage.md:93 msgid "建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize" msgstr "" "It is recommended that users refer to alpaca_tokenizer.py to write new " "scripts to tokenize their own datasets" -#: ../../../usage.md:92 +#: ../../../usage.md:95 msgid "训练配置" msgstr "Training Configuration" -#: ../../../usage.md:94 +#: ../../../usage.md:97 msgid "以 7B Demo 的配置文件`configs/7B_sft.py`为例：" msgstr "" "Taking the configuration file `configs/7B_sft.py` for the 7B demo as an " "example," -#: ../../../usage.md:307 +#: ../../../usage.md:310 msgid "接下来将详细介绍启动一个模型训练所需要进行的数据、模型、并行和监控等相关的配置。" msgstr "" "let's discuss the data, model, parallel and monitoring configurations " "required to start a model training." -#: ../../../usage.md:309 +#: ../../../usage.md:312 msgid "数据配置" msgstr "Data Configuration" -#: ../../../usage.md:310 +#: ../../../usage.md:313 msgid "数据相关的关键参数配置及释义如下所示：" msgstr "Here are the key parameters and their explanations for data configuration:" -#: ../../../usage.md:325 +#: ../../../usage.md:328 msgid "![pack_into_one](./imgs/pack_into_one.png)" msgstr "" -#: ../../../usage.md:325 +#: ../../../usage.md:328 msgid "pack_into_one" msgstr "" -#: ../../../usage.md:328 +#: ../../../usage.md:331 msgid "目前支持传入数据集文件路径`train_folder`，且要求文件格式如下：" msgstr "" "Currently, it supports passing the dataset file path `train_folder`, and " "the file format is required to be as follows:" -#: ../../../usage.md:335 +#: ../../../usage.md:338 msgid "数据集的详细内容可参考``数据准备``模块相关的介绍。" msgstr "" "For detailed information about the dataset, please refer to the \"Data " "Preparation\" section." -#: ../../../usage.md:337 +#: ../../../usage.md:340 msgid "同时，也支持huggingface格式的数据集处理。" msgstr "" "Additionally, it supports processing of datasets in the Hugging Face " "format." -#: ../../../usage.md:339 +#: ../../../usage.md:342 msgid "train_folder设置为从huggingface上下载的本地数据集路径，如：\"/mnt/petrelfs/hf-TinyStories\"" msgstr "" "Set the train_folder to the local path of the dataset downloaded from " "Hugging Face, for example: \"/mnt/petrelfs/hf-TinyStories\"." -#: ../../../usage.md:341 +#: ../../../usage.md:344 msgid "在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如：" msgstr "" "In the data section, you need to add new fields for type and " "tokenizer_path to indicate that the dataset is in Hugging Face format and" " to specify the path of the tokenizer, for example:" -#: ../../../usage.md:359 +#: ../../../usage.md:362 msgid "模型配置" msgstr "Model Configuration" -#: ../../../usage.md:361 +#: ../../../usage.md:364 msgid "如果在启动训练时要加载模型 `checkpoint`，可进行如下相关配置：" msgstr "" "If you want to load a model checkpoint when starting the training, you " "can configure it as follows:" -#: ../../../usage.md:385 +#: ../../../usage.md:388 msgid "注意：" msgstr "Note:" -#: ../../../usage.md:386 +#: ../../../usage.md:389 msgid "" "路径若以 `local:` 为前缀，则存储在本地文件系统；若以 `boto3:` 为前缀，则存储在远程 oss " "上；若无前缀，为huggingface上可以直接下载的模型路径。" @@ -309,11 +313,11 @@ msgstr "" "local file system. If it starts with `boto3:`, it means the file is " "stored in the remote OSS." -#: ../../../usage.md:388 +#: ../../../usage.md:391 msgid "模型相关关键参数配置如下所示：" msgstr "The configuration for the model is as follows:" -#: ../../../usage.md:412 +#: ../../../usage.md:415 msgid "注意：用户可自定义模型类型名和模型结构，并配置相对应的模型参数。通过`internlm/model/registry.py`下的`model_initializer`对象进行模型初始化函数接口注册，在训练主函数`train.py`中初始化模型时，可通过`model_type`配置获取指定的模型初始化接口函数。" msgstr "" "Note: Users can customize the model type name and model structure, and " @@ -324,7 +328,7 @@ msgstr "" "interface function can be obtained through the `model_type` " "configuration." -#: ../../../usage.md:414 +#: ../../../usage.md:417 msgid "" "*如果基于 InternLM 7B继续训练，可以参考 " "[ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo) 中 " @@ -334,130 +338,130 @@ msgstr "" "OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-" "zoo) to download weights*." -#: ../../../usage.md:416 +#: ../../../usage.md:419 msgid "并行配置" msgstr "Parallel Configuration" -#: ../../../usage.md:418 +#: ../../../usage.md:421 msgid "训练并行配置样例如下：" msgstr "Training parallel configuration example:" -#: ../../../usage.md:427 +#: ../../../usage.md:430 msgid "zero1（字典）：" msgstr "zero1 (dict): " -#: ../../../usage.md:428 +#: ../../../usage.md:431 msgid "size: 整数" msgstr "size: int " -#: ../../../usage.md:429 +#: ../../../usage.md:432 msgid "当`zero1 <= 0`，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" msgstr "" "When `zero1 <= 0` , the size of the zero1 process group is equal to the " "size of the data parallel process group, so the optimizer state " "parameters will be split within the data parallel range." -#: ../../../usage.md:430 +#: ../../../usage.md:433 msgid "当`zero1 == 1`，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" msgstr "" "When `zero1 == 1`, zero1 is not used, and all data parallel groups retain" " the complete optimizer state parameters." -#: ../../../usage.md:431 +#: ../../../usage.md:434 msgid "当`zero1 > 1`且`zero1 <= data_parallel_world_size`，则 zero1 进程组是数据并行进程组的子集" msgstr "" "When `zero1 > 1` and `zero1 <= data_parallel_world_size`, the zero1 " "process group is a subset of the data parallel process group." -#: ../../../usage.md:432 +#: ../../../usage.md:435 msgid "fsdp: 布尔值，启用/禁用torch的完全分片数据并行，默认为False。" msgstr "" "fsdp: A boolean value that enables or disables fully sharded data " "parallelism in torch, with the default being False." -#: ../../../usage.md:433 +#: ../../../usage.md:436 msgid "tensor（字典）：" msgstr "tensor (dict): " -#: ../../../usage.md:434 +#: ../../../usage.md:437 msgid "size: 整数，张量并行的大小。" msgstr "size: int, size of tensor parallem" -#: ../../../usage.md:435 +#: ../../../usage.md:438 msgid "mode: 字符串，张量并行模式，应该是 ['mtp', 'msp', 'fsp', 'isp'] 中的一个，" msgstr "" "mode: string, tensor parallel mode, should be one of ['mtp', 'msp', " "'fsp', 'isp'] " -#: ../../../usage.md:436 +#: ../../../usage.md:439 msgid "默认为 'mtp'，意味着没有序列并行的纯Megatron张量并行。" msgstr "" "Default is 'mtp', which means there is no sequence parallelism, just pure" " tensor parallelism for Megatron." -#: ../../../usage.md:437 +#: ../../../usage.md:440 msgid "msp: 带序列并行的Megatron张量并行，序列并行大小 = 张量并行大小。" msgstr "" "msp: Megatron Tensor Parallelism with Sequence Parallelism, where the " "size of sequence parallelism is equal to the size of tensor parallelism." -#: ../../../usage.md:438 +#: ../../../usage.md:441 msgid "fsp: 通过flash-attn带序列并行的张量并行，序列并行大小 = 张量并行大小。" msgstr "" "fsp: Tensor Parallelism with Sequence Parallelism facilitated by flash-" "attn, where the size of sequence parallelism is equal to the size of " "tensor parallelism." -#: ../../../usage.md:439 +#: ../../../usage.md:442 msgid "isp: 定制的内部序列并行，不带张量并行，可以与权重并行一起使用。" msgstr "" "isp: Custom internal sequence parallelism, without tensor parallelism, " "which can be used in conjunction with weight parallelism." -#: ../../../usage.md:440 +#: ../../../usage.md:443 msgid "pipeline（字典）：" msgstr "pipeline: pipeline parallel strategy" -#: ../../../usage.md:441 +#: ../../../usage.md:444 msgid "size: 整数，流水线并行的大小。" msgstr "size: int, size of pipeline parallel" -#: ../../../usage.md:442 +#: ../../../usage.md:445 msgid "interleaved_overlap: 布尔值，启用/禁用在使用交错流水线调度器时的通信重叠，默认为False。" msgstr "" "interleaved_overlap: A boolean value that enables or disables " "communication overlapping when using an interleaved pipeline scheduler, " "with the default being False." -#: ../../../usage.md:443 +#: ../../../usage.md:446 msgid "weight（字典）：" msgstr "weight (dict):" -#: ../../../usage.md:444 +#: ../../../usage.md:447 msgid "size: 整数，权重并行的大小。" msgstr "size: int, size of weight parallel" -#: ../../../usage.md:445 +#: ../../../usage.md:448 msgid "overlap: 布尔值，启用/禁用all_gather/reduce_scatter通信重叠，默认为False。" msgstr "" "overlap: bool, enable/disable all_gather/reduce_scatter communication " "overlap, default is False" -#: ../../../usage.md:446 +#: ../../../usage.md:449 msgid "memory_pool: 布尔值，启用/禁用内存池，默认为False。" msgstr "memory_pool: bool, enable/disable memory pool, default is False" -#: ../../../usage.md:448 +#: ../../../usage.md:451 msgid "注意：`数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小`" msgstr "" "Note: `Data parallel size = Total number of GPUs / Pipeline parallel size" " / Tensor parallel size`" -#: ../../../usage.md:450 +#: ../../../usage.md:453 msgid "启动训练" msgstr "Start Training" -#: ../../../usage.md:452 +#: ../../../usage.md:455 msgid "完成了以上数据集准备和相关训练配置后，可启动 Demo 训练。接下来分别以 slurm 和 torch 环境为例，介绍训练启动方式。" msgstr "" "After completing the data preparation and relevant training " @@ -465,41 +469,41 @@ msgstr "" "following examples demonstrate how to start the training in both slurm " "and torch environments." -#: ../../../usage.md:454 ../../../usage.md:489 +#: ../../../usage.md:457 ../../../usage.md:492 msgid "若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on slurm with 16 GPUs across " "multiple nodes, use the following command:" -#: ../../../usage.md:459 +#: ../../../usage.md:462 msgid "若在 torch 上启动分布式运行环境，单节点 8 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on torch with 8 GPUs on a " "single node, use the following command:" -#: ../../../usage.md:464 +#: ../../../usage.md:467 msgid "运行结果" msgstr "Training Results" -#: ../../../usage.md:466 +#: ../../../usage.md:469 msgid "以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下：" msgstr "" "Taking the configuration of the demo training on a single machine with 8 " "GPUs on slurm as an example, the training result log is shown below:" -#: ../../../usage.md:487 +#: ../../../usage.md:490 msgid "加载训练的checkpoint并生成" msgstr "Load the training checkpoint and generate." -#: ../../../usage.md:494 +#: ../../../usage.md:497 msgid "在配置文件中添加`generation`配置" msgstr "Add generation configuration to the configuration file." -#: ../../../usage.md:512 +#: ../../../usage.md:515 msgid "长文本生成" msgstr "Long Text Generation" -#: ../../../usage.md:514 +#: ../../../usage.md:517 msgid "" "在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K " "的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK " @@ -511,7 +515,7 @@ msgstr "" "the use of Dynamic NTK RoPE in models formatted in both Hugging Face " "format and InternLM's native format." -#: ../../../usage.md:517 +#: ../../../usage.md:520 msgid "" "对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 " "`config.json` 中的 `rotary.type` 修改为 `origin`；" @@ -520,7 +524,7 @@ msgstr "" "by default. If users wish to disable this behavior, they can modify " "`rotary.type` in the `config.json` file to `origin`." -#: ../../../usage.md:518 +#: ../../../usage.md:521 msgid "" "对于 InternLM " "本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。" @@ -529,7 +533,7 @@ msgstr "" "this behavior by adding use_dynamic_ntk_rope=True to the configuration " "dictionary when initializing the model." -#: ../../../usage.md:520 +#: ../../../usage.md:523 msgid "" "用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE " "是如何生效的。例如文件[长文本示例](../../aux_materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用" @@ -543,23 +547,23 @@ msgstr "" "text. However, after applying Dynamic NTK RoPE, the response from the " "InternLM Chat 7B v1.1 model is as follows:" -#: ../../../usage.md:523 +#: ../../../usage.md:526 msgid "![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png)" msgstr "" -#: ../../../usage.md:523 +#: ../../../usage.md:526 msgid "dynamic_ntk_answer" msgstr "" -#: ../../../usage.md:525 +#: ../../../usage.md:528 msgid "关于 Dyanmic NTK 的原理，详细请参考" msgstr "Regarding the principle of Dyanmic NTK, please refer to" -#: ../../../usage.md:526 +#: ../../../usage.md:529 msgid "[dynamically_scaled_rope_further_increases](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases)" msgstr "" -#: ../../../usage.md:527 +#: ../../../usage.md:530 msgid "[https://kexue.fm/archives/9675](https://kexue.fm/archives/9675)" msgstr "" @@ -615,3 +619,6 @@ msgstr "" #~ "在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如：" #~ msgstr "" +#~ msgid "数据准备 （预训练）" +#~ msgstr "Dataset Preparation (Pre-training)" + diff --git a/doc/en/install.md b/doc/en/install.md index eae4a12c6..5d4c983fa 100644 --- a/doc/en/install.md +++ b/doc/en/install.md @@ -23,13 +23,23 @@ pip install InternEvo ``` Install flash-attention (version v2.2.1): + +If you need to use flash-attention to accelerate training, and it is supported in your environment, install as follows: ```bash pip install flash-attn==2.2.1 ``` Install Apex (version 23.05): + Apex is an optional package; If you choose to install it, follow the instructions in Install through source code. +Install megablocks（version 0.3.2）: + +If the model you are training is MoE releated, install as follows: +```bash +pip install git+https://github.com/databricks/megablocks@v0.3.2 # MOE相关 +``` + ### Install through source code #### Required Packages The required packages and corresponding version are shown as follows: @@ -85,10 +95,10 @@ pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation - cd ../../ ``` -### Additional Installation +Install megablocks（version 0.3.2）: ```bash -pip install git+https://github.com/databricks/megablocks@v0.3.2 # MOE need -``` +pip install git+https://github.com/databricks/megablocks@v0.3.2 # MOE releated +``` ### Environment Image Users can use the provided dockerfile combined with docker.Makefile to build their own images, or obtain images with InternEvo runtime environment installed from https://hub.docker.com/r/internlm/internevo/tags. @@ -122,17 +132,17 @@ The default directory in the container is `/InternEvo`, please start training ac torchrun --nproc_per_node=8 --nnodes=1 train.py --config configs/7B_sft.py --launcher torch ``` -## Environment Installation (NPU) +### NPU Environment Installation For machines with NPU, the version of the installation environment can refer to that of GPU. Use Ascend's torch_npu instead of torch on NPU machines. Additionally, Flash-Attention and Apex are no longer supported for installation on NPU. The corresponding functionalities have been internally implemented in the InternEvo codebase. The following tutorial is only for installing torch_npu. Official documentation for torch_npu: https://gitee.com/ascend/pytorch -### Example Installation of Environment +#### NPU Environment Preparation - Linux OS - torch_npu: v2.1.0-6.0.rc1 - NPU card: 910B -#### Installing torch_run +##### Installing torch_run Refer to the documentation: https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/ You can try installing according to the methods in the documentation or download the specified version of torch_npu from https://gitee.com/ascend/pytorch/releases for installation, as shown below: diff --git a/doc/en/usage.md b/doc/en/usage.md index b701a5bbc..416fbda66 100644 --- a/doc/en/usage.md +++ b/doc/en/usage.md @@ -6,9 +6,11 @@ To start a demo model training, you need to prepare three things: **installation Please refer to the [installation guide](./install.md) for instructions on how to install the necessary dependencies. -### Dataset Preparation (Pre-training) +### Dataset Preparation -#### Using Hugging Face Format Dataset +#### Pre-training + +##### Using Hugging Face Format Dataset If you are using a Hugging Face dataset, you will first need to download the dataset and the required tokenizer to your local machine. @@ -34,7 +36,7 @@ data = dict( The type defaults to "tokenized", but here it needs to be changed to "streaming". Also, you need to specify the `tokenizer_path`. If you are using the dataset after tokenization as described below, you do not need to set this field. `TRAIN_FOLDER` specifies the local path of the dataset. -#### Using a Dataset After Tokenization +##### Using a Dataset After Tokenization The dataset for the InternEvo training task includes a series of `bin` and `meta` files. A `tokenizer` is used to generate the training dataset from the original text files. The tokenizer model is imported by specifying the model parameter path in `tools/tokenizer.py`. Currently, `tokenizer_internlm.model` is provided to generate tokens. If you want to use a different model, you can directly modify the model parameter path in `tokenizer.py`. @@ -86,7 +88,7 @@ For example, the first `sequence` starts at index 0 and has 16 `tokens`. The sec The `bin` and `meta` file formats for `json` and `jsonl` type files are the same as for `txt`, so we won't go over them here. -### Data Preparation (Fine-tuning) +#### Fine-tuning The data format for fine-tuning tasks is the same as for pre-training tasks, which consists of a series of `bin` and `meta` files. Let's take the Alpaca dataset as an example to explain the data preparation process for fine-tuning. diff --git a/doc/install.md b/doc/install.md index f14934726..d48fc4a04 100644 --- a/doc/install.md +++ b/doc/install.md @@ -23,13 +23,23 @@ pip install InternEvo ``` 安装 flash-attention (version v2.2.1)： + +如果需要使用flash-attention加速训练，且环境中支持，安装方式如下： ```bash pip install flash-attn==2.2.1 ``` 安装 Apex (version 23.05)： + apex为非必须安装包，如果安装，参考下述源码方式安装。 +安装megablocks（version 0.3.2）: + +如果是MoE相关模型，需要安装。 +```bash +pip install git+https://github.com/databricks/megablocks@v0.3.2 # MOE相关 +``` + ### 源码方式安装 #### 依赖包 首先，需要安装的依赖包及对应版本列表如下： @@ -85,10 +95,10 @@ pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation - cd ../../ ``` -### 额外安装 +安装megablocks（version 0.3.2）: ```bash pip install git+https://github.com/databricks/megablocks@v0.3.2 # MOE相关 -``` +``` ### 环境镜像 用户可以使用提供的 dockerfile 结合 docker.Makefile 来构建自己的镜像，或者也可以从 https://hub.docker.com/r/internlm/internevo/tags 获取安装了 InternEvo 运行环境的镜像。 @@ -122,18 +132,18 @@ docker run --gpus all -it -m 500g --cap-add=SYS_PTRACE --cap-add=IPC_LOCK --shm- torchrun --nproc_per_node=8 --nnodes=1 train.py --config configs/7B_sft.py --launcher torch ``` -## 环境安装（NPU） +### NPU环境安装 在搭载NPU的机器上安装环境的版本可参考GPU，在NPU上使用昇腾torch_npu代替torch，同时Flash-Attention和Apex不再支持安装，相应功能已由InternEvo代码内部实现。以下教程仅为torch_npu安装。 torch_npu官方文档：https://gitee.com/ascend/pytorch -### 环境安装样例 +#### NPU环境准备 - Linux OS - torch_npu: v2.1.0-6.0.rc1 - NPU显卡：910B -#### 安装torch_run +##### 安装torch_run 参考文档：https://gitee.com/ascend/pytorch/tree/v2.1.0-6.0.rc1/ diff --git a/doc/usage.md b/doc/usage.md index 98852100d..96d19887b 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1,12 +1,15 @@ -# 使用教程 +## 使用教程 启动一个 Demo 模型训练，需要进行三项准备，**安装**，**数据集准备**和**模型训练配置**。接下来，首先会介绍数据准备相关的操作，再简要描述模型训练配置相关的内容。 -## 安装 +### 安装 请参考[安装文档](./install.md)进行安装。 -## 数据准备 （预训练） -### 使用huggingface格式数据集 +### 数据准备 + +#### 预训练 + +##### 使用huggingface格式数据集 如果使用huggingface数据集，需要先将数据集和需要使用的tokenizer下载到本地。 @@ -28,7 +31,7 @@ data = dict( ``` type默认为"tokenized"，这里需要改为"streaming"类型。同时需要指定`tokenizer_path`, 如果使用下述tokenized之后的数据集，则不需要设置该字段。`TRAIN_FOLDER`指定本地数据集路径。 -### 使用tokenized之后数据集 +##### 使用tokenized之后数据集 InternEvo训练任务的数据集包括一系列的`bin`和`meta`文件。使用`tokenizer`从原始文本文件生成训练用数据集。通过在`tools/tokenizer.py`中指定模型参数路径的方式来导入tokenizer模型。目前提供`V7_sft.model`来生成tokens。若想使用不同的模型，可直接修改`tokernizer.py`中的模型参数路径。 @@ -74,7 +77,7 @@ $ python tools/tokenizer.py --text_input_path raw_data.txt --bin_output_path cn/ `json`和`jsonl`类型的文件的`bin`和`meta`文件格式和`txt`一致，此处不再赘叙。 -## 数据准备 （微调） +#### 微调 微调任务的数据集格式与预训练任务保持一致，生成的数据格式为一系列的`bin`和`meta`文件。以下以 Alpaca 数据集为例，介绍微调的数据准备流程。 @@ -88,7 +91,7 @@ python tools/alpaca_tokenizer.py /path/to/alpaca_dataset /path/to/output_dataset 建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize -## 训练配置 +### 训练配置 以 7B Demo 的配置文件`configs/7B_sft.py`为例： ```python @@ -305,7 +308,7 @@ monitor = dict( ``` 接下来将详细介绍启动一个模型训练所需要进行的数据、模型、并行和监控等相关的配置。 -### 数据配置 +#### 数据配置 数据相关的关键参数配置及释义如下所示： ```python TRAIN_FOLDER = "/path/to/dataset" @@ -355,7 +358,7 @@ data = dict( ) ``` -### 模型配置 +#### 模型配置 如果在启动训练时要加载模型 `checkpoint`，可进行如下相关配置： ```python @@ -412,7 +415,7 @@ model = dict( *如果基于 InternLM 7B继续训练，可以参考 [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo) 中 OpenXLab 链接下载权重* -### 并行配置 +#### 并行配置 训练并行配置样例如下： ```python @@ -446,7 +449,7 @@ parallel = dict( 注意：`数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小` -## 启动训练 +### 启动训练 完成了以上数据集准备和相关训练配置后，可启动 Demo 训练。接下来分别以 slurm 和 torch 环境为例，介绍训练启动方式。 @@ -460,7 +463,7 @@ $ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python trai $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" ``` -## 运行结果 +### 运行结果 以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下： ```bash @@ -483,7 +486,7 @@ $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py - 2023-07-07 12:29:16,994 INFO train.py:323 in record_current_batch_training_metrics -- tflops=189.3109313713174,step=5,loss=9.822169303894043,tgs (tokens/gpu/second)=4262.67,lr=1.4000000000000001e-06,loss_scale=65536.0,grad_norm=47.10386835560855,micro_num=4,num_consumed_tokens=786432,inf_nan_skip_batches=0,num_samples_in_batch=17,largest_length=2048,largest_batch=6,smallest_batch=3,adam_beta2=0.95,fwd_bwd_time=3.69 ``` -## 加载训练的checkpoint并生成 +### 加载训练的checkpoint并生成 若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示： ```bash @@ -508,7 +511,7 @@ generation = dict( ) ``` -## 长文本生成 +### 长文本生成 在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K 的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK RoPE。 From f10e3ccd6f324410f6875e211f7e377ac75d566a Mon Sep 17 00:00:00 2001 From: sallyjunjun Date: Fri, 30 Aug 2024 19:25:00 +0800 Subject: [PATCH 4/5] add train script instruction and fix doc --- .../locales/en/LC_MESSAGES/data_flow.po | 217 +++++++++-------- .../locales/en/LC_MESSAGES/example/index.po | 9 +- doc/code-docs/locales/en/LC_MESSAGES/index.po | 35 ++- .../locales/en/LC_MESSAGES/initialize.po | 135 +++++++---- .../locales/en/LC_MESSAGES/mixed_precision.po | 54 +++-- .../locales/en/LC_MESSAGES/monitor.po | 49 ++-- .../locales/en/LC_MESSAGES/parallel.po | 199 +++++++-------- .../locales/en/LC_MESSAGES/profiler.po | 16 +- .../locales/en/LC_MESSAGES/training.po | 229 ++++++++++++------ doc/code-docs/locales/en/LC_MESSAGES/usage.po | 40 +-- doc/code-docs/source/checkpoint.rst | 6 +- doc/code-docs/source/data_flow.rst | 29 +-- doc/code-docs/source/example/index.rst | 2 +- doc/code-docs/source/index.rst | 7 +- doc/code-docs/source/initialize.rst | 150 ++++++------ doc/code-docs/source/mixed_precision.rst | 10 +- doc/code-docs/source/monitor.rst | 12 +- doc/code-docs/source/parallel.rst | 38 +-- doc/code-docs/source/profiler.rst | 4 +- doc/code-docs/source/training.rst | 69 +++++- doc/en/usage.md | 2 + doc/usage.md | 2 + 22 files changed, 776 insertions(+), 538 deletions(-) diff --git a/doc/code-docs/locales/en/LC_MESSAGES/data_flow.po b/doc/code-docs/locales/en/LC_MESSAGES/data_flow.po index 84164214f..8ada8c824 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/data_flow.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/data_flow.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternEvo \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-26 15:00+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -19,8 +19,8 @@ msgstr "" "Generated-By: Babel 2.15.0\n" #: ../../source/data_flow.rst:2 -msgid "数据格式" -msgstr "Data Format" +msgid "数据加载与流程" +msgstr "Data Load and Procedure" #: ../../source/data_flow.rst:7 msgid "Dataloader加载数据" @@ -51,7 +51,7 @@ msgstr "" "construction logic is exemplified as follows:" #: ../../source/data_flow.rst:15 ../../source/data_flow.rst:40 -#: ../../source/data_flow.rst:208 +#: ../../source/data_flow.rst:206 msgid "假设 ``micro_bsz`` 为2， ``SEQ_LEN`` 为8，输入数据格式如下：" msgstr "" "Assuming ``micro_bsz`` is set to 2 and ``SEQ_LEN`` is set to 8, the input" @@ -125,7 +125,7 @@ msgstr "" "Following the aforementioned rules, the format of the data after packing " "is as follows:" -#: ../../source/data_flow.rst:65 ../../source/data_flow.rst:221 +#: ../../source/data_flow.rst:65 ../../source/data_flow.rst:219 msgid "``label`` 的值如下：" msgstr "the value of ``label`` is as follows:" @@ -154,7 +154,7 @@ msgstr "" msgid "从Dataloader中取出数据" msgstr "Data Fetching" -#: ../../source/data_flow.rst:88 +#: ../../source/data_flow.rst:87 msgid "" "这里 ``batch_data`` 的类型为 ``list`` ，其中包含两个元素，第一个元素为 ``dict`` 类型的数据 ``data`` " "，第二个元素为 ``torch.Tensor`` 类型的标签 ``label`` 。" @@ -164,7 +164,7 @@ msgstr "" "and the second element is a ``torch.Tensor`` type of label named " "``label`` ." -#: ../../source/data_flow.rst:90 +#: ../../source/data_flow.rst:89 msgid "" "其中，第一个元素 ``data`` 包含 ``input_ids`` 、 ``cu_seqlens`` 、 ``indexes`` " "三个字段，其类型及形状分别为：" @@ -173,11 +173,11 @@ msgstr "" "contains three fields: ``input_ids`` , ``cu_seqlens`` , and ``indexes`` ." " The types and shapes of these fields are as follows:" -#: ../../source/data_flow.rst:98 +#: ../../source/data_flow.rst:97 msgid "第二个元素 ``label`` 的形状为：" msgstr "The shape of the second element ``label`` is:" -#: ../../source/data_flow.rst:104 +#: ../../source/data_flow.rst:103 msgid "" "``micro_num`` 在 ``config.py`` 配置文件中设置，为梯度累计的大小，即经过 ``micro_num`` 次 " "``forward`` + ``backward`` 之后，进行梯度更新。 ``micro_bsz * SEQ_LEN`` 为 ``pack " @@ -191,26 +191,26 @@ msgstr "" "of length ``micro_bsz * SEQ_LEN`` , thereby enhancing the training " "efficiency." -#: ../../source/data_flow.rst:107 +#: ../../source/data_flow.rst:106 msgid "举例： 假设 ``micro_num`` 为2， ``micro_bsz`` 为2， ``SEQ_LEN`` 为8" msgstr "" "For example, assuming micro_num is set to 2, micro_bsz is 2, and SEQ_LEN " "is 8" -#: ../../source/data_flow.rst:116 +#: ../../source/data_flow.rst:115 msgid "其中第一个batch由长度分别为4,7,5的子句拼接而成，第二个batch由长度分别为3,5的子句拼接而成，则：" msgstr "" "In the example, the first batch is composed of sub-sentences with lengths" " of 4, 7, and 5, respectively, and the second batch is composed of sub-" "sentences with lengths of 3 and 5, respectively. Then:" -#: ../../source/data_flow.rst:124 +#: ../../source/data_flow.rst:123 msgid "其中，每相邻两个数字的差值，为当前子句的长度。" msgstr "" "In this case, the difference between each pair of adjacent numbers " "represents the length of the current sub-sentence." -#: ../../source/data_flow.rst:132 +#: ../../source/data_flow.rst:131 msgid "其中，每一个数字表示了token在当前子句中的位置。如果最后一句存在padding，则indexes依然按照从0递增的方式直至padding结束。" msgstr "" "In this context, each number represents the position of a token within " @@ -218,15 +218,15 @@ msgstr "" "sentence, the indexes still increment from 0 up to and including the end " "of the padding." -#: ../../source/data_flow.rst:140 +#: ../../source/data_flow.rst:139 msgid "这里为对应的label的数值。" msgstr "Here are the corresponding values for the label." -#: ../../source/data_flow.rst:144 +#: ../../source/data_flow.rst:143 msgid "处理数据" msgstr "Data Processing" -#: ../../source/data_flow.rst:150 +#: ../../source/data_flow.rst:148 msgid "" "首先，通过 ``_load_micro_batch`` 函数，将 ``data`` 和 ``label`` 中数据的第一个维度 " "``micro_num`` 转化为1，并通过更新 ``offset`` 的值，依次获取每个微批次的数据。" @@ -236,13 +236,13 @@ msgstr "" " 1. By updating the value of offset, data for each micro-batch is " "retrieved sequentially." -#: ../../source/data_flow.rst:152 +#: ../../source/data_flow.rst:150 msgid "其次，通过注册 ``data_process_func`` 对数据做进一步处理。" msgstr "" "Secondly, further processing of the data is carried out by registering a " "``data_process_func`` ." -#: ../../source/data_flow.rst:154 +#: ../../source/data_flow.rst:152 msgid "" "当 ``config.py`` 中设置 ``use_packed_dataset`` 为 ``True`` 时， " "``data_process_func`` 中的流程如下:" @@ -250,7 +250,7 @@ msgstr "" "When ``use_packed_dataset`` is set to True in the config.py, the process " "within the ``data_process_func`` is as follows:" -#: ../../source/data_flow.rst:156 +#: ../../source/data_flow.rst:154 msgid "" "通过 ``packed_data_normalizer`` 函数，对 ``data['indexes']`` 和 " "``data['cu_seqlens']`` 做降维处理，去掉size为1的第一维，并通过 ``data['cu_seqlens']`` " @@ -262,14 +262,14 @@ msgstr "" "individual sub-sentences is calculated using the values in " "data['cu_seqlens'], and this value is recorded in data['max_seqlen']." -#: ../../source/data_flow.rst:158 +#: ../../source/data_flow.rst:156 msgid "按照上述举例，假设加载第一个批次的数据，经过 ``_load_accum_batch`` 处理后的 ``data`` 和 ``label`` 如下：" msgstr "" "Following the example provided, assuming the first batch of data is " "loaded, the data and label after processing by ``_load_accum_batch`` " "would be as follows:" -#: ../../source/data_flow.rst:174 +#: ../../source/data_flow.rst:172 msgid "" "如果张量并行模式为 ``isp`` ，且tp size（即序列化并行大小）大于1，则会在 ``data_process_func`` 中注册 " "``split_data_sequence_parallel`` 函数，对数据的 ``sequence`` 维度进行切分。" @@ -279,7 +279,7 @@ msgstr "" "``split_data_sequence_parallel`` function will be registered within the " "``data_process_func`` to split the data along the sequence dimension." -#: ../../source/data_flow.rst:176 +#: ../../source/data_flow.rst:174 msgid "" "假设tp size为2，则对上述数据 ``data['input_ids']`` 、 ``data['indexes']`` 和 " "``label`` 切分之后的结果如下：" @@ -288,15 +288,15 @@ msgstr "" "data data['input_ids'], data['indexes'], and label using the " "``split_data_sequence_parallel`` function would be as follows:" -#: ../../source/data_flow.rst:178 ../../source/data_flow.rst:243 +#: ../../source/data_flow.rst:176 ../../source/data_flow.rst:241 msgid "tp rank0 中的数据：" msgstr "Data in tp rank0" -#: ../../source/data_flow.rst:189 ../../source/data_flow.rst:255 +#: ../../source/data_flow.rst:187 ../../source/data_flow.rst:253 msgid "tp rank1 中的数据：" msgstr "Data in tp rank1" -#: ../../source/data_flow.rst:200 +#: ../../source/data_flow.rst:198 msgid "" "当 ``config.py`` 中设置 ``use_packed_dataset`` 为 ``False`` 时， " "``data_process_func`` 中的流程如下:" @@ -304,7 +304,7 @@ msgstr "" "When ``use_packed_dataset`` is set to False in the config.py, the process" " within the ``data_process_func`` is as follows:" -#: ../../source/data_flow.rst:202 +#: ../../source/data_flow.rst:200 msgid "" "通过 ``unpack_data`` 函数对数据做unpack处理，将 ``data[\"input_ids\"]`` 和 ``label`` " "的数据恢复到unpack的格式，并从data中去除掉\"cu_seqlens\"和\"indexes\"字段。" @@ -314,7 +314,7 @@ msgstr "" "format, and removing the \"cu_seqlens\" and \"indexes\" fields from the " "data." -#: ../../source/data_flow.rst:204 +#: ../../source/data_flow.rst:202 msgid "" "unpack之后 ``data[\"input_ids\"]`` 和 ``label`` 的形状为 " "``torch.Size([micro_bsz, SEQ_LEN])`` 。" @@ -322,21 +322,21 @@ msgstr "" "After unpacking, the shapes of data[\"input_ids\"] and label are " "torch.Size([micro_bsz, SEQ_LEN])." -#: ../../source/data_flow.rst:206 +#: ../../source/data_flow.rst:204 msgid "按照上述数据举例：" msgstr "Following the example of the data provided:" -#: ../../source/data_flow.rst:215 +#: ../../source/data_flow.rst:213 msgid "pack之后的数据格式如下：" msgstr "The packed data format is as follows:" -#: ../../source/data_flow.rst:227 +#: ../../source/data_flow.rst:225 msgid "经过 ``unpack_data`` 处理之后， ``data[\"input_ids\"]`` 和 ``label`` 分别如下：" msgstr "" "After processing with unpack_data, data[\"input_ids\"] and label are as " "follows:" -#: ../../source/data_flow.rst:239 +#: ../../source/data_flow.rst:237 msgid "" "如果 ``tp`` 并行模式为 ``isp`` ，且tp size（即序列化并行大小）大于1，则会在 ``data_process_func`` " "中注册 ``split_data_sequence_parallel`` 函数，对数据的 ``sequence`` 维度进行切分。" @@ -346,72 +346,70 @@ msgstr "" "``split_data_sequence_parallel`` function will be registered within the " "``data_process_func`` to split the data along the sequence dimension." -#: ../../source/data_flow.rst:241 +#: ../../source/data_flow.rst:239 msgid "假设tp size为2，则对上述数据 ``data['input_ids']`` 和 ``label`` 切分之后的结果如下：" msgstr "" "Assuming the tp size is 2, the result of splitting the aforementioned " "data data['input_ids'] and label would be as follows:" -#: ../../source/data_flow.rst:269 +#: ../../source/data_flow.rst:267 msgid "Forward过程数据格式" msgstr "During the Forward process, the data format is:" -#: ../../source/data_flow.rst:271 +#: ../../source/data_flow.rst:269 msgid "以internlm2模型为例，详细介绍一下整个模型在不同并行模式下的权重情况，以及在运行过程中数据的流动过程。" msgstr "" "Using the internlm2 model as an example, I will detail the weight " "situation of the entire model under different parallelism modes and the " "flow of data during operation." -#: ../../source/data_flow.rst:273 +#: ../../source/data_flow.rst:271 msgid "首先，介绍模型在不同并行模式下，权重切分的过程。" msgstr "" "Firstly, the process of weight partitioning in the model under different " "parallel modes is introduced." -#: ../../source/data_flow.rst:276 +#: ../../source/data_flow.rst:274 msgid "ISP并行模式下权重切分" msgstr "weight partitioning in ISP parallel mode" -#: ../../source/data_flow.rst:278 +#: ../../source/data_flow.rst:275 msgid "" -"ISP并行的具体原理请参见： `并行训练 " -"`_" +"ISP并行的具体原理请参见： `并行训练 `_" msgstr "" "For the specific principles of ISP parallelism, please refer " -"to:`_" +"to:`_" -#: ../../source/data_flow.rst:280 +#: ../../source/data_flow.rst:277 msgid "internlm2模型中，涉及weight切分的参数为：\"wqkv\"、\"wo\"、\"w1\"、\"w2\"、\"w3\"、\"output\"，通过new_linear函数进行切分。" msgstr "" "In the internlm2 model, the parameters that involve weight partitioning " "are: \"wqkv\", \"wo\", \"w1\", \"w2\", \"w3\", \"output\". These are " "partitioned using the new_linear function." -#: ../../source/data_flow.rst:282 +#: ../../source/data_flow.rst:279 msgid "假设配置文件中设置的weight并行大小为 ``wp_size`` ，初始化之后的模型结构及权重如下：" msgstr "" "Assuming the configuration file sets the weight parallelism size to " "wp_size, the model structure and weights after initialization are as " "follows:" -#: ../../source/data_flow.rst:321 +#: ../../source/data_flow.rst:318 msgid "MTP/MSP/FSP并行模式下权重切分" msgstr "weight partitioning in MTP/MSP/FSP parallel mode" -#: ../../source/data_flow.rst:323 +#: ../../source/data_flow.rst:319 msgid "" -"``MTP/MSP/FSP`` 并行的具体原理请参见： `并行训练 " -"`_" +"``MTP/MSP/FSP`` 并行的具体原理请参见： `并行训练 `_" msgstr "" "For the specific principles of MTP/MSP/FSP parallelism, please refer " -"to:`_" +"to:`_" -#: ../../source/data_flow.rst:325 +#: ../../source/data_flow.rst:321 msgid "" "与 ``ISP`` 并行模式相比， ``MSP`` 并行切分权重的参数是一样的，但是切分的方式不同，在 ``ISP`` " "并行模式中，所有切分参数采用列切方式，而 ``MSP`` 并行模式中，\"wo\"和\"w2\"参数采用行切方式进行切分。" @@ -422,7 +420,7 @@ msgstr "" "partitioning, while in the MSP parallel mode, the \"wo\" and \"w2\" " "parameters are partitioned using row-wise partitioning." -#: ../../source/data_flow.rst:327 +#: ../../source/data_flow.rst:323 msgid "" "假设配置文件中设置的tensor并行大小为 ``tp_size`` ，初始化之后的模型结构及权重与ISP中列出的权重结果基本一致，ISP模式中的 " "``wp_size`` 对应MSP模式下的 ``tp_size`` ，有差异的\"wo\"和\"w2\"参数如下：" @@ -433,66 +431,66 @@ msgstr "" " corresponds to tp_size in MSP mode. The parameters \"wo\" and \"w2\" " "that differ are as follows:" -#: ../../source/data_flow.rst:358 +#: ../../source/data_flow.rst:354 msgid "Forward整体流程" msgstr "Forward Procedure" -#: ../../source/data_flow.rst:360 +#: ../../source/data_flow.rst:355 msgid "internlm2模型中，forward整体流程如下图所示：" msgstr "" "In the internlm2 model, the overall forward process is shown in the " "figure below:" -#: ../../source/data_flow.rst:366 +#: ../../source/data_flow.rst:361 msgid "下面介绍不同并行模式下，数据在上图forward流程过程中的变化过程。" msgstr "" "Below is an introduction to the changes in data during the forward " "process in the diagram above, under different parallel modes." -#: ../../source/data_flow.rst:369 +#: ../../source/data_flow.rst:364 msgid "ISP并行模式下数据流程" msgstr "Data Procedure in ISP parallelism" -#: ../../source/data_flow.rst:371 +#: ../../source/data_flow.rst:365 msgid "假设配置文件中设置的tensor并行大小为 ``sp_size`` （在ISP模式下，张量并行大小即为序列化并行大小）" msgstr "" "Assuming the configuration file sets the tensor parallelism size to " "sp_size (in ISP mode, the tensor parallelism size is the same as the " "sequence parallelism size)." -#: ../../source/data_flow.rst:373 +#: ../../source/data_flow.rst:367 msgid "展开介绍每一步计算过程中，数据维度的变化情况。" msgstr "" "Expanding on the introduction, here is a description of the changes in " "data dimensions at each step of the computation process:" -#: ../../source/data_flow.rst:377 ../../source/data_flow.rst:526 +#: ../../source/data_flow.rst:371 ../../source/data_flow.rst:511 msgid "tok_embeddings计算过程" msgstr "tok_embeddings Calculation Procedure" -#: ../../source/data_flow.rst:379 +#: ../../source/data_flow.rst:372 msgid "在embedding的计算过程中，对数据的seq_len维度做了切分。" msgstr "" "During the embedding computation process, the seq_len dimension of the " "data is partitioned." -#: ../../source/data_flow.rst:381 +#: ../../source/data_flow.rst:374 msgid "输入参数及权重：" msgstr "input parameters and weight:" -#: ../../source/data_flow.rst:391 +#: ../../source/data_flow.rst:384 msgid "输出结果：" msgstr "output result:" -#: ../../source/data_flow.rst:400 ../../source/data_flow.rst:554 +#: ../../source/data_flow.rst:393 ../../source/data_flow.rst:538 msgid "attention计算过程" msgstr "attention Calculation Procedure" -#: ../../source/data_flow.rst:403 ../../source/data_flow.rst:561 +#: ../../source/data_flow.rst:395 ../../source/data_flow.rst:544 msgid "qkv准备" msgstr "qkv preparation" -#: ../../source/data_flow.rst:409 +#: ../../source/data_flow.rst:400 msgid "" "这里计算过程，会通过 ``weight_hook`` 对之前被权重并行切分的权重做 ``All-Gather`` 操作，最终输出结果 " "``qkv`` 的最后一个维度为 ``self.wqkv`` 中 ``out_features`` 经过 ``All-Gather`` " @@ -504,7 +502,7 @@ msgstr "" "dimension as the out_features dimension from self.wqkv after the All-" "Gather operation." -#: ../../source/data_flow.rst:411 +#: ../../source/data_flow.rst:402 msgid "" "注：后续所有通过new_linear函数创建的被切分过weight的权重，在forward过程中计算都会通过 ``weight_hook`` 做 " "``All-Gather`` 操作。" @@ -513,7 +511,7 @@ msgstr "" "and used in subsequent computations will undergo an All-Gather operation " "via weight_hook during the forward process." -#: ../../source/data_flow.rst:418 +#: ../../source/data_flow.rst:409 msgid "" "之后将qkv拆分为 ``[batch_size, seq_len, num_head, group_size, head_dim]`` " "维度，并分别计算q、k、v的值：" @@ -522,28 +520,28 @@ msgstr "" "num_head, group_size, head_dim], and the values for q (query), k (key), " "and v (value) are calculated respectively:" -#: ../../source/data_flow.rst:431 ../../source/data_flow.rst:587 +#: ../../source/data_flow.rst:422 ../../source/data_flow.rst:569 msgid "之后，将kv的值组合在一起，以便进行后续的attention计算：" msgstr "" "Afterward, the values of k and v are combined for subsequent attention " "computation." -#: ../../source/data_flow.rst:439 ../../source/data_flow.rst:595 +#: ../../source/data_flow.rst:430 ../../source/data_flow.rst:577 msgid "计算attention" msgstr "calculate attention" -#: ../../source/data_flow.rst:441 ../../source/data_flow.rst:597 +#: ../../source/data_flow.rst:431 ../../source/data_flow.rst:578 msgid "attention计算的过程如下：" msgstr "The process of attention calculation is as follows:" -#: ../../source/data_flow.rst:447 +#: ../../source/data_flow.rst:437 msgid "这里通过dispatch的形式，根据q、k、v是分离还是组合在一起的状态，找到对应的forward函数进行attention计算。" msgstr "" "Here, the dispatch mechanism is used to determine whether the q, k, and v" " are separate or combined, and then the corresponding forward function is" " called to perform the attention calculation." -#: ../../source/data_flow.rst:449 +#: ../../source/data_flow.rst:439 msgid "" "在计算attention之前，通过 ``AllToAll`` 通信，对q和kv的 ``num_head`` 维度做 ``scatter`` ， " "``seq_len`` 维度做 ``gather`` 。" @@ -552,13 +550,13 @@ msgstr "" "scatter the num_head dimension of q and kv, and to gather the seq_len " "dimension." -#: ../../source/data_flow.rst:458 +#: ../../source/data_flow.rst:448 msgid "调用 ``context = self.local_attn(q, kv)`` 函数进行attention计算，计算结果的维度为：" msgstr "" "The function context = self.local_attn(q, kv) is called to perform the " "attention computation. The resulting dimension of the computation is:" -#: ../../source/data_flow.rst:465 +#: ../../source/data_flow.rst:455 msgid "" "在计算attention之后，再通过 ``AllToAll`` 通信，对q和kv的 ``num_head`` 维度做 ``gather`` ， " "``seq_len`` 维度做 ``scatter`` 。" @@ -567,53 +565,53 @@ msgstr "" "to gather the num_head dimension of q and kv, and to scatter the seq_len " "dimension." -#: ../../source/data_flow.rst:473 ../../source/data_flow.rst:613 +#: ../../source/data_flow.rst:463 ../../source/data_flow.rst:594 msgid "输出变换" msgstr "Output transformation" -#: ../../source/data_flow.rst:475 +#: ../../source/data_flow.rst:464 msgid "通过调用 \"wo\" 对attention计算的输出结果做变换，输出结果的维度如下：" msgstr "" "The output of the attention computation is transformed by calling \"wo\"," " and the dimensions of the output result are as follows:" -#: ../../source/data_flow.rst:482 ../../source/data_flow.rst:630 +#: ../../source/data_flow.rst:471 ../../source/data_flow.rst:610 msgid "feed_forward计算过程" msgstr "feed_forward Calculation Procedure" -#: ../../source/data_flow.rst:484 +#: ../../source/data_flow.rst:472 msgid "在feed_forward前馈网络层，通过\"w1\"、\"w2\"、\"w3\"对输出结果做线性变换。变换之后的结果如下：" msgstr "" "In the feed-forward network layer, linear transformations are applied to " "the output results using \"w1\", \"w2\", and \"w3\". The results after " "the transformation are as follows:" -#: ../../source/data_flow.rst:500 ../../source/data_flow.rst:663 +#: ../../source/data_flow.rst:488 ../../source/data_flow.rst:642 msgid "norm计算过程" msgstr "norm Calculation Procedure" -#: ../../source/data_flow.rst:502 +#: ../../source/data_flow.rst:489 msgid "经过norm层计算之后的结果维度保持不变，为：" msgstr "" "The result dimensions remain unchanged after the computation through the " "normalization layer, and are as follows:" -#: ../../source/data_flow.rst:510 ../../source/data_flow.rst:682 +#: ../../source/data_flow.rst:497 ../../source/data_flow.rst:660 msgid "output计算过程" msgstr "output Calculation Procedure" -#: ../../source/data_flow.rst:512 ../../source/data_flow.rst:684 +#: ../../source/data_flow.rst:498 ../../source/data_flow.rst:661 msgid "最后，经过output层将模型的最后一层输出转换为适合最终任务的格式，结果如下：" msgstr "" "Finally, the last layer of the model is transformed into a format " "suitable for the final task through the output layer, with the results as" " follows:" -#: ../../source/data_flow.rst:521 +#: ../../source/data_flow.rst:507 msgid "MTP/MSP/FSP并行模式下数据流程" msgstr "Data Procedure in MTP/MSP/FSP parallelism" -#: ../../source/data_flow.rst:523 +#: ../../source/data_flow.rst:508 msgid "" "在 ``MTP`` 并行模式中，只有张量并行对模型权重进行切分，不涉及对数据的seq_len维度进行切分。而 ``MSP`` 和 ``FSP`` " "并行模式中，均会涉及对数据进行序列化并行切分，且序列化并行与张量并行大小相同，两者共用通信组。" @@ -625,21 +623,21 @@ msgstr "" "parallelism is the same as that of tensor parallelism; both share the " "same communication group." -#: ../../source/data_flow.rst:528 +#: ../../source/data_flow.rst:512 msgid "在embedding的计算过程中，embedding的权重会进行切分：" msgstr "" "During the computation process of the embedding, the embedding weights " "will be partitioned:" -#: ../../source/data_flow.rst:535 +#: ../../source/data_flow.rst:519 msgid "``MTP`` 张量并行模式的输入输出结果如下：" msgstr "The input and output results of the MTP mode are as follows:" -#: ../../source/data_flow.rst:544 +#: ../../source/data_flow.rst:528 msgid "``MSP/FSP`` 的输入输出结果如下：" msgstr "The input and output results for MSP and FSP are as follows:" -#: ../../source/data_flow.rst:556 +#: ../../source/data_flow.rst:539 msgid "" "在进入attention计算之前，如果是 ``MSP/FSP`` 并行模式，会通过 ``All-Gather`` " "通信，将经过序列化并行切分后的数据聚集起来。因此，整个attention计算过程中， ``MTP/MSP/FSP`` 三种并行模式的参数维度一致。" @@ -650,7 +648,7 @@ msgstr "" "attention computation process, the parameter dimensions of the " "MTP/MSP/FSP three parallel modes are consistent." -#: ../../source/data_flow.rst:558 +#: ../../source/data_flow.rst:541 msgid "" "在attention计算完成之后， ``wo`` 层中做线性变换时，如果是 ``MSP/FSP`` 并行模式，会通过 ``Reduce-" "Scatter`` 通信，将linear变换行切的结果整合，同时做序列化并行操作。" @@ -661,11 +659,11 @@ msgstr "" "linear transformation, while also performing serialized parallel " "operations." -#: ../../source/data_flow.rst:567 +#: ../../source/data_flow.rst:549 msgid "计算后的qkv维度如下：" msgstr "The dimensions of the computed qkv are as follows:" -#: ../../source/data_flow.rst:574 +#: ../../source/data_flow.rst:556 msgid "" "之后将qkv拆分为 ``[batch_size, seq_len, num_head, group_size, head_dim]`` " "维度，并分别计算q、k、v的值，这里会对 ``num_head`` 维度做张量并行切分：" @@ -675,43 +673,43 @@ msgstr "" " respectively, where tensor parallelism partitioning is performed on the " "num_head dimension." -#: ../../source/data_flow.rst:603 +#: ../../source/data_flow.rst:584 msgid "这里直接进行attention计算，不需要像 ``ISP`` 模式中做 ``AllToAll`` 通信。" msgstr "" "Here, the attention calculation is performed directly, without the need " "for AllToAll communication as required in the \"ISP\" mode." -#: ../../source/data_flow.rst:605 +#: ../../source/data_flow.rst:586 msgid "计算结果的维度为：" msgstr "The dimensions of the computation result are:" -#: ../../source/data_flow.rst:615 +#: ../../source/data_flow.rst:595 msgid "通过调用 \"wo\" 对attention计算的输出结果做变换" msgstr "" "The \"wo\" is called to transform the output results of the attention " "computation." -#: ../../source/data_flow.rst:617 ../../source/data_flow.rst:646 -#: ../../source/data_flow.rst:667 +#: ../../source/data_flow.rst:597 ../../source/data_flow.rst:625 +#: ../../source/data_flow.rst:645 msgid "``MTP`` 并行模式，输出结果的维度如下：" msgstr "" "In the MTP parallel mode, the dimensions of the output results are as " "follows:" -#: ../../source/data_flow.rst:623 ../../source/data_flow.rst:654 -#: ../../source/data_flow.rst:674 +#: ../../source/data_flow.rst:603 ../../source/data_flow.rst:633 +#: ../../source/data_flow.rst:652 msgid "``MSP/FSP`` 并行模式，输出结果的维度如下：" msgstr "" "In the MSP/FSP parallel modes, the dimensions of the output results are " "as follows:" -#: ../../source/data_flow.rst:632 +#: ../../source/data_flow.rst:611 msgid "在feed_forward前馈网络层，通过\"w1\"、\"w2\"、\"w3\"对输出结果做线性变换。" msgstr "" "In the feed-forward network layer, linear transformations are applied to " "the output results using \"w1\", \"w2\", and \"w3\"." -#: ../../source/data_flow.rst:634 +#: ../../source/data_flow.rst:613 msgid "" "``MSP/FSP`` 并行模式下，在 ``w1`` 和 ``w3`` 线性变换层之前，需要进行 ``All-Gather`` 通信。因此， " "``MTP/MSP/FSP`` :w 并行模式下的输出维度相同：" @@ -720,15 +718,28 @@ msgstr "" "before the linear transformation layers of w1 and w3. Therefore, the " "output dimensions are the same across the MTP/MSP/FSP parallel modes." -#: ../../source/data_flow.rst:644 +#: ../../source/data_flow.rst:623 msgid "在经过\"w2\"层做线性变换之后，如果是 ``MSP/FSP`` 并行模式，需要进行 ``Reduce-Scatter`` 通信。" msgstr "" "After the linear transformation through the \"w2\" layer, in the MSP/FSP " "parallel modes, a Reduce-Scatter communication is necessary." -#: ../../source/data_flow.rst:665 +#: ../../source/data_flow.rst:643 msgid "经过norm层计算之后的结果维度保持不变。" msgstr "" "The result dimensions remain unchanged after the computation through the " "norm layer." +#~ msgid "数据格式" +#~ msgstr "Data Format" + +#~ msgid "" +#~ "ISP并行的具体原理请参见： `并行训练 " +#~ "`_" +#~ msgstr "" +#~ "For the specific principles of ISP " +#~ "parallelism, please refer " +#~ "to:`_" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/example/index.po b/doc/code-docs/locales/en/LC_MESSAGES/example/index.po index 48b8d0b81..80eb22519 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/example/index.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/example/index.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2023-11-10 11:54+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,11 +16,11 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/example/index.rst:2 -msgid "训练样例" -msgstr "Training Example" +msgid "训练样例介绍" +msgstr "Training Example Introduction" #: ../../source/example/index.rst:5 msgid "7B Demo" @@ -29,3 +29,4 @@ msgstr "" #: ../../source/example/index.rst:13 msgid "20B Demo" msgstr "" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/index.po b/doc/code-docs/locales/en/LC_MESSAGES/index.po index 5d802c702..09824605d 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/index.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/index.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-26 16:38+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -34,59 +34,58 @@ msgstr "Quickstart Guide" msgid "训练构建" msgstr "Model Setup" -#: ../../source/index.rst:35 -msgid "训练 API" -msgstr "Training API" - -#: ../../source/index.rst:43 +#: ../../source/index.rst:40 msgid "并行训练" msgstr "Parallel Training" -#: ../../source/index.rst:51 +#: ../../source/index.rst:48 msgid "数据格式" msgstr "Data Format" -#: ../../source/index.rst:59 +#: ../../source/index.rst:56 msgid "混合精度" msgstr "Mixed Precision" -#: ../../source/index.rst:67 +#: ../../source/index.rst:64 msgid "混合专家模型" msgstr "Mixture-of-Experts" -#: ../../source/index.rst:75 +#: ../../source/index.rst:72 msgid "模型备份" msgstr "Model Checkpointing" -#: ../../source/index.rst:83 +#: ../../source/index.rst:80 msgid "性能分析" msgstr "Profiler" -#: ../../source/index.rst:91 +#: ../../source/index.rst:88 msgid "训练监控" msgstr "Monitor" -#: ../../source/index.rst:99 +#: ../../source/index.rst:96 msgid "训练样例" msgstr "Example" -#: ../../source/index.rst:107 +#: ../../source/index.rst:104 msgid "常见问题" msgstr "Q&A" -#: ../../source/index.rst:115 +#: ../../source/index.rst:112 msgid "索引和表格" msgstr "Indices and tables" -#: ../../source/index.rst:117 +#: ../../source/index.rst:114 msgid ":ref:`genindex`" msgstr "" -#: ../../source/index.rst:118 +#: ../../source/index.rst:115 msgid ":ref:`modindex`" msgstr "" -#: ../../source/index.rst:119 +#: ../../source/index.rst:116 msgid ":ref:`search`" msgstr "" +#~ msgid "训练 API" +#~ msgstr "Training API" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/initialize.po b/doc/code-docs/locales/en/LC_MESSAGES/initialize.po index 99edfa825..b94457a99 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/initialize.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/initialize.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-27 10:37+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: zh_CN\n" @@ -19,8 +19,8 @@ msgstr "" "Generated-By: Babel 2.15.0\n" #: ../../source/initialize.rst:2 -msgid "训练构建" -msgstr "Training Setup" +msgid "训练初始化" +msgstr "Training Initialization" #: ../../source/initialize.rst:4 msgid "InternEvo 的训练流程可以归纳为两个步骤：" @@ -112,83 +112,129 @@ msgstr "" #: ../../source/initialize.rst:79 msgid "值得注意的是，用户可以定义新的模型类型，并通过 ``register_module`` 注册模型的初始化函数，示例如下所示：" -msgstr "It is worth noting that users can define new model types and register the model initialization functions via register_module. An example is shown as follows:" - -#: ../../source/initialize.rst:90 -msgid "其中，\"INTERNLM\"为新的模型类型，InternLM1为新模型的入口函数。" -msgstr "In this context, "INTERNLM" is the new model type, and InternLM1 is the entry function for the new model." - -#: ../../source/initialize.rst:93 -msgid "优化器初始化" -msgstr "Optimizer Initialization" - -#: ../../source/initialize.rst:97 -msgid "" -"InternEvo 在配置文件中使用字段 ``grad_scaler`` 、 ``hybrid_zero_optimizer`` 、 " -"``adam`` 、 ``lr_scheduler`` 和 ``beta2_scheduler`` " -"来控制优化器初始化过程。示例优化器初始化配置定义如下：" msgstr "" -"InternEvo utilizes the fields grad_scaler, hybrid_zero_optimizer, adam, lr_scheduler, and beta2_scheduler in the configuration file to control the initialization process of the optimizer. An example configuration for initializing the optimizer is defined as follows:" +"It is worth noting that users can define new model types and register the" +" model initialization functions via register_module. An example is shown " +"as follows:" -#: ../../source/initialize.rst:158 -msgid "" -"用户通过 ``initialize_optimizer`` 函数初始化优化器，并传入 ``isp_communicator`` 参数，以便处理 " -"ISP 并行模式下的通信。" +#: ../../source/initialize.rst:88 +msgid "其中，\"INTERNLM\"为新的模型类型，InternLM1为新模型的入口函数。" msgstr "" -"Users initialize the optimizer through the function initialize_optimizer and pass in the isp_communicator parameter to handle communication in the ISP parallel mode." +"In this context, \"INTERNLM\" is the new model type, and InternLM1 is the" +" entry function for the new model." -#: ../../source/initialize.rst:163 +#: ../../source/initialize.rst:93 msgid "数据加载器初始化" msgstr "Dataloader Initialization" -#: ../../source/initialize.rst:167 +#: ../../source/initialize.rst:97 msgid "InternEvo 在配置文件中使用字段 ``data`` 来控制数据加载器初始化过程。示例数据加载器初始化配置定义如下：" msgstr "" -"InternEvo uses the field data in the configuration file to control the initialization process of the data loader. The example configuration for initializing the data loader is defined as follows:" +"InternEvo uses the field data in the configuration file to control the " +"initialization process of the data loader. The example configuration for " +"initializing the data loader is defined as follows:" -#: ../../source/initialize.rst:204 +#: ../../source/initialize.rst:134 msgid "这里支持三种数据集的初始化，包括模拟数据集、已分词数据集和流式数据集。" -msgstr "This section supports the initialization of three types of datasets, including dummy datasets, tokenized datasets, and streaming datasets." +msgstr "" +"This section supports the initialization of three types of datasets, " +"including dummy datasets, tokenized datasets, and streaming datasets." -#: ../../source/initialize.rst:206 +#: ../../source/initialize.rst:136 msgid "模拟数据集" msgstr "dummy dataset" -#: ../../source/initialize.rst:208 +#: ../../source/initialize.rst:138 msgid "如果设置TRAIN_FOLDER为None，则随机生成模拟数据集，如果设置的随机种子一样，生成的数据集保持一致。" -msgstr "If the TRAIN_FOLDER is set to None, a dummy dataset will be randomly generated, and if the random seed is the same, the generated dataset will remain consistent." +msgstr "" +"If the TRAIN_FOLDER is set to None, a dummy dataset will be randomly " +"generated, and if the random seed is the same, the generated dataset will" +" remain consistent." -#: ../../source/initialize.rst:210 +#: ../../source/initialize.rst:140 msgid "已分词数据集" msgstr "tokenized dataset" -#: ../../source/initialize.rst:212 +#: ../../source/initialize.rst:142 msgid "如果设置TRAIN_FOLDER为本地指定路径，路径中保存经过分词之后的.bin和.meta文件，则加载已分词数据集。" -msgstr "If the TRAIN_FOLDER is set to a local path where the .bin and .meta files after tokenization are stored, then a tokenized dataset will be loaded." +msgstr "" +"If the TRAIN_FOLDER is set to a local path where the .bin and .meta files" +" after tokenization are stored, then a tokenized dataset will be loaded." -#: ../../source/initialize.rst:214 +#: ../../source/initialize.rst:144 msgid "流式数据集" msgstr "streaming dataset" -#: ../../source/initialize.rst:216 +#: ../../source/initialize.rst:146 msgid "如果设置TRAIN_FOLDER为本地指定路径，路径中保存从huggingface下载的数据集，同时在data配置中，新增如下type和tokenizer_path字段，则加载流式数据集。" -msgstr "If the TRAIN_FOLDER is set to a specified local path that contains the dataset downloaded from HuggingFace, and new fields type and tokenizer_path are added in the data configuration, then a streaming dataset will be loaded." +msgstr "" +"If the TRAIN_FOLDER is set to a specified local path that contains the " +"dataset downloaded from HuggingFace, and new fields type and " +"tokenizer_path are added in the data configuration, then a streaming " +"dataset will be loaded." -#: ../../source/initialize.rst:225 +#: ../../source/initialize.rst:153 msgid "" "已分词数据集和流式数据集格式的详细说明，参见 `使用教程 " "`_" msgstr "" -"For detailed instructions on the formats for tokenized datasets and streaming datasets, please refer to the `User Guide " +"For detailed instructions on the formats for tokenized datasets and " +"streaming datasets, please refer to the `User Guide " "`_" -#: ../../source/initialize.rst:228 +#: ../../source/initialize.rst:158 +msgid "并行通信初始化" +msgstr "Parallel Communication Initialization" + +#: ../../source/initialize.rst:162 +msgid "通过 ``initialize_parallel_communicator`` 函数，初始化不同模式并行下的通信状态。" +msgstr "Initialize the communication status under different parallel modes using the ``initialize_parallel_communicator`` function." + +#: ../../source/initialize.rst:164 +msgid "" +"在 ``ISP`` 并行模式下，处理overlap优化，以及注册linear层的All_Gather通信。 在 ``MTP`` " +"并行模式下，分别注册被行切以及列切的权重的通信函数。 在 ``MSP`` 以及 ``FSP`` 并行模式下，注册序列并行的通信函数。 在 " +"``MoE`` 模型中，注册MoE序列化并行通信函数。" +msgstr "" +"In the ``ISP`` parallel mode, handle overlap optimization and register the All_Gather communication for linear layers." +" In the ``MTP`` parallel mode, register communication functions for weights that are row-wise and column-wise partitioned." +" In ``MSP`` and ``FSP`` parallel modes, register communication functions for sequence parallelism." +" In the ``MoE`` model, register the MoE serialized parallel communication function." + +#: ../../source/initialize.rst:172 +msgid "优化器初始化" +msgstr "Optimizer Initialization" + +#: ../../source/initialize.rst:176 +msgid "" +"InternEvo 在配置文件中使用字段 ``grad_scaler`` 、 ``hybrid_zero_optimizer`` 、 " +"``adam`` 、 ``lr_scheduler`` 和 ``beta2_scheduler`` " +"来控制优化器初始化过程。示例优化器初始化配置定义如下：" +msgstr "" +"InternEvo utilizes the fields grad_scaler, hybrid_zero_optimizer, adam, " +"lr_scheduler, and beta2_scheduler in the configuration file to control " +"the initialization process of the optimizer. An example configuration for" +" initializing the optimizer is defined as follows:" + +#: ../../source/initialize.rst:237 +msgid "" +"用户通过 ``initialize_optimizer`` 函数初始化优化器，并传入 ``isp_communicator`` 参数，以便处理 " +"ISP 并行模式下的通信。" +msgstr "" +"Users initialize the optimizer through the function initialize_optimizer " +"and pass in the isp_communicator parameter to handle communication in the" +" ISP parallel mode." + +#: ../../source/initialize.rst:242 msgid "Trainer 初始化" msgstr "Trainer Initialization" -#: ../../source/initialize.rst:232 +#: ../../source/initialize.rst:246 msgid "通过initialize_trainer函数，初始化训练过程，需要输入创建好的模型、初始化的优化器以及调度器等参数。" -msgstr "The initialize_trainer function is used to initiate the training process, requiring parameters such as the created model, initialized optimizer, scheduler, and other related parameters." +msgstr "" +"The initialize_trainer function is used to initiate the training process," +" requiring parameters such as the created model, initialized optimizer, " +"scheduler, and other related parameters." #~ msgid "Initialize model with Automatic Mixed Precision." #~ msgstr "" @@ -291,3 +337,6 @@ msgstr "The initialize_trainer function is used to initiate the training process #~ "``trainer`` could not be None." #~ msgstr "" +#~ msgid "训练构建" +#~ msgstr "Training Setup" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/mixed_precision.po b/doc/code-docs/locales/en/LC_MESSAGES/mixed_precision.po index 8cac83722..ea3b158b2 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/mixed_precision.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/mixed_precision.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,25 +16,35 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/mixed_precision.rst:2 -msgid "混合精度" -msgstr "Mixed Precision" +msgid "混合精度训练" +msgstr "Mixed Precision Training" -#: ../../source/mixed_precision.rst:3 -msgid "" -"混合精度是指在模型训练的过程中同时使用16位和32位浮点数类型，是一种在最小化精度损失的前提下加速模型训练的方法。 " -"混合精度通过让模型的某些部分使用32位浮点数以保持数值稳定性，并在其余部分利用半精度浮点数加速训练并可以减少内存使用，在评估指标（如准确率）方面仍可以获得同等的训练效果。" +#: ../../source/mixed_precision.rst:5 +msgid "实现说明" +msgstr "Implementation Instructions" + +#: ../../source/mixed_precision.rst:6 +msgid "混合精度是指在模型训练的过程中同时使用16位和32位浮点数类型，是一种在最小化精度损失的前提下加速模型训练的方法。" msgstr "" "Mixed precision refers to using both 16-bit and 32-bit floating-point " "types to train model, which can accelerate the model training while " -"minimizing the accuracy loss. Mixed precision training uses 32-bit " -"floating-point types in certain parts of the model to maintain numerical " +"minimizing the accuracy loss." + +#: ../../source/mixed_precision.rst:8 +msgid "混合精度通过让模型的某些部分使用32位浮点数以保持数值稳定性，并在其余部分利用半精度浮点数加速训练并可以减少内存使用，在评估指标（如准确率）方面仍可以获得同等的训练效果。" +msgstr "" +"Mixed precision training uses 32-bit floating-point types in certain parts of the model to maintain numerical " "stability, and accelerate training and reduce memory usage by using " "16-bit floating-point types in other parts. Mixed precision can achieve " "the same training effect in evaluating indicators such as accuracy." +#: ../../source/mixed_precision.rst:10 +msgid "通过 ``NaiveAMPModel`` 类实现混合精度，该类的具体说明如下：" +msgstr "Implementing mixed-precision through the class NaiveAMPModel, the specific description is as follows:" + #: internlm.core.naive_amp.NaiveAMPModel:1 of msgid "" "This is a wrapper class for a model that automatically casts the model, " @@ -42,9 +52,9 @@ msgid "" "output back to fp32 and to synchronize buffers." msgstr "" -#: internlm.core.naive_amp.NaiveAMPModel of +#: ../../source/mixed_precision.rst msgid "参数" -msgstr "" +msgstr "Parameters" #: internlm.core.naive_amp.NaiveAMPModel:4 of msgid "The model to be wrapped and cast into fp16." @@ -64,7 +74,7 @@ msgstr "" msgid "If True, the buffers are synchronized. Defaults to True." msgstr "" -#: ../../source/mixed_precision.rst:8 +#: ../../source/mixed_precision.rst:14 msgid "InternEvo默认将模型转换为16位浮点数类型进行训练（在配置文件中可以设置默认类型为其他数据类型）。在使用混合精度时，需要在构建模型时使用" msgstr "" "InternEvo converts the model to 16-bit floating-point types for model " @@ -72,49 +82,49 @@ msgstr "" "the configuration file). When using mixed precision, it is necessary to " "use " -#: ../../source/mixed_precision.rst:14 +#: ../../source/mixed_precision.rst:20 msgid "将模型的某个子模块设置为32位浮点数类型进行训练，InternEvo会在模型训练时自动将数据类型转换成需要的精度。" msgstr "" "to set a sub-module of the model to 16-bit floating-point types for " "training, and InternEvo will automatically convert the data type to the " "required precision during model training." -#: ../../source/mixed_precision.rst:16 +#: ../../source/mixed_precision.rst:22 msgid "例如：" msgstr "For example:" -#: ../../source/mixed_precision.rst:40 +#: ../../source/mixed_precision.rst:46 msgid "TF32训练" msgstr "TF32 Training" -#: ../../source/mixed_precision.rst:41 +#: ../../source/mixed_precision.rst:47 msgid "TensorFloat-32（TF32）是Nvidia在Ampere架构GPU上推出的专门运用于TensorCore的一种计算格式。其与其他常用数据格式的比较如下图：" msgstr "" "TensorFloat-32 (TF32) is a computational format introduced by Nvidia on " "Ampere Architecture GPUs for TensorCore. A comparison with other data " "formats is shown below." -#: ../../source/mixed_precision.rst:47 +#: ../../source/mixed_precision.rst:53 msgid "使用TF32的前置条件：" msgstr "Prerequisites for using TF32." -#: ../../source/mixed_precision.rst:49 +#: ../../source/mixed_precision.rst:55 msgid "输入数据类型为FP32，且计算为矩阵乘法及卷积相关运算，才可以使用TF32作为TensorCore的中间计算类型。" msgstr "" "The input data type should be FP32 and TF32 is designed for matrix " "multiplication, convolutions, and other relative computations." -#: ../../source/mixed_precision.rst:51 +#: ../../source/mixed_precision.rst:57 msgid "Ampere架构的GPU。" msgstr "Ampere Architecture GPU" -#: ../../source/mixed_precision.rst:53 +#: ../../source/mixed_precision.rst:59 msgid "InternEvo支持使用TF32训练模型，允许用户在config文件中将 ``dtype`` 设置为 ``torch.tf32``。" msgstr "" "InternEvo supports training model in TF32 and allows user to set the " "``dtype`` in config as ``torch.tf32``." -#: ../../source/mixed_precision.rst:75 +#: ../../source/mixed_precision.rst:81 msgid "" "值得注意的是，TF32仅仅是在使用TensorCore时的一种中间计算格式，并不是一个完全的数据类型。因此，在InternEvo中，尽管用户将 " "``dtype`` 设置成了 ``torch.tf32``，模型的数据类型依旧是 ``torch.float32``。InternEvo会针对 " diff --git a/doc/code-docs/locales/en/LC_MESSAGES/monitor.po b/doc/code-docs/locales/en/LC_MESSAGES/monitor.po index 55f0b6d16..4d61a4353 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/monitor.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/monitor.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-23 20:19+0800\n" +"POT-Creation-Date: 2024-08-30 16:07+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -28,11 +28,11 @@ msgstr "Monitoring" #: ../../source/monitor.rst:7 msgid "" -"InternEvo 使用 ``internlm.monitor.monitor.initialize_monitor_manager()`` " +"InternEvo 使用 ``internlm.monitor.initialize_monitor_manager()`` " "来初始化上下文监控管理。其中，一个实例化的单例对象 ``internlm.monitor.monitor.MonitorManager`` " "将管理监控线程并使用 ``internlm.monitor.monitor.MonitorTracker`` 来跟踪模型训练生命周期和训练状态。" msgstr "" -"InternEvo uses ``internlm.monitor.monitor.initialize_monitor_manager()`` " +"InternEvo uses ``internlm.monitor.initialize_monitor_manager()`` " "to initialize context monitor. During this time, a singleton " "``internlm.monitor.monitor.MonitorManager`` will manage monitoring thread" " and track training status with " @@ -84,36 +84,25 @@ msgstr "" "`_." " Here is an example of a monitoring configuration:" -#: ../../source/monitor.rst:41 +#: ../../source/monitor.rst:42 msgid "enable_feishu_alert (bool)：是否启用飞书告警。默认值：False。" msgstr "enable_feishu_alert: Whether to enable Feishu alerts. Defaults: False." -#: ../../source/monitor.rst:42 +#: ../../source/monitor.rst:43 msgid "feishu_alert_address (str)：飞书告警的 Webhook 地址。默认值：None。" msgstr "" "feishu_alert_address: The webhook address for Feishu alerts. Defaults: " "None." -#: ../../source/monitor.rst:43 +#: ../../source/monitor.rst:44 msgid "light_monitor_address (str)：轻量监控的地址。默认值：None。" msgstr "" "light_monitor_address: The address for lightweight monitoring. Defaults: " "None." #: ../../source/monitor.rst:45 -msgid "" -"InternEvo 使用 ``internlm.monitor.alert.initialize_light_monitor`` " -"来初始化轻量监控客户端。一旦初始化完成，它会建立与监控服务器的连接。在训练过程中，使用 " -"``internlm.monitor.alert.send_heartbeat`` " -"来发送不同类型的心跳信息至监控服务器。监控服务器会根据这些心跳信息来检测训练是否出现异常，并在需要时发送警报消息。" -msgstr "" -"InternEvo uses ``internlm.monitor.alert.initialize_light_monitor`` to " -"initialize the lightweight monitoring client. Once initialization is " -"complete, it establishes a connection with the monitoring server. During " -"the training process, it uses ``internlm.monitor.alert.send_heartbeat`` " -"to send various types of heartbeat messages to the monitoring server. The" -" monitoring server uses these heartbeat messages to detect if the " -"training encounters any abnormalities and sends alert messages as needed." +msgid "alert_file_path (str)：告警存储路径。默认值：None。" +msgstr "alert_file_path: path of alert. Defaults: None." #~ msgid "" #~ "InternLM轻量监控基于心跳机制来监控训练过程中是否出现 " @@ -259,3 +248,25 @@ msgstr "" #~ " 2.3, \"bwd_time\": 6.2})``" #~ msgstr "" +#~ msgid "" +#~ "InternEvo 使用 " +#~ "``internlm.monitor.alert.initialize_light_monitor`` " +#~ "来初始化轻量监控客户端。一旦初始化完成，它会建立与监控服务器的连接。在训练过程中，使用 " +#~ "``internlm.monitor.alert.send_heartbeat`` " +#~ "来发送不同类型的心跳信息至监控服务器。监控服务器会根据这些心跳信息来检测训练是否出现异常，并在需要时发送警报消息。" +#~ msgstr "" +#~ "InternEvo uses " +#~ "``internlm.monitor.alert.initialize_light_monitor`` to " +#~ "initialize the lightweight monitoring client." +#~ " Once initialization is complete, it " +#~ "establishes a connection with the " +#~ "monitoring server. During the training " +#~ "process, it uses " +#~ "``internlm.monitor.alert.send_heartbeat`` to send " +#~ "various types of heartbeat messages to" +#~ " the monitoring server. The monitoring " +#~ "server uses these heartbeat messages to" +#~ " detect if the training encounters " +#~ "any abnormalities and sends alert " +#~ "messages as needed." + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/parallel.po b/doc/code-docs/locales/en/LC_MESSAGES/parallel.po index c32bc4a6e..b948e4f9b 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/parallel.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/parallel.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-23 14:49+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -19,8 +19,8 @@ msgstr "" "Generated-By: Babel 2.15.0\n" #: ../../source/parallel.rst:2 -msgid "并行训练" -msgstr "Parallel Training" +msgid "并行模式与原理" +msgstr "Parallel Mode and Principle" #: ../../source/parallel.rst:6 msgid "" @@ -51,22 +51,22 @@ msgstr "" "the default value is -1" #: ../../source/parallel.rst:21 -msgid "当 ``size <= 0``，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" +msgid "当 ``size <= 0`` ，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" msgstr "" -"When ``size <= 0``, the size of the zero1 process group is equal to the " +"When ``size <= 0`` , the size of the zero1 process group is equal to the " "size of the data parallel process group, so the optimizer state " "parameters will be split within the data parallel range." #: ../../source/parallel.rst:22 -msgid "当 ``size == 1``，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" +msgid "当 ``size == 1`` ，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" msgstr "" -"When ``size == 1``, zero1 is not used, and all data parallel groups " +"When ``size == 1`` , zero1 is not used, and all data parallel groups " "retain the complete optimizer state parameters." #: ../../source/parallel.rst:23 -msgid "当 ``size > 1`` 且 ``zero1 <= data_parallel_size``，则 zero1 进程组是数据并行进程组的子集" +msgid "当 ``size > 1`` 且 ``zero1 <= data_parallel_size`` ，则 zero1 进程组是数据并行进程组的子集" msgstr "" -"When ``size > 1`` and ``size <= data_parallel_size``, the zero1 process " +"When ``size > 1`` and ``size <= data_parallel_size`` , the zero1 process " "group is a subset of the data parallel process group." #: ../../source/parallel.rst:25 @@ -147,11 +147,11 @@ msgstr "" "Note: `Data parallel size = Total number of GPUs / Pipeline parallel size" " / Tensor parallel size`" -#: ../../source/parallel.rst:49 +#: ../../source/parallel.rst:51 msgid "张量并行" msgstr "Tensor Parallel" -#: ../../source/parallel.rst:51 +#: ../../source/parallel.rst:53 msgid "" "InternEvo 系统 ``v0.3.0`` 版本在张量并行策略上有较大更新，目前的张量并行支持四种模式配置['mtp', 'msp', " "'fsp', 'isp']，前三种模式为基于 Megatron-LM 的张量并行和序列化并行的策略实现，最后一种模式为 InternEvo " @@ -166,11 +166,11 @@ msgstr "" "with weight parallelism. The following provides a detailed explanation of" " the differences among these tensor parallelism modes." -#: ../../source/parallel.rst:53 +#: ../../source/parallel.rst:55 msgid "MTP" msgstr "" -#: ../../source/parallel.rst:55 +#: ../../source/parallel.rst:57 msgid "" "MTP(Megatron-LM Tensor Parallel)， 为默认的张量并行模型，引用自 `Megatron-LM Tensor " "Parallel `_ 并行方案，如下图所示，带有张量并行的 " @@ -182,26 +182,26 @@ msgstr "" "`_. The following diagram illustrates " "the ``Transformer`` layer with tensor parallelism:" -#: ../../source/parallel.rst:61 +#: ../../source/parallel.rst:63 msgid "Transformer layer with tensor parallelism." msgstr "" -#: ../../source/parallel.rst:63 +#: ../../source/parallel.rst:65 msgid "" "MTP 主要对 `attention " "`_" " 和 `linear " "`_" -" 这两个模块进行张量并行操作。假设张量并行大小为 ``tp``，输入数据的序列长度为 ``seqlen``，隐藏层大小为 ``hidden " -"size``，则张量并行期间产生的激活值 ``shape`` 为 ``[seqlen, hidden_size/tp]``。" +" 这两个模块进行张量并行操作。假设张量并行大小为 ``tp`` ，输入数据的序列长度为 ``seqlen`` ，隐藏层大小为 ``hidden " +"size`` ，则张量并行期间产生的激活值 ``shape`` 为 ``[seqlen, hidden_size/tp]`` 。" msgstr "" "MTP primarily applies tensor parallelism operations to the attention and " "the linear module. Assuming the tensor parallelism size is tp, the " "sequence length of input data is seqlen, and the hidden layer size is " "hidden size, then the shape of the activation values generated during " -"tensor parallelism is ``[seqlen, hidden_size/tp]``." +"tensor parallelism is ``[seqlen, hidden_size/tp]`` ." -#: ../../source/parallel.rst:65 +#: ../../source/parallel.rst:67 msgid "" "MTP 张量并行引入的通信如上图所示，其中 ``f`` 和 ``f̄`` 是共轭的。在前向传递中 ``f`` 是无操作，而在反向传递中进行 " "``all-reduce`` 操作。而 ``f̄`` 在前向传递中进行 ``all-reduce`` 操作，在反向传递中是无操作。" @@ -213,11 +213,11 @@ msgstr "" "reduce`` operation in the forward pass and is a no-operation in the " "backward pass." -#: ../../source/parallel.rst:67 +#: ../../source/parallel.rst:69 msgid "MSP" msgstr "" -#: ../../source/parallel.rst:69 +#: ../../source/parallel.rst:71 msgid "" "MSP(Megatron-LM Sequence Parallel)，引用自 `Megatron-LM Sequence Parallel " "`_ 并行方案，如下图所示，带有张量并行和序列化并行的 " @@ -227,28 +227,28 @@ msgstr "" "Sequence Parallel parallel scheme. The following diagram illustrates the " "Transformer layer with both tensor parallelism and sequence parallelism:" -#: ../../source/parallel.rst:75 +#: ../../source/parallel.rst:77 msgid "Transformer layer with tensor and sequence parallelism." msgstr "" -#: ../../source/parallel.rst:77 +#: ../../source/parallel.rst:79 msgid "" "与 MTP 对比，我们可以发现，MSP 主要针对未进行张量并行的模块，如 ``LayerNorm`` 和 ``Dropout`` " -"等模型进行序列化并行操作。需要注意的是，序列化并行大小与张量并行大小相等，且共用通信组。假设张量并行大小为 ``tp``，输入数据的序列长度为 " -"``seqlen``，隐藏层大小为 ``hidden size``，则序列化并行期间产生的激活值形状为 ``[seqlen/tp, " -"hidden_size]``，张量并行期间产生的激活值形状为 ``[seqlen, hidden_size/tp]``。" +"等模型进行序列化并行操作。需要注意的是，序列化并行大小与张量并行大小相等，且共用通信组。假设张量并行大小为 ``tp`` ，输入数据的序列长度为 " +"``seqlen`` ，隐藏层大小为 ``hidden size`` ，则序列化并行期间产生的激活值形状为 ``[seqlen/tp, " +"hidden_size]`` ，张量并行期间产生的激活值形状为 ``[seqlen, hidden_size/tp]`` 。" msgstr "" "Compared to MTP, it is evident that MSP primarily focuses on modules " -"without tensor parallelism, such as ``LayerNorm`` and ``Dropout``, and " +"without tensor parallelism, such as ``LayerNorm`` and ``Dropout`` , and " "performs sequence parallelism operations. It is important to note that " "the size of sequence parallelism is equal to the size of tensor " "parallelism, and they share the same communication group. Assuming the " "tensor parallelism size is tp, the input data has a sequence length of " "seqlen, and the hidden layer size is hidden size, the shape of activation" -" values during sequence parallelism is ``[seqlen/tp, hidden_size]``, " -"while during tensor parallelism, it is ``[seqlen, hidden_size/tp]``." +" values during sequence parallelism is ``[seqlen/tp, hidden_size]`` , " +"while during tensor parallelism, it is ``[seqlen, hidden_size/tp]`` ." -#: ../../source/parallel.rst:79 +#: ../../source/parallel.rst:81 msgid "" "MSP与MTP相比，通信原语有所变化，如上图所示 ``g`` 和 ``ḡ`` 是共轭的。在前向传递中 ``g`` 进行 ``all-" "gather`` 操作，而在反向传递中进行 ``reduce-scatter`` 操作。而 ``ḡ`` 在前向传递中进行 ``reduce-" @@ -262,33 +262,33 @@ msgstr "" "scatter`` operation in the forward pass and an ``all-gather`` operation " "in the backward pass." -#: ../../source/parallel.rst:81 +#: ../../source/parallel.rst:83 msgid "" "在前向传递中 ``g`` 通信处于序列化并行和张量并行的交接处，进行的是激活值在 ``seqlen`` 维度的 ``all-gather`` " -"操作，该通信完成后，激活值形状变成完整的 ``[seqlen, hidden_size]``，然后进入张量并行模块范围。``ḡ`` " -"通信处于张量并行和序列化并行的交接处，需要把 MTP 中的 ``all-reduce`` 通信操作变成 ``reduce-" -"scatter``，才能完成 ``seqlen`` 维度的切分，激活值形状变成 ``[seqlen/tp, " -"hidden_size]``，从而正常进入序列化并行的阶段。而反向传递时，则是同样的道理。" +"操作，该通信完成后，激活值形状变成完整的 ``[seqlen, hidden_size]`` ，然后进入张量并行模块范围。 ``ḡ`` " +"通信处于张量并行和序列化并行的交接处，需要把 MTP 中的 ``all-reduce`` 通信操作变成 ``reduce-scatter`` " +"，才能完成 ``seqlen`` 维度的切分，激活值形状变成 ``[seqlen/tp, hidden_size]`` " +"，从而正常进入序列化并行的阶段。而反向传递时，则是同样的道理。" msgstr "" "In the forward pass, the communication of ``g`` occurs at the junction of" " sequence parallelism and tensor parallelism, performing an ``all-" "gather`` operation along the seqlen dimension of activation values. After" " this communication is completed, the shape of the activation values " -"becomes the full ``[seqlen, hidden_size]``, and then it enters the scope " +"becomes the full ``[seqlen, hidden_size]`` , and then it enters the scope " "of the tensor parallelism module. The communication of ``ḡ`` is situated " "at the junction of tensor parallelism and sequence parallelism, requiring" " the transformation of the ``all-reduce`` communication operation from " "MTP into a ``reduce-scatter`` operation to achieve the split along the " "seqlen dimension. This results in the activation values having a shape of" -" ``[seqlen/tp, hidden_size]``, enabling a smooth transition into the " +" ``[seqlen/tp, hidden_size]`` , enabling a smooth transition into the " "sequence parallelism phase. The same principles apply during the backward" " pass." -#: ../../source/parallel.rst:83 +#: ../../source/parallel.rst:85 msgid "FSP" msgstr "" -#: ../../source/parallel.rst:85 +#: ../../source/parallel.rst:87 msgid "" "FSP(Flash-Attn Sequence Parallel)，引用自 `flash attention " "`_ 的序列化并行实现方案。该实现方案与 MSP " @@ -306,7 +306,7 @@ msgstr "" "computation, an additional ``all-gather`` operation is needed to retrieve" " the complete input data." -#: ../../source/parallel.rst:87 +#: ../../source/parallel.rst:89 msgid "因此，FSP 与 MSP 性能对比的话，会有更小的显存占用，但是由于引入额外的 ``all-gather`` 通信，会导致训练速度 TGS 降低。" msgstr "" "Therefore, in terms of performance comparison between FSP and MSP, FSP " @@ -314,11 +314,11 @@ msgstr "" "additional ``all-gather`` communication can lead to a reduction in the " "training speed, denoted as TGS." -#: ../../source/parallel.rst:90 +#: ../../source/parallel.rst:92 msgid "ISP" msgstr "" -#: ../../source/parallel.rst:92 +#: ../../source/parallel.rst:94 msgid "" "ISP(Intern Sequence Parallel)，InternEvo " "系统自研的灵活可扩展序列化并行方案，支持张量并行与序列化并行解耦，通过计算和通信的overlap提高训练性能，并基于显存池管理降低显存碎片化的可能性，提高显存利用率。" @@ -331,15 +331,15 @@ msgstr "" "reduce the likelihood of memory fragmentation, thereby improving memory " "utilization." -#: ../../source/parallel.rst:94 +#: ../../source/parallel.rst:96 msgid "" "以 `configs/7B_isp_sft.py " "`_" -" 配置文件为例，将 ``tensor.mode`` 字段设置为 ``isp``，而 ``tensor.size`` 字段代表的是数据 " +" 配置文件为例，将 ``tensor.mode`` 字段设置为 ``isp`` ，而 ``tensor.size`` 字段代表的是数据 " "``seqlen`` 维度切分大小。ISP 算法可与 ``weight parallel`` 结合使用，其中 ``weight.size`` " -"字段代表的是模型权重切分大小，将 ``weight.overlap`` 字段设置为 ``True`` 即为开启计算与通信的 " -"``overlap``，可提高训练性能。将 ``weight.memory_pool`` 字段设置为 ``True`` " -"即为开启显存池管理功能，可一定程度降低 GPU 显存碎片的可能性，提高显存利用率。" +"字段代表的是模型权重切分大小，将 ``weight.overlap`` 字段设置为 ``True`` 即为开启计算与通信的 ``overlap``" +" ，可提高训练性能。将 ``weight.memory_pool`` 字段设置为 ``True`` 即为开启显存池管理功能，可一定程度降低 GPU" +" 显存碎片的可能性，提高显存利用率。" msgstr "" "Taking the configuration file `configs/7B_isp_sft.py " "`_" @@ -352,22 +352,22 @@ msgstr "" " management feature, which helps to some extent in reducing the " "likelihood of GPU memory fragmentation and improving memory utilization." -#: ../../source/parallel.rst:105 +#: ../../source/parallel.rst:107 msgid "如下图所示，带有序列化并行和权重并行的 ``Transformer`` 层：" msgstr "" "As illustrated in the diagram below, there is a Transformer layer with " "both sequence parallelism and weight parallelism:" -#: ../../source/parallel.rst:111 +#: ../../source/parallel.rst:113 msgid "" "如图所示，ISP 的序列化并行范围覆盖整个 ``Transformer`` 模型层，模型权重并行主要针对 ``Attention`` 和 " -"``MLP Block`` 的 ``Linear module``。" +"``MLP Block`` 的 ``Linear module`` 。" msgstr "" "As shown in the figure, the sequence parallelism scope of ISP covers the " "entire Transformer model layer, while the weight parallelism primarily " "targets the Linear module within the Attention and MLP Block." -#: ../../source/parallel.rst:113 +#: ../../source/parallel.rst:115 msgid "" "通信原语变化情况为，在前向传递时，每个 ``Linear`` 需要进行模型权重的 ``all-gather`` 通信；在后向传递时，每个 " "``Linear`` 在进行后向计算前需要进行模型权重的 ``all-gather`` 通信，在后向计算后，需要进行模型权重的梯度的 " @@ -380,7 +380,7 @@ msgstr "" " model weight. After the backward computation, there is a ``reduce-" "scatter`` communication operation for the gradients of model weights." -#: ../../source/parallel.rst:115 +#: ../../source/parallel.rst:117 msgid "" "需要注意的是，与 MSP 和 FSP 相比，在进行 ``attention score`` 计算时，ISP 也有通信原语的一些变化，如 " "``Self-Atten`` 前后各增加了一个 ``all-to-all`` 通信操作，用于完成激活值形状的转置，目的是在进行 " @@ -393,22 +393,22 @@ msgstr "" "shape of activation values. The purpose is to maintain the original " "tensor parallelism pattern during the attention score calculation." -#: ../../source/parallel.rst:117 +#: ../../source/parallel.rst:119 msgid "" "关于 ISP 算法更多的设计思路和性能评测，请参考论文 `InternEvo: Efficient Long-sequence Large " "Language Model Training via Hybrid Parallelism and Redundant Sharding " -"`_。" +"`_ 。" msgstr "" "For more design details and performance evaluation of the ISP algorithm, " "please refer to the paper `InternEvo: Efficient Long-sequence Large " "Language Model Training via Hybrid Parallelism and Redundant Sharding " "`_." -#: ../../source/parallel.rst:121 +#: ../../source/parallel.rst:123 msgid "流水线并行" msgstr "Pipeline Parallel" -#: ../../source/parallel.rst:123 +#: ../../source/parallel.rst:125 msgid "" "InternEvo 在流水线并行中使用 `1F1B `_ " "（1F1B，一次前向传递后跟一次反向传递）策略。对于 1F1B 策略，有两种实现方式：" @@ -417,26 +417,26 @@ msgstr "" "forward pass followed by one backward pass) for pipeline parallel. For " "1F1B strategy, there are two implementations:" -#: ../../source/parallel.rst:125 +#: ../../source/parallel.rst:127 msgid "非交错调度器，内存高效。" msgstr "non-interleaved scheduler, which is memory-efficient" -#: ../../source/parallel.rst:126 +#: ../../source/parallel.rst:128 msgid "交错调度器，内存高效且时间高效（GPU空泡较少）。" msgstr "interleaved scheduler, which is both memory-efficient and time-efficient." -#: ../../source/parallel.rst:132 +#: ../../source/parallel.rst:134 msgid "1F1B 流水线并行调度器，采用自 `Megatron-LM `_" msgstr "" "Non-interleaved and interleaved scheduler for 1F1B pipeline parallelism, " "adopted from `Megatron-LM `_" -#: ../../source/parallel.rst:135 +#: ../../source/parallel.rst:137 msgid "非交错式流水线调度" msgstr "scheduler for non-interleaved 1F1B strategy" -#: ../../source/parallel.rst:136 -msgid "如果要使用非交错式调度, 需要设置 ``model.num_chunks = 1``。" +#: ../../source/parallel.rst:138 +msgid "如果要使用非交错式调度, 需要设置 ``model.num_chunks = 1`` 。" msgstr "" "To use non-interleaved pipeline scheduler, users need to set " "``model.num_chunks = 1`` in the config file." @@ -450,7 +450,7 @@ msgstr "" #: ../../source/parallel.rst msgid "参数" -msgstr "" +msgstr "Parameter" #: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler:5 of msgid "The number of microbatches." @@ -561,7 +561,7 @@ msgstr "" #: ../../source/parallel.rst msgid "返回类型" -msgstr "" +msgstr "Return type" #: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler.forward_backward_step:19 #: internlm.core.scheduler.pipeline_scheduler.PipelineScheduler.forward_backward_step:19 @@ -569,12 +569,12 @@ msgstr "" msgid "Tuple[:class:`torch.Tensor`]" msgstr "" -#: ../../source/parallel.rst:142 +#: ../../source/parallel.rst:144 msgid "交错式流水线调度" msgstr "scheduler for interleaved 1F1B strategy" -#: ../../source/parallel.rst:143 -msgid "如果要使用交错式调度, 需要设置 ``model.num_chunks > 1``。" +#: ../../source/parallel.rst:145 +msgid "如果要使用交错式调度, 需要设置 ``model.num_chunks > 1`` 。" msgstr "" "To use interleaved pipeline scheduler, users need to set " "``model.num_chunks > 1`` in the config file." @@ -590,49 +590,49 @@ msgid "" "communication between pipeline stages as needed." msgstr "" -#: ../../source/parallel.rst:148 +#: ../../source/parallel.rst:150 msgid "值得注意的是，在使用交错式流水线调度器时可启用通信优化功能，即在 1F1B 阶段启用异步通信，以充分利用上行/下行带宽并实现通信与计算重叠。" msgstr "" "Asynchronous communication will be enabled in 1F1B stage to make full use" " of uplink/downlink bandwidth and achieve communication overlap. " -#: ../../source/parallel.rst:150 +#: ../../source/parallel.rst:152 msgid "" -"用户需要在配置文件中设置 ``parallel.pipeline.interleaved_overlap = " -"True``。该功能启用后，将调用函数 " -"``InterleavedPipelineScheduler._run_1f1b_loop_with_overlap``，并创建 " +"用户需要在配置文件中设置 ``parallel.pipeline.interleaved_overlap = True`` " +"。该功能启用后，将调用函数 " +"``InterleavedPipelineScheduler._run_1f1b_loop_with_overlap`` ，并创建 " "``internlm.core.communication.AsynCommunicator`` 以管理异步通信。" msgstr "" -"When ``parallel.pipeline.interleaved_overlap = True``, function " +"When ``parallel.pipeline.interleaved_overlap = True`` , function " "``InterleavedPipelineScheduler._run_1f1b_loop_with_overlap`` will be " "called and ``internlm.core.communication.AsynCommunicator`` will be " "created for managing async communication." -#: ../../source/parallel.rst:152 +#: ../../source/parallel.rst:154 msgid "``1F1B-without-overlap`` 和 ``1F1B-with-overlap`` 的区别如下所示：" msgstr "" "The difference between 1F1B stage without overlap and 1F1B stage with " "overlap is shown as follows:" -#: ../../source/parallel.rst:172 +#: ../../source/parallel.rst:174 msgid "数据并行" msgstr "Data Parallel" -#: ../../source/parallel.rst:174 +#: ../../source/parallel.rst:176 msgid "InternEvo 支持数据并行。数据并行大小为:" msgstr "InternEvo supports data parallel. For data parallel:" -#: ../../source/parallel.rst:176 +#: ../../source/parallel.rst:178 msgid "" "`Data parallel size = Total number of GPUs / Pipeline parallel size / " "Tensor parallel size`" msgstr "" -#: ../../source/parallel.rst:179 +#: ../../source/parallel.rst:181 msgid "ZeRO1.5" msgstr "" -#: ../../source/parallel.rst:181 +#: ../../source/parallel.rst:183 msgid "" "ZeRO1.5 的实现使用了分层分片的概念，通过配置值 ``parallel.zero1`` " "启用了本地节点内的分片。这个方法有助于有效管理和分配模型参数和梯度，以减少内存使用并提高训练效率。" @@ -641,30 +641,30 @@ msgstr "" "via config value ``parallel.zero1``, which enables sharding within local " "nodes." -#: ../../source/parallel.rst:183 -msgid "当 ``parallel.zero1 <= 0``，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" +#: ../../source/parallel.rst:185 +msgid "当 ``parallel.zero1 <= 0`` ，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" msgstr "" -"If ``parallel.zero1 <= 0``, the size of the zero process group is equal " +"If ``parallel.zero1 <= 0`` , the size of the zero process group is equal " "to the size of the dp process group, so parameters will be divided within" " the range of dp." -#: ../../source/parallel.rst:184 -msgid "当 ``parallel.zero1 == 1``，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" +#: ../../source/parallel.rst:186 +msgid "当 ``parallel.zero1 == 1`` ，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" msgstr "" -"If ``parallel.zero1 == 1``, zero is not used, and all dp groups retain " +"If ``parallel.zero1 == 1`` , zero is not used, and all dp groups retain " "the full amount of model parameters." -#: ../../source/parallel.rst:185 +#: ../../source/parallel.rst:187 msgid "" -"当 ``parallel.zero1 > 1`` 且 ``parallel.zero1 <= " -"data_parallel_world_size``，则 zero1 进程组是数据并行进程组的子集" +"当 ``parallel.zero1 > 1`` 且 ``parallel.zero1 <= data_parallel_world_size``" +" ，则 zero1 进程组是数据并行进程组的子集" msgstr "" -"If ``parallel.zero1 > 1`` and ``parallel.zero1 <= dp world size``, the " +"If ``parallel.zero1 > 1`` and ``parallel.zero1 <= dp world size`` , the " "world size of zero is a subset of dp world size. For smaller models, it " "is usually a better choice to split the parameters within nodes with a " -"setting ``parallel.zero1 <= 8``." +"setting ``parallel.zero1 <= 8`` ." -#: ../../source/parallel.rst:187 +#: ../../source/parallel.rst:189 msgid "" "此外，用户可以在配置文件中通过 ``hybrid_zero_optimizer`` " "字段启用优化器的通信优化功能，设置桶大小，以及梯度剪裁等参数。这些设置有助于优化训练过程中的通信和计算效率，以及梯度的处理方式。" @@ -672,27 +672,27 @@ msgstr "" "Furthermore, you can enable communication-computation overlap, set bucket" " reduce size, gradient clipping parameters in the config file." -#: ../../source/parallel.rst:201 +#: ../../source/parallel.rst:203 msgid "这里有两个值得关注的通信优化点：" msgstr "There are two communication optimizations worth paying attention to here:" -#: ../../source/parallel.rst:203 +#: ../../source/parallel.rst:205 msgid "" -"overlap_sync_grad: 如果设置为 ``True``，则将训练的 ``backward pass`` 与梯度的 ``all-" +"overlap_sync_grad: 如果设置为 ``True`` ，则将训练的 ``backward pass`` 与梯度的 ``all-" "reduce`` 通信重叠" msgstr "" "overlap_sync_grad: If set True, overlapping training backward pass with " "gradients' all-reduce communication." -#: ../../source/parallel.rst:204 +#: ../../source/parallel.rst:206 msgid "" -"overlap_sync_param: 如果设置为 ``True``，则将参数的 ``broadcast`` 通信与下一步的 ``forward " -"pass`` 进行重叠" +"overlap_sync_param: 如果设置为 ``True`` ，则将参数的 ``broadcast`` 通信与下一步的 ``forward" +" pass`` 进行重叠" msgstr "" "overlap_sync_param: If set True, overlapping parameters' broadcast " "communication with next step's forward pass." -#: ../../source/parallel.rst:206 +#: ../../source/parallel.rst:208 msgid "这些优化可以加速训练过程，提高训练效率。" msgstr "" "These optimizations can speed up the training process and improve " @@ -951,3 +951,6 @@ msgstr "" #~ msgid "Whether the gradient is success updated, and the gradient." #~ msgstr "" +#~ msgid "并行训练" +#~ msgstr "Parallel Training" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/profiler.po b/doc/code-docs/locales/en/LC_MESSAGES/profiler.po index a6582200a..2bde9609a 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/profiler.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/profiler.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-27 10:51+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -19,8 +19,8 @@ msgstr "" "Generated-By: Babel 2.15.0\n" #: ../../source/profiler.rst:2 -msgid "性能分析" -msgstr "Profiler" +msgid "性能分析工具" +msgstr "Performance Analysis Tool" #: ../../source/profiler.rst:7 msgid "Torch Profiler" @@ -33,9 +33,9 @@ msgid "" "`_ ，输出的性能分析 trace 文件可以使用 " "`tensorboard `_ 进行可视化。" msgstr "" -"InternLM uses ``internlm.train.pipeline.initialize_llm_profile()`` to profile " -"performance data, execution time duration and breakdown analysis of step " -"time. The implementation is based on `torch.profiler " +"InternLM uses ``internlm.train.pipeline.initialize_llm_profile()`` to " +"profile performance data, execution time duration and breakdown analysis " +"of step time. The implementation is based on `torch.profiler " "`_ and output tracing " "files can be visualized with `tensorboard " "`_." @@ -124,7 +124,7 @@ msgstr "" #: ../../source/profiler.rst msgid "参数" -msgstr "" +msgstr "Parameter" #: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler:3 of msgid "The model to profile." @@ -156,7 +156,7 @@ msgstr "" #: ../../source/profiler.rst msgid "返回" -msgstr "" +msgstr "Return" #: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler.point:8 #: internlm.utils.simple_memory_profiler.SimpleMemoryProfiler.step:3 of diff --git a/doc/code-docs/locales/en/LC_MESSAGES/training.po b/doc/code-docs/locales/en/LC_MESSAGES/training.po index 692103b37..7ee2c17aa 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/training.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/training.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-01-23 18:01+0800\n" +"POT-Creation-Date: 2024-08-30 16:09+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -16,106 +16,128 @@ msgstr "" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" -"Generated-By: Babel 2.12.1\n" +"Generated-By: Babel 2.15.0\n" #: ../../source/training.rst:2 -msgid "训练 API" -msgstr "Training API" +msgid "启动训练脚本" +msgstr "Script of Starting Training" #: ../../source/training.rst:4 msgid "" -"InternEvo 的训练 API 由 ``internlm.core.trainer.Trainer`` " -"管理。在定义了训练引擎和调度器之后，我们可以调用 Trainer API 来执行模型训练、评估、梯度清零和参数更新等。" +"用户在安装了InternEvo之后，需要自行编写训练启动脚本，请参考： `train.py " +"`_" msgstr "" -"InternEvo training API is managed in ``internlm.core.trainer.Trainer``. " -"After defining the training engine and runtime scheduler, we can call " -"training API to perform training, evaluation, zero gradients and " -"parameter update steps." +"After installing InternEvo, users need to write their own training startup scripts. Please refer to:" +" `train.py `_ " #: ../../source/training.rst:6 -msgid "有关详细用法，请参阅 Trainer API 文档和示例。" -msgstr "" -"For detailed usage, please refer to Trainer API documentation and " -"examples." - -#: internlm.core.trainer.Trainer:1 of msgid "" -"This is a class tending for easy deployments of users' training and " -"evaluation instead of writing their own scripts." +"脚本中的流程可以分为三步：参数解析、初始化、启动训练。其中参数解析和初始化过程的具体原理参见： `训练初始化 " +"`_" msgstr "" +"The process in the script can be divided into three steps: parameter parsing, initialization, and starting training." +" For the specific principles of parameter parsing and initialization, please refer to: `Training Initialization " +"`_" -#: internlm.core.trainer.Trainer internlm.core.trainer.Trainer.execute_schedule -#: of -msgid "参数" -msgstr "" - -#: internlm.core.trainer.Trainer:4 of -msgid "Engine responsible for the process function." -msgstr "" +#: ../../source/training.rst:9 +msgid "配置参数解析" +msgstr "Configuration Parameter Parsing" -#: internlm.core.trainer.Trainer:6 of -msgid "Runtime schedule. Defaults to None." -msgstr "" - -#: internlm.core.trainer.Trainer.engine:1 of +#: ../../source/training.rst:14 msgid "" -"Returns the engine that responsible for managing the training and " -"evaluation process." +"调用 ``parse_args`` 函数，解析在启动训练传入的配置文件中设置的参数。详见： `命令行参数解析 " +"`_" msgstr "" +"Call the parse_args function to parse the parameters set in the configuration file when starting the training. For more details, see:" +" `Argument Parsing `_" -#: internlm.core.trainer.Trainer.schedule:1 of -msgid "Returns the runtime scheduler." -msgstr "" +#: ../../source/training.rst:17 +msgid "初始化过程" +msgstr "Initialization process" -#: internlm.core.trainer.Trainer.uses_pipeline:1 of -msgid "Returns whether the pipeline parallel is used or not." -msgstr "" +#: ../../source/training.rst:18 +msgid "初始化分布式训练环境" +msgstr "Initialize Distributed Training Environment" -#: internlm.core.trainer.Trainer.train:1 of -msgid "Sets the model to training mode." +#: ../../source/training.rst:23 +msgid "" +"调用 ``initialize_distributed_env`` 函数，支持通过 slurm 或 torch " +"方式启动训练脚本，并传入配置文件、端口号、进程随机种子等信息。函数详细说明如下：" msgstr "" +"Call the initialize_distributed_env function, which supports launching the training script through Slurm or Torch," +" and pass in information such as the configuration file, port number, and process random seed. Detailed description of the function is as follows:" -#: internlm.core.trainer.Trainer.eval:1 of -msgid "Sets the model to evaluation mode." -msgstr "" +#: ../../source/training.rst:27 +msgid "初始化模型" +msgstr "Initialize Model" -#: internlm.core.trainer.Trainer.zero_grad:1 of -msgid "Sets the gradient of all parameters in the model to zero." -msgstr "" +#: ../../source/training.rst:32 +msgid "" +"详细介绍请参考： `模型初始化 `_" +msgstr "Detailed introduction refer to: `Model Initialization `_" -#: internlm.core.trainer.Trainer.step:1 of -msgid "Executes the parameter update step." -msgstr "" +#: ../../source/training.rst:34 +msgid "初始化训练数据加载器" +msgstr "Initialize Training Dataloader" -#: internlm.core.trainer.Trainer.execute_schedule:1 of +#: ../../source/training.rst:39 msgid "" -"Runs the forward, loss computation, and backward for the model. Returns a" -" tuple of (output, label, loss)." -msgstr "" +"详细介绍请参考： `数据加载器初始化 `_" +msgstr "Detailed introduction refer to: `Dataloader Initialization `_" -#: internlm.core.trainer.Trainer.execute_schedule:4 of -msgid "The data iterator." -msgstr "" +#: ../../source/training.rst:41 +msgid "初始化验证数据加载器" +msgstr "Initialize Validation Dataloader" -#: internlm.core.trainer.Trainer.execute_schedule:6 of -msgid "Additional keyword arguments." -msgstr "" +#: ../../source/training.rst:46 +msgid "初始化验证数据加载器，加载过程与训练数据加载类似，通过配置文件中的 ``VALID_FOLDER `` 字段设置验证数据集路径。" +msgstr "Initialize the validation data loader, which has a loading process similar to that of the training data. The path to the validation dataset is set through the VALID_FOLDER field in the configuration file." -#: internlm.core.trainer.Trainer.execute_schedule of -msgid "返回" -msgstr "" +#: ../../source/training.rst:48 +msgid "初始化Trainer" +msgstr "Initialize Trainer" -#: internlm.core.trainer.Trainer.execute_schedule:8 of -msgid "A tuple of (output, label, loss, moe_loss)." +#: ../../source/training.rst:53 +msgid "" +"这里 ``TrainerBuilder`` 接口继承自 ``Trainer`` 类，InternEvo 的训练 API 由 " +"``internlm.core.trainer.Trainer`` 管理。在定义了训练引擎和调度器之后，我们可以调用 Trainer API " +"来执行模型训练、评估、梯度清零和参数更新等。" msgstr "" +"The TrainerBuilder interface inherits from the Trainer class, and the training API of InternEvo is managed by internlm.core.trainer.Trainer." +" After defining the training engine and scheduler, we can call the Trainer API to perform model training, evaluation, gradient clearing, and parameter updating, etc." -#: internlm.core.trainer.Trainer.execute_schedule of -msgid "返回类型" +#: ../../source/training.rst:55 +msgid "有关详细用法，请参阅 Trainer API 文档和示例。" msgstr "" +"For detailed usage, please refer to Trainer API documentation and " +"examples." -#: internlm.core.trainer.Trainer.execute_schedule:9 of -msgid "Tuple[:class:`torch.Tensor`]" -msgstr "" +#: ../../source/training.rst:61 +msgid "启动训练过程" +msgstr "Start Training Process" + +#: ../../source/training.rst:66 +msgid "首先，通过 ``self.train()`` 方法，将模型设置为training状态。" +msgstr "Firstly, by using the self.train() method, the model is set to training mode." + +#: ../../source/training.rst:68 +msgid "" +"在每一步训练的过程中，通过 ``load_new_batch`` 加载数据集。 然后通过 ``execute_schedule`` " +"调度器启动训练，调用 ``forward_backward_step`` 开始forward及backward训练过程。 之后，通过 " +"``self.step()`` 更新参数，并返回梯度值。 如果达到了需要验证的step数，则通过 ``evaluate_on_val_dls`` " +"对模型训练结果进行评估。 最后，如果开启了保存ckpt功能，通过 ``try_save_checkpoint`` " +"函数保留训练中间状态以及最终训练结果。" +msgstr "" +"During each step of the training process, the load_new_batch function is used to load the dataset." +" Then, the execute_schedule scheduler is used to initiate training, and the forward_backward_step " +"begins the forward and backward training process. Afterwards, the self.step() updates the parameters" +" and returns the gradient values. If the step count reaches the number required for validation," +" the model's training results are evaluated using evaluate_on_val_dls. Finally, if the checkpoint" +" saving function is enabled, the intermediate training state and the final training results are" +" saved using the try_save_checkpoint function." #~ msgid "InternLM 的训练流程可以归纳为两个步骤：" #~ msgstr "The training process of InternLM can be summarized into two steps: " @@ -162,3 +184,70 @@ msgstr "" #~ msgid "A tuple of (output, label, loss)." #~ msgstr "" +#~ msgid "训练 API" +#~ msgstr "Training API" + +#~ msgid "" +#~ "This is a class tending for easy" +#~ " deployments of users' training and " +#~ "evaluation instead of writing their own" +#~ " scripts." +#~ msgstr "" + +#~ msgid "参数" +#~ msgstr "" + +#~ msgid "Engine responsible for the process function." +#~ msgstr "" + +#~ msgid "Runtime schedule. Defaults to None." +#~ msgstr "" + +#~ msgid "" +#~ "Returns the engine that responsible for" +#~ " managing the training and evaluation " +#~ "process." +#~ msgstr "" + +#~ msgid "Returns the runtime scheduler." +#~ msgstr "" + +#~ msgid "Returns whether the pipeline parallel is used or not." +#~ msgstr "" + +#~ msgid "Sets the model to training mode." +#~ msgstr "" + +#~ msgid "Sets the model to evaluation mode." +#~ msgstr "" + +#~ msgid "Sets the gradient of all parameters in the model to zero." +#~ msgstr "" + +#~ msgid "Executes the parameter update step." +#~ msgstr "" + +#~ msgid "" +#~ "Runs the forward, loss computation, and" +#~ " backward for the model. Returns a" +#~ " tuple of (output, label, loss)." +#~ msgstr "" + +#~ msgid "The data iterator." +#~ msgstr "" + +#~ msgid "Additional keyword arguments." +#~ msgstr "" + +#~ msgid "返回" +#~ msgstr "" + +#~ msgid "A tuple of (output, label, loss, moe_loss)." +#~ msgstr "" + +#~ msgid "返回类型" +#~ msgstr "" + +#~ msgid "Tuple[:class:`torch.Tensor`]" +#~ msgstr "" + diff --git a/doc/code-docs/locales/en/LC_MESSAGES/usage.po b/doc/code-docs/locales/en/LC_MESSAGES/usage.po index a5fd94407..fea1cb69c 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/usage.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/usage.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-29 16:25+0800\n" +"POT-Creation-Date: 2024-08-30 15:51+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -335,8 +335,8 @@ msgid "" "OpenXLab 链接下载权重*" msgstr "" "*If you want to start training based on InternLM 7B, you can refer to " -"OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-" -"zoo) to download weights*." +"OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo)" +" to download weights*." #: ../../../usage.md:419 msgid "并行配置" @@ -469,7 +469,7 @@ msgstr "" "following examples demonstrate how to start the training in both slurm " "and torch environments." -#: ../../../usage.md:457 ../../../usage.md:492 +#: ../../../usage.md:457 ../../../usage.md:494 msgid "若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on slurm with 16 GPUs across " @@ -482,28 +482,34 @@ msgstr "" "single node, use the following command:" #: ../../../usage.md:467 +msgid "" +"其中，train.py文件的内容，请参考： [训练脚本](https://internevo.readthedocs.io/zh-" +"cn/latest/training.html)" +msgstr "The content of train.py, please refer to: [training script](https://internevo.readthedocs.io/en/latest/training.html) " + +#: ../../../usage.md:469 msgid "运行结果" msgstr "Training Results" -#: ../../../usage.md:469 +#: ../../../usage.md:471 msgid "以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下：" msgstr "" "Taking the configuration of the demo training on a single machine with 8 " "GPUs on slurm as an example, the training result log is shown below:" -#: ../../../usage.md:490 +#: ../../../usage.md:492 msgid "加载训练的checkpoint并生成" msgstr "Load the training checkpoint and generate." -#: ../../../usage.md:497 +#: ../../../usage.md:499 msgid "在配置文件中添加`generation`配置" msgstr "Add generation configuration to the configuration file." -#: ../../../usage.md:515 +#: ../../../usage.md:517 msgid "长文本生成" msgstr "Long Text Generation" -#: ../../../usage.md:517 +#: ../../../usage.md:519 msgid "" "在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K " "的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK " @@ -515,7 +521,7 @@ msgstr "" "the use of Dynamic NTK RoPE in models formatted in both Hugging Face " "format and InternLM's native format." -#: ../../../usage.md:520 +#: ../../../usage.md:522 msgid "" "对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 " "`config.json` 中的 `rotary.type` 修改为 `origin`；" @@ -524,7 +530,7 @@ msgstr "" "by default. If users wish to disable this behavior, they can modify " "`rotary.type` in the `config.json` file to `origin`." -#: ../../../usage.md:521 +#: ../../../usage.md:523 msgid "" "对于 InternLM " "本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。" @@ -533,7 +539,7 @@ msgstr "" "this behavior by adding use_dynamic_ntk_rope=True to the configuration " "dictionary when initializing the model." -#: ../../../usage.md:523 +#: ../../../usage.md:525 msgid "" "用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE " "是如何生效的。例如文件[长文本示例](../../aux_materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用" @@ -547,23 +553,23 @@ msgstr "" "text. However, after applying Dynamic NTK RoPE, the response from the " "InternLM Chat 7B v1.1 model is as follows:" -#: ../../../usage.md:526 +#: ../../../usage.md:528 msgid "![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png)" msgstr "" -#: ../../../usage.md:526 +#: ../../../usage.md:528 msgid "dynamic_ntk_answer" msgstr "" -#: ../../../usage.md:528 +#: ../../../usage.md:530 msgid "关于 Dyanmic NTK 的原理，详细请参考" msgstr "Regarding the principle of Dyanmic NTK, please refer to" -#: ../../../usage.md:529 +#: ../../../usage.md:531 msgid "[dynamically_scaled_rope_further_increases](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases)" msgstr "" -#: ../../../usage.md:530 +#: ../../../usage.md:532 msgid "[https://kexue.fm/archives/9675](https://kexue.fm/archives/9675)" msgstr "" diff --git a/doc/code-docs/source/checkpoint.rst b/doc/code-docs/source/checkpoint.rst index 4170d420c..64a70f2e6 100644 --- a/doc/code-docs/source/checkpoint.rst +++ b/doc/code-docs/source/checkpoint.rst @@ -12,6 +12,8 @@ CheckpointManager - ``enable_save_ckpt`` : 是否开启检查点存储功能（不影响检查点加载）。参数类型 ``bool`` ，必选参数。 +- ``enable_internevo2hf_ckpt`` : 是否同时保存huggingface格式的权重。如果开启，会将被并行切分后的权重整合起来保存成huggingface格式，方便后续使用hf方式加载权重，避免因并行方式不同导致的权重转换问题。默认值为： ``False`` 。 + - ``save_ckpt_folder`` : 检查点存储路径，参数类型 ``str`` ，默认为： ``None`` ，在开启检查点存储功能时为必选参数。 - ``checkpoint_every`` : 检查点存储频率，参数类型 ``int`` ，默认为： ``50`` 。 @@ -28,8 +30,6 @@ CheckpointManager - ``stop_file_path`` : 检查点存储控制文件的路径，默认值为： ``None`` ，详见 :ref: `stopfile` 。 -- ``enable_internevo2hf_ckpt`` : 是否同时保存huggingface格式的权重。如果开启，会将被并行切分后的权重整合起来保存成huggingface格式，方便后续使用hf方式加载权重，避免因并行方式不同导致的权重转换问题。默认值为： ``False`` 。 - 下面给出config文件的参数设置例子： @@ -37,6 +37,7 @@ CheckpointManager ckpt = dict( enable_save_ckpt=False, # enable ckpt save. + enable_internevo2hf_ckpt=True, # save huggingface format ckpt at the same time. save_ckpt_folder=SAVE_CKPT_FOLDER, # Path to save training ckpt. load_ckpt_info=dict(path="local:/mnt/mfs/ckpt", content=("all",), ckpt_type="internevo"), auto_resume=False, # disable auto-resume, internlm will load model checkpoint from the path of 'load_ckpt_info'. @@ -44,7 +45,6 @@ CheckpointManager async_upload=True, # async ckpt upload. (only work for boto3, volc and oss2 ckpt) async_upload_tmp_folder="/dev/shm/internlm_tmp_ckpt/", # path for temporarily files during asynchronous upload. oss_snapshot_freq=int(CHECKPOINT_EVERY / 2), # snapshot ckpt save frequency. - enable_internevo2hf_ckpt=True, # save huggingface format ckpt at the same time. ) .. autoclass:: internlm.checkpoint.checkpoint_manager.CheckpointManager diff --git a/doc/code-docs/source/data_flow.rst b/doc/code-docs/source/data_flow.rst index 8eed41d68..6db415026 100644 --- a/doc/code-docs/source/data_flow.rst +++ b/doc/code-docs/source/data_flow.rst @@ -1,4 +1,4 @@ -数据格式 +数据加载与流程 ================== .. Introduction to how data is constructed in dataloader and how data changes in forward procedure @@ -80,7 +80,6 @@ Dataloader加载数据 从Dataloader中取出数据 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - .. code-block:: python batch_data, actual_batch_size = engine.load_batch(data_iter) @@ -142,7 +141,6 @@ Dataloader加载数据 处理数据 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - .. code-block:: python _data, _label = self._load_accum_batch(data, label) @@ -274,8 +272,7 @@ Forward过程数据格式 ISP并行模式下权重切分 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -ISP并行的具体原理请参见： `并行训练 `_ +ISP并行的具体原理请参见： `并行训练 `_ internlm2模型中，涉及weight切分的参数为："wqkv"、"wo"、"w1"、"w2"、"w3"、"output"，通过new_linear函数进行切分。 @@ -319,8 +316,7 @@ internlm2模型中，涉及weight切分的参数为："wqkv"、"wo"、"w1"、"w2 MTP/MSP/FSP并行模式下权重切分 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -``MTP/MSP/FSP`` 并行的具体原理请参见： `并行训练 `_ +``MTP/MSP/FSP`` 并行的具体原理请参见： `并行训练 `_ 与 ``ISP`` 并行模式相比， ``MSP`` 并行切分权重的参数是一样的，但是切分的方式不同，在 ``ISP`` 并行模式中，所有切分参数采用列切方式，而 ``MSP`` 并行模式中，"wo"和"w2"参数采用行切方式进行切分。 @@ -356,7 +352,6 @@ MTP/MSP/FSP并行模式下权重切分 Forward整体流程 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - internlm2模型中，forward整体流程如下图所示： .. figure:: ../../imgs/forward_flow.png @@ -367,7 +362,6 @@ internlm2模型中，forward整体流程如下图所示： ISP并行模式下数据流程 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 假设配置文件中设置的tensor并行大小为 ``sp_size`` （在ISP模式下，张量并行大小即为序列化并行大小） 展开介绍每一步计算过程中，数据维度的变化情况。 @@ -375,7 +369,6 @@ ISP并行模式下数据流程 tok_embeddings计算过程 ```````````````````````````````````````` - 在embedding的计算过程中，对数据的seq_len维度做了切分。 输入参数及权重： @@ -398,10 +391,8 @@ tok_embeddings计算过程 attention计算过程 ``````````````````````````````````````` - qkv准备 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - .. code-block:: python qkv = self.wqkv(x) @@ -437,7 +428,6 @@ qkv准备 计算attention ^^^^^^^^^^^^^^^^^^^^^^^^^^ - attention计算的过程如下： .. code-block:: python @@ -471,7 +461,6 @@ attention计算的过程如下： 输出变换 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - 通过调用 "wo" 对attention计算的输出结果做变换，输出结果的维度如下： .. code-block:: python @@ -480,7 +469,6 @@ attention计算的过程如下： feed_forward计算过程 `````````````````````````````````````` - 在feed_forward前馈网络层，通过"w1"、"w2"、"w3"对输出结果做线性变换。变换之后的结果如下： .. code-block:: python @@ -498,7 +486,6 @@ feed_forward计算过程 norm计算过程 ````````````````````````` - 经过norm层计算之后的结果维度保持不变，为： .. code-block:: python @@ -508,7 +495,6 @@ norm计算过程 output计算过程 ````````````````````````` - 最后，经过output层将模型的最后一层输出转换为适合最终任务的格式，结果如下： .. code-block:: python @@ -519,12 +505,10 @@ output计算过程 MTP/MSP/FSP并行模式下数据流程 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 在 ``MTP`` 并行模式中，只有张量并行对模型权重进行切分，不涉及对数据的seq_len维度进行切分。而 ``MSP`` 和 ``FSP`` 并行模式中，均会涉及对数据进行序列化并行切分，且序列化并行与张量并行大小相同，两者共用通信组。 tok_embeddings计算过程 ```````````````````````````````` - 在embedding的计算过程中，embedding的权重会进行切分： .. code-block:: python @@ -552,14 +536,12 @@ tok_embeddings计算过程 attention计算过程 `````````````````````````````````````` - 在进入attention计算之前，如果是 ``MSP/FSP`` 并行模式，会通过 ``All-Gather`` 通信，将经过序列化并行切分后的数据聚集起来。因此，整个attention计算过程中， ``MTP/MSP/FSP`` 三种并行模式的参数维度一致。 在attention计算完成之后， ``wo`` 层中做线性变换时，如果是 ``MSP/FSP`` 并行模式，会通过 ``Reduce-Scatter`` 通信，将linear变换行切的结果整合，同时做序列化并行操作。 qkv准备 ^^^^^^^^^^^^^^^^^^ - .. code-block:: python qkv = self.wqkv(x) @@ -593,7 +575,6 @@ qkv准备 计算attention ^^^^^^^^^^^^^^^^^^^^^^^^^^ - attention计算的过程如下： .. code-block:: python @@ -611,7 +592,6 @@ attention计算的过程如下： 输出变换 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - 通过调用 "wo" 对attention计算的输出结果做变换 ``MTP`` 并行模式，输出结果的维度如下： @@ -628,7 +608,6 @@ attention计算的过程如下： feed_forward计算过程 `````````````````````````````````````` - 在feed_forward前馈网络层，通过"w1"、"w2"、"w3"对输出结果做线性变换。 ``MSP/FSP`` 并行模式下，在 ``w1`` 和 ``w3`` 线性变换层之前，需要进行 ``All-Gather`` 通信。因此， ``MTP/MSP/FSP`` :w @@ -661,7 +640,6 @@ feed_forward计算过程 norm计算过程 ````````````````````````` - 经过norm层计算之后的结果维度保持不变。 ``MTP`` 并行模式，输出结果的维度如下： @@ -680,7 +658,6 @@ norm计算过程 output计算过程 ````````````````````````` - 最后，经过output层将模型的最后一层输出转换为适合最终任务的格式，结果如下： .. code-block:: python diff --git a/doc/code-docs/source/example/index.rst b/doc/code-docs/source/example/index.rst index 136752db4..e42d50ce6 100644 --- a/doc/code-docs/source/example/index.rst +++ b/doc/code-docs/source/example/index.rst @@ -1,4 +1,4 @@ -训练样例 +训练样例介绍 ================ 7B Demo diff --git a/doc/code-docs/source/index.rst b/doc/code-docs/source/index.rst index 6d435e8e5..49e8b041c 100644 --- a/doc/code-docs/source/index.rst +++ b/doc/code-docs/source/index.rst @@ -5,7 +5,7 @@ InternEvo -======== +============= 环境构建 ------------------- @@ -31,9 +31,6 @@ InternEvo initialize -训练 API -------------------- - .. toctree:: :maxdepth: 2 @@ -51,7 +48,7 @@ InternEvo ------------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 2 data_flow diff --git a/doc/code-docs/source/initialize.rst b/doc/code-docs/source/initialize.rst index 311ad5265..bcfe67d1a 100644 --- a/doc/code-docs/source/initialize.rst +++ b/doc/code-docs/source/initialize.rst @@ -1,4 +1,4 @@ -训练构建 +训练初始化 ============== InternEvo 的训练流程可以归纳为两个步骤： @@ -85,10 +85,89 @@ InternEvo 在配置文件中使用字段 ``model_type`` 和 ``model`` 来控制 def register_model_initializer() -> None: model_initializer.register_module("INTERNLM", InternLM1) -.. _InternLM-optim-init: - 其中，"INTERNLM"为新的模型类型，InternLM1为新模型的入口函数。 +.. _InternLM-dl-init: + +数据加载器初始化 +------------------------- + +.. autofunction:: internlm.data.build_train_loader_with_data_type + +InternEvo 在配置文件中使用字段 ``data`` 来控制数据加载器初始化过程。示例数据加载器初始化配置定义如下： + +.. code-block:: python + + TRAIN_FOLDER = None # "/path/to/dataset" + VALID_FOLDER = None # "/path/to/dataset" + data = dict( + seq_len=SEQ_LEN, + # micro_num means the number of micro_batch contained in one gradient update + micro_num=4, + # packed_length = micro_bsz * SEQ_LEN + micro_bsz=2, + # defaults to the value of micro_num + valid_micro_num=4, + # defaults to 0, means disable evaluate + valid_every=50, + pack_sample_into_one=False, + total_steps=50000, + skip_batches="", + # rampup_batch_size (str): A string with three space-separated integers representing the + # starting batch size, the increment, and the number of steps between + # each increment. For example, "192 24 8" means that the batch size (micro_num) + # starts at 192 and increases by 24 every 8 steps. Defaults to None. + # (IMPORTANT): The interval step size is 'micro_bsz'. + rampup_batch_size="", + # Datasets with less than 50 rows will be discarded + min_length=50, + train_folder=TRAIN_FOLDER, + valid_folder=VALID_FOLDER, + empty_cache_and_diag_interval=200, + diag_outlier_ratio=1.1, + # whether use shared memory to load meta files + use_shm=False, + # when use shm, the default shm_path is "/dev/shm/metacache" + # shm_path="/dev/shm/metacache" + ) + +这里支持三种数据集的初始化，包括模拟数据集、已分词数据集和流式数据集。 + +- 模拟数据集 + + 如果设置TRAIN_FOLDER为None，则随机生成模拟数据集，如果设置的随机种子一样，生成的数据集保持一致。 + +- 已分词数据集 + + 如果设置TRAIN_FOLDER为本地指定路径，路径中保存经过分词之后的.bin和.meta文件，则加载已分词数据集。 + +- 流式数据集 + + 如果设置TRAIN_FOLDER为本地指定路径，路径中保存从huggingface下载的数据集，同时在data配置中，新增如下type和tokenizer_path字段，则加载流式数据集。 + +.. code-block:: python + + type="streaming", + tokenizer_path="/path/to/tokenizer", + +已分词数据集和流式数据集格式的详细说明，参见 `使用教程 `_ + +.. _InternLM-comm-init: + +并行通信初始化 +---------------- + +.. autofunction:: internlm.train.initialize_parallel_communicator + +通过 ``initialize_parallel_communicator`` 函数，初始化不同模式并行下的通信状态。 + +在 ``ISP`` 并行模式下，处理overlap优化，以及注册linear层的All_Gather通信。 +在 ``MTP`` 并行模式下，分别注册被行切以及列切的权重的通信函数。 +在 ``MSP`` 以及 ``FSP`` 并行模式下，注册序列并行的通信函数。 +在 ``MoE`` 模型中，注册MoE序列化并行通信函数。 + +.. _InternLM-optim-init: + 优化器初始化 ------------------------- @@ -157,73 +236,8 @@ InternEvo 在配置文件中使用字段 ``grad_scaler`` 、 ``hybrid_zero_optim 用户通过 ``initialize_optimizer`` 函数初始化优化器，并传入 ``isp_communicator`` 参数，以便处理 ISP 并行模式下的通信。 -.. _InternLM-dl-init: - -数据加载器初始化 -------------------------- - -.. autofunction:: internlm.train.build_train_loader_with_data_type - -InternEvo 在配置文件中使用字段 ``data`` 来控制数据加载器初始化过程。示例数据加载器初始化配置定义如下： - -.. code-block:: python - - TRAIN_FOLDER = None # "/path/to/dataset" - VALID_FOLDER = None # "/path/to/dataset" - data = dict( - seq_len=SEQ_LEN, - # micro_num means the number of micro_batch contained in one gradient update - micro_num=4, - # packed_length = micro_bsz * SEQ_LEN - micro_bsz=2, - # defaults to the value of micro_num - valid_micro_num=4, - # defaults to 0, means disable evaluate - valid_every=50, - pack_sample_into_one=False, - total_steps=50000, - skip_batches="", - # rampup_batch_size (str): A string with three space-separated integers representing the - # starting batch size, the increment, and the number of steps between - # each increment. For example, "192 24 8" means that the batch size (micro_num) - # starts at 192 and increases by 24 every 8 steps. Defaults to None. - # (IMPORTANT): The interval step size is 'micro_bsz'. - rampup_batch_size="", - # Datasets with less than 50 rows will be discarded - min_length=50, - train_folder=TRAIN_FOLDER, - valid_folder=VALID_FOLDER, - empty_cache_and_diag_interval=200, - diag_outlier_ratio=1.1, - # whether use shared memory to load meta files - use_shm=False, - # when use shm, the default shm_path is "/dev/shm/metacache" - # shm_path="/dev/shm/metacache" - ) - -这里支持三种数据集的初始化，包括模拟数据集、已分词数据集和流式数据集。 - -- 模拟数据集 - - 如果设置TRAIN_FOLDER为None，则随机生成模拟数据集，如果设置的随机种子一样，生成的数据集保持一致。 - -- 已分词数据集 - - 如果设置TRAIN_FOLDER为本地指定路径，路径中保存经过分词之后的.bin和.meta文件，则加载已分词数据集。 - -- 流式数据集 - - 如果设置TRAIN_FOLDER为本地指定路径，路径中保存从huggingface下载的数据集，同时在data配置中，新增如下type和tokenizer_path字段，则加载流式数据集。 - -.. code-block:: python - - type="streaming", - tokenizer_path="/path/to/tokenizer", - .. _InternLM-trainer-init: -已分词数据集和流式数据集格式的详细说明，参见 `使用教程 `_ - Trainer 初始化 ------------------------- diff --git a/doc/code-docs/source/mixed_precision.rst b/doc/code-docs/source/mixed_precision.rst index fef85efc9..bbada7f77 100644 --- a/doc/code-docs/source/mixed_precision.rst +++ b/doc/code-docs/source/mixed_precision.rst @@ -1,8 +1,14 @@ -混合精度 -============ +混合精度训练 +============== + +实现说明 +------------------- 混合精度是指在模型训练的过程中同时使用16位和32位浮点数类型，是一种在最小化精度损失的前提下加速模型训练的方法。 + 混合精度通过让模型的某些部分使用32位浮点数以保持数值稳定性，并在其余部分利用半精度浮点数加速训练并可以减少内存使用，在评估指标（如准确率）方面仍可以获得同等的训练效果。 +通过 ``NaiveAMPModel`` 类实现混合精度，该类的具体说明如下： + .. autoclass:: internlm.core.naive_amp.NaiveAMPModel InternEvo默认将模型转换为16位浮点数类型进行训练（在配置文件中可以设置默认类型为其他数据类型）。在使用混合精度时，需要在构建模型时使用 diff --git a/doc/code-docs/source/monitor.rst b/doc/code-docs/source/monitor.rst index 568dad285..e6a872162 100644 --- a/doc/code-docs/source/monitor.rst +++ b/doc/code-docs/source/monitor.rst @@ -4,9 +4,9 @@ 监控 ----------------- -InternEvo 使用 ``internlm.monitor.monitor.initialize_monitor_manager()`` 来初始化上下文监控管理。其中，一个实例化的单例对象 ``internlm.monitor.monitor.MonitorManager`` 将管理监控线程并使用 ``internlm.monitor.monitor.MonitorTracker`` 来跟踪模型训练生命周期和训练状态。 +InternEvo 使用 ``internlm.monitor.initialize_monitor_manager()`` 来初始化上下文监控管理。其中，一个实例化的单例对象 ``internlm.monitor.monitor.MonitorManager`` 将管理监控线程并使用 ``internlm.monitor.monitor.MonitorTracker`` 来跟踪模型训练生命周期和训练状态。 -.. autofunction:: internlm.monitor.monitor.initialize_monitor_manager +.. autofunction:: internlm.monitor.initialize_monitor_manager .. autoclass:: internlm.monitor.monitor.MonitorManager :members: @@ -35,15 +35,11 @@ InternEvo轻量级监控工具采用心跳机制实时监测训练过程中的 enable_feishu_alert=False, feishu_alert_address=None, light_monitor_address=None, + alert_file_path=f"llm_alter/{JOB_NAME}_alert.log", ), ) - enable_feishu_alert (bool)：是否启用飞书告警。默认值：False。 - feishu_alert_address (str)：飞书告警的 Webhook 地址。默认值：None。 - light_monitor_address (str)：轻量监控的地址。默认值：None。 - -InternEvo 使用 ``internlm.monitor.alert.initialize_light_monitor`` 来初始化轻量监控客户端。一旦初始化完成，它会建立与监控服务器的连接。在训练过程中，使用 ``internlm.monitor.alert.send_heartbeat`` 来发送不同类型的心跳信息至监控服务器。监控服务器会根据这些心跳信息来检测训练是否出现异常，并在需要时发送警报消息。 - -.. autofunction:: internlm.monitor.alert.initialize_light_monitor - -.. autofunction:: internlm.monitor.alert.send_heartbeat +- alert_file_path (str)：告警存储路径。默认值：None。 diff --git a/doc/code-docs/source/parallel.rst b/doc/code-docs/source/parallel.rst index 496b7be64..1ad9ff639 100644 --- a/doc/code-docs/source/parallel.rst +++ b/doc/code-docs/source/parallel.rst @@ -1,4 +1,4 @@ -并行训练 +并行模式与原理 ================== .. Brief introduction to training parallelism, and how-to guide about config setting @@ -18,9 +18,9 @@ InternEvo 的并行设置由配置文件中的 ``parallel`` 字段指定，用 - zero1：zero 并行策略，分如下三种情况，默认值为 -1 - - 当 ``size <= 0``，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配 - - 当 ``size == 1``，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数 - - 当 ``size > 1`` 且 ``zero1 <= data_parallel_size``，则 zero1 进程组是数据并行进程组的子集 + - 当 ``size <= 0`` ，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配 + - 当 ``size == 1`` ，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数 + - 当 ``size > 1`` 且 ``zero1 <= data_parallel_size`` ，则 zero1 进程组是数据并行进程组的子集 - tensor：张量并行策略 @@ -45,6 +45,8 @@ InternEvo 的并行设置由配置文件中的 ``parallel`` 字段指定，用 注意：数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小 +.. _InternLM-tensor-parallel: + 张量并行 ----------------- @@ -60,7 +62,7 @@ MTP(Megatron-LM Tensor Parallel)， 为默认的张量并行模型，引用自 ` Transformer layer with tensor parallelism. -MTP 主要对 `attention `_ 和 `linear `_ 这两个模块进行张量并行操作。假设张量并行大小为 ``tp``，输入数据的序列长度为 ``seqlen``，隐藏层大小为 ``hidden size``，则张量并行期间产生的激活值 ``shape`` 为 ``[seqlen, hidden_size/tp]``。 +MTP 主要对 `attention `_ 和 `linear `_ 这两个模块进行张量并行操作。假设张量并行大小为 ``tp`` ，输入数据的序列长度为 ``seqlen`` ，隐藏层大小为 ``hidden size`` ，则张量并行期间产生的激活值 ``shape`` 为 ``[seqlen, hidden_size/tp]`` 。 MTP 张量并行引入的通信如上图所示，其中 ``f`` 和 ``f̄`` 是共轭的。在前向传递中 ``f`` 是无操作，而在反向传递中进行 ``all-reduce`` 操作。而 ``f̄`` 在前向传递中进行 ``all-reduce`` 操作，在反向传递中是无操作。 @@ -74,11 +76,11 @@ MSP(Megatron-LM Sequence Parallel)，引用自 `Megatron-LM Sequence Parallel `_ 配置文件为例，将 ``tensor.mode`` 字段设置为 ``isp``，而 ``tensor.size`` 字段代表的是数据 ``seqlen`` 维度切分大小。ISP 算法可与 ``weight parallel`` 结合使用，其中 ``weight.size`` 字段代表的是模型权重切分大小，将 ``weight.overlap`` 字段设置为 ``True`` 即为开启计算与通信的 ``overlap``，可提高训练性能。将 ``weight.memory_pool`` 字段设置为 ``True`` 即为开启显存池管理功能，可一定程度降低 GPU 显存碎片的可能性，提高显存利用率。 +以 `configs/7B_isp_sft.py `_ 配置文件为例，将 ``tensor.mode`` 字段设置为 ``isp`` ，而 ``tensor.size`` 字段代表的是数据 ``seqlen`` 维度切分大小。ISP 算法可与 ``weight parallel`` 结合使用，其中 ``weight.size`` 字段代表的是模型权重切分大小，将 ``weight.overlap`` 字段设置为 ``True`` 即为开启计算与通信的 ``overlap`` ，可提高训练性能。将 ``weight.memory_pool`` 字段设置为 ``True`` 即为开启显存池管理功能，可一定程度降低 GPU 显存碎片的可能性，提高显存利用率。 .. code-block:: python @@ -108,13 +110,13 @@ ISP(Intern Sequence Parallel)，InternEvo 系统自研的灵活可扩展序列 :scale: 30% :class: with-border -如图所示，ISP 的序列化并行范围覆盖整个 ``Transformer`` 模型层，模型权重并行主要针对 ``Attention`` 和 ``MLP Block`` 的 ``Linear module``。 +如图所示，ISP 的序列化并行范围覆盖整个 ``Transformer`` 模型层，模型权重并行主要针对 ``Attention`` 和 ``MLP Block`` 的 ``Linear module`` 。 通信原语变化情况为，在前向传递时，每个 ``Linear`` 需要进行模型权重的 ``all-gather`` 通信；在后向传递时，每个 ``Linear`` 在进行后向计算前需要进行模型权重的 ``all-gather`` 通信，在后向计算后，需要进行模型权重的梯度的 ``reduce-scatter`` 通信操作。 需要注意的是，与 MSP 和 FSP 相比，在进行 ``attention score`` 计算时，ISP 也有通信原语的一些变化，如 ``Self-Atten`` 前后各增加了一个 ``all-to-all`` 通信操作，用于完成激活值形状的转置，目的是在进行 ``attention score`` 计算时能保持原有的张量并行的模式。 -关于 ISP 算法更多的设计思路和性能评测，请参考论文 `InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding `_。 +关于 ISP 算法更多的设计思路和性能评测，请参考论文 `InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding `_ 。 流水线并行 @@ -133,21 +135,21 @@ InternEvo 在流水线并行中使用 `1F1B 1``。 +如果要使用交错式调度, 需要设置 ``model.num_chunks > 1`` 。 .. autoclass:: internlm.core.scheduler.pipeline_scheduler.InterleavedPipelineScheduler :members: 值得注意的是，在使用交错式流水线调度器时可启用通信优化功能，即在 1F1B 阶段启用异步通信，以充分利用上行/下行带宽并实现通信与计算重叠。 -用户需要在配置文件中设置 ``parallel.pipeline.interleaved_overlap = True``。该功能启用后，将调用函数 ``InterleavedPipelineScheduler._run_1f1b_loop_with_overlap``，并创建 ``internlm.core.communication.AsynCommunicator`` 以管理异步通信。 +用户需要在配置文件中设置 ``parallel.pipeline.interleaved_overlap = True`` 。该功能启用后，将调用函数 ``InterleavedPipelineScheduler._run_1f1b_loop_with_overlap`` ，并创建 ``internlm.core.communication.AsynCommunicator`` 以管理异步通信。 ``1F1B-without-overlap`` 和 ``1F1B-with-overlap`` 的区别如下所示： @@ -180,9 +182,9 @@ ZeRO1.5 ZeRO1.5 的实现使用了分层分片的概念，通过配置值 ``parallel.zero1`` 启用了本地节点内的分片。这个方法有助于有效管理和分配模型参数和梯度，以减少内存使用并提高训练效率。 -1. 当 ``parallel.zero1 <= 0``，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配 -2. 当 ``parallel.zero1 == 1``，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数 -3. 当 ``parallel.zero1 > 1`` 且 ``parallel.zero1 <= data_parallel_world_size``，则 zero1 进程组是数据并行进程组的子集 +1. 当 ``parallel.zero1 <= 0`` ，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配 +2. 当 ``parallel.zero1 == 1`` ，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数 +3. 当 ``parallel.zero1 > 1`` 且 ``parallel.zero1 <= data_parallel_world_size`` ，则 zero1 进程组是数据并行进程组的子集 此外，用户可以在配置文件中通过 ``hybrid_zero_optimizer`` 字段启用优化器的通信优化功能，设置桶大小，以及梯度剪裁等参数。这些设置有助于优化训练过程中的通信和计算效率，以及梯度的处理方式。 @@ -200,8 +202,8 @@ ZeRO1.5 的实现使用了分层分片的概念，通过配置值 ``parallel.zer 这里有两个值得关注的通信优化点： -- overlap_sync_grad: 如果设置为 ``True``，则将训练的 ``backward pass`` 与梯度的 ``all-reduce`` 通信重叠 -- overlap_sync_param: 如果设置为 ``True``，则将参数的 ``broadcast`` 通信与下一步的 ``forward pass`` 进行重叠 +- overlap_sync_grad: 如果设置为 ``True`` ，则将训练的 ``backward pass`` 与梯度的 ``all-reduce`` 通信重叠 +- overlap_sync_param: 如果设置为 ``True`` ，则将参数的 ``broadcast`` 通信与下一步的 ``forward pass`` 进行重叠 这些优化可以加速训练过程，提高训练效率。 diff --git a/doc/code-docs/source/profiler.rst b/doc/code-docs/source/profiler.rst index babd9646c..db527b4ad 100644 --- a/doc/code-docs/source/profiler.rst +++ b/doc/code-docs/source/profiler.rst @@ -1,5 +1,5 @@ -性能分析 -======== +性能分析工具 +================== .. Mainly about the usage of torch profiler and memory profiler diff --git a/doc/code-docs/source/training.rst b/doc/code-docs/source/training.rst index ce7dae569..a0b4c2288 100644 --- a/doc/code-docs/source/training.rst +++ b/doc/code-docs/source/training.rst @@ -1,9 +1,72 @@ -训练 API -============ +启动训练脚本 +=================== -InternEvo 的训练 API 由 ``internlm.core.trainer.Trainer`` 管理。在定义了训练引擎和调度器之后，我们可以调用 Trainer API 来执行模型训练、评估、梯度清零和参数更新等。 +用户在安装了InternEvo之后，需要自行编写训练启动脚本，请参考： `train.py `_ + +脚本中的流程可以分为三步：参数解析、初始化、启动训练。其中参数解析和初始化过程的具体原理参见： `训练初始化 `_ + +配置参数解析 +------------------------ +.. code-block:: python + + args = parse_args() + +调用 ``parse_args`` 函数，解析在启动训练传入的配置文件中设置的参数。详见： `命令行参数解析 `_ + +初始化过程 +------------------------ +- 初始化分布式训练环境 +.. code-block:: python + + initialize_distributed_env(config=args.config, launcher=args.launcher, master_port=args.port, seed=args.seed) + +调用 ``initialize_distributed_env`` 函数，支持通过 slurm 或 torch 方式启动训练脚本，并传入配置文件、端口号、进程随机种子等信息。函数详细说明如下： + +.. autofunction:: internlm.initialize.initialize_distributed_env + +- 初始化模型 +.. code-block:: python + + model = initialize_model() + +详细介绍请参考： `模型初始化 `_ + +- 初始化训练数据加载器 +.. code-block:: python + + train_dl, dataset_types = build_train_loader_with_data_type() + +详细介绍请参考： `数据加载器初始化 `_ + +- 初始化验证数据加载器 +.. code-block:: python + + val_dls = build_valid_loader_with_data_type() + +初始化验证数据加载器，加载过程与训练数据加载类似，通过配置文件中的 ``VALID_FOLDER `` 字段设置验证数据集路径。 + +- 初始化Trainer +.. code-block:: python + + trainer = TrainerBuilder(model, train_dl, val_dls, **kwargs) + +这里 ``TrainerBuilder`` 接口继承自 ``Trainer`` 类，InternEvo 的训练 API 由 ``internlm.core.trainer.Trainer`` 管理。在定义了训练引擎和调度器之后，我们可以调用 Trainer API 来执行模型训练、评估、梯度清零和参数更新等。 有关详细用法，请参阅 Trainer API 文档和示例。 .. autoclass:: internlm.core.trainer.Trainer :members: + +启动训练过程 +------------------ +.. code-block:: python + + trainer.fit() + +首先，通过 ``self.train()`` 方法，将模型设置为training状态。 + +在每一步训练的过程中，通过 ``load_new_batch`` 加载数据集。 +然后通过 ``execute_schedule`` 调度器启动训练，调用 ``forward_backward_step`` 开始forward及backward训练过程。 +之后，通过 ``self.step()`` 更新参数，并返回梯度值。 +如果达到了需要验证的step数，则通过 ``evaluate_on_val_dls`` 对模型训练结果进行评估。 +最后，如果开启了保存ckpt功能，通过 ``try_save_checkpoint`` 函数保留训练中间状态以及最终训练结果。 diff --git a/doc/en/usage.md b/doc/en/usage.md index 416fbda66..fa05710bf 100644 --- a/doc/en/usage.md +++ b/doc/en/usage.md @@ -414,6 +414,8 @@ If you want to start distributed training on torch with 8 GPUs on a single node, $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" ``` +The content of train.py, please refer to [Training Scrip](https://internevo.readthedocs.io/en/latest/training.html) + ### Training Results Taking the configuration of the demo training on a single machine with 8 GPUs on slurm as an example, the training result log is shown below: diff --git a/doc/usage.md b/doc/usage.md index 96d19887b..7759e5da2 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -463,6 +463,8 @@ $ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python trai $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" ``` +其中，train.py文件的内容，请参考： [训练脚本](https://internevo.readthedocs.io/zh-cn/latest/training.html) + ### 运行结果 以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下： From 5e0ac248b1ad4eded9d63d335584acf71a172105 Mon Sep 17 00:00:00 2001 From: sallyjunjun Date: Mon, 2 Sep 2024 11:17:33 +0800 Subject: [PATCH 5/5] fix README --- README-ja-JP.md | 137 +++++++++++++++-- README-zh-Hans.md | 137 +++++++++++++++-- README.md | 137 +++++++++++++++-- doc/code-docs/locales/en/LC_MESSAGES/usage.po | 138 +++++++++--------- doc/en/usage.md | 4 +- doc/usage.md | 4 +- 6 files changed, 456 insertions(+), 101 deletions(-) diff --git a/README-ja-JP.md b/README-ja-JP.md index 679e2a25f..a9327eccc 100644 --- a/README-ja-JP.md +++ b/README-ja-JP.md @@ -34,27 +34,142 @@ InternEvoは、広範な依存関係を必要とせずにモデルの事前ト InternEvoトレーニングフレームワークを基に、当社はInternLM-7BシリーズやInternLM-20Bシリーズを含むさまざまな大規模言語モデルを継続的にリリースしています。これらのモデルは、LLaMAのような数多くの有名なオープンソースの大規模言語モデルや、その他の業界をリードするモデルを大きく上回る性能を発揮しています。 +## インストール + +まず、指定バージョンのtorch、torchvision、torchaudio、およびtorch-scatterをインストールしてください。 +たとえば： +```bash +pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.1.0+cu118 torchvision==0.16.0+cu118 torchaudio==2.1.0+cu118 +pip install torch-scatter -f https://data.pyg.org/whl/torch-2.1.0+cu118.html +``` + +InternEvoをインストールします： +```bash +pip install InternEvo +``` + +flash-attention（バージョンv2.2.1）をインストールします： + +もしflash-attentionを使用してトレーニングを加速する必要があり、あなたの環境でサポートされている場合は、以下の通りインストールしてください： +```bash +pip install flash-attn==2.2.1 +``` + +インストール環境やソースコードインストールに関するより詳細な情報については、以下のリンクを参照してください [インストールチュートリアル](https://internevo.readthedocs.io/en/latest/install.html#) ## クイックスタート -InternEvoのインストール、データ処理、事前トレーニング、およびファインチューニングを開始するためには、[使用チュートリアル](./doc/en/usage.md) を参照してください。 +### トレーニングスクリプト -詳細については、以下をご確認ください: [internevo.readthedocs.io](https://internevo.readthedocs.io/en/latest/?badge=latest) +まず、トレーニングスクリプトを準備してください [train.py](https://github.com/InternLM/InternEvo/blob/develop/train.py) -## システムアーキテクチャ +より詳細な説明については、以下を参照してください [トレーニングチュートリアル](https://internevo.readthedocs.io/en/latest/training.html#) -アーキテクチャの詳細については、[システムアーキテクチャドキュメント](./doc/en/structure.md)を参照してください。 +### データの準備 + +次に、トレーニングまたはファインチューニングのためのデータを準備してください。 + +データセットをHuggingfaceからダウンロードし、roneneldan/TinyStories データセットを例にとって説明します： +```bash +huggingface-cli download --repo-type dataset --resume-download "roneneldan/TinyStories" --local-dir "/mnt/petrelfs/hf-TinyStories" +``` + +トークナイザーをローカルパスに配置してください。例として、https://huggingface.co/internlm/internlm2-7b/tree/main から special_tokens_map.json、tokenizer.model、tokenizer_config.json、tokenization_internlm2.py、tokenization_internlm2_fast.py をローカルの /mnt/petrelfs/hf-internlm2-tokenizer にダウンロードしてください。 + +次に、設定ファイルを以下のように変更します： +```bash +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" +data = dict( + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", +) +``` + +他のタイプのデータセットの準備については、以下を参照してください [使用方法のチュートリアル](https://internevo.readthedocs.io/en/latest/usage.html#) + +### Configuration File + +設定ファイルの内容は以下の通りです：[7B_sft.py](https://github.com/InternLM/InternEvo/blob/develop/configs/7B_sft.py) -## トレーニングパフォーマンス +より詳細な紹介については、以下を参照してください [使用方法のチュートリアル](https://internevo.readthedocs.io/en/latest/usage.html#) -InternEvoは、Flash-Attention、Apexなどの高性能モデルオペレーターを深く統合してトレーニング効率を向上させています。Hybrid Zeroテクニックを構築することにより、計算と通信の効率的な重複を実現し、トレーニング中のクロスノード通信トラフィックを大幅に削減します。InternEvoは、7Bモデルを8つのGPUから1024個のGPUに拡張することをサポートし、千のGPUスケールで最大90%の加速効率、180 TFLOPSを超えるトレーニングスループット、そしてGPUあたり秒間3600トークン以上の平均を実現します。以下の表は、異なる構成でのInternEvoのスケーラビリティテストデータを示しています： +### トレーニング開始 -| GPU 番号 | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | -| ---------------- | ---- | ---- | ---- | ---- | ----- | ----- | ----- | ------ | -| TGS | 4078 | 3939 | 3919 | 3944 | 3928 | 3920 | 3835 | 3625 | -| TFLOPS | 193 | 191 | 188 | 188 | 187 | 185 | 186 | 184 | +トレーニングは、slurmまたはtorch distributed環境で開始できます。 + +Slurm環境で2ノード16カードを使用する場合、コマンドは以下の通りです： +```bash +$ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python train.py --config ./configs/7B_sft.py +``` + +torchを使用し、1ノード8カードで実行する場合、コマンドは以下の通りです： +```bash +$ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" +``` -TGSは、GPUごとの秒間平均処理トークン数を表しています。より多くのパフォーマンステストデータについては、[トレーニングパフォーマンスドキュメント](./doc/en/train_performance.md) をご参照ください。 +## システムアーキテクチャ + +アーキテクチャの詳細については、[システムアーキテクチャドキュメント](./doc/en/structure.md)を参照してください。 + +## フィーチャーズー + +
+ InternEvo フィーチャーズー +
+ + + + + + + + + + + + + + + + + +
+ データ + + モデル + + 並列 + + ツール +
+
    +
  • Tokenized
    • +
  • Streaming
    • +
+ 		+ + +
    +
  • ZeRO 1.5
    • +
  • 1F1B Pipeline Parallel
    • +
  • PyTorch FSDP Training
    • +
  • Megatron-LM Tensor Parallel (MTP)
    • +
  • Megatron-LM Sequence Parallel (MSP)
    • +
  • Flash-Attn Sequence Parallel (FSP)
    • +
  • Intern Sequence Parallel (ISP)
    • +
  • Memory Profiling
    • +
+ 		+ +
## コントリビュート diff --git a/README-zh-Hans.md b/README-zh-Hans.md index fc9e78d4d..582dfd8ac 100644 --- a/README-zh-Hans.md +++ b/README-zh-Hans.md @@ -34,27 +34,142 @@ InternEvo是一个开源的轻量级训练框架，旨在支持无需大量依 基于InternEvo训练框架，我们累计发布了一系列大语言模型，包括InternLM-7B系列和InternLM-20B系列，这些模型在性能上显著超越了许多知名的开源LLMs，如LLaMA和其他模型。 +## 安装 + +首先，安装指定版本的torch, torchvision, torchaudio, and torch-scatter. +例如: +```bash +pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.1.0+cu118 torchvision==0.16.0+cu118 torchaudio==2.1.0+cu118 +pip install torch-scatter -f https://data.pyg.org/whl/torch-2.1.0+cu118.html +``` + +安装InternEvo: +```bash +pip install InternEvo +``` + +安装flash-attention (version v2.2.1): + +如果需要使用flash-attention加速训练, 并且环境中支持, 按如下方式安装: +```bash +pip install flash-attn==2.2.1 +``` + +有关安装环境以及源码方式安装的更多详细信息，请参考[安装文档](https://internevo.readthedocs.io/zh-cn/latest/install.html#) + ## 快速开始 -请查看 [用户说明](./doc/usage.md) 来开始InternEvo的安装、数据处理、预训练与微调。 +### 训练脚本 -更多细节请查看文档 [internevo.readthedocs.io](https://internevo.readthedocs.io/zh_CN/latest/?badge=latest) +首先，准备训练脚本，参考：[train.py](https://github.com/InternLM/InternEvo/blob/develop/train.py) -## 系统架构 +有关训练脚本的更多详细解释，请参考[训练文档](https://internevo.readthedocs.io/zh-cn/latest/training.html#) -系统架构细节请参考：[系统架构文档](./doc/structure.md) +### 数据准备 + +其次，准备训练或者微调的数据。 + +从huggingface下载数据集，以 `roneneldan/TinyStories` 数据集为例: +```bash +huggingface-cli download --repo-type dataset --resume-download "roneneldan/TinyStories" --local-dir "/mnt/petrelfs/hf-TinyStories" +``` + +获取分词器到本地路径。例如，从 `https://huggingface.co/internlm/internlm2-7b/tree/main` 下载special_tokens_map.json、tokenizer.model、tokenizer_config.json、tokenization_internlm2.py和tokenization_internlm2_fast.py文件，并保存到本地路径： `/mnt/petrelfs/hf-internlm2-tokenizer` 。 -## 框架性能 +然后，修改配置文件： +```bash +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" +data = dict( + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", +) +``` + +对于其他数据集类型的准备方式，请参考：[用户文档](https://internevo.readthedocs.io/zh-cn/latest/usage.html#) + +### 配置文件 + +配置文件的内容，请参考：[7B_sft.py](https://github.com/InternLM/InternEvo/blob/develop/configs/7B_sft.py) -InternEvo深度集成了Flash-Attention、Apex等高性能计算库，以提高训练效率。通过构建Hybrid Zero技术，InternEvo可在训练过程中实现计算和通信的有效重叠，显著降低跨节点通信流量。InternEvo支持将7B模型从8个GPU扩展到1024个GPU，在千卡规模下可实现高达90%的加速效率，超过180 TFLOPS的训练吞吐量，平均每个GPU每秒可处理超过3600个tokens。下表展示了InternEvo在不同配置下的可扩展性测试数据： +关于配置文件更多详细的说明，请参考：[用户文档](https://internevo.readthedocs.io/zh-cn/latest/usage.html#) -| GPU Number | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | -| ---------------- | ---- | ---- | ---- | ---- | ----- | ----- | ----- | ------ | -| TGS | 4078 | 3939 | 3919 | 3944 | 3928 | 3920 | 3835 | 3625 | -| TFLOPS | 193 | 191 | 188 | 188 | 187 | 185 | 186 | 184 | +### 开启训练 -TGS表示每张GPU每秒可处理的平均Tokens数量。更多模型性能测试数据细节请查看 [训练性能文档](./doc/train_performance.md) +可以在 slurm 或者 torch 分布式环境中开始训练。 + +slurm环境，双机16卡，启动训练命令如下： +```bash +$ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python train.py --config ./configs/7B_sft.py +``` + +torch环境，单机8卡，启动训练命令如下： +```bash +$ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" +``` +## 系统架构 + +系统架构细节请参考：[系统架构文档](./doc/structure.md) + +## 特性列表 + +
+ InternEvo 特性列表 +
+ + + + + + + + + + + + + + + + + +
+ 数据集 + + 模型 + + 并行模式 + + 工具 +
+
    +
  • 已分词数据集
    • +
  • 流式数据集
    • +
+ 		+ + +
    +
  • ZeRO 1.5
    • +
  • 1F1B 流水线并行
    • +
  • PyTorch FSDP 训练
    • +
  • Megatron-LM 张量并行 (MTP)
    • +
  • Megatron-LM 序列化并行 (MSP)
    • +
  • Flash-Attn 序列化并行 (FSP)
    • +
  • Intern 序列化并行 (ISP)
    • +
  • 内存性能分析
    • +
+ 		+ +
## 贡献 diff --git a/README.md b/README.md index 4fd668408..1a22f4198 100644 --- a/README.md +++ b/README.md @@ -34,27 +34,142 @@ InternEvo is an open-sourced lightweight training framework aims to support mode Based on the InternEvo training framework, we are continually releasing a variety of large language models, including the InternLM-7B series and InternLM-20B series, which significantly outperform numerous renowned open-source LLMs such as LLaMA and other leading models in the field. +## Installation + +First, install the specified versions of torch, torchvision, torchaudio, and torch-scatter. +For example: +```bash +pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.1.0+cu118 torchvision==0.16.0+cu118 torchaudio==2.1.0+cu118 +pip install torch-scatter -f https://data.pyg.org/whl/torch-2.1.0+cu118.html +``` + +Install InternEvo: +```bash +pip install InternEvo +``` + +Install flash-attention (version v2.2.1): + +If you need to use flash-attention to accelerate training, and it is supported in your environment, install as follows: +```bash +pip install flash-attn==2.2.1 +``` + +For more detailed information about installation environment or source code installation, please refer to [Install Tutorial](https://internevo.readthedocs.io/en/latest/install.html#) ## Quick Start -Please refer to [Usage Tutorial](./doc/en/usage.md) to start InternEvo installation, data processing, pre-training and fine-tuning. +### Train Script -For more details, please check [internevo.readthedocs.io](https://internevo.readthedocs.io/en/latest/?badge=latest) +Firstly, prepare training script as [train.py](https://github.com/InternLM/InternEvo/blob/develop/train.py) -## System Architecture +For more detailed explanation, please refer to [Training Tutorial](https://internevo.readthedocs.io/en/latest/training.html#) -Please refer to the [System Architecture document](./doc/en/structure.md) for architecture details. +### Data Preparation + +Secondly, prepare data for training or fine-tuning. + +Download dataset from huggingface, take `roneneldan/TinyStories` dataset as example: +```bash +huggingface-cli download --repo-type dataset --resume-download "roneneldan/TinyStories" --local-dir "/mnt/petrelfs/hf-TinyStories" +``` + +Achieve tokenizer to local path. For example, download special_tokens_map.json、tokenizer.model、tokenizer_config.json、tokenization_internlm2.py and tokenization_internlm2_fast.py from `https://huggingface.co/internlm/internlm2-7b/tree/main` to local `/mnt/petrelfs/hf-internlm2-tokenizer` . + +Then modify configuration file as follows: +```bash +TRAIN_FOLDER = "/mnt/petrelfs/hf-TinyStories" +data = dict( + type="streaming", + tokenizer_path="/mnt/petrelfs/hf-internlm2-tokenizer", +) +``` + +For other type dataset preparation, please refer to [Usage Tutorial](https://internevo.readthedocs.io/en/latest/usage.html#) + +### Configuration File + +The content of configuration file is as [7B_sft.py](https://github.com/InternLM/InternEvo/blob/develop/configs/7B_sft.py) -## Performance +For more detailed introduction, please refer to [Usage Tutorial](https://internevo.readthedocs.io/en/latest/usage.html#) -InternEvo deeply integrates Flash-Attention, Apex and other high-performance model operators to improve training efficiency. By building the Hybrid Zero technique, it achieves efficient overlap of computation and communication, significantly reducing cross-node communication traffic during training. InternEvo supports expanding the 7B model from 8 GPUs to 1024 GPUs, with an acceleration efficiency of up to 90% at the thousand-GPU scale, a training throughput of over 180 TFLOPS, and an average of over 3600 tokens per GPU per second. The following table shows InternEvo's scalability test data at different configurations: +### Train Start -| GPU Number | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | -| ---------------- | ---- | ---- | ---- | ---- | ----- | ----- | ----- | ------ | -| TGS | 4078 | 3939 | 3919 | 3944 | 3928 | 3920 | 3835 | 3625 | -| TFLOPS | 193 | 191 | 188 | 188 | 187 | 185 | 186 | 184 | +Training can be started on slurm or torch distributed environment. + +On slurm, using 2 nodes and 16 cards, the command is as follows: +```bash +$ srun -p internllm -N 2 -n 16 --ntasks-per-node=8 --gpus-per-task=1 python train.py --config ./configs/7B_sft.py +``` + +On torch, using 1 node and 8 cards, the command is as follows: +```bash +$ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py --launcher "torch" +``` -TGS represents the average number of tokens processed per GPU per second. For more performance test data, please refer to the [Training Performance document](./doc/en/train_performance.md) for further details. +## System Architecture + +Please refer to the [System Architecture document](./doc/en/structure.md) for architecture details. + +## Feature Zoo + +
+ InternEvo Feature Zoo +
+ + + + + + + + + + + + + + + + + +
+ Data + + Model + + Parallel + + Tool +
+
    +
  • Tokenized
    • +
  • Streaming
    • +
+ 		+ + +
    +
  • ZeRO 1.5
    • +
  • 1F1B Pipeline Parallel
    • +
  • PyTorch FSDP Training
    • +
  • Megatron-LM Tensor Parallel (MTP)
    • +
  • Megatron-LM Sequence Parallel (MSP)
    • +
  • Flash-Attn Sequence Parallel (FSP)
    • +
  • Intern Sequence Parallel (ISP)
    • +
  • Memory Profiling
    • +
+ 		+ +
## Contribution diff --git a/doc/code-docs/locales/en/LC_MESSAGES/usage.po b/doc/code-docs/locales/en/LC_MESSAGES/usage.po index fea1cb69c..615fabf77 100644 --- a/doc/code-docs/locales/en/LC_MESSAGES/usage.po +++ b/doc/code-docs/locales/en/LC_MESSAGES/usage.po @@ -7,7 +7,7 @@ msgid "" msgstr "" "Project-Id-Version: InternLM \n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2024-08-30 15:51+0800\n" +"POT-Creation-Date: 2024-09-02 11:13+0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language: en\n" @@ -222,89 +222,93 @@ msgid "对 Alpaca 数据进行 tokenize，使用以下命令" msgstr "Tokenize the Alpaca dataset using the following command:" #: ../../../usage.md:93 -msgid "建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize" +msgid "建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize。" msgstr "" "It is recommended that users refer to alpaca_tokenizer.py to write new " -"scripts to tokenize their own datasets" +"scripts to tokenize their own datasets." #: ../../../usage.md:95 +msgid "微调任务中，也同样可以使用huggingface格式数据集，与预训练中的准备过程一致。" +msgstr "In fine-tuning tasks, Hugging Face formatted datasets can also be used, consistent with the preparation process in pre-training." + +#: ../../../usage.md:97 msgid "训练配置" msgstr "Training Configuration" -#: ../../../usage.md:97 +#: ../../../usage.md:99 msgid "以 7B Demo 的配置文件`configs/7B_sft.py`为例：" msgstr "" "Taking the configuration file `configs/7B_sft.py` for the 7B demo as an " "example," -#: ../../../usage.md:310 +#: ../../../usage.md:312 msgid "接下来将详细介绍启动一个模型训练所需要进行的数据、模型、并行和监控等相关的配置。" msgstr "" "let's discuss the data, model, parallel and monitoring configurations " "required to start a model training." -#: ../../../usage.md:312 +#: ../../../usage.md:314 msgid "数据配置" msgstr "Data Configuration" -#: ../../../usage.md:313 +#: ../../../usage.md:315 msgid "数据相关的关键参数配置及释义如下所示：" msgstr "Here are the key parameters and their explanations for data configuration:" -#: ../../../usage.md:328 +#: ../../../usage.md:330 msgid "![pack_into_one](./imgs/pack_into_one.png)" msgstr "" -#: ../../../usage.md:328 +#: ../../../usage.md:330 msgid "pack_into_one" msgstr "" -#: ../../../usage.md:331 +#: ../../../usage.md:333 msgid "目前支持传入数据集文件路径`train_folder`，且要求文件格式如下：" msgstr "" "Currently, it supports passing the dataset file path `train_folder`, and " "the file format is required to be as follows:" -#: ../../../usage.md:338 +#: ../../../usage.md:340 msgid "数据集的详细内容可参考``数据准备``模块相关的介绍。" msgstr "" "For detailed information about the dataset, please refer to the \"Data " "Preparation\" section." -#: ../../../usage.md:340 +#: ../../../usage.md:342 msgid "同时，也支持huggingface格式的数据集处理。" msgstr "" "Additionally, it supports processing of datasets in the Hugging Face " "format." -#: ../../../usage.md:342 +#: ../../../usage.md:344 msgid "train_folder设置为从huggingface上下载的本地数据集路径，如：\"/mnt/petrelfs/hf-TinyStories\"" msgstr "" "Set the train_folder to the local path of the dataset downloaded from " "Hugging Face, for example: \"/mnt/petrelfs/hf-TinyStories\"." -#: ../../../usage.md:344 +#: ../../../usage.md:346 msgid "在data中，需要新增type及tokenizer_path字段，标示数据集是huggingface格式，并指定tokenizer路径，如：" msgstr "" "In the data section, you need to add new fields for type and " "tokenizer_path to indicate that the dataset is in Hugging Face format and" " to specify the path of the tokenizer, for example:" -#: ../../../usage.md:362 +#: ../../../usage.md:364 msgid "模型配置" msgstr "Model Configuration" -#: ../../../usage.md:364 +#: ../../../usage.md:366 msgid "如果在启动训练时要加载模型 `checkpoint`，可进行如下相关配置：" msgstr "" "If you want to load a model checkpoint when starting the training, you " "can configure it as follows:" -#: ../../../usage.md:388 +#: ../../../usage.md:390 msgid "注意：" msgstr "Note:" -#: ../../../usage.md:389 +#: ../../../usage.md:391 msgid "" "路径若以 `local:` 为前缀，则存储在本地文件系统；若以 `boto3:` 为前缀，则存储在远程 oss " "上；若无前缀，为huggingface上可以直接下载的模型路径。" @@ -313,11 +317,11 @@ msgstr "" "local file system. If it starts with `boto3:`, it means the file is " "stored in the remote OSS." -#: ../../../usage.md:391 +#: ../../../usage.md:393 msgid "模型相关关键参数配置如下所示：" msgstr "The configuration for the model is as follows:" -#: ../../../usage.md:415 +#: ../../../usage.md:417 msgid "注意：用户可自定义模型类型名和模型结构，并配置相对应的模型参数。通过`internlm/model/registry.py`下的`model_initializer`对象进行模型初始化函数接口注册，在训练主函数`train.py`中初始化模型时，可通过`model_type`配置获取指定的模型初始化接口函数。" msgstr "" "Note: Users can customize the model type name and model structure, and " @@ -328,140 +332,140 @@ msgstr "" "interface function can be obtained through the `model_type` " "configuration." -#: ../../../usage.md:417 +#: ../../../usage.md:419 msgid "" "*如果基于 InternLM 7B继续训练，可以参考 " "[ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo) 中 " "OpenXLab 链接下载权重*" msgstr "" "*If you want to start training based on InternLM 7B, you can refer to " -"OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-zoo)" -" to download weights*." +"OpenXLab [ModelZoo](https://github.com/InternLM/InternLM/tree/main#model-" +"zoo) to download weights*." -#: ../../../usage.md:419 +#: ../../../usage.md:421 msgid "并行配置" msgstr "Parallel Configuration" -#: ../../../usage.md:421 +#: ../../../usage.md:423 msgid "训练并行配置样例如下：" msgstr "Training parallel configuration example:" -#: ../../../usage.md:430 +#: ../../../usage.md:432 msgid "zero1（字典）：" msgstr "zero1 (dict): " -#: ../../../usage.md:431 +#: ../../../usage.md:433 msgid "size: 整数" msgstr "size: int " -#: ../../../usage.md:432 +#: ../../../usage.md:434 msgid "当`zero1 <= 0`，则 zero1 进程组的大小等于数据并行进程组的大小，因此优化器状态参数将在数据并行范围内分配" msgstr "" "When `zero1 <= 0` , the size of the zero1 process group is equal to the " "size of the data parallel process group, so the optimizer state " "parameters will be split within the data parallel range." -#: ../../../usage.md:433 +#: ../../../usage.md:435 msgid "当`zero1 == 1`，则不使用 zero1 ，所有数据并行组保留完整的优化器状态参数" msgstr "" "When `zero1 == 1`, zero1 is not used, and all data parallel groups retain" " the complete optimizer state parameters." -#: ../../../usage.md:434 +#: ../../../usage.md:436 msgid "当`zero1 > 1`且`zero1 <= data_parallel_world_size`，则 zero1 进程组是数据并行进程组的子集" msgstr "" "When `zero1 > 1` and `zero1 <= data_parallel_world_size`, the zero1 " "process group is a subset of the data parallel process group." -#: ../../../usage.md:435 +#: ../../../usage.md:437 msgid "fsdp: 布尔值，启用/禁用torch的完全分片数据并行，默认为False。" msgstr "" "fsdp: A boolean value that enables or disables fully sharded data " "parallelism in torch, with the default being False." -#: ../../../usage.md:436 +#: ../../../usage.md:438 msgid "tensor（字典）：" msgstr "tensor (dict): " -#: ../../../usage.md:437 +#: ../../../usage.md:439 msgid "size: 整数，张量并行的大小。" msgstr "size: int, size of tensor parallem" -#: ../../../usage.md:438 +#: ../../../usage.md:440 msgid "mode: 字符串，张量并行模式，应该是 ['mtp', 'msp', 'fsp', 'isp'] 中的一个，" msgstr "" "mode: string, tensor parallel mode, should be one of ['mtp', 'msp', " "'fsp', 'isp'] " -#: ../../../usage.md:439 +#: ../../../usage.md:441 msgid "默认为 'mtp'，意味着没有序列并行的纯Megatron张量并行。" msgstr "" "Default is 'mtp', which means there is no sequence parallelism, just pure" " tensor parallelism for Megatron." -#: ../../../usage.md:440 +#: ../../../usage.md:442 msgid "msp: 带序列并行的Megatron张量并行，序列并行大小 = 张量并行大小。" msgstr "" "msp: Megatron Tensor Parallelism with Sequence Parallelism, where the " "size of sequence parallelism is equal to the size of tensor parallelism." -#: ../../../usage.md:441 +#: ../../../usage.md:443 msgid "fsp: 通过flash-attn带序列并行的张量并行，序列并行大小 = 张量并行大小。" msgstr "" "fsp: Tensor Parallelism with Sequence Parallelism facilitated by flash-" "attn, where the size of sequence parallelism is equal to the size of " "tensor parallelism." -#: ../../../usage.md:442 +#: ../../../usage.md:444 msgid "isp: 定制的内部序列并行，不带张量并行，可以与权重并行一起使用。" msgstr "" "isp: Custom internal sequence parallelism, without tensor parallelism, " "which can be used in conjunction with weight parallelism." -#: ../../../usage.md:443 +#: ../../../usage.md:445 msgid "pipeline（字典）：" msgstr "pipeline: pipeline parallel strategy" -#: ../../../usage.md:444 +#: ../../../usage.md:446 msgid "size: 整数，流水线并行的大小。" msgstr "size: int, size of pipeline parallel" -#: ../../../usage.md:445 +#: ../../../usage.md:447 msgid "interleaved_overlap: 布尔值，启用/禁用在使用交错流水线调度器时的通信重叠，默认为False。" msgstr "" "interleaved_overlap: A boolean value that enables or disables " "communication overlapping when using an interleaved pipeline scheduler, " "with the default being False." -#: ../../../usage.md:446 +#: ../../../usage.md:448 msgid "weight（字典）：" msgstr "weight (dict):" -#: ../../../usage.md:447 +#: ../../../usage.md:449 msgid "size: 整数，权重并行的大小。" msgstr "size: int, size of weight parallel" -#: ../../../usage.md:448 +#: ../../../usage.md:450 msgid "overlap: 布尔值，启用/禁用all_gather/reduce_scatter通信重叠，默认为False。" msgstr "" "overlap: bool, enable/disable all_gather/reduce_scatter communication " "overlap, default is False" -#: ../../../usage.md:449 +#: ../../../usage.md:451 msgid "memory_pool: 布尔值，启用/禁用内存池，默认为False。" msgstr "memory_pool: bool, enable/disable memory pool, default is False" -#: ../../../usage.md:451 +#: ../../../usage.md:453 msgid "注意：`数据并行大小 = 总的 GPU 数目 / 流水线并行大小 / 张量并行大小`" msgstr "" "Note: `Data parallel size = Total number of GPUs / Pipeline parallel size" " / Tensor parallel size`" -#: ../../../usage.md:453 +#: ../../../usage.md:455 msgid "启动训练" msgstr "Start Training" -#: ../../../usage.md:455 +#: ../../../usage.md:457 msgid "完成了以上数据集准备和相关训练配置后，可启动 Demo 训练。接下来分别以 slurm 和 torch 环境为例，介绍训练启动方式。" msgstr "" "After completing the data preparation and relevant training " @@ -469,47 +473,49 @@ msgstr "" "following examples demonstrate how to start the training in both slurm " "and torch environments." -#: ../../../usage.md:457 ../../../usage.md:494 +#: ../../../usage.md:459 ../../../usage.md:496 msgid "若在 slurm 上启动分布式运行环境，多节点 16 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on slurm with 16 GPUs across " "multiple nodes, use the following command:" -#: ../../../usage.md:462 +#: ../../../usage.md:464 msgid "若在 torch 上启动分布式运行环境，单节点 8 卡的运行命令如下所示：" msgstr "" "If you want to start distributed training on torch with 8 GPUs on a " "single node, use the following command:" -#: ../../../usage.md:467 +#: ../../../usage.md:469 msgid "" "其中，train.py文件的内容，请参考： [训练脚本](https://internevo.readthedocs.io/zh-" "cn/latest/training.html)" -msgstr "The content of train.py, please refer to: [training script](https://internevo.readthedocs.io/en/latest/training.html) " +msgstr "" +"The content of train.py, please refer to: [training " +"script](https://internevo.readthedocs.io/en/latest/training.html) " -#: ../../../usage.md:469 +#: ../../../usage.md:471 msgid "运行结果" msgstr "Training Results" -#: ../../../usage.md:471 +#: ../../../usage.md:473 msgid "以 slurm 上单机 8 卡的 Demo 训练配置为例，训练结果日志展示如下：" msgstr "" "Taking the configuration of the demo training on a single machine with 8 " "GPUs on slurm as an example, the training result log is shown below:" -#: ../../../usage.md:492 +#: ../../../usage.md:494 msgid "加载训练的checkpoint并生成" msgstr "Load the training checkpoint and generate." -#: ../../../usage.md:499 +#: ../../../usage.md:501 msgid "在配置文件中添加`generation`配置" msgstr "Add generation configuration to the configuration file." -#: ../../../usage.md:517 +#: ../../../usage.md:519 msgid "长文本生成" msgstr "Long Text Generation" -#: ../../../usage.md:519 +#: ../../../usage.md:521 msgid "" "在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K " "的外推效果。 目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK " @@ -521,7 +527,7 @@ msgstr "" "the use of Dynamic NTK RoPE in models formatted in both Hugging Face " "format and InternLM's native format." -#: ../../../usage.md:522 +#: ../../../usage.md:524 msgid "" "对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 " "`config.json` 中的 `rotary.type` 修改为 `origin`；" @@ -530,7 +536,7 @@ msgstr "" "by default. If users wish to disable this behavior, they can modify " "`rotary.type` in the `config.json` file to `origin`." -#: ../../../usage.md:523 +#: ../../../usage.md:525 msgid "" "对于 InternLM " "本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。" @@ -539,7 +545,7 @@ msgstr "" "this behavior by adding use_dynamic_ntk_rope=True to the configuration " "dictionary when initializing the model." -#: ../../../usage.md:525 +#: ../../../usage.md:527 msgid "" "用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE " "是如何生效的。例如文件[长文本示例](../../aux_materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用" @@ -553,23 +559,23 @@ msgstr "" "text. However, after applying Dynamic NTK RoPE, the response from the " "InternLM Chat 7B v1.1 model is as follows:" -#: ../../../usage.md:528 +#: ../../../usage.md:530 msgid "![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png)" msgstr "" -#: ../../../usage.md:528 +#: ../../../usage.md:530 msgid "dynamic_ntk_answer" msgstr "" -#: ../../../usage.md:530 +#: ../../../usage.md:532 msgid "关于 Dyanmic NTK 的原理，详细请参考" msgstr "Regarding the principle of Dyanmic NTK, please refer to" -#: ../../../usage.md:531 +#: ../../../usage.md:533 msgid "[dynamically_scaled_rope_further_increases](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases)" msgstr "" -#: ../../../usage.md:532 +#: ../../../usage.md:534 msgid "[https://kexue.fm/archives/9675](https://kexue.fm/archives/9675)" msgstr "" diff --git a/doc/en/usage.md b/doc/en/usage.md index fa05710bf..8e1670c2f 100644 --- a/doc/en/usage.md +++ b/doc/en/usage.md @@ -100,7 +100,9 @@ The data format for fine-tuning tasks is the same as for pre-training tasks, whi python tools/alpaca_tokenizer.py /path/to/alpaca_dataset /path/to/output_dataset /path/to/tokenizer --split_ratio 0.1 ``` -It is recommended that users refer to alpaca_tokenizer.py to write new scripts to tokenize their own datasets +It is recommended that users refer to alpaca_tokenizer.py to write new scripts to tokenize their own datasets. + +In fine-tuning tasks, Hugging Face formatted datasets can also be used, consistent with the preparation process in pre-training. ### Training Configuration diff --git a/doc/usage.md b/doc/usage.md index 7759e5da2..a3da69e72 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -89,7 +89,9 @@ $ python tools/tokenizer.py --text_input_path raw_data.txt --bin_output_path cn/ python tools/alpaca_tokenizer.py /path/to/alpaca_dataset /path/to/output_dataset /path/to/tokenizer --split_ratio 0.1 ``` -建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize +建议用户参考 alpaca_tokenizer.py 编写新的脚本对自己的数据集进行 tokenize。 + +微调任务中，也同样可以使用huggingface格式数据集，与预训练中的准备过程一致。 ### 训练配置