pure-text data format

[hongjian]

I'm curious about the data format of pure-text used in pre-training in DeepSeek-VL2 but I haven't found specific details.
In LLM pre-training, the loss is typically computed over all tokens. However, in traditional MLLM pre-training, the data format often resembles a multi-turn QA setup, and only the answer tokens are used for loss computation — the system prompt and question are usually masked.
So, what approach does DeepSeek-VL2 use?

[zhaozhipeng1997]

       conversation = [
        {
            "role": "<|User|>",
            "content": "\n<|ref|>" + sample['question'] + "<|/ref|>.",
            "images": []
            
        },
        {
            "role": "<|Assistant|>",
            "content": ""
        }
        ]
        prepare_inputs = vl_chat_processor(
                conversations=conversation,
                images=[],
                force_batchify=True,
                system_prompt="",
            ).to(dtype=torch.bfloat16,device=vl_gpt.device)
        inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)