1.5B accuracy 0 after SFT

[alvinshao0313]

Dear author, hello,
I encountered a problem while experimenting with the 1.5b model; I used the following script for training.

uid="$(date +%Y%m%d_%H%M%S)"
base_model=\"Qwen/Qwen2.5-1.5B-Instruct\"
lr=1e-5
min_lr=0
epochs=5
weight_decay=1e-4 # -> the same training pipe as slurm_training
micro_batch_size=1 # -> batch_size will be 16 if 16 gpus
gradient_accumulation_steps=8 # requires more GPU memory, default is 1
max_steps=-1
gpu_count=2 #$(nvidia-smi -L | wc -l)
push_to_hub=false

torchrun --nproc-per-node ${gpu_count} --master_port 12345 \\
    train/sft.py \"
    --block_size=12000 \
    --per_device_train_batch_size=${micro_batch_size} \
    --per_device_eval_batch_size=${micro_batch_size} \
    --gradient_accumulation_steps=${gradient_accumulation_steps} \"
    --num_train_epochs=${epochs} \
    --train_file_path=\"simplescaling/s1K-1.1_tokenized\" \\
    --model_name=${base_model} \\
    --warmup_ratio=0.05 \\
    --fsdp=\"full_shard auto_wrap\" \\
    --fsdp_config=\"train/fsdp_config_qwen.json\" \\
    --bf16=True \\
    --eval_strategy=\"no\" \\
    --logging_steps=1 \
    --save_strategy=\"no\" \
    --lr_scheduler_type=\"cosine\" \
    --learning_rate=${lr} \"
    --weight_decay=${weight_decay} \
    --adam_beta1=0.9 \
    --adam_beta2=0.95 \
    --output_dir=\"ckpts/s1-$(basename \"$base_model\")-${uid}\" \"
    --push_to_hub=${push_to_hub} \\
    --save_only_model=True \\

Since I only have two A800s, I set the gradient accumulation to 8 to ensure batch consistency. Since a block size of 20k would cause an OOM, I set it to 12k.

但是不论我使用一下那个测试命令：

CUDA_VISIBLE_DEVICES=5 lm_eval --model vllm --model_args pretrained=/s1-Qwen2.5-1.5B-Instruct-20250419_021608,dtype=bfloat16,tensor_parallel_size=1 --tasks aime24_figures,aime24_nofigures --batch_size auto --output_path dummy --log_samples --gen_kwargs "max_gen_toks=12000

CUDA_VISIBLE_DEVICES=5 lm_eval --model vllm --model_args pretrained=/s1-Qwen2.5-1.5B-Instruct-20250419_021608,dtype=float32,tensor_parallel_size=1 --tasks aime24_nofigures --batch_size auto --apply_chat_template --output_path s1.1forcingignore1wait --log_samples --gen_kwargs "max_gen_toks=12000,max_tokens_thinking=auto,thinking_n_ignore=1,thinking_n_ignore_str=Wait"

CUDA_VISIBLE_DEVICES=5 lm_eval --model vllm --model_args pretrained=/s1-Qwen2.5-1.5B-Instruct-20250419_021608,dtype=float32,tensor_parallel_size=1 --tasks aime25_nofigures,aime24_nofigures --batch_size auto --apply_chat_template --output_path s1.1forcingignore2wait --log_samples --gen_kwargs "max_gen_toks=12000,max_tokens_thinking=auto,thinking_n_ignore=2,thinking_n_ignore_str=Wait"

The accuracy on AIME is 0.

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
aime24_figures	1	none	0	exact_match	↑	0	±	N/A
		none	0	extracted_answers	↑	-1	±	N/A
aime24_nofigures	1	none	0	exact_match	↑	0	±	N/A
		none	0	extracted_answers	↑	-1	±	N/A

I wonder if you have any help regarding my current issue.

[Muennighoff]

12k block size is pretty bad, definitely recommend >=20k

[alvinshao0313]

@Muennighoff I tried using deepspeed to increase my block size. Here is my deepspeed_zero3.yaml:

compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  deepspeed_multinode_launcher: standard
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2 # 默认 8 卡
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

I will modify the corresponding script to:

# Reference Running: bash train/sft.sh
# {'train_runtime': 5268.8407, 'train_samples_per_second': 0.949, 'train_steps_per_second': 0.119, 'train_loss': 0.1172730620391667, 'epoch': 5.0}
uid="$(date +%Y%m%d_%H%M%S)"
base_model="Qwen/Qwen2.5-1.5B-Instruct"
lr=1e-5
min_lr=0
epochs=5
weight_decay=1e-4 # -> the same training pipe as slurm_training
micro_batch_size=1 # -> batch_size will be 16 if 16 gpus
gradient_accumulation_steps=8 # requires more GPU memory, default is 1
max_steps=-1
gpu_count=2 #$(nvidia-smi -L | wc -l)
push_to_hub=false

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file=./train/deepspeed_zero3.yaml ./train/sft.py \
    --block_size=20000 \
    --per_device_train_batch_size=${micro_batch_size} \
    --per_device_eval_batch_size=${micro_batch_size} \
    --gradient_accumulation_steps=${gradient_accumulation_steps} \
    --gradient_checkpointing=True \
    --num_train_epochs=${epochs} \
    --max_steps=${max_steps} \
    --train_file_path="simplescaling/s1K-1.1_tokenized" \
    --model_name=${base_model} \
    --warmup_ratio=0.05 \
    --bf16=True \
    --eval_strategy="steps" \
    --eval_steps=50 \
    --logging_steps=1 \
    --lr_scheduler_type="cosine" \
    --learning_rate=${lr} \
    --weight_decay=${weight_decay} \
    --adam_beta1=0.9 \
    --adam_beta2=0.95 \
    --output_dir="ckpts/s1_${uid}" \
    --save_only_model=True \
    --save_strategy=no \
    --push_to_hub=${push_to_hub} \

Test command:

 lm_eval --model vllm --model_args pretrained=./ckpts/s1_20250421_030714-20k,tokenizer=./ckpts/s1_20250421_030714-20k,dtype=float16,tensor_parallel_size=1 --tasks aime24_figures,aime24_nofigures --batch_size auto --apply_chat_template --output_path qwen-eval/ --log_samples --gen_kwargs "max_gen_toks=20000"

But the result is still 0.

vllm (pretrained=./ckpts/s1_20250421_030714-20k,tokenizer=/home/shaoyuantian/program/D2M/s1/ckpts/s1_20250421_030714-20k,dtype=float16,tensor_parallel_size=1), gen_kwargs: (max_gen_toks=20000), limit: None, num_fewshot: None, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
aime24_figures	1	none	0	exact_match	↑	0	±	N/A
		none	0	extracted_answers	↑	-1	±	N/A
aime24_nofigures	1	none	0	exact_match	↑	0	±	N/A
		none	0	extracted_answers	↑	-1	±	N/A

May I ask if it's my testing problem or a training problem?

In addition, I compared the Wandb logs you posted and found that we only have two differences, dataset batch_2 and x_seq length.

My dataset batch_stize: 1000 max_deq_1ength: 2000 https://wandb.ai/alvin-shao/s1/runs/p4vluv76/overview

Your dataset batch size: null max_seq_length:23000 https://wandb.ai/hashimoto-group/o1/runs/stcs4jwi/overview

I have reviewed the output of the model and found that there are many repetitions.

"resps": [["<|im_start|>think\nOkay, let's see. So, Aya goes for a 9-kilometer walk every morning. When she walks at a speed of s km/h, the walk takes her 4 hours, including t minutes spent in the coffee shop. When she increases her speed to s + 2 km/h, the walk takes her 2 hours and 24 minutes, which is 2 + 24/60 = 2.4 hours. We need to find the time it takes her to walk at s + 0.5 km/h, including the t minutes in the coffee shop. Hmm.\n\nFirst, let me try to set up the equations based on the information given. Let's denote the time spent walking as W and the time spent in the coffee shop as t. So, for the first scenario:\n\nDistance = Speed * Time\n\nSo, 9 km = s * (4 - t/60) km/h, because 4 hours minus t minutes is 4 - t/60 hours. Wait, no. Wait, the total time for the first scenario is 4 hours including t minutes in the coffee shop. So, if the walk takes 4 hours, then the walking time is 4 - t/60 hours. So, 9 = s * (4 - t/60). Similarly, for the second scenario, when she walks at s + 2 km/h, the time is 2.4 hours, so:\n\n9 = (s + 2) * (2 + 24/60) = (s + 2) * (2 + 0.4) = (s + 2) * 2.4\n\nSo, let's write those equations:\n\n1) 9 = s * (4 - t/60)\n\n2) 9 = (s + 2) * 2.4\n\nSo, from equation 2, we can solve for s:\n\n9 = (s + 2) * 2.4\n\nDivide both sides by 2.4:\n\n9 / 2.4 = s + 2\n\nCalculate 9 / 2.4: 9 / 2.4 = 90 / 24 = 15 / 4 = 3.75. So, 3.75 = s + 2. Therefore, s = 3.75 - 2 = 1.75 km/h. So, s = 7/8 km/h, which is 0.875 km/h or 45/50 km/h or 9/16 km/h. Wait, 1.75 is 7/8, which is 0.875. So, s = 7/8 km/h.\n\nNow, we need to find the time it takes her to walk at s + 0.5 km/h, which is 1.375 km/h. Let's denote this time as T. The total time including the t minutes in the coffee shop would be T + t. So, we need to find T.\n\nBut we need to find t first. From equation 1: 9 = s * (4 - t/60). Let's substitute s = 7/8 into this equation.\n\nSo, 9 = (7/8) * (4 - t/60)\n\nMultiply both sides by 8/7 to solve for t:\n\n9 * (8/7) = 4 - t/60\n\nCalculate 9 * 8/7 = 72/7 ≈ 10.2857. So,\n\n72/7 = 4 - t/60\n\nSubtract 4 from both sides:\n\n72/7 - 28/7 = -t/60\n\n(72 - 28)/7 = -t/60\n\n44/7 = -t/60\n\nMultiply both sides by -60:\n\n-60 * (44/7) = t\n\nSo, t = -60 * 44 / 7 ≈ -2640 / 7 ≈ -377.142857. Wait, that can't be. Negative time? That can't be right. Wait, hold on. Wait, t is the time spent in the coffee shop, which can't be negative. So, there must be an error in the calculation.\n\nWait, let's check the steps again.\n\nStarting from equation 1: 9 = (7/8)(4 - t/60)\n\nMultiply both sides by 8/7:\n\n9 * (8/7) = (7/8)(4 - t/60) * (8/7)\n\nSimplify: 9 * (8/7) = 4 - t/60\n\nCompute 9 * 8/7: 72/7\n\nSo, 72/7 = 4 - t/60\n\nSubtract 4: 72/7 - 28/7 = -t/60\n\n(72 - 28)/7 = -t/60\n\n44/7 = -t/60\n\nMultiply both sides by -60:\n\n-60 * (44/7) = t\n\nSo, t = -2640/7 ≈ -377.142857. That can't be right. So, there must be an error in the setup.\n\nWait, let's check the equations again.\n\nEquation 1: 9 = s*(4 - t/60)\n\nEquation 2: 9 = (s + 2)2.4\n\nSo, from equation 2, we have s + 2 = 9 / 2.4 = 15/4 = 3.75. So, s = 3.75 - 2 = 1.75. So, s = 7/8 km/h. Then, substituting s into equation 1:\n\n9 = (7/8)(4 - t/60)\n\nMultiply both sides by 8/7:\n\n9*(8/7) = 4 - t/60\n\nCompute 98/7 = 72/7\n\nSo, 72/7 = 4 - t/60\n\nSubtract 4: 72/7 - 28/7 = 44/7\n\nSo, 44/7 = -t/60\n\nMultiply both sides by -60:\n\n-60(44/7) = t\n\nSo, t = -2640/7 ≈ -377.142857. That can't be. So, there must be an error in the problem statement or in my interpretation.\n\nWait, let me check the problem again. The problem says: "When she walks at a constant speed of s kilometers per hour, the walk takes her 4 hours, including t minutes spent in the coffee shop." So, the total time is 4 hours, which is 240 minutes. So, 4 hours is 240 minutes. So, the walking time is 240 - t minutes. Wait, but the problem says "the walk takes her 4 hours, including t minutes spent in the coffee shop." So, the total time is 4 hours, which is 240 minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes. So, the walking time is 240 - t minutes...

I'm not sure if this is the reason for the evaluation result being 0, and I wonder if you know how to avoid this phenomenon.

[TikaToka]

I also encountered the same results. might be the limitation of 1.5B?

[KunyangHan]

I also encountered the same results. might be the limitation of 1.5B?

Similar here, I run an experiment on 7b model, and it degrades 3.5% on diamond (compared to qwen-7b).

[TikaToka]

I also encountered the same results. might be the limitation of 1.5B?

Similar here, I run an experiment on 7b model, and it degrades 3.5% on diamond (compared to qwen-7b).

Same for me. All my experiment on qwen2.5 family from 0.5b to 7b had no performance improvement.

However, as Qwen3 family supports thinking, using qwen3 might be a naive solution....

[SondosBsharat]

@KunyangHan Same here , for Aime24 :exact match =10% where qwen-7b=13.3%

[ivnle]

@Muennighoff

12k block size is pretty bad, definitely recommend >=20k

What's the rationale for the larger block size during training? The longest example in s1k is 9,109 tokens using Qwen 2.5's tokenizer.

[Muennighoff]

simplescaling/s1K-1.1_tokenized should be much longer