When using llamafactory-cli for training inside the container, the GPU memory usage shown by nvidia-smi inside the container is inconsistent with that on the host machine, and the GPU memory usage cannot be controlled.

[yc-01]

Please provide an in-depth description of the question you have:

When using llamafactory-cli for training inside the container, the GPU memory usage shown by nvidia-smi inside the container is inconsistent with that on the host machine, and the GPU memory usage cannot be controlled.

In addition, I don’t see the training PID inside the container. Could you tell me why this is happening?

container：

node：

Environment:

• HAMi version: v2.6.1
• Kubernetes version: v1.33.2

[archlitchi]

hi. could you provide more detailed information? how can we reproduce this issue?

[yc-01]

hi. could you provide more detailed information? how can we reproduce this issue?

Hi.

test image : ubuntu22.04 install vllm cuda12.8 llamafactory-cli 0.9.3

image ENTRYPOINT：

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

cat entrypoint.sh
#!/bin/bash
set -e
printenv | sed 's/^(.*)$/export \1/g' > /etc/container.env
if ! grep -q "source /etc/container.env" /etc/bash.bashrc; then
echo "source /etc/container.env" >> /etc/bash.bashrc
fi
if ! grep -q "source /etc/container.env" /root/.bashrc; then
echo "source /etc/container.env" >> /root/.bashrc
fi
tail -f /dev/null

my k8s deployment yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
name: test-vcuda-v1
namespace: test
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: test-vcuda-v1
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
nvidia.com/use-gpuuuid: GPU-a26829b8-2fa8-50dd-8199-e8844d22a22d
creationTimestamp: null
labels:
app: test-vcuda-v1
spec:
containers:
- image: harbor-app.gczx.cn/common/vllm0.10-cuda12.8-llamafactory0.9.3-ubuntu22.04:v1.0.0
imagePullPolicy: IfNotPresent
name: test-vcuda-v1
resources:
limits:
cpu: "24"
memory: 128Gi
nvidia.com/gpu: "1"
nvidia.com/gpucores: "50"
nvidia.com/gpumem-percentage: "50"
requests:
cpu: "24"
memory: 128Gi
nvidia.com/gpu: "1"
nvidia.com/gpucores: "50"
nvidia.com/gpumem-percentage: "50"
securityContext:
seccompProfile:
type: Unconfined
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /data/Qwen3-8B
name: model-data
- mountPath: /data/Qwen3-4B-Instruct-2507
name: model-data-1
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- emptyDir:
medium: Memory
sizeLimit: 32Gi
name: dshm
- hostPath:
path: /data/wenxiaohui-h100/Qwen3-8B
type: Directory
name: model-data
- hostPath:
path: /data/Qwen3-4B-Instruct-2507
type: Directory
name: model-data-1

my service yaml：

apiVersion: v1
kind: Service
metadata:
name: test-vcuda-v1
namespace: test
spec:
ports:

• nodePort: 31596
  port: 33
  protocol: TCP
  targetPort: 33
  selector:
  app: test-vcuda-v1
  sessionAffinity: None
  type: NodePort

When I access this container via SSH or kubectl exec, I run the training command,The situation mentioned above occurs.

training command:
FORCE_TORCHRUN=1 CUDA_VISIBLE_DEVICES=0 llamafactory-cli train --stage sft --do_train True --model_name_or_path /data/Qwen3-8B --preprocessing_num_workers 4 --finetuning_type lora --template qwen3 --flash_attn auto --dataset_dir data --dataset bi_sql_0221 --cutoff_len 1024 --learning_rate 2e-05 --num_train_epochs 10.0 --max_samples 100000 --per_device_train_batch_size 8 --gradient_accumulation_steps 2 --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 5 --save_steps 500 --warmup_steps 500 --packing False --report_to none --output_dir saves/Qwen3-8B/lora/train_2025-09-02-16-55_qwen3_8b_r8_a16_dp01_sql --bf16 True --plot_loss True --trust_remote_code True --ddp_timeout 180000000 --include_num_input_tokens_seen True --optim adamw_torch --lora_rank 8 --lora_alpha 16 --lora_dropout 0.1 --lora_target all

In addition, when I start inference using vLLM, the limiting policy takes effect and the GPU memory is displayed correctly, but nvidia-smi does not show the related PID.

my vllm command:
vllm serve /data/Qwen3-8B --max-model-len 32000 --tensor-parallel-size 1 --served-model-name Qwen3-chat --max-num-seqs 25 --disable-log-requests --host 0.0.0.0 --port 5001

container:

node: