运行llamafactory-cli, log的info太啰嗦, 干扰debug, 如何屏蔽?

[sisrfeng]

Reminder

• I have read the above rules and searched the existing issues.

System Info

略

Reproduction

llamafactory-cli export conf/train_config/merge/qwen2_5_vl_3b_lora_sft.yaml

Others

能否参考:
https://huggingface.co/docs/transformers/v4.36.1/zh/main_classes/logging
提供类似TRANSFORMERS_VERBOSITY=error和TRANSFORMERS_NO_ADVISORY_WARNINGS=1的开关?

另外 现在就算设置了上述2个开关, llamafactory-cli运行后, transformers还是会输出很多info.
llamafactory里的log的设置, 把TRANSFORMERS_VERBOSITY等给覆盖了么?

[hiyouga]

export LLAMAFACTORY_VERBOSITY=WARN

[sisrfeng]

感谢!
LLaMA-Factory/src/llamafactory/init.py里找到:

Disable version checking: DISABLE_VERSION_CHECK=1
Enable VRAM recording: RECORD_VRAM=1
Force using torchrun: FORCE_TORCHRUN=1
Set logging verbosity: LLAMAFACTORY_VERBOSITY=WARN
Use modelscope: USE_MODELSCOPE_HUB=1
Use openmind: USE_OPENMIND_HUB=1

建议后续加到README里.
我赶完个deadline后 也可以参与贡献

[sisrfeng]

LLaMA-Factory/src/llamafactory/data/processor/supervised.py里

    def print_data_example(self, example: dict[str, list[int]]) -> None:
        valid_labels = list(filter(lambda x: x != IGNORE_INDEX, example["labels"]))
        print("input_ids:\n{}".format(example["input_ids"]))
        print("inputs:\n{}".format(self.tokenizer.decode(example["input_ids"], skip_special_tokens=False)))
        print("label_ids:\n{}".format(example["labels"]))
        print(f"labels:\n{self.tokenizer.decode(valid_labels, skip_special_tokens=False)}")

会打印一大堆token, 有点弄脏屏幕, 我后续看看能不能用类似下面的方法, 只打印头尾的token:

option4 = {'precision' : 2,
           'threshold' : 50,
           'linewidth' : 90,
           'edgeitems' : 3,
          }

import numpy as np
np.set_printoptions( **option4 )
# 无法影响print(..., file=...)


import torch
torch.set_printoptions( **option4 )