Question: Where is the setting to freeze the backbone LLM like LLaVA?

[Kurt232]

# LLaVA
if model_args.freeze_backbone:
        model.model.requires_grad_(False)

In LTU code, only note the LLM has already frozen.

    # for audio params, lora always trainable, llama always frozen
    for name, param in model.named_parameters():
        if trainable_params == 'all':
            if "audio" in name:
                param.requires_grad = True
                #print(f"Parameter: {name}, requires_grad: {param.requires_grad}")
        if trainable_params == 'proj':
            if "audio_proj" in name:
                param.requires_grad = True
                #print(f"Parameter: {name}, requires_grad: {param.requires_grad}")

[YuanGongND]

can you be more specific on the question.

if I understand the question correctly, the freeze is in here

ltu/src/ltu_as/peft-main/src/peft/tuners/lora.py

Lines 363 to 377 in 2002aad

	def mark_only_lora_as_trainable(model: nn.Module, bias: str = "none") -> None:
	for n, p in model.named_parameters():
	if "lora_" not in n:
	p.requires_grad = False
	if bias == "none":
	return
	elif bias == "all":
	for n, p in model.named_parameters():
	if "bias" in n:
	p.requires_grad = True
	elif bias == "lora_only":
	for m in model.modules():
	if isinstance(m, LoraLayer) and hasattr(m, "bias") and m.bias is not None:
	m.bias.requires_grad = True
	else:

, which happens when you do:

ltu/src/ltu_as/finetune.py

Line 191 in 2002aad

model = get_peft_model(model, config)

so we are actually adding trainable parameters back rather than explicitly freeze some parameters.

-Yuan