generat关于eembedding输入相关问题

[bingwork]

我没看到具体generate方法代码，就先用prepare_inputs_for_generation分析。
如上图，llama的prepare_inputs_for_generation可以支持embedding输入，但是chatglm没有。
请问chatglm的generate方法是否不支持embedding输入？
如果理解错误，还望见谅。
@xunkai55 @davidlvxin @duzx16

[LittleGreenYuan]

请问，你有任何新的想法吗？我在源文件中找到了‘PrefixEncoder’的类，似乎被用在了P-TuningV2里
定义在：

65: class PrefixEncoder(torch.nn.Module)

使用在：

736: class ChatGLMModel(ChatGLMPreTrainedModel):
789:      def forward():

在这一过程中，官方是将这里的emmbedding作为past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None
送入模型的，不知道能不能帮到你，我也恰好正在研究这里

[zRzRzRzRzRzRzR]

不确定，将跟算法同学进行讨论

[Junjie-Chu]

I'm trying to use GCG with ChatGLM3.

After I read the code carefully, I think generate() actually supports inputs_embeds, which may solve the issue.
I found that input_ids is only used to provide size to create attention_mask and position_ids. When inputs_embeds is passed in, according to the code

if inputs_embeds is None:
    inputs_embeds = self.embedding(input_ids)

the parameter input_ids does not actually affect the inference results?

So in fact, to use inputs_embeds as input, we only needmodel(input_ids, inputs_embeds)

Not sure if my understanding is correct?

And I find, when run model(input_ids=input_ids.unsqueeze(0),inputs_embeds=full_embeds), the output dimensions of ChatGLM3 seem to be different with those of Llama2 or Vicuna? Need to use something like .permute(1, 0, 2)?

Not sure about my understanding, thanks a lot in advance for your support!

[LittleGreenYuan]

Following the code below does pass embedding as an input, but when using model.generate(), it will prompt an error:"You passed inputs_embeds to .generate(), but the model class ChatGLMForConditionalGeneration doesn't have its forwarding implemented. See the GPT2 implementation for an example (huggingface/transformers#21405), and feel free to open a PR with it!"

inputs  = tokenizer(MutilTalk_Prompt,padding = 'max_length',max_length = 99)
tensor_input_ids = torch.tensor(inputs['input_ids']+[2])
tensor_input_ids = tensor_input_ids.cuda()
print(tensor_input_ids)
input_embeds = model.transformer.embedding(tensor_input_ids.unsqueeze(0))

outputs = model(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)

#error
outputs = model.generate(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)

[Junjie-Chu]

May I ask, if simlply following GPT-2 example, modify the prepare_inputs_for_generation function, could make it run normally?

[Junjie-Chu]

Following the code below does pass embedding as an input, but when using model.generate(), it will prompt an error:"You passed inputs_embeds to .generate(), but the model class ChatGLMForConditionalGeneration doesn't have its forwarding implemented. See the GPT2 implementation for an example (huggingface/transformers#21405), and feel free to open a PR with it!"

inputs  = tokenizer(MutilTalk_Prompt,padding = 'max_length',max_length = 99)
tensor_input_ids = torch.tensor(inputs['input_ids']+[2])
tensor_input_ids = tensor_input_ids.cuda()
print(tensor_input_ids)
input_embeds = model.transformer.embedding(tensor_input_ids.unsqueeze(0))

outputs = model(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)

#error
outputs = model.generate(input_ids=tensor_input_ids.unsqueeze(0),inputs_embeds=input_embeds)
logits_output = tokenizer.batch_decode(torch.argmax(outputs['logits'], -1).detach().cpu().numpy(), skip_special_tokens=True)
print(logits_output)

Oh, I get what u mean now, actually I do not use generate(), instead I just use model().logits. And in this case it runs well. But output has a different dimension with that of llama2 or vicuna XD

[lqr2018]

Hello, I am trying to use GCG with chatglm3 too. I have tried what you said here, but I meet something wrong when the model is optimizing. It seems that my inputs_embeds have to use something like .permute(1, 0, 2). Have you met this problem?Any information will be helpful.

[lh-kyf]

你们好，请问遇到这个 model.resize_token_embeddings 时报错：NotImplementedError后，是如何解决的呢？不使用吗？还是说要修改代码？本人不太会，希望能有大哥帮忙指导一下

[CRonaldo1997]

解决了吗大佬

[chenzebiaohub]

@CRonaldo1997 解决了？

[lh-kyf]

你好，可以看看我在discussions里说的：
AttributeError: 'ChatGLMForConditionalGeneration' object has no attribute 'add_adapter' #498

[heavenhellchen]

我也遇到这种问题，模型中没有set_input_embedding的接口，导致运行的时候报NotImplementedError，这个该怎么解决呀

[CRonaldo1997]

@heavenhellchen 遇到同样问题，请问解决了吗大佬

[zRzRzRzRzRzRzR]

更新huggingface

[chenzebiaohub]

@CRonaldo1997 您好 解决了 留下联系方式 chenzebiaohub@126.com 欢迎沟通！

[chenzebiaohub]

@zRzRzRzRzRzRzR 能提供解决方法？您好

[CLIsVeryOK]

did you solve it?

[louiss007]

更新huggingface
你好，您是指更新transformers版本，是吧。有具体的版本号吗，我更新到4.39.3后，依然报错~

[CLIsVeryOK]

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university