Question about emb.py

[Hechriver]

Hi, thank you for your interesting research.

I have some questions when running the emb.py:

1. I get a warning : Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
2. I also get a error : RuntimeError: The expanded size of the tensor (638) must match the existing size (514) at non-singleton dimension 1. Target sizes: [16, 638]. Tensor sizes: [1, 514].

[reviverkey]

Hello,

Thank you for your questions. Language models have a limitation on the length of the input language information. If the input exceeds this limit, it may result in the error you are encountering. If you are applying the model to a new dataset, you might consider reducing the length of the input language information. You can also refer to the documentation on padding and truncation provided by Hugging Face at https://huggingface.co/docs/transformers/pad_truncation.

Best Regards

[Hechriver]

Thank you for your reply. But I didn't change the dataset. I only changed the card in the code because the server I used can't connect to the huggingface. So I downloaded the model, distilroberta-base, from the hugging face and used absolute path instead. And then I got the warning and error when I ran the emb.py. I also read the documentation but it didn't work.

[yang71]

Hi,
I've encountered the same issue. Have you resolved it?

[reviverkey]

Hello,
it's possible that there may be discrepancies between the environment settings of the model downloaded offline and the model loaded online, which could result in an inability to automatically truncate when the number of tokens exceeds the limit. You can consider setting up a filtering mechanism to filter out those text that exceed the token limit, which may resolve this issue.
Best Regards

[yang71]

Hi,
I have resolved this question:
def tokenize_function(samples):
# return tokenizer(samples["text"], truncation=True)
return tokenizer(samples["text"], max_length=512, truncation=True)