Loading pretrained weight from any model

[SeriousJ55]

Thank you again @rasbt for this amazing book!!! I'm still reading and re-reading it.

In order to use weights from a model like GPT-2 or Llama, you need to:

1. define a GPTModel with the model's architecture, and
2. create a function like load_weights_into_gpt or load_weights_into_llama that matches the weights to your architecture

I'm amazed that you could do that! It's not an easy task.

My question are:

• Can it be done for more recent models? For instance, could it be done with Gemma or Deepseek? From what I understand, they don't published their exact architecture.
• If not, can those models be fine-tuned?

[rasbt]

Thanks for the kind words! And yes you can! Actually, I helped doing that for many models in the context of LitGPT ( https://github.com/Lightning-AI/litgpt)

[SeriousJ55]

Wow! LitGPT is impressive and very useful!!! Just out of curiosity, can you briefly explain how you can fine tune a model without knowing the full architecture? From your book, in chapters 5 and 7, it that you need to know the architecture.

…

________________________________ De : Sebastian Raschka ***@***.***> Envoyé : Tuesday, February 4, 2025 8:35:43 PM À : rasbt/LLMs-from-scratch ***@***.***> Cc : SeriousJ55 ***@***.***>; Author ***@***.***> Objet : Re: [rasbt/LLMs-from-scratch] Loading pretrained weight from any model (Discussion #514) Thanks for the kind words! And yes you can! Actually, I helped doing that for many models in the context of LitGPT ( https://github.com/Lightning-AI/litgpt) — Reply to this email directly, view it on GitHub<#514 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGNJVK7XE7YCI2UOIMZX52L2OEJA7AVCNFSM6AAAAABWPEWWEGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBVHA4TOOA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rasbt]

Yes, we had to know the architectures to implement them in LitGPT and to be also able to load the pretrained weights. This is all done in these 2 files:

• https://github.com/Lightning-AI/litgpt/blob/main/litgpt/model.py
• https://github.com/Lightning-AI/litgpt/blob/main/litgpt/config.py

Often, it was a lot of work to find out the implementation details. I remember spending days to a week on some.

[SeriousJ55]

In understand. But the question remains: how do you find out the architecture? Is it always published in research papers? Do you reverse engineer the weights?

…

________________________________ De : Sebastian Raschka ***@***.***> Envoyé : Wednesday, February 5, 2025 1:38:51 PM À : rasbt/LLMs-from-scratch ***@***.***> Cc : SeriousJ55 ***@***.***>; Author ***@***.***> Objet : Re: [rasbt/LLMs-from-scratch] Loading pretrained weight from any model (Discussion #514) Yes, we had to know the architectures to implement them in LitGPT and to be also able to load the pretrained weights. This is all done in these 2 files: * https://github.com/Lightning-AI/litgpt/blob/main/litgpt/model.py * https://github.com/Lightning-AI/litgpt/blob/main/litgpt/config.py Often, it was a lot of work to find out the implementation details. I remember spending days to a week on some. — Reply to this email directly, view it on GitHub<#514 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGNJVKZKRW7WUH26W66JFD32OIA5XAVCNFSM6AAAAABWPEWWEGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBWHEYDSNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[rasbt]

That's the challenging part. The answer is: it depends. Sometimes, if you are lucky, there's a detailed model card with that information. Ideally also a research paper. And often you still end up looking at the weight tensors and try to reverse-engineer it. That's why it is often hard and can take a long time.

[d-kleine]

When it's about visualizing a model's architecture, I came across this post some time ago. The cool thing about torchview is that can visualize models that are hosted on HF (which torchviz can't do). It uses graphviz, just like sklearn does for visualizing decision trees. Personally, I found this very useful understanding a model architecture, but it's very technical (and takes a few minutes to visualize for more complex architectures

Example:

# !pip install pytorch transformers graphviz torchview

from transformers import AutoModel, AutoTokenizer
from torchview import draw_graph
import torch
device = "cuda"
model = torch.nn.Linear(in_features=1, out_features=1, device=device)
model_graph = draw_graph(model, input_size=(32, 1), device=device)

model = AutoModel.from_pretrained("openai-community/gpt2")
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")

inputs = tokenizer("Hello world!", return_tensors="pt")

model_graph = draw_graph(model, input_data=inputs, device=device, expand_nested=True)

model_graph.visual_graph.node_attr["fontname"] = "Helvetica"
model_graph.visual_graph

[SeriousJ55]

LitGPT is amazing! Thank you very much for your great tool and your support!

I have a few questions regarding LitGPT:

• I get the following warning. How can I make it run on a GPU?

Using a model with more than 1B parameters on a CPU can be slow, it is recommended to switch to a GPU.

• My end goal is to get the logits (or the prob distribution) before the next token in case of a completion? Is there a way to access the logits?

[rasbt]

Glad to hear! By the way, I think you commented in the wrong repo :).
In any case, I think your issue may be that your computer doesn't have or cannot detect the GPU? You can do a rough check by executing the following in Python:

import torch
torch.cuda.is_available()

if it returns False, then your computer may not have a GPU, your nvidia drivers are not installed correctly, or PyTorch has some issue detecting the GPU.

[SeriousJ55]

Yes, you were right, cuda was not detected. I had to reinstall pytorch and it worked fine.

I'm asking my second question in LitGPT repo.

[rasbt]

glad it worked!