Add granite #1882
Add granite #1882
Conversation
There was a problem hiding this comment.
Thanks for putting together this PR for Granite! I'm one of the Granite team leads for open source integrations, so I've got a couple of notes inline. Also, in addition to pulling head_dim from config and applying the embedding_multiplier, there are three more scalars that are used in the GraniteForCausalLM architecture that differentiate it from llama:
- attention_multiplier: This should be applied as a scaling after the self attention block of each layer (here)
- residual_multiplier: This should be applied when recombining with the residual after the self attention block (here) and the FFN (here) in each layer
- logits_scaling: This should be applied to the output
logitsbefore returning them from the model forward pass (here)
| rotary_interleave=False, | ||
| rotary_base=getattr(model.config, "rope_theta", 5000000.0), | ||
| num_heads_kv=num_heads_kv, | ||
| head_dim=model.config.hidden_size // model.config.num_attention_heads, |
There was a problem hiding this comment.
In Granite, the head_dim can be pulled directly from config with the fallback being this calculation (here)
| self.set_linear(spec.decoder.projection, model.lm_head) | ||
|
|
||
| if hasattr(model.config, "embedding_multiplier") and model.config.embedding_multiplier: | ||
| spec.decoder.embeddings.multiply_by_sqrt_depth = model.config.embedding_multiplier |
There was a problem hiding this comment.
Without unboxing the repo further, I can't tell exactly whether this is the right place for embedding_multipler, but it looks right. It should simply be applied as a scaling after the embeddings are computed (here)
Should add the really cool granite models by IBM.