feat(transformers): add EXAONE 4 (v4.54.1)

[alien-0119]

What does this PR do?

Adds # (feature)
Add EXAONE 4 model and fast ut.

Usage Example:

from mindone.transformers import AutoModelForCausalLM
from transformers import AutoTokenizer
import mindspore as ms

model = AutoModelForCausalLM.from_pretrained("LGAI-EXAONE/EXAONE-4.0-1.2B")
tokenizer = AutoTokenizer.from_pretrained("LGAI-EXAONE/EXAONE-4.0-1.2B")

prompt = "Explain how wonderful you are"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="np",
    enable_thinking=False,
)
input_ids = ms.tensor(input_ids)

output = model.generate(input_ids, max_new_tokens=128)
tokenizer.decode(output[0], skip_special_tokens=False)
# As EXAONE, I am designed to assist you effectively. My purpose is to understand and execute instructions accurately, providing helpful responses based on extensive training data. While I don't have personal feelings, I am here to enhance your experience through clear, precise, and efficient communication. Let me know how I can make your interactions better![|endofturn|]

Performance:
Experiments were tested on Ascend Atlas 800T A2 machines with mindspore 2.7.0 pynative mode.

model	precision	weight load(s)	s/step
LGAI-EXAONE/EXAONE-4.0-1.2B	fp32	48.403	0.410
LGAI-EXAONE/EXAONE-4.0-1.2B	fp16	49.171	0.447
LGAI-EXAONE/EXAONE-4.0-1.2B	bf16	33.916	0.407

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
  documentation guidelines
• Did you build and run the code without any errors?
• Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

[gemini-code-assist]

Summary of Changes

Hello @alien-0119, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the mindone.transformers library by integrating the EXAONE 4 model. This addition provides a new powerful language model and its derivatives for various NLP tasks, making it available within the MindSpore ecosystem. The changes also include minor but important infrastructure improvements for tensor handling and parallelism tracking, ensuring better compatibility and future scalability.

Highlights

• New Model Integration: Introduced full support for the EXAONE 4 (v4.54.1) model, including its base model, causal language model, and various task-specific heads (sequence classification, token classification, question answering).
• Tensor Parallelism Support: Added a _tp_size attribute and tp_size property to the PreTrainedModel class to track the tensor parallelism degree, enhancing model sharding capabilities.
• Masking Utility Improvement: Implemented a fix in masking_utils.py to correctly handle padding_length when it is a mindspore.Tensor, ensuring robust padding operations.
• Comprehensive Testing: Included new unit tests for all EXAONE 4 model variants, verifying their functionality and numerical precision against PyTorch implementations across different data types.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the EXAONE 4 model, including its implementation, configuration, and integration with auto-model classes, along with corresponding tests. The changes largely follow the standard procedure for adding a new model. My review has identified a potential high-severity issue in the application of Rotary Position Embeddings (RoPE) for hybrid attention configurations, which could impact model correctness. Additionally, a medium-severity typo in a code example within a docstring has been noted, which could cause confusion for users.

[gemini-code-assist]

The logic for applying Rotary Position Embeddings (RoPE) appears incorrect for hybrid attention models. When self.sliding_window is not None, RoPE is not applied to full attention layers (self.is_sliding is False). This would leave these layers without positional information, likely causing incorrect model behavior. Positional embeddings are typically required for all attention layers to process token order correctly.

The condition should likely be removed to apply RoPE to all layers. Additionally, the comment on line 220 seems to have a typo and should likely be 'RoPE' instead of 'NoPE'.

        # We use global RoPE for hybrid attention model
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)

[gemini-code-assist]

There is a typo in the example code within the docstring. The variable out putcontains a backtick and extra spaces, which will cause a syntax error if a user copies and pastes this example. It should beoutput`.

        >>> tokenizer.decode(output[0], skip_special_tokens=False)