Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver

Dataset: 🤗 SynthGeo-228k | Diagram Formalizer: 🤗 Diagram Formalizer | Reasoning Model: 🤗 DFE-GPS-9B | 🤗 DFE-GPS-34B

Click to expand the detailed description

In this study, we introduce the Diagram Formalization Enhanced Geometry Problem Solver (DFE-GPS), a multi-modal architecture with three core components: a Diagram Formalizer, a Projection module, and a LLM. The LLM processes three types of inputs: diagram features $\mathcal{F}_{D}$ from the Diagram Encoder, formal diagram language representations (ConsCDL and ImgCDL) from the Diagram Formalizer, and natural language inputs containing problem statements and instructions. The Projection module aligns this information in the LLM's semantic space, enabling effective fusion. The LLM then refines the formal representations and generates reasoning steps for problem-solving. We used pre-trained SigLIP-0.4B as the Vision Encoder, Qwen2-0.5B-Instruct as the Lightweight LLM, and Yi-1.5-Chat (9B or 34B) as the primary LLM. The training process is divided into three stages, all of which focus on auto-regressive generation tasks.

Stage 1: The first stage focuses on training the Diagram Formalizer module, with training objective of generating formalized language descriptions corresponding to geometric diagrams. During this stage, all parameters of the Vision Encoder and part of the parameters of the Lightweight LLM (via the LoRA) are trainable to enhance the ability to extract visual features.
Stage 2: The second stage emphasizes training the Projection modules, aligning vision feature $\mathcal{F}_{D}$ with the semantic space of the LLM by generating natural language descriptions and formalized language expressions for the geometric diagrams. During training, the parameters of the Diagram Encoder and LLM are frozen, with only the MLP parameters connecting the visual features and the language model being trainable.
Stage 3: In the third stage, instruction fine-tuning enables the model to calibrate formalized diagram representations and solve problems. The input consists of geometric diagrams, formalized descriptions with random perturbations simulating Diagram Formalizer errors, and problem text accompanied by calibration and reasoning directives. The model learns to refine ConsCDL and ImgCDL, generating coherent natural language reasoning. During this phase, the parameters of the Diagram Encoder remain fixed, while the MLP and LLM parameters are trainable. Full parameter tuning is applied to the 9B model, whereas LoRA tuning is employed for the 34B model.

Quick Start

Before running the script, install the following necessary dependencies.

pip install torch==2.4.0 transformers==4.40.0 accelerate pillow sentencepiece

You can solve geometric problems using the following script. First, formalize the geometric images with the diagram formalizer, and then use the multi-modal reasing model for problem-solving:

python simple_infer.py
# image path: ./images/6173.png
# question: As shown in the diagram, ∠DOB=130°, and the center of circle O is O. Find the measure of ∠BCD.

Case Study

This section lists some test cases that feature a single problem with solutions generated by different models to illustrate the evaluation process using GPT-4o-mini. Based on experimental results and case studies, it is found that utilizing large language models (LLMs) to assess the solution generation process is reliable.

Problem Statement

As shown in the diagram, ∠DOB=130°, and the center of circle O is O. Find the measure of ∠BCD.

Reference Answer

由题意得，∠DOB=130°（1），O是圆O的圆心（2）；由弧的性质（圆内接四边形对角互补）可得，∠BCD = -⌒OBD/2 + 180°（3）；已知条件（2），由常识可得，∠DOB = ⌒OBD（4）；已知条件（1）（4），计算可得，⌒OBD = 130°（5）；已知条件（3）（5），计算可得，∠BCD = 115°（6）；完成解题。

Evaluation Instruction

Please first check the reference answer and then the student answer, and make sure you carefully examine each reasoning step with your reason.

1. **Check both the reference answer and the student's answer:**
   - Initially verify if the student’s response aligns with the reference answer, focusing particularly on key steps and outcomes

2. **Carefully review each reasoning step:**
   - **Accuracy of Calculations (0-10 points):** Examine the accuracy of the computational steps in the student's answer. Adjust the score based on the number and severity of errors.
   - **Logical Reasoning (0-10 points):** Assess whether the logic in the student's answer is sound and coherent, and if it correctly transitions from one step to the next.
   - **Redundancy (0-10 points):** Analyze whether the answer contains unnecessary repetitive steps or explanations. More concise and elegant answers should score higher.

3. **Verify each step:** Check each step to ensure it complies with mathematical principles and logic. If any step contains logical or computational errors, mark it as “INCORRECT.” If all steps are correct, mark it as “CORRECT.”

4. **Output Format: Score Summary:**
   - Calculation Accuracy: [Score]/10
   - Logical Reasoning: [Score]/10
   - Redundancy: [Score]/10

5. **Final Judgment:** Either 'CORRECT' or 'INCORRECT' based on the step verifications.