Mathematical Handwriting Recognition

Introduction

This project is a web application that allows users to write mathematical expressions by hand and see them instantly converted into both LaTeX code and beautifully formatted mathematical notation. It was developed as a submission for StackUp February 2025 Coding Challenge and addresses the challenge of making mathematical input more intuitive and efficient for students, educators, and anyone working with mathematical content. The application leverages the MyScript Interactive Ink Web SDK for handwriting recognition and MathJax for rendering mathematical expressions.

Handwriting Recognition Model Selection and Justification

My primary focus was on identifying and integrating an existing, high-performing handwriting recognition model specifically designed for mathematical notation. Building and training a custom model from scratch was deemed infeasible due to the extensive data requirements and computational resources needed.

Candidate Models Considered:

Several options were considered for the handwriting recognition component:

• MyScript Interactive Ink Web SDK: A commercial handwriting recognition engine specifically designed for mathematical expressions. It offers high accuracy, a web-based SDK for easy integration, and a free tier suitable for development and testing.
• Tesseract.js: A JavaScript port of the popular Tesseract OCR engine. While Tesseract is powerful and versatile, it is primarily designed for general text recognition. Adapting it to accurately recognize complex mathematical notations would require significant custom training data and potentially complex pre- and post-processing, making it less suitable for a rapid prototyping project.
• Google Cloud Vision API: Google's Cloud Vision API can recognize handwritten text. However, it is not specifically optimized for mathematical symbols and equations. Achieving high accuracy for mathematical input would likely require significant post-processing of the recognized text and potentially custom symbol mapping.
• Mathpix Snip: This is also a dedicated tool and API, but after brief review of the two, the MyScript seems more promising to my project.
• Custom Model Training (CNN/RNN): Ideally, a custom Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) trained on a large dataset of handwritten mathematical expressions would offer the highest potential for accuracy and customization. However, creating such a model from scratch, or even fine-tuning a pre-trained model extensively, was not feasible within the for me due to the significant data collection, annotation, and training time required.

Model Selection Rationale:

MyScript Interactive Ink Web SDK was selected as the optimal solution for this project due to the following reasons:

• Specialization: MyScript has specific usecase designed for mathematical handwriting recognition, unlike general-purpose OCR engines. This specialization translates to higher accuracy and better handling of complex mathematical notations.
• Ease of Integration: The MyScript Web SDK provides a straightforward JavaScript API for integrating handwriting recognition into web applications. This significantly reduced development time compared to building a custom solution or adapting a general-purpose OCR engine.
• Accuracy: MyScript is known for its high accuracy in recognizing mathematical handwriting, making it suitable for educational and professional applications where precision is critical.
• Free Tier: The availability of a free tier allowed for development and testing without incurring costs.
• Real-time Recognition: Supports real-time conversion.

Model Evaluation:

The application itself serves as a real-time evaluation of the MyScript model's performance. By writing various mathematical expressions on the canvas, users (and judges) can directly assess the accuracy and responsiveness of the recognition engine.

Technical Details

The application is built using the following technologies:

• Frontend: HTML, CSS, JavaScript
• Handwriting Recognition: MyScript Interactive Ink Web SDK
• Math Rendering: MathJax
• Build Tool: Vite

The application's structure is straightforward:

1. Canvas Input: An <editor> element powered by iink-ts library captures user input in real-time as they write mathematical expressions.

2. MyScript Integration: The MyScript Web SDK (iink-ts) is initialized with:

   • WebSocket protocol configuration
   • Environment-based API keys (applicationKey and hmacKey)
   • Math recognition type with LaTeX output format

3. Event Handling: The application manages multiple event listeners:

   • changed: Updates button states (undo/redo/clear) based on editor state
   • exported: Processes recognized math expressions

4. Output Processing:

   • LaTeX Recognition: Captured in a variable when MyScript returns results
   • Text Display: Shows raw LaTeX
   • MathJax Rendering: Converts LaTeX to beautifully formatted mathematical notation with error handling

5. User Interface Controls:

   • Clear Button: Resets the editor and clears all outputs
   • Undo/Redo Buttons: Manages drawing history with dynamic enabling/disabling
   • Copy Button: Copies LaTeX to clipboard with visual feedback ("Copied!" message)

Instructions

To run this project locally:

1. Clone the Repository:

   git clone 

2. Obtain MyScript API Keys:

   • Sign up for a free MyScript developer account .
   • Obtain your application key and HMAC key.

3. Install Dependencies:

    pnpm install

4. Insert API Keys:

   • Create a .env file in the root directory of the project.
   • Add your MyScript application key and HMAC key to the .env file in the following format:

   VITE_MYSRIPT_APP_KEY=YOUR_APPLICATION_KEY
   VITE_MYSRIPT_HMAC_KEY=YOUR_HMAC_KEY

5. Run the Application:

   • Using Vite, run this command.

   pnpm run dev

   • Open your web browser and navigate to the URL provided by Vite (usually http://localhost:5173 or similar).

Dependencies

• MyScript Web SDK
• MathJax
• Vite

Known Limitations

• Requires internet connectivity
• MyScript may have challenges with extremely obscure or non-standard mathematical notations.
• MyScript is a black-box model, so detailed insights into its inner workings are limited.
• The free tier of MyScript has usage limits, which may affect the application's availability under heavy usage.