OpenQASM 3 Parser Benchmarks

This project compares the performance and capabilities of three different OpenQASM 3 parsers:

1. Qiskit ANTLR Parser (Reference Implementation)
2. Qiskit Rust Parser (Experimental)
3. Qasm-ts

• Parser Implementations
  • Qiskit ANTLR Parser
  • Qiskit Rust Parser
  • Qasm-ts
• Usage
  • Setup
  • Running the Benchmarks
• Benchmarking Implementation Notes

---

Parser Implementations

Qiskit ANTLR Parser

The ANTLR-based parser is qiskit's reference implementation for parsing OpenQASM 3. It is accessible through the loads() and load() methods, which require installing the qiskit-qasm3-import package. This parser uses ANTLR4 to generate a parser from the official OpenQASM 3 grammar.

The general workflow for parsing OpenQASM 3 from the qiskit reference parser is as follows:

1. OpenQASM 3 String - Loaded script or string of OpenQASM
2. qiskit.qasm3.loads() - Entry point in qiskit
3. qiskit_qasm3_import.parse() - Calls openqasm3 package
4. openqasm3.parse() - Entry point for the ANTLR parser
5. qasm3Lexer - ANTLR lexer to tokenize input string
6. qasm3Parser - ANTLR parser to generate the abstract syntax tree

Note: Qiskit provides further downstream steps such as converting the resulting AST to a Circuit. For sake of benchmarking, we are only concerned up to the AST generation.

Qiskit Rust Parser

The qiskit rust parser is a native high performance experimental parser that is built as a native extension through the main qiskit Python package (through PyO3 bindings) and exposed through qiskit._accerlate. The rust parser is used when using the loads_experimental() and load_experimental() methods. The actual building of the qiskit._accelerate Python extension module is done through the qiskit-pyext crate. The Python bindings are built through the qiskit._qasm3 crate, the rust level qiskit interface to the native rust parser.

The general workflow for parsing OpenQASM 3 from the experimental rust parser is as follows:

1. OpenQASM 3 String - Loaded script or string of OpenQASM
2. qiskit.qasm3.loads_experimental() - Entry point in qiskit
3. qiskit._accelerate_qasm3.loads() - The PyO3 binding that connects Python to Rust
4. oq3_lexer - Tokenizes the input string
5. oq3_parser - Creates the abstract syntax tree

Note: The rust rust parser provides further downstream steps such as performing semantic analysis on the resulting AST to build an abstract semantic graph (ASG). For sake of benchmarking, we are only concerned up to the AST generation.

QASM-TS

The qasm-ts parser is an external typescript library for usage in web applications.

The general workflow for parsing OpenQASM 3 from the qasm-ts parser is as follows:

1. OpenQASM 3 String - Loaded script or string of OpenQASM
2. parseString() - Entry point in qasm-ts
3. lex() - Tokenizes the input string
4. parse() - Creates the abtract syntax tree

Usage

Setup

It is recommended to create a python virtual environment.

python3 -m env env
source env/bin/activate   # For Linux/macOS
env\Script\activate       # For Windows

Next, install project dependencies.

For Python dependencies:

pip install -r requirements.txt

For Node.js dependencies:

npm install

Running the Benchmarks

python benchmarks/parse_bench.py

Benchmarking Implementation Notes

• The OpenQASM 3 Python bindings provided to the Rust parser do not expose a way to just generate an AST. Rather, the exposed loads() and load() functions also perform semantic analysis to generate abstract semantic graphs. In order to benchmark just to the AST generation, the Rust crates were used directly rather than through the Python bindings.
• Similarly, the qiskit-qasm3-import module only exposes functions that perform ANTLR parsing plus downstream conversion of the AST to a Qiskit circuit. To avoid this, the openqasm library is used directly.
• In order to avoid including Nodejs and Rust compilation in the benchmark times, a non-timed warm up run is used to initialize system resources. In the Rust instance, the warmup run is used to compile the release binary and the path is cached using an LRU cache for retrieval on subsequent timed iterations. For the Typescript library, the warm up run is used to initialize a server where the QASM snippets to be parsed will be sent.