NL4OPT- Code and Dataset

This repo contains updates on NL4Opt Competition at Neurips 2022 and Its dataset.

News

• (Dec, 2024) Our new benchmark dataset ORQA is now published for public access. The paper was accepted at AAAI 2025: Evaluating LLM Reasoning in the Operations Research Domain with ORQA. Link
• (Jun, 2024) New paper is published at INFOR: Optimization modeling and verification from problem specifications using a multi-agent multi-stage LLM framework. Link
• (May, 2024) New paper is published at COLLING: Towards Human-aligned Evaluation for Linear Programming Word Problems. Link
• (July, 2023) New paper is published at ACL: LaTeX2Solver: a Hierarchical Semantic Parsing of LaTeX Document into Code for an Assistive Optimization Modeling Application. Link
• (Dec, 2022) Competition report has been published: NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions. Link
• (July, 2022) Following the discussions with Karthik (#8) and Sam (#15), we have decided to remove all lines containing in-sentence "white space tokens" from the NER dataset. The test set has also been revised to the same format, where it will be used evaluate your submissions for Subtask-1.

Subtasks

Baselines for each subtask are located in the following respositories

• Subtask 1 - NER
• Subtask 2 - generation

Dataset

Datasets for the respective tasks are contained in this repo.

Citation

Please cite our papers below if our dataset or competition aids you in your research.

@InProceedings{NL4OPT,
  title = 	 {NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions},
  author =       {Ramamonjison, Rindranirina and Yu, Timothy and Li, Raymond and Li, Haley and Carenini, Giuseppe and Ghaddar, Bissan and He, Shiqi and Mostajabdaveh, Mahdi and Banitalebi-Dehkordi, Amin and Zhou, Zirui and Zhang, Yong},
  booktitle = 	 {Proceedings of the NeurIPS 2022 Competitions Track},
  pages = 	 {189--203},
  year = 	 {2022},
  editor = 	 {Ciccone, Marco and Stolovitzky, Gustavo and Albrecht, Jacob},
  volume = 	 {220},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {28 Nov--09 Dec},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v220/ramamonjison23a.html},
}

@inproceedings{augmentingORModeling,
  title = "Augmenting Operations Research with Auto-Formulation of Optimization Models From Problem Descriptions",
  author = "Ramamonjison et al,.",
  booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track",
  month = dec,
  year = "2022",
  address = "Abu Dhabi, UAE",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2022.emnlp-industry.4",
  pages = "29--62",
}