Consistency is a critical concern in knowledge engineering (KE), particularly given the increasing reliance on Large Language Models (LLMs) in various tasks. This work introduces a framework to assess the consistency of large language models when used to support KE (as depicted the following figure).
The CoLLM, is designed to assess whether a system or process produces consistent results in LLM-based KE tasks through three tests:
- LLM Reproducibility Test: What is the extent of the stochasticity or non-determinism of the results of previous scientific studies?
- LLM Update Impact Test: To what extent do updates of LLMs affect the results of existing studies?
- LLM Replacement Test: To what extent does an alternative LLM achieve similar performance to those used in previous studies, or are certain studies dependent on a specific LLM?
These tests empirically validate the consistency of the results in KE studies under varying conditions to ensure the reliability of the reported findings of research efforts. The framework is validated by means of extensive experimentation using five recent research papers to define the research work, and leveraging various LLMs and datasets.
case-studies: you need to create your case-study-specific directory here (if it is not there already). Please organize your code to be executable via the same directory as a root.images: directory contains any general images/diagrams related to the project. e.g. architecture diagram.
The case studies of this work are as follows:
| Title | Directory | |
|---|---|---|
| 1 | Navigating Ontology Development with Large Language Models | OntologyDevelopement |
| 2 | Ontology generation with metacognitive prompting and LLMs | Ontogenia |
| 3 | LLMs4OL: Large Language Models for Ontology Learning | LLMs4OL |
| 4 | LLMs4OM: Maching Ontologies with Large Language Models | LLMs4OM |
| 5 | Retrofitting CQs from Existing Ontologies Using LLMs | RETROFIT-CQs |
This software is licensed under the MIT License.
