GenDI is a research prototype of an interactive system to create standardized data products. This readme describes the workflow to setup the system, generate the necessary data, use the graphical user interface, and conduct experiments to reproduce the published results.
GenDI Prototype
File: README.MD
Authors: Tobias Jacobs (tobias.jacobs@neclab.eu)
NEC Laboratories Europe GmbH, Copyright (c) 2025, All rights reserved.
THIS HEADER MAY NOT BE EXTRACTED OR MODIFIED IN ANY WAY.
PROPRIETARY INFORMATION ---
SOFTWARE LICENSE AGREEMENT
ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY
BY USING OR DOWNLOADING THE SOFTWARE, YOU ARE AGREEING TO THE TERMS OF THIS
LICENSE AGREEMENT. IF YOU DO NOT AGREE WITH THESE TERMS, YOU MAY NOT USE OR
DOWNLOAD THE SOFTWARE.
This is a license agreement ("Agreement") between your academic institution
or non-profit organization or self (called "Licensee" or "You" in this
Agreement) and NEC Laboratories Europe GmbH (called "Licensor" in this
Agreement). All rights not specifically granted to you in this Agreement
are reserved for Licensor.
RESERVATION OF OWNERSHIP AND GRANT OF LICENSE: Licensor retains exclusive
ownership of any copy of the Software (as defined below) licensed under this
Agreement and hereby grants to Licensee a personal, non-exclusive,
non-transferable license to use the Software for noncommercial research
purposes, without the right to sublicense, pursuant to the terms and
conditions of this Agreement. NO EXPRESS OR IMPLIED LICENSES TO ANY OF
LICENSOR'S PATENT RIGHTS ARE GRANTED BY THIS LICENSE. As used in this
Agreement, the term "Software" means (i) the actual copy of all or any
portion of code for program routines made accessible to Licensee by Licensor
pursuant to this Agreement, inclusive of backups, updates, and/or merged
copies permitted hereunder or subsequently supplied by Licensor, including
all or any file structures, programming instructions, user interfaces and
screen formats and sequences as well as any and all documentation and
instructions related to it, and (ii) all or any derivatives and/or
modifications created or made by You to any of the items specified in (i).
CONFIDENTIALITY/PUBLICATIONS: Licensee acknowledges that the Software is
proprietary to Licensor, and as such, Licensee agrees to receive all such
materials and to use the Software only in accordance with the terms of this
Agreement. Licensee agrees to use reasonable effort to protect the Software
from unauthorized use, reproduction, distribution, or publication. All
publication materials mentioning features or use of this software must
explicitly include an acknowledgement the software was developed by NEC
Laboratories Europe GmbH.
COPYRIGHT: The Software is owned by Licensor.
PERMITTED USES: The Software may be used for your own noncommercial
internal research purposes. You understand and agree that Licensor is not
obligated to implement any suggestions and/or feedback you might provide
regarding the Software, but to the extent Licensor does so, you are not
entitled to any compensation related thereto.
DERIVATIVES: You may create derivatives of or make modifications to the
Software, however, You agree that all and any such derivatives and
modifications will be owned by Licensor and become a part of the Software
licensed to You under this Agreement. You may only use such derivatives and
modifications for your own noncommercial internal research purposes, and you
may not otherwise use, distribute or copy such derivatives and modifications
in violation of this Agreement.
BACKUPS: If Licensee is an organization, it may make that number of copies
of the Software necessary for internal noncommercial use at a single site
within its organization provided that all information appearing in or on the
original labels, including the copyright and trademark notices are copied
onto the labels of the copies.
USES NOT PERMITTED: You may not distribute, copy or use the Software except
as explicitly permitted herein. Licensee has not been granted any trademark
license as part of this Agreement. Neither the name of NEC Laboratories
Europe GmbH nor the names of its contributors may be used to endorse or
promote products derived from this Software without specific prior written
permission.
You may not sell, rent, lease, sublicense, lend, time-share or transfer, in
whole or in part, or provide third parties access to prior or present
versions (or any parts thereof) of the Software.
ASSIGNMENT: You may not assign this Agreement or your rights hereunder
without the prior written consent of Licensor. Any attempted assignment
without such consent shall be null and void.
TERM: The term of the license granted by this Agreement is from Licensee's
acceptance of this Agreement by downloading the Software or by using the
Software until terminated as provided below.
The Agreement automatically terminates without notice if you fail to comply
with any provision of this Agreement. Licensee may terminate this Agreement
by ceasing using the Software. Upon any termination of this Agreement,
Licensee will delete any and all copies of the Software. You agree that all
provisions which operate to protect the proprietary rights of Licensor shall
remain in force should breach occur and that the obligation of
confidentiality described in this Agreement is binding in perpetuity and, as
such, survives the term of the Agreement.
FEE: Provided Licensee abides completely by the terms and conditions of this
Agreement, there is no fee due to Licensor for Licensee's use of the
Software in accordance with this Agreement.
DISCLAIMER OF WARRANTIES: THE SOFTWARE IS PROVIDED "AS-IS" WITHOUT WARRANTY
OF ANY KIND INCLUDING ANY WARRANTIES OF PERFORMANCE OR MERCHANTABILITY OR
FITNESS FOR A PARTICULAR USE OR PURPOSE OR OF NON- INFRINGEMENT. LICENSEE
BEARS ALL RISK RELATING TO QUALITY AND PERFORMANCE OF THE SOFTWARE AND
RELATED MATERIALS.
SUPPORT AND MAINTENANCE: No Software support or training by the Licensor is
provided as part of this Agreement.
EXCLUSIVE REMEDY AND LIMITATION OF LIABILITY: To the maximum extent
permitted under applicable law, Licensor shall not be liable for direct,
indirect, special, incidental, or consequential damages or lost profits
related to Licensee's use of and/or inability to use the Software, even if
Licensor is advised of the possibility of such damage.
EXPORT REGULATION: Licensee agrees to comply with any and all applicable
export control laws, regulations, and/or other laws related to embargoes and
sanction programs administered by law.
SEVERABILITY: If any provision(s) of this Agreement shall be held to be
invalid, illegal, or unenforceable by a court or other tribunal of competent
jurisdiction, the validity, legality and enforceability of the remaining
provisions shall not in any way be affected or impaired thereby.
NO IMPLIED WAIVERS: No failure or delay by Licensor in enforcing any right
or remedy under this Agreement shall be construed as a waiver of any future
or other exercise of such right or remedy by Licensor.
GOVERNING LAW: This Agreement shall be construed and enforced in accordance
with the laws of Germany without reference to conflict of laws principles.
You consent to the personal jurisdiction of the courts of this country and
waive their rights to venue outside of Germany.
ENTIRE AGREEMENT AND AMENDMENTS: This Agreement constitutes the sole and
entire agreement between Licensee and Licensor as to the matter set forth
herein and supersedes any previous agreements, understandings, and
arrangements between the parties relating hereto.
THIS HEADER MAY NOT BE EXTRACTED OR MODIFIED IN ANY WAY.
The GenDI research prototype consists of the following components:
- The GenDI Server, serving
- the GenDI web UI,
- the GenDI API.
- Ipython Notebooks for
- Vector Database Creation,
- Smart Datamodel Cache Creation,
- Benchmark Dataset Generation,
- Benchmark Testing.
The GenDI research prototype is a suite of Python programs that has been tested on Python 3.9.13 for Windows as well as Python 3.10.12 on Linux based systems. The required libraries are listed in the requirements.txt file; usage of a virtual environment (conda, virtualenv) is highly recommended. Additionally, Jupyter is required to run the Ipython notebooks, with a Python Kernel that includes the required libraries.
Furthermore, GenDI uses Azure OpenAI as LLM provider. An Azure OpenAI endpoint with at least one chat completion LLM model deployment as as well as one embedding model deployment is required to use GenDI.
Before starting the server, the vector database has to be created and populated with information about the desired target datamodels. The Ipython notebook stored at notebooks/Populate_Vector_Store.ipynb contains the program code for this task. As it relies on Azure OpenAI embedding models, credentials need to be provided as variables in the appropriate cells of the notebook.
While example objects for Smart Data Models are available in principle via the pysmartdatamodels package, there are sometimes problems with inaccessible retrieval URLs. For this reason, the GenDI Server instead relies on a cache of example objects. To populate the cache, open and run the Ipython notebook notebooks/CacheSmartDMExamples.ipynb.
Warning: the GenDI Server of this research prototype should never be run on a publicly accessible machine without appropriate security measures. The GenDI server allows execution of arbitrary Python code on the Server, including potentially harmful code.
To configure the GenDI server, open src/api/config.py and edit the configuration parameters in the dictionary declared therein:
- server: The address of the GenDI server. localhost will do if the server will be accessed locally.
- port: The port under which the GenDI server shall be reachable.
- vdb_loc: The path to the vector database.
- subjects: The list of subjects that shall be selectable in the dropdown menu of the web UI.
- default_llm_deployment: The name of the Azure OpenAI deployment to be used as the default LLM (that is, when no other deployment is specified as a parameter of individual requests).
To start the GenDI Server, first activate the virtual Python environment (if applicable).
After that, the Azure OpenAI credentials need to be specified as the environment variables AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_KEY. Under Linux, this is achieved by typing
- export AZURE_OPENAI_ENDPOINT=(your endpoint)
- export AZURE_OPENAI_API_KEY=(your key)
Finally, navigate to the src folder and start the server with the command python -m api.api_server.
Once the GenDI Server has been started, the Web UI is accessible from a web browser under the server root address. For example, when the server is running at localhost on port 7861, the web UI can be accessed by opening a browser and entering http://localhost:7861 in the address bar.
To generate a data transformation function using the Web UI, the following workflow is recommended:
- Select a data source. This automatically loads a data sample into the Source Data Sample field. In this prototype, only JSON-formatted data is supported.
- Select a subject and datamodel for the data. To get assistance from the AI, click the AI-predict datamodels button. This will populate the dropdown menu with a number of datamodel suggestions, sorted by likelihood.
- Transform the data sample into the target datamodel. You can click the Fill example target data button to fill the text area with an example object from the target, which you can manually edit to make the values match with the source sample. Alternatively, the AI-transform source sample will trigger the AI function to translate the source sample to the target datamodel and fill the result into the Transformed Data Sample field. The button Apply transformation function will attempt to execute the transformation program code and populate the Transformed Data Sample field with the result. This will fail as long as the transformation program code has not been written. Finally, you can provide instructions about how the source data should be transformed to the target in the corresponding field.
- Code the transformation function. The button AI-code will trigger the AI to generate the program code using the above sample transformation as an example to guide the code generation. The generated code can be tested on the Source Data Sample using the Apply transformation function button. Code generation will take into account the instructions or feedback provided in the instructions field.
The RESTful API of the GenDI server supports the following functions via POST requests to the respective endpoints with JSON payloads. All functions return a JSON object with the two keys result and log, where the latter contains a server log for further analysis.
- /select_data_model: Retrieve a set of recommended datamodels. Request parameters are
- data: The input data sample for which a matching datamodel is requested.
- no_results (default:1): The number of datamodels to retrieve. If this parameter is 2 or larger, the datamodels will be returned as a ranked list.
- llm-rerank (default:False): Whether to re-rank the datamodels after retrieval from the vector store.
- /transform_data: Transform a sample to the target datamodel. Request parameters are
- model: Name of the datamodel to transform to.
- data: The data sample to be transformed.
- /get_code: Generate the transformation function Python code. Request parameters are
- src_data: The source sample to be used as part of the one-shot example.
- tgt_transformation: The transformed source sample.
- /get_code_directly: A function to directly generate the transformation code from the source sample and the target datamodel, to be used for ablation studies. Request parameters are src_data and model.
In addition to the above individual parameters, all functions above support a common parameter llm to change the Azure OpenAI deployment to a different model.
The Ipython Notebook benchmark/creation/generateBenchmarkFromSmartDMv2.ipynb contains the program code to generate a testing benchmark from Smart Data Models. The notebook accesses the library smartDM2inputv2.py provided in the benchmark/creation folder as well. Additionally, as the generation process uses a chat completion LLM provided via Azure OpenAI, credentials have to be provided in the form of files named 'open_ai_endpoint.txt' and 'open_ai_key.txt' placed in the same folder. If necessary, also adapt the deployment name and API version inside smartDM2inputv2.py.
The Ipython Notebook notebooks/LLM4DT Benchmark Test.ipynb contains the program code to evaluate GenDI on the generated benchmark. At the top of the notebook, all parameters for the test can be set, including the function to test, the LLM deployment, whether to use the full benchmark or a subset, and the performance metrics to evaluate. Once the parameters have been set, the benchmark test and evaluation can be run by executing all cells in top-down order. This will generate a file with detailed results for each test sample as well as aggregated metrics.