Semantic annotation standards and tools

[hampusnasstrom]

This is a discussion thread for the work group on semantic annotation standards and tools.

[hampusnasstrom]

Tools

Languages

• Python
• Javascript (frontend)
• Java

Python packages

• PyLD for JSON-LD in Python

Other packages

• dlite

Standards

• JSON Schema
• RDF
• JSON LD
• XML (Generally not liked...)

"Transport of RDF should be through JSON-LD"

YAML can be used but should be sent and saved in JSON

There are tools for converting pydantic to JSON Schema. FastAPI can generate a JSON Schema from the pydantic model returned at an endpoint.

Questions

"Is it enough to send a JSON-LD with a JSON Schema?"

• One problem is units, there are a lot of different ontologies for defining units

"How do you deal with versions of schemas?"

• We should have clear instructions on how to version schemas
• Is there a tool for checking compatibility of schema versions?

Tasks

1. Convert .csv to JSON(-LD) and to another ELN

• NOMAD tabular parser

TODO: Write a short (as short as possible) (python/pseudo) script to read and compare two different files that are semantically annotated.
GOAL: Figure out how the semantic annotations need to look

• Identify datasets for semantic annotation/transformation
• Create exemplars for creating a semantic metadata record in JSON-LD, and fully transforming data into JSON-LD
• Identify what aspects of these exemplars can be easily automated in python and what is more complex
• create some simple automation scripts

2. (Graphical) Tool for writing a schema

3. Units

• Gather ontologies for units
• Gather example files
• Decide on tool name
• Create repository
• Create package on PyPI

[samikanza]

.csv & JSON-LD Tools

• https://github.com/ali1k/csv-to-jsonLD
• http://ld-r.org/docs/csvJSONLD.html
• https://pypi.org/project/csv-ld/

[samikanza]

some potentially useful datasets in JSON-LD for reference: https://scidata.unf.edu/tranche/trc/

[simontaurus]

@hampusnasstrom
Work on Units in OpenSemanticLab:
https://github.com/OpenSemanticLab/osw-python/blob/13576c4c2aac045de91923757b3a703ef5c2b161/nb/quantities/Quantities.ipynb

[hampusnasstrom]

https://scidata.unf.edu/tranche/trc/jced/acs.jced.5b00005_8.jsonld

[hampusnasstrom]

playground_link_v0.0.1

[giovannipizzi]

• About converting actual data thanks to (meta)data semantic annotations: A software to undertand semantically annotated data in various formats (csv, ...) ad even convert units: https://github.com/SINTEF/dlite
• About schem versioning, and specifically if there is a tool to check if a new schema is backward-compatible: I found this tool, is relatively recent: https://github.com/IBM/jsonsubschema

[hujo91]

may be of interest:

• https://metadatacenter.org/#about
• https://blog.rwth-aachen.de/forschungsdaten/en/2022/03/24/aims-in-coscine-neues-feature-22-maerz-2022/

[cbarillari]

Additional discussion points:

• how much metadata should be used?
• how do you deal with missing data?
• tools for writing schema (UI-based for regular users). Need to limit options for users.
• how to treat units for measures. At which level should they be included?
• Automation is our goal, but seems very far. Agree on best practices woudl alreday be an achievement.
• recommendation on how to do schema versions

[cbarillari]

https://stuchalk.github.io/scidata/

[fabioacl]

Tool for finding ontologies and terms: https://www.ebi.ac.uk/ols4/

[samikanza]

https://bioportal.bioontology.org/
https://matportal.org/ontologies
https://terminology.nfdi4chem.de/ts/

[samikanza]

Generic Ontologies for generic metadata

• Dublin Core - https://www.dublincore.org/specifications/dublin-core/dcmi-terms/
• DCAT - https://www.w3.org/TR/vocab-dcat-2/ (uses Dublin Core)
• Schema.org - https://schema.org/
• DataCite - https://sparontologies.github.io/datacite/current/datacite.html

[samikanza]

Python Library

• https://github.com/digitalbazaar/pyld

[samikanza]

Potential tasks

• Unit work - gather information about unit implementations and find some example documents and create package on PyPi
• Identify datasets for semantic annotation/transformation
• Create exemplars for creating a semantic metadata record in JSON-LD, and fully transforming data into JSON-LD
• Identify what aspects of these exemplars can be easily automated in python and what is more complex
• create some simple automation scripts

[samikanza]

Useful Literature Articles

• https://www.sciencedirect.com/science/article/pii/S1570826822000373?casa_token=LpWIzFH6yxMAAAAA:eZrJnqc578rGVKJVpRUggqkkHdCWdwHsgpEgzAvBIXARsl8JPx3Z5u_H4gqhprsHrU4KINBy2io
• https://www.researchgate.net/profile/Research-Publications-2/publication/353971524_Review_of_Various_Techniques_of_Automatic_Metadata_Extraction_from_Digital_Documents/links/611ca0331e95fe241adbdce9/Review-of-Various-Techniques-of-Automatic-Metadata-Extraction-from-Digital-Documents.pdf

[despadam]

Link to available datasets: https://zenodo.org/records/10990048

[simontaurus]

Reference dataset (@context is added to the original data)

{
  "@context": {
    "mad": "https://madices.github.io/",
    "entry_id": "@id",
    "archive": "@nest",
    "data": "@nest",
    "samples": {
      "@id": "mad:hasInput",
      "@type": "@id"
    },
    "name": {
      "@id": "mad:name"
    },
    "lab_id": "@id",
    "reaction_results": "@nest",
    "reactants_conversions": {
      "@id": "mad:hasReactant",
      "@type": "@id",
      "@context": {
        "name": "@id"
      }
    }
  },
  "entry_id": "3fe5dguZxs7e7bVbfAdjmHXnCI8u",
  "parser_name": "parsers/archive",
  "archive": {
    "data": {
      "m_def": "../upload/raw/measurement_entry_schema.archive.yaml#/definitions/section_definitions/0",
      "name": "MnWO4_C3performance",
      "datetime": "2024-04-16T11:38:18.967172+00:00",
      "location": "FHI/AC department",
      "reaction_class": "Oxidation",
      "reaction_name": "Oxidation of Propane",
      "experimenter": "P. Kube",
      "data_file": "C3_performance_MnWO4.xlsx",
      "samples": [
        {
          "m_def": "nomad.datamodel.metainfo.basesections.CompositeSystemReference",
          "name": "32024",
          "reference": "../uploads/YdPbMF9bT0621URM2mboUg/archive/pw50pT2_2tTTJMbiDphASGqcSAuX#/data",
          "lab_id": "32024"
        }
      ],
      "reactor_filling": {
        "catalyst_mass": 231.1
      },
      "reaction_results": {
        "reactants_conversions": [
          {
            "name": "oxygen"
          },
          {
            "name": "propane"
          }
        ]
      }
    }
  }
}

[simontaurus]

JSON-LD playground with mapping: link

[samikanza]

https://json-ld.org/playground/#startTab=tab-expanded&json-ld=%7B%22%40context%22%3A%7B%22mad%22%3A%22https%3A%2F%2Fmadices.github.io%22%2C%22samples%22%3A%22mad%3AhasInput%22%2C%22name%22%3A%22mad%3Aname%22%2C%22lab_id%22%3A%22%40id%22%2C%22ncit%22%3A%7B%22%40id%22%3A%22http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fncit%2F%22%7D%2C%22biochemical_reaction%22%3A%7B%22%40id%22%3A%22ncit%3Abiochemical_reaction%22%2C%22%40type%22%3A%22%40id%22%7D%7D%2C%22m_def%22%3A%22..%2Fupload%2Fraw%2Fmeasurement_entry_schema.archive.yaml%23%2Fdefinitions%2Fsection_definitions%2F0%22%2C%22name%22%3A%22MnWO4_C3performance%22%2C%22datetime%22%3A%222024-04-16T11%3A38%3A18.967172%2B00%3A00%22%2C%22location%22%3A%22FHI%2FAC%20department%22%2C%22biochemical_reaction%22%3A%22ncit%3Aoxidation%22%2C%22reaction_name%22%3A%22Oxidation%20of%20Propane%22%2C%22experimenter%22%3A%22P.%20Kube%22%2C%22data_file%22%3A%22C3_performance_MnWO4.xlsx%22%2C%22samples%22%3A%5B%7B%22m_def%22%3A%22nomad.datamodel.metainfo.basesections.CompositeSystemReference%22%2C%22name%22%3A%2232024%22%2C%22reference%22%3A%22..%2Fuploads%2FYdPbMF9bT0621URM2mboUg%2Farchive%2Fpw50pT2_2tTTJMbiDphASGqcSAuX%23%2Fdata%22%2C%22lab_id%22%3A%2232024%22%7D%5D%2C%22reactor_filling%22%3A%7B%22catalyst_mass%22%3A231.1%7D%2C%22reaction_results%22%3A%7B%22reactants_conversions%22%3A%5B%7B%22name%22%3A%22oxygen%22%7D%2C%7B%22name%22%3A%22propane%22%7D%5D%7D%7D

[simontaurus]

Framing example

link

[simontaurus]

Schema Editor and Registry
Demo https://osl-sandbox.big-map.eu/wiki/Main_Page
based on https://github.com/OpenSemanticLab

[ml-evs]

OPTIMADE unit system that may be helpful to you at some point: https://schemas.optimade.org/defs/v1.2/unitsystems/optimade/optimade.html

[PeterKraus]

Discussion on annotating xarray dumps.
geis.json

[simontaurus]

JSON-LD playground

link

[samikanza]

Playground Links

• SPK Playground - SPK

[samikanza]

Latest Link

[hampusnasstrom]

I now added the demo of the ontopint to the semantics repo: github.com/MADICES/semantic-annotation/blob/main/ontopint_demo.ipynb. @PeterKraus, @ml-evs, and @edan-bainglass, please let me know if you want me to post it somewhere else as well. I'll add some more info to the package readme in the coming days.

[dalito]

Hi @hampusnasstrom . On pypi the current version of ontopint is 0.0.5. Was that used? You could add a note about the required (minimum) version of ontopint to the notebook.

[hampusnasstrom]

@dalito yes, the 0.0.5 includes the ucumvert dependency. I'll add a note :)