Merge pull request '2.1.0' (#211) from develop into master

Author: strangetom

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ repos:
     rev: v5.0.0
     hooks:
       - id: check-added-large-files
+        args: ['--maxkb=5000']
         stages: [pre-commit]
       - id: check-ast
         stages: [pre-commit]
@@ -15,7 +16,7 @@ repos:
       - id: debug-statements
         stages: [pre-commit]
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.6
+    rev: v0.11.6
     hooks:
       - id: ruff
         args: ["--fix"]

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 2.10
+
+> [!WARNING]
+>
+> This version replaces the floret dependency with numpy.
+> 
+> Numpy was already a dependency of floret, so if you are upgrading from v2.0.0 there should be little impact.
+
+* This release overhauls the foundation foods functionality so that ingredient names are matched to entries in the [FoodData Central](https://fdc.nal.usda.gov/) (FDC) database.
+
+  * This update does not change the API. It adds additional fields to `FoundationFood` objects for FDC ID, category and data type. The `text` field now returns the description for the matching FDC entry.
+
+  * Beware that enabling this functionality causes the `parse_ingredient` function to be much slower than when disabled (default).
+
+    |                      | foundation_foods=False (default) | foundation_foods=True |
+    | -------------------- | -------------------------------- | --------------------- |
+    | Sentences per second | ~1500                            | ~20                   |
+
+  * This functionality works entirely offline.
+  
+  * See the [foundation foods](https://ingredient-parser.readthedocs.io/en/latest/explanation/foundation.html) page of the docs for specifics.
+
 ## 2.0.0
 
 > [!Caution]

MANIFEST.in

@@ -1,8 +1,7 @@
 include ingredient_parser/density_context.txt
 include ingredient_parser/en/model.en.crfsuite
 include ingredient_parser/en/ModelCard.en.md
-include ingredient_parser/en/ff_model.en.crfsuite
-include ingredient_parser/en/FF_ModelCard.en.md
-include ingredient_parser/en/embeddings.floret.bin
+include ingredient_parser/en/ingredient_embeddings.25d.glove.txt.gz
+include ingredient_parser/en/fdc_ingredients.csv.gz
 global-exclude test*
 prune */__pycache__

README.md

@@ -42,19 +42,19 @@ Refer to the documentation [here](https://ingredient-parser.readthedocs.io/en/la
 
 ## Model
 
-The core of the library is a sequence labelling model that is used to label each token in the sentence with the part of the sentence it belongs to. A data set of 75,000 example sentences is used to train and evaluate the model. See the [Model Guide](https://ingredient-parser.readthedocs.io/en/latest/guide/index.html) in the documentation for mode details.
+The core of the library is a sequence labelling model that is used to label each token in the sentence with the part of the sentence it belongs to. A data set of 81,000 example sentences is used to train and evaluate the model. See the [Model Guide](https://ingredient-parser.readthedocs.io/en/latest/guide/index.html) in the documentation for mode details.
 
 The model has the following accuracy on a test data set of 20% of the total data used:
 
 ```
 Sentence-level results:
-	Accuracy: 94.72%
+	Accuracy: 94.66%
 
 Word-level results:
 	Accuracy 97.82%
-	Precision (micro) 97.80%
+	Precision (micro) 97.81%
 	Recall (micro) 97.82%
-	F1 score (micro) 97.80%
+	F1 score (micro) 97.81%
 ```
 
 ## Development

benchmark.py

@@ -53,4 +53,4 @@
     duration = time.time() - start
     print(f"Elapsed time: {duration:.2f} s")
     print(f"{1e6 * duration / total_sentences:.2f} us/sentence")
-    print(f"{int(total_sentences / duration)} sentences/second")
+    print(f"{total_sentences / duration:.2f} sentences/second")