Extract data from proof images using AI

[raphodn]

There is lots of interesting info in the proof images :

• price tag : price, barcode
• receipt : multiple prices, no barcodes nor EAN but a "product name", date, location
• exif data : date & location ?

What if the server was able to do some image recognition, and return to the frontend all of that info ? 🤩

How ?

Using GenAI

• extract data from price tags (using Google's Gemini) (feat(proofs): New endpoint to extract price data from images (with Gemini) #557)

@TTalex has built a prototype, using Google's Gemini API.

Using an external library

• Proof image / Price tags: barcode scanner library to extract the EAN #670

Our own trained model

• Proof classification : integrated into Robotoff (not yet into OP) (Proof : auto classification on user upload #394)
• Google Cloud vision : we call the API on each new image : useful for moderation (e.g. detect inappropriate images) (Proof: send to Google Cloud Vision, and store the matching JSON output #320)
• Extract price tags (bounding boxes) from Proof images : Add a price_tags table to store predictions #611
• extract data from price tags

@raphael0202 is working on building a long-term solution based on training data.

Challenges

Price tag

Receipt

Without the product EAN or barcode, how to link to the exact product in the OFF database ?

Related discussions

• Leverage Hunger Games to crowdsource price addition (from price tag images) #523
• Speed up proof & price addition #526 (comment)

Related issues

• Proof: send to Google Cloud Vision, and store the matching JSON output #320
• Proof : auto classification on user upload #394
• Use hunger games to take advantage of historical price data present in OFF product's picture  #308
• Create a proof_prediction table to store predictions from ML models #511
• Proof: use GenAI to extract price tag data #569
• Add a price_tags table to store predictions #611

[teolemon]

On this one, it's probably about building the endpoints

• EXIF to (timestamp + lists of candidate stores)
• Image to (list of candidates barcodes + candidate prices)

Then those endpoints can be reused on proof addition, or for retro-processing

[raphodn]

On Slack you mentionned a tool/workflow you used to extract barcode info for instance.

We could use this issue to list available tools ?

I've been thinking of ways to accelerate data collection, taking pictures of shelves, and having an image recognition tool that would return a list of product codes+prices.

Then would come a step to integrate the "chosen" tool in the backend ?

[teolemon]

• There are Tensorflow.JS based solutions
• https://blog.tensorflow.org/2022/06/ocr-in-browser-using-tensorflowjs.html
• https://demo-doctr-tensorflowjs.mindee.com/
• Tesseract.js https://tesseract.projectnaptha.com/
• We can offload processing to Robotoff, which has all the tooling in place
• We can do it in the Open Prices backend, it's not rocket-science (possibly with the Robotoff lib)

[teolemon]

For EXIF to stores

• https://github.com/ozgecinko/image-metadata-extractor
• https://codereview.stackexchange.com/questions/221809/get-nearest-major-city-of-an-image-using-exif-gps-data
• https://medium.com/geekculture/extract-gps-information-from-photos-using-python-79288c58ccd9
• Client side: https://github.com/exif-js/exif-js

[alexfauquette]

I gave a try few days ago with terseract, but surprisingly it has lots of false positive, and is not perfect on barcode (some digits are missing)

I need to try again with different resolution

[teolemon]

Or we could cloudvision our way out of it, it's not like we'll be out of credits this year.

[teolemon]

• We could just use Cloud Vision Proof: send to Google Cloud Vision, and store the matching JSON output #320

[raphodn]

Image examples

Price tag	Multiple price tags	Receipt