feature: AI-Enabled Food Search #2325

taylorpatterson-T1D · 2025-07-05T22:31:32Z

Initial commit

taylorpatterson-T1D · 2025-07-05T23:44:39Z

Could I please get a review from the maintainers? @ps2, @marionbarker, @loudnate

This is my submission for a new AI-enabled food search capability for Loop. Loop has helped me become a better diabetic and I would love to give back to the community by contributing this new feature for Loop, which I believe will benefit all Loop users.

PROBLEM WE'RE SOLVING: Diabetics lack consistent nutrition knowledge when estimating carb intake. And we need that knowledge right in the moment, when we’re making dosing decisions - this knowledge is the single most important tool we have to improve our Time in Range and A1C.

NEW FEATURE'S IMPACT: The Food Search system is a comprehensive food analysis and nutrition tracking solution integrated into Loop for improved diabetes management. It provides multiple search options including text, barcode scanning, voice, and AI-powered image analysis that produces comprehensive nutrition data and dosing suggestions via diabetic notes. This let's us enter a meal bolus with MUCH greater accuracy and improve Time in Range and A1C. The AI-enabled analysis engine can even read menu items from an image captured, and provide guidance in multiple languages!!!

Video Demo: https://youtu.be/L0LD8AxNX0Q

Review the docs in “//LoopWorkspace/Loop/Documentation/FoodSearch Docs” for details. API setup guides are provided so the user can enter their your own AI API keys for AI image analysis.

New Search for Food section on the Add Carb Entry view:

Text search:

Barcode scan search for high accuracy:

AI-enabled food image analysis engine in action:

AI-Analysis results with comprehensive analysis details including an item by item breakdown of the meal:

Even more detail about how Portions and Servings were calculated and diabetic specific notes to advise during dosing:

New Setup screen. Pick the AI service you prefer, enter your own API keys:

☐ Added network quality detection ☐ Implement safer timeout handling ☐ Add fast mode fallback logic 1. On Good WiFi: Fast parallel processing (unchanged performance) 2. On Restaurant WiFi: Sequential processing prevents freezing 3. On Cellular: Conservative mode with longer timeouts 4. On Poor Networks: Automatic fallback to safer processing

Improved AI prompt to include glycemic index analysis and strengthen diabetic notes section. Removed excessive DEBUG logging for better overall performance.

marionbarker

This is the first of several comments. It includes 2 specific lines that need to be removed from Loop.xcodeproj/project.pbxproj. The 2 lines contain Taylor's Developer ID. They break the automatic signing configuration for Loop and are not required - they can be deleted.

In addition to those 2 lines in Loop.xcodeproj/project.pbxproj, I do not know why the LibreTransmitter information was touched. I think that is a mistake. (It might not harm anything, but we prefer not to make changes unless they are necessary.)

Loop.xcodeproj/project.pbxproj

marionbarker · 2025-07-08T04:27:28Z

Loop.xcodeproj/project.pbxproj

@@ -3,7 +3,7 @@
 	archiveVersion = 1;
 	classes = {
 	};
-	objectVersion = 54;
+	objectVersion = 70;


Does the objectVersion need to be raised? (This means Xcode 16.0 or later, whereas objectVersion of 54 is Xcode 12 or later.)

I don't know the answer to the question, but I did restore it to 54 in my local clone and it built and seemed to run the same.

reverted to 54

I did some research and testing on this. Because I'm using newer xCode it automatically updates this. Older xCode users will be unable to use my repo unless they manually make the change you did first, which appears safe to do since my updates do not leverage an 70-specific capabilities. My options are:

Always develop in the oldest xcode version supported

Clarify instructions stating one must manually update prior to build

Include manual adjustment in the build script we might create when attempting to add this to the Customizations build script

Provide instructions to update to latest version of xCode before building. (which LoopDocs already does).

When testing, it did NOT add back my Dev Team details. That is happening when I build on my machine for some reason. Still tracking down why and will fix that if I can.

marionbarker · 2025-07-08T05:24:21Z

More general comments:

This was pretty slick. The bar code reader was the one I played with the most. The general food search seemed to spend a long time loading, but maybe my network was having an issue.

I will do more testing and report back.

Regarding this addition, I'm not sure of the best way to incorporate it. I'm sure it is user specific whether they want this level of detail or if they go with a few Favorite Food listings and use a SWAG for the carbs to enter (which is my approach). We might want to move the position of the food search/service lower on the screen into its own FOOD SEARCH area (like FAVORITE FOODS).

You might want to upload the images in your comment as png files - they are distorted and hard to read on my phone and distorted and impossible to read on my computer browser.

The following graphic was made as a composite from my test phone (an SE 2nd gen) - the display is taller than will fit on the phone with the keyboard deployed.

taylorpatterson-T1D · 2025-07-09T18:52:23Z

@marionbarker - Thank you for your kind help and suggestion. I have repaired the Taylor-specific issues and reverted the version uptick.

If we make this functionality a new Customization option from the main Loop build script then the user is saying that they want to prioritize the value delivered by making food search results their new priority (versus anything they may have used in the past). So keeping the functionality at the top of the Add Carb Entry view seems appropriate.

If they don't like the functionality then it can be removed returning Add Carb Entry to its default.

If they make the choice to add Food Search, then my POV is that it should remain at the top of the Add Carb Entry section and not be moved below (near Favorite Foods) where it may go overlooked and therefore unused.

Your thoughts?

Also reverted object version to 54

…o dev

marionbarker

I did another review and there are still some changes that need to be removed.

Please note: this only builds with Mac-Xcode. I am unable to build with GitHub Actions (at this time). I tested both the version in PR 2325, commit feb6f26, and the version with the items removed as described below.

For GitHub errors, please see: https://github.com/docs-test/LoopWorkspace/actions/runs/16239378975

Modification Summary

I removed these items and tested that Loop with food search still builds correctly with Mac (macOS 15.5) and Xcode (16.3):

I removed all references to LibreTransmitter from Loop.xcodeproj/project.pbxproj
I removed Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
I removed the entire folder Loop/Loop.xcodeproj (the Loop.xcodeproj folder should only be at the top level, this was duplicated at a lower level)

I pushed my version to the loopandlearn github account so I could link some of the changes in the details below and to support testing the build with GitHub actions.

Modification Details

Before making these changes

Be sure to clean build folder and close the workspace in Xcode
Only open Xcode workspace again after changes are made

Item 1:

see this commit: loopandlearn@f2874d2

Item 2:

command line to take this action:

% git rm Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
rm 'Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved'

see this commit: loopandlearn@aa67ab8

Item 3:

command line to take this action:

% git rm -r Loop/Loop.xcodeproj 
rm 'Loop/Loop.xcodeproj/project.pbxproj'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/contents.xcworkspacedata'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/DoseMathTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Intent Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Status Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/LoopTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/SmallStatusWidgetExtension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/WatchApp.xcscheme'

see this commit: loopandlearn@b40a220

marionbarker · 2025-07-12T18:38:07Z

Regarding not being able to build with GitHub actions. The modification in this link fixed the error.

loopandlearn@b40cb3c

Required change:

add missing arguments as indicated in the error message

Other change:

be consistent using imageUrl instead of imageURL in all arguments and labels
- the imageURL version was found in some places with imageUrl in most places

Error reported using Xcode: Product->Archive was:

/Users/marion/Downloads/ManualClones/Loop_dev_updates/LoopWorkspace/
 Loop/Loop/Services/FoodSearchRouter.swift:
 240:58: error: missing arguments for parameters
 'servingQuantity', 'imageFrontUrl', 'code' in call

            let placeholderProduct = OpenFoodFactsProduct(

Initial Commit

1dff182

taylorpatterson-T1D mentioned this pull request Jul 6, 2025

Integrate OpenFoodFacts food search API into Loop's "Add Carb Entry" screen (+barcode scan +voice dictation) #1939

Closed

taylorpatterson-T1D changed the title ~~AI-enabled food search~~ AI-Enabled Food Search Jul 6, 2025

taylorpatterson-T1D added 2 commits July 6, 2025 14:28

AI Prompt Improvements

9287c97

Improved AI prompt to include glycemic index analysis and strengthen diabetic notes section. Removed excessive DEBUG logging for better overall performance.

taylorpatterson-T1D changed the title ~~AI-Enabled Food Search~~ feature: AI-Enabled Food Search Jul 7, 2025

marionbarker requested changes Jul 8, 2025

View reviewed changes

taylorpatterson-T1D added 3 commits July 9, 2025 11:58

Removed developer-specific team refs in Loop.xcodeproj/project.pbxproj

e78c3a1

Also reverted object version to 54

Merge branch 'dev' of https://github.com/taylorpatterson-T1D/Loop int…

82348a6

…o dev

Repaired object version (again)

feb6f26

marionbarker requested changes Jul 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feature: AI-Enabled Food Search #2325

feature: AI-Enabled Food Search #2325

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading

Uh oh!

marionbarker left a comment

Uh oh!

Uh oh!

Uh oh!

marionbarker Jul 8, 2025

Uh oh!

taylorpatterson-T1D Jul 9, 2025

Uh oh!

taylorpatterson-T1D Jul 10, 2025

Uh oh!

marionbarker commented Jul 8, 2025

Uh oh!

taylorpatterson-T1D commented Jul 9, 2025

Uh oh!

marionbarker left a comment

Uh oh!

marionbarker commented Jul 12, 2025

Uh oh!

Uh oh!

feature: AI-Enabled Food Search #2325

Are you sure you want to change the base?

feature: AI-Enabled Food Search #2325

Uh oh!

Conversation

taylorpatterson-T1D commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marionbarker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

marionbarker Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

taylorpatterson-T1D Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

taylorpatterson-T1D Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

marionbarker commented Jul 8, 2025

Uh oh!

taylorpatterson-T1D commented Jul 9, 2025

Uh oh!

marionbarker left a comment

Choose a reason for hiding this comment

Modification Summary

Modification Details

Item 1:

Item 2:

Item 3:

Uh oh!

marionbarker commented Jul 12, 2025

Uh oh!

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading