screen-grep

A privacy-oriented, OS-agnostic, open-source alternative to Windows Recall, crafted for simplicity, and centered on local data storage and model inference.

This solution consists of the following components:

• screenshot: script that captures screenshots of your active window on a regular interval
• caption: run image-to-text and OCR models locally to create text captions from the screenshots
• elasticsearch: text search engine
• search: webapp to find content from previous screenshots

How to run

Run the screenshot capturing script:

./src/screenshot/screenshot.sh 10  # Screenshot every 10 seconds

Check the "Screenshot" section for support on different OSes such as Windows or macOS.

This scrip should start collecting screenshots from your active window under data/screenshots.

TIP: you can stop this script to stop capturing new screenshots while still running the search service.

Next, run the search app:

docker compose up -d

You can search for previous screenshots in http://localhost:5000/

Note: by default, docker compose will look for screenshots under ./data/screenshots/, if your screenshot script is running in a separate location, define the SCREENSHOTS environment variable before running compose:

export SCREENSHOTS=/path/to/data/screenshots

Minimum Requirements

The default image-to-text model, microsoft/Florence-2-large, requires at least 16GB of RAM for inference. If you have less available memory, consider using a smaller variant, such as microsoft/Florence-2-base, which only needs 8GB. To use this variant, replace this line with microsoft/Florence-2-base.

Local development setup

Run the screenshot capturing script:

./src/screenshot/screenshot.sh

Run an elasticsearch container in docker to store processed captions:

docker run --rm -it -e discovery.type=single-node -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:7.10.1

Run the service that generates captions from the screenshots and stores them in the search index:

pip install -r src/caption/requirements.txt
export HF_HOME=data/models  # Optionally store huggingface cache in current dir
python src/caption/main.py

Start the webapp that searches for previous screenshots:

pip install -r src/search/requirements.txt
python src/search/app.py

Screenshot

Support for this component throughout different OSes is still a work in progress.

Linux

The screenshot.sh script supports a variety of different environments:

• For KDE Plasma, the script utilizes the spectacle app CLI, so please ensure it is installed.
• In other X11 window sessions, you need to have imagemagick and xdottool installed on your system.
• For Wayland support, the script uses grim, swaymsg, and jq.

Windows

To capture screenshots in Windows, use this PowerShell script. In order to run it, you should change the default PowerShell's execution policy, which restricts the running of scripts for security reasons:

1. Search for "PowerShell" in the Start menu.
2. Right-click on "Windows PowerShell" and select "Run as administrator."
3. Run the following command to set the execution policy to allow running scripts:

   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned

Once the policy is set, right-click on the PowerShell script and select "Run with PowerShell".

macOS

By default, the screenshot.sh script takes a screenshot of the entire screen on macOS using screencapture -x. To take a screenshot of only the active window, create a special shortcut with the Shortcuts app. Follow these steps:

1. Open the Shortcuts app and create a new shortcut
2. Use the "Find Windows" action, set the following options to "Find All Windows":
   • Sort by: Window Index
   • Order: Smallest First
   • Limit: Checked
   • Windows: 1
3. Use the "Save File" action, set the following options to "Save Windows to Documents":
   • Ask Where To Save: Unchecked (then select the Documents directory instead of Shortcuts)
   • Subpath: ./active_window.png
   • Overwrite If File Exists: Checked
4. Name your shortcut: "Screenshot active window"

Your custom Shortcut should look like this:

When a shortcut named "Screenshot active window" is present, the script will use it instead of screencapture. The first time you run it, it will request user authorization. Select "Always Allow":

Roadmap of features

• Regex to isolate hyperlinks from screenshots
• Handle similar screenshots (find and remove duplicates)
• Improve search algorithm (use embeddings combined with fuzzy search)
• Option to remove specific screenshots
• Option to remove history
• Expand Linux screenshot support
• Windows screenshot support
• macOS screenshot support
• Simplify and move the screenshot script to docker
• Add LLM chat support (eg: be able to ask "how much time have I been working on my IDE today?")