Updating paper

Author: Erotemic

README.rst

@@ -92,6 +92,18 @@ illustrates an example of one of these "triples".
 The name "ShitSpotter" is an homage to my earlier work: `HotSpotter <https://github.com/Erotemic/hotspotter>`_, which later became `IBEIS <https://github.com/Erotemic/ibeis>`_ This is work on individual animal identification, particularly Zebras. After 2017, this work was continued in Wildbook by `WildMe <https://www.wildme.org/>`_, which merged with `Conservation X Labs <https://www.conservationxlabs.com/>`_ in 2024.
 
 
+Experiments
+===========
+
+We are working on polishing docker images to reproduce our existing experiments
+and serve as a foundation for new experiments.
+
+Initial images are available on dockerhub:
+https://hub.docker.com/repository/docker/erotemic/shitspotter/general
+
+Documentation "coming soon"™
+
+
 Downloading the Data
 ====================
 
@@ -153,6 +165,7 @@ Recent Updates
 Check back for updates, but because this is a personal project, it might take
 some time for it to fully drop.
 
+* 2025-08-02 - Paper is under peer review with slightly positive reviews, grounding dino and YOLO models now trainable. Small test dataset is main limitation - working to rectify with roboflow data (note: a subset of our dataset is on there, but it has the incorrect license). Initial docker image to reproduce experiments is published.
 * 2025-07-04 - Releasing new data on IPFS. The growth seems to be increasing. Will take 7-9 more years to get 30k images.
 * 2025-04-20 - The number of images is now over 9000! The dataset is now `mirrored on hugging face <https://huggingface.co/datasets/erotemic/shitspotter>`__.
 * 2025-03-09 - Bunch of new images, with somewhat of a domain shift. The detectron model is good at annotating new images, but still not good enough. More work to be done.

dockerfiles/shitspotter.dockerfile

@@ -1,26 +1,17 @@
 # syntax=docker/dockerfile:1.5
 FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
 
-ENV PIP_ROOT_USER_ACTION=ignore
-
-# Control which python version we are using
-ARG PYTHON_VERSION=3.13
-
-# Control the version of uv
-ARG UV_VERSION=0.7.19
 
 # ------------------------------------
 # Step 1: Install System Prerequisites
 # ------------------------------------
-RUN <<EOF
+
+RUN --mount=type=cache,target=/var/cache/apt \
+    --mount=type=cache,target=/var/lib/apt/lists <<EOF
 #!/bin/bash
 set -e
 apt update -q
 DEBIAN_FRONTEND=noninteractive apt install -q -y --no-install-recommends \
-    bzip2 \
-    rsync \
-    tmux \
-    fd-find jq htop tree \
     curl \
     wget \
     git \
@@ -32,23 +23,29 @@ apt clean
 rm -rf /var/lib/apt/lists/*
 EOF
 
-# Set the shell to bash to auto-activate enviornments
+# Set the shell to bash to auto-activate environments
 SHELL ["/bin/bash", "-l", "-c"]
 
-# ------------------
+
 # Step 2: Install uv
 # ------------------
 # Here we take a few extra steps to pin to a verified version of the uv
 # installer. This increases reproducibility and security against the main
 # astral domain, but not against those linked in the main installer.
 # The "normal" way to install the latest uv is:
 # curl -LsSf https://astral.sh/uv/install.sh | bash
-RUN <<EOF
+
+# Control the version of uv
+ARG UV_VERSION=0.8.4
+
+RUN --mount=type=cache,target=/root/.cache <<EOF
 #!/bin/bash
 set -e
 mkdir /bootstrap
 cd /bootstrap
+# For new releases see: https://github.com/astral-sh/uv/releases
 declare -A UV_INSTALL_KNOWN_HASHES=(
+    ["0.8.4"]="601321180a10e0187c99d8a15baa5ccc11b03494c2ca1152fc06f5afeba0a460"
     ["0.7.20"]="3b7ca115ec2269966c22201b3a82a47227473bef2fe7066c62ea29603234f921"
     ["0.7.19"]="e636668977200d1733263a99d5ea66f39d4b463e324bb655522c8782d85a8861"
 )
@@ -59,11 +56,14 @@ if [[ -z "$EXPECTED_SHA256" ]]; then
     exit 1
 fi
 curl -LsSf https://astral.sh/uv/$UV_VERSION/install.sh > $DOWNLOAD_PATH
-echo "$EXPECTED_SHA256  $DOWNLOAD_PATH" | sha256sum --check
+report_bad_checksum(){
+    echo "Got unexpected checksum"
+    sha256sum "$DOWNLOAD_PATH"
+    exit 1
+}
+echo "$EXPECTED_SHA256  $DOWNLOAD_PATH" | sha256sum --check || report_bad_checksum
 # Run the install script
 bash /bootstrap/uv-install-v${UV_VERSION}.sh
-# Cleanup for smaller images
-rm -rf /root/.cache/
 EOF
 
 
@@ -73,7 +73,13 @@ EOF
 # This step mirrors a normal virtualenv development environment inside the
 # container, which can prevent subtle issues due when running as root inside
 # containers. 
-RUN <<EOF
+
+# Control which python version we are using
+ARG PYTHON_VERSION=3.10
+
+ENV PIP_ROOT_USER_ACTION=ignore
+
+RUN --mount=type=cache,target=/root/.cache <<EOF
 #!/bin/bash
 export PATH="$HOME/.local/bin:$PATH"
 # Use uv to install the requested python version and seed the venv
@@ -83,27 +89,69 @@ BASHRC_CONTENTS='
 export HOME="/root"
 export PATH="$HOME/.local/bin:$PATH"
 # Auto-activate the venv on login
-source /root/venv'$PYTHON_VERSION'/bin/activate
+source $HOME/venv'$PYTHON_VERSION'/bin/activate
 '
 # It is important to add the content to both so 
-# subsequent run commands use the the context we setup here.
+# subsequent run commands use the context we setup here.
 echo "$BASHRC_CONTENTS" >> $HOME/.bashrc
 echo "$BASHRC_CONTENTS" >> $HOME/.profile
+echo "$BASHRC_CONTENTS" >> $HOME/.bash_profile
 EOF
 
 
-RUN mkdir -p /root/code/shitspotter
+# -----------------------------------
+# Step 4: Ensure venv auto-activation
+# -----------------------------------
+# This step creates an entrypoint script that ensures any command passed to
+# `docker run` is executed inside a login shell where the virtual environment
+# is auto-activated. It handles complex cases like multi-arg commands and
+# ensures quoting is preserved accurately.
+RUN <<EOF
+#!/bin/bash
+set -e
+
+# We use a quoted heredoc to write the entrypoint script literally, with no variable expansion.
+cat <<'__EOSCRIPT__' > /entrypoint.sh
+#!/bin/bash
+set -e
+
+# Reconstruct the full command line safely, quoting each argument
+args=()
+for arg in "$@"; do
+  args+=("$(printf "%q" "$arg")")
+done
+
+# Join arguments into a command string that can be executed by bash -c
+# This preserves exact argument semantics (including quotes, spaces, etc.)
+cmd="${args[*]}"
+
+# Execute the reconstructed command inside a login shell
+# This ensures virtualenv activation via .bash_profile
+exec bash -l -c "$cmd"
+__EOSCRIPT__
+
+# Print the script at build time for visibility/debugging
+cat /entrypoint.sh
+
+chmod +x /entrypoint.sh
+EOF
+
+# Set the entrypoint to our script that activates the virtual environment first
+ENTRYPOINT ["/entrypoint.sh"]
 
-# Control the version of REPO (by default uses the current branch)
-ARG REPO_GIT_HASH=HEAD
 
 # ---------------------------------
-# Step 4: Checkout and install REPO
+# Step 5: Checkout and install REPO
 # ---------------------------------
 # Based on the state of the repo this copies the host .git data over and then
 # checks out the extact version of REPO requested by REPO_GIT_HASH. It then
 # performs a basic install of shitspotter into the virtual environment.
 
+RUN mkdir -p /root/code/shitspotter
+
+# Control the version of REPO (by default uses the current branch)
+ARG REPO_GIT_HASH=HEAD
+
 # NOTE: our .dockerignore file prevents us from copying in populated secrets /
 # credentials
 COPY .git /root/code/shitspotter/.git
@@ -147,35 +195,6 @@ cd  /root/code/YOLO-v9
 uv pip install -e .
 EOF
 
-
-
-# -----------------------------------
-# Step 5: Ensure venv auto-activation
-# -----------------------------------
-# This final steps ensures that commands the user provides to docker run
-# will always run in in the context of the virtual environment. 
-RUN  <<EOF
-#!/bin/bash
-set -e
-# write the entrypoint script
-echo '#!/bin/bash
-set -e
-# Build the escaped command string
-cmd=""
-for arg in "$@"; do
-  # Use printf %q to properly escape each argument for bash
-  cmd+=$(printf "%q " "$arg")
-done
-# Remove trailing space
-cmd=${cmd% }
-exec bash -lc "$cmd"
-' > entrypoint.sh
-chmod +x /entrypoint.sh
-EOF
-
-# Set the entrypoint to our script that activates the virtual enviornment first
-ENTRYPOINT ["/entrypoint.sh"]
-
 # Set the default workdir to the shitspotter code repo
 WORKDIR /root/code/shitspotter
 
@@ -200,18 +219,27 @@ REPO_GIT_HASH=$(git rev-parse --short=12 HEAD)
 
 python ./dockerfiles/setup_staging.py
 
-# Build REPO in a reproducible way.
+# Determine version of repo, uv, and python to use
+export REPO_GIT_HASH=$(git rev-parse --short=12 HEAD)
+export UV_VERSION=0.8.4
+export PYTHON_VERSION=3.11
+
+# Build the image with version-specific tags
 DOCKER_BUILDKIT=1 docker build --progress=plain \
-    -t shitspotter:$REPO_GIT_HASH-uv0.7.29-python3.11 \
-    --build-arg PYTHON_VERSION=3.11 \
-    --build-arg UV_VERSION=0.7.19 \
+    -t shitspotter:${REPO_GIT_HASH}-uv${UV_VERSION}-python${PYTHON_VERSION} \
+    --build-arg PYTHON_VERSION=$PYTHON_VERSION \
+    --build-arg UV_VERSION=$UV_VERSION \
     --build-arg REPO_GIT_HASH=$REPO_GIT_HASH \
     -f ./dockerfiles/shitspotter.dockerfile .
 
-
-# Add latest tags for convinience
-docker tag shitspotter:$REPO_GIT_HASH-uv0.7.29-python3.11 shitspotter:latest-uv0.7.29-python3.11
-docker tag shitspotter:$REPO_GIT_HASH-uv0.7.29-python3.11 shitspotter:latest
+# Add concise tags for easier reuse
+export IMAGE_QUALNAME=shitspotter:${REPO_GIT_HASH}-uv${UV_VERSION}-python${PYTHON_VERSION}
+export NAME1=shitspotter:latest-uv${UV_VERSION}-python${PYTHON_VERSION}
+export NAME2=shitspotter:latest-python${PYTHON_VERSION}
+export NAME3=shitspotter:latest
+docker tag $IMAGE_QUALNAME $NAME1
+docker tag $IMAGE_QUALNAME $NAME2
+docker tag $IMAGE_QUALNAME $NAME3
 
 # Verify that GPUs are visible and that each shitspotter command works
 docker run --gpus=all -it shitspotter:latest nvidia-smi
@@ -220,6 +248,31 @@ docker run --gpus=all -it shitspotter:latest nvidia-smi
 # (TODO: show how to replicate experiments)
 docker run --gpus=all -it shitspotter:latest bash
 
+# 1) Authenticate (recommended: use a Docker Hub access token)
+#    Create a token in Docker Hub -> Account Settings -> Security
+#    Then run:
+# echo "<your-access-token>" | docker login --username "$DOCKERHUB_USER" --password-stdin
+#
+# If you must, you can use interactive login:
+docker login
+
+export DH_USER="erotemic"
+
+# 3) Create remote-qualified tags
+docker tag $IMAGE_QUALNAME $DH_USER/$IMAGE_QUALNAME
+docker tag $NAME1  $DH_USER/$NAME1
+docker tag $NAME2  $DH_USER/$NAME2
+docker tag $NAME3  $DH_USER/$NAME3
+
+# 4) Push the tags
+docker push $DH_USER/$IMAGE_QUALNAME
+docker push $DH_USER/$NAME1
+docker push $DH_USER/$NAME2
+docker push $DH_USER/$NAME3
+docker push $DH_USER:latest-uv0.7.29-python3.11
+docker push $DH_USER:latest
+
+
 ' > /dev/null
 
 EOF

experiments/grounding-dino-experiments/run_grounding_dino_experiments_v1.sh

@@ -1,6 +1,6 @@
 __doc__='
 SeeAlso, for tuned version of grounding dino:
-    ~/code/shitspotter/dev/poc/tune_grounding_dino.sh
+    ~/code/shitspotter/experiments/grounding-dino-experiments/tune_grounding_dino.sh
 '
 
 #