new ipfs data drop

Author: Erotemic

README.rst

@@ -86,7 +86,7 @@ illustrates an example of one of these "triples".
 .. image:: https://i.imgur.com/NnEC8XZ.jpg
 
 
-The name "ShitSpotter" is an homage to my earlier work: `HotSpotter <https://github.com/Erotemic/hotspotter>`_, which later became `IBEIS <https://github.com/Erotemic/ibeis>`_ This is work on individual animal identification, particularly Zebras. This work is continued by `WildMe <https://www.wildme.org/>`_.
+The name "ShitSpotter" is an homage to my earlier work: `HotSpotter <https://github.com/Erotemic/hotspotter>`_, which later became `IBEIS <https://github.com/Erotemic/ibeis>`_ This is work on individual animal identification, particularly Zebras. After 2017, this work was continued in Wildbook by `WildMe <https://www.wildme.org/>`_, which merged with `Conservation X Labs <https://www.conservationxlabs.com/>`_ in 2024.
 
 
 Downloading the Data
@@ -108,7 +108,7 @@ The following `IPNS <https://docs.ipfs.tech/concepts/ipns/>`_ address should alw
 `/ipns/k51qzi5uqu5dje1ees96dtsoslauh124drt5ajrtr85j12ae7fwsfhxb07shit <https://ipfs.io/ipns/k51qzi5uqu5dje1ees96dtsoslauh124drt5ajrtr85j12ae7fwsfhxb07shit>`_.
 
 This should resolve to the most recent `IPFS CID <https://docs.ipfs.tech/concepts/content-addressing/>`_ (Content Identifier) of:
-`/ipfs/bafybeihsd6rwjha4kbeluwdjzizxshrkcsynkwgjx7fipm5pual6eexax4 <https://ipfs.io/ipfs/QmQonrckXZq37ZHDoRGN4xVBkqedvJRgYyzp2aBC5Ujpyp?redirectURL=bafybeihsd6rwjha4kbeluwdjzizxshrkcsynkwgjx7fipm5pual6eexax4&autoadapt=0&requiresorigin=0&web3domain=0&immediatecontinue=1&magiclibraryconfirmation=0>`_.
+`/ipfs/bafybeidfxayyacq4jbbhjcxbxumqlspmsmkj52nq2ns4vfew5udgysocoy <https://ipfs.io/ipfs/QmQonrckXZq37ZHDoRGN4xVBkqedvJRgYyzp2aBC5Ujpyp?redirectURL=bafybeidfxayyacq4jbbhjcxbxumqlspmsmkj52nq2ns4vfew5udgysocoy&autoadapt=0&requiresorigin=0&web3domain=0&immediatecontinue=1&magiclibraryconfirmation=0>`_.
 
 This can be viewed in a webbrowser through an IPFS gateway:
 
@@ -150,6 +150,7 @@ Recent Updates
 Check back for updates, but because this is a personal project, it might take
 some time for it to fully drop.
 
+* 2025-07-04 - Releasing new data on IPFS. The growth seems to be increasing. Will take 7-9 more years to get 30k images.
 * 2025-04-20 - The number of images is now over 9000! The dataset is now `mirrored on hugging face <https://huggingface.co/datasets/erotemic/shitspotter>`__.
 * 2025-03-09 - Bunch of new images, with somewhat of a domain shift. The detectron model is good at annotating new images, but still not good enough. More work to be done.
 * 2024-12-31 - It is the end of 2024, lots has changed: new varied images, new privacy policy, and new contributions. Happy new year 🎊, all new 2025 images will go into the train set.
@@ -376,7 +377,8 @@ registration via the SIFT+RANSAC algorithm.
 +-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
 | 2025-04-20  | 9175     | ~3189               | 2316                  | 3198                  | bafybeia2uv3ea3aoz27ytiwbyudrjzblfuen47hm6tyfrjt6dgf6iadta4  |
 +-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
-
+| 2025-07-04  | 9790     | ~3394               | 2444                  | 3406                  | bafybeidfxayyacq4jbbhjcxbxumqlspmsmkj52nq2ns4vfew5udgysocoy  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
 
 
 
@@ -567,6 +569,7 @@ Current IPFS addresses for each top-level asset group are:
     bafybeihgnfs6hku3xlqa7fnoqujkmq2ezk7lvidw32dvgjhsmk7wpk72cy - shitspotter_dvc/assets/poop-2024-12-30-T212347
     bafybeidvaphwcib2qezdcey4cj3a2r7r7oxskl56yaccgdi75pdou4ggmm - shitspotter_dvc/assets/poop-2025-03-08-T224918
     bafybeieyl6yzi6cyz3minjyvmz53ydbpxmljxs5gytv6cwu6ci7tmwyvjq - shitspotter_dvc/assets/poop-2025-04-20-T172113
+    bafybeicz7kxvmxojmu33pskfrkglosz3tndgsmpzz6cmmcxynaau5xzfeu - shitspotter_dvc/assets/poop-2025-07-03-T230656
 
 
 

dev/journal.txt

@@ -278,3 +278,129 @@ be starting now.
 
 Also going to try forwarding 58334 to my box running deluge, and that seemed to
 work.
+
+
+# 2025-07-02
+
+Annotating a bunch of images taken since the 2024-04-20 release, have ~600
+images, so ~200 with poop.  Its interesting to see the MaskRCNN failure cases.
+It certainly gets it right a lot of the time, but there are still some (what
+seem to be easy) misses. There is definately an issue with leafs, helicopter
+seeds, and sticks. But now those will be false positive annotations. 
+
+Missing cases where the poop is in a strong light gradient. Occluded cases.
+Generally noisy images cases. Some cases, which are probably due to NMS issues,
+but not 100% sure.
+
+Pretty good at cases in beds of clover or in short grass.
+Rocks with strong shadows are causing false positives.
+
+I should probably retrain the model its at least 6 months out of date.
+
+What is my image capture rate? / Dataset growth rate? A: 170-215 images /
+month.
+
+Should make a wordcloud of negative annotations.
+
+Current tag / cat histograms are:
+
+cat_hist = {
+    'mulch': 1,
+    'yellowsnow': 1,
+    'acorn-cap': 1,
+    'cigarette': 1,
+    'wood': 1,
+    'tree-trunk': 1,
+    'plant-matter': 1,
+    'root': 1,
+    'acorn': 1,
+    'dogtoy': 1,
+    'nut': 2,
+    'treebark': 2,
+    'treeroot': 2,
+    'background': 2,
+    'residue': 2,
+    'pine-leafs': 2,
+    'pipe': 2,
+    'bottle-cap': 2,
+    'shadow': 2,
+    'woodpost': 2,
+    'wood-chip': 3,
+    'leafs': 4,
+    'dirt': 4,
+    'crushed-nut': 4,
+    'bark': 5,
+    'helicopter-seed': 5,
+    'grass-patch': 6,
+    'rock': 10,
+    'ignore': 11,
+    'stick': 13,
+    'helicopterseed': 13,
+    'unknown': 14,
+    'pine-cone': 16,
+    'leaf': 134,
+    'poop': 6817,
+}
+tag_hist = {
+    'from-roadie': 1,
+    'sqashed': 1,
+    'blury': 1,
+    'dark': 1,
+    'blurry': 1,
+    'big-pile': 1,
+    'stringy': 1,
+    'polygon-hole': 1,
+    'hard': 1,
+    'partial-occlusion': 1,
+    'wet': 1,
+    'digestedtoy': 1,
+    'frozen': 1,
+    'shiney': 1,
+    'maybe-sick': 1,
+    'confusor': 1,
+    'crumbling': 1,
+    'residue': 1,
+    'liquidy': 1,
+    'some crud in it, maybe a toy': 1,
+    'liquid': 1,
+    'unsure': 1,
+    'soft': 1,
+    'inside': 1,
+    'ambiguous': 1,
+    'hardcase': 1,
+    'hairy': 1,
+    'difficult': 2,
+    'smeared': 2,
+    'splatter': 2,
+    'occlusion': 2,
+    'gross': 3,
+    'decomposing': 3,
+    'residual': 3,
+    'minor-occlusion': 3,
+    'splat': 3,
+    'camo': 4,
+    'hidden': 5,
+    'camoflauged': 5,
+    'squashed': 6,
+    'messy': 7,
+    'sick': 9,
+    'diarrhea': 13,
+    'snowcovered': 13,
+    'occluded': 17,
+    'crumbly': 19,
+    'smashed': 19,
+    'fresh': 25,
+    'old': 105,
+}
+
+
+I need to think about this project's end state.
+
+When is the dataset big enought? I'm thinking I want about 10,000 images with
+good poop annotations as an arbitrary number. That will take about 7-10 more
+years at current growth rates.
+
+But maybe the end state is really:
+
+* There exists a good phone app that has a very high detection F1 to the point
+  of being useful.

experiments/geowatch-experiments/run_pixel_eval_on_test_pipeline.sh

@@ -177,7 +177,7 @@ python -m geowatch.mlops.schedule_evaluation \
     --root_dpath="$EVAL_PATH" \
     --devices="0,1," --tmux_workers=2 \
     --backend=tmux --skip_existing=1 \
-    --run=1
+    --run=0
 
 
 # Simple no-dependency result readout

secrets/privacy.1.data

@@ -1,24 +1,70 @@
-U2FsdGVkX1+6s+zCoTMGQcZO8WEhXDO3HwBuKJaGWgNB18MNSBbqOnoIjOr7UAjD
-jtKF2Hj/GzBM91Xi+DYPUFve+SShVk8QUopVMlj+BlOShC1As3VNGOzLtlLsgIWH
-EE7rcenTs9A13o7UoKbf6KBViajqqSUD6QmfgEXk5QZwsUgCXkrqN/9BPT7oyXd1
-aSidMNLlZx6XO+lwvdbgAejQT90VqgEo235FvUz32ITaptygqJiDDBOm1MJDPNeF
-thO+bgXvxDUNjRFG8ZE99dEdbAEJXKdJHKDJCLiOtZYQ6/6lDPWnEQYbXx0AGIzZ
-/NioUFaCxOFWPmraD9Q4T3o44nFrBkAiiwOOkROPrc686UIYx+8T+sZPzVTltQYt
-hVlDbVglZ7R4a1T4TmBYnyoOzS4wn8+eQT3lDPWk8SkoOuh17EjeETG8+HPTl4vr
-lD6wfnXWTsmmqzr3GfNzJqQ0UKiveVwBS+zVutdz8OlbQBO63waLQ+iBvON8xYfj
-x+h3s+hrVybwy+BWSoSqLBLbHcQSqeW6QV+dy6I8DpG9UJfRkZlYFzgL1dZ0Ns7e
-HJq9hE/dXDqm5wv93jpNox4Qd8J3uiBWxbpzNjPm8tXbnxX1CiKt5cun87DaGCEi
-mYmpnOUbOGuWBeHOV8INTDY9sqHnnMWymq+LGK2fzvfmCemPuBgczUdXvA7gakIK
-BbDztNAVyyXZx0WIBScEGD7zJjuT1JJMl9ceW4k1Kfbb0WdnIJtH/WvxNahOgE1R
-N1qHNaYpTV6rN76zFtcUEjgKRvxd9OshBeaL/HUf+rabhY1plmIKbSn8/geGoODX
-IuUNOC0gnWGtnF9H8WdK5yiEbh+1RD2DUQpB7YTiLrJ/JbLNGSqAriiOtJDpX6eK
-Gw0gLpCw+povZGol4EJpInZJOGNq7+6nLKqaOk2aibqlWiQum5hVCBnJzL4aTvFX
-c6axLDZRkLepGCMA/ycPdytm26TyOGO6wdDPVcoiliLSlZ2XG6bW0AnAu+j6wUqm
-5Mu7kJK4NIa9lLmScjb41QfWS8GXX73om1/wbkW9ebx9jZkLZgEN0cicJh1vi8JB
-K5yzqrjnRlxzRKos8+8id5ZhoeSaIfXHYaE+1ktV9/AF1Y1MVd4Y0AWp5b0RiUDe
-xtYx7fDFi3t3utI75wYQr0A7slw1NZY2xznG+B44KP9CV+HLGXartz1UM6N63Ljd
-I1N/VMu3iS+t6pLHI4Po+hjBcd+JgF2MlNvOXd7aSb7vyaZovYsLlxAlDI9MP8uR
-2UmD56K4AvjSfdIcYJJpEtNae+cF12FVNrzHG1w6IgJjIx0Ew0tyT5EJ6tYxGWly
-njsPBV2hcth4IpPJ5h0QN+5RaJu/gp5fFOHRM7K53d/iJ1ya5f1ir5BhPB6JVW1M
-KfQvWzkcvBHaadEVoy5RFIoKDK2fZYQxXiM157G+BxdppMdBqPEt3H/ICV2DSr+h
-De0LOn1lV5Kwy+YY2gMCW/h+l3bllWPVQJgRQ7v5elI=
+U2FsdGVkX1+O1XqSXMPEYSRjaGrubZbJC6NezgmEumDsdKQiErox/mHSFOjPQ6eW
+zFa4Eqx3NMRDTVPu1rlwXorkut36jWuuNiYKTo5JvmSCaF1wdvKs6VGhPH8UhdmU
+p/shZFPNeRKtnv/zhZV16/kWZxSFJJcu+iB43bYh4TK/RLcUC8QlKOzYdVxVF042
+jpEQvnN7f8RzQ9NVTUExhDOY9N8HDroyaPR1Df7MDX5j1/Xuhbt7Mb0typP9tSlA
+mLpoCsJTRZ+9zy+Se2YMzZJvHnoN/fLCtwkw7LYpY3wtLD9EiloFO22GfhHh660j
+nj5MIBnnWc4+ibNZ5z57ecXJx4Ik8Hqb5GnepR46n36yREGIin/g8RQDSqjxmQ+X
+P4rYM9GRvbfULm+VRwNBhHbw3cqbbwsEDaatC5kWGOxi+E135sbtyHZD/4fuGnzN
+D5wVakPX//bpITxeAXXOthoB7gcz1aVZzbNyfirMxQsHgNKfWa2OBLwDnJ18vs/e
+UVvbuCJcC+tlfaAB/c4S60YOeqqrjm1X3BDIIrX1h1674CdymKbMY4PUV8OqANtb
+onjRtNHuND2rAfm85wV1JtgSygIvbG2zL/kARCNjUS3p2PWUqvZwBjM9gveSvgLa
+VYIpjAPQ026JqPWJQ0DkGyuDdsrCRQP/Jbo2+34xxuLggiMXS+nl+li5LaUZ/ix4
+KeSF0Eum6I81WcABEmzctC0Q+fyI62pb3lJuYZJYiwQiS7mgzm2TQBnuiLeIdXPy
+A4YoXVuF5UU3AyZEYh+PFyA0GwmG/zBH3wUnVIX5G5YjQsNZUeJM6ivyrLBvC3Lf
+E9pl/xKP3/lFyFctzfoGubMltHP6H5PQWZmjaTnkIuq6IIyuKLfADa/w2/aLf7cg
+Xk14HgFbiJ6rqDW015vj1JLPT9d0B3fcwrD3TdHYnc2QMP18TMfSTzFmVxfeeca0
+TBVj09FPcvByiQ5I1JEIoaW8lAvF+BHJ2gH0uBYZqQsJL+/I4is8wz9eCIA+YRvE
+LVEvqZlGffcNQb+WhWdq0MgcAmv0Xi8hvEtMiRd0Crz0aqTjLBcvEohCvpvaqj3D
+eqtefd8ETZ031+YNDPxlXDP0vDbGrkrDgOTPRu9YbgqWbredmsNv7eQ/7+w+TnZA
+JjdwsmsPeeCOHzKV6sdE+hL6skv2uDfFm6K+UFugNeNe2Nhu/oqE/EDoNKfkHsO7
+RU8/jfxqZ0LLgtOILCgLGX6tIcmIS4g8o4BsaqlOHFS1nzsfRjJASdUYCLG5egAj
+k4y+R/YmfRkKLYpThQ/OMXaecuY8l15kUeRx6CERguSFgr2VhNmZGGlQ+TLSZf9X
+s6kRpy937e2lmPrj4vvXTaB0G6KNMInOOl6Bwoz4piqHlsP34PLilYA5sGDMOjKa
+mJhGy2nVoutWA8vnMottIWypmJlp6CJedx6sW3i5nlyv5Qppmnh/nKbNW6PC7zNg
+x2aBIwKugnmo5i4iPMODF0NIn9+yvvoIPDHlVpyZPNAGegyjfKOslrWnLokfEetm
+AMG3ON1Af4Jl3/Y+VRjC/1+oKPjdNSkATisIECtUhk2kbuw6dcHmuM7ZmWNYbdFa
+OWzN97+1pwxPKhI+CJCjpVyJ+bdkLPlak12ED9RFdY7lJ1enPV0bXFqx9V8JQOeR
+IMetcwS3h43sqcsIjF/TRa6b4Y+UNjh5pjsX8zsy0qWCs4qxXpkS6dPp3Dg22PYz
+0wobYiXRxhdn/Xtdck+J7185MQ/gB6DpkJatSPD+TgS/vsV1KPhaMNn8W4ceKhj+
+RMu17Jw2/ynf1SLLDVxDk5N9kmkdWB5yWluI3GneFH8fyWbv2ymv+5ELy5bHx0Lb
+dmAU3cZ+1QEBLcPCYAl18pPRnrt5OS4VHuJaZfMz0hxMDo7q3wAn10w9wUFqa3yY
+Ig8cdblTnn4F8K0AHXnbZPIhhl3t/iKMdiOL+Y/ugvs3jkju5I7zX2MGQJY6NR+b
+S2LH84jcJH8WfEcigESrn3Wu5Tq/sAYhdb6U2QS6j3e838ZNdLwrAEGmcYftfIQd
+fLxLkjlJcxwbFOB7RFUVkmo3JIKlG5VoVANdtPvPKdT1maeYNxnYL6bS1cdPG12N
+9fQEFumG1gr7ew/WYkS3h8IMN1m/ztf6ZpwjB/JJQ23NphYNRWhbJHB7o9I9bCa5
+10//krepHAkJ1uwPXRY6CQY2j7c7J/PanzSNvm+e7c/E1ft+H5YBwJApakZAsUh1
+xd4ipQYsQQ/CC7B3d4gat2TEfRgVonYjUeDy6oyDqVmU2vIInJKNj5Yc0KBKZGEE
+BOZ4HvnBCwGWCNttypPEqSFgAH4qJFCTT+ysoCpuNxWdPG0xyitq/5Mlc/LtGl0Q
+1KsnlFoOVyLjOL2lpQITP60zl02vJ36PxptR9qfCWxBsTwFYqCOsOYha9Rcxzj7v
+XiVTo4CcTxWnZgcWyVk4XJdmnDmnIrZC7c3ms6+FrK1/23LKUpkHXFm6V8mrCsFi
+YzAwYXRtNTCpkSyiZj+o2Kapx7Q0VyAxgVxC2hp/JepLJfwuWzRZAo2PEplN15RI
+5FhbKoqwNcdZrrhzqwQl9LeBPW683oK4R7R6u/SG26ozcqeMFIJnCOyjFTU/+Ql1
+AcCSBPmGUgYrEUxoZOIyWS0G/cNHzOeSLMDHeij8w0ACrm3AWJ3Ae91lHZK47g3e
+euuY2ffzcWFw8fmS9oXd1eYhhAtY1BnHq3jks7YlslZ93/hQAXVG1N1E3iLXQQmh
+v6L1BW+uPz7z1cC+X3ViLPh/kDphkPQChsEZzNcpJPiiEupA3HQHkq/nBGrt8WVm
+FdI2jin9OTvR+pmO4pIZmOjZVhchUBUt5Jkm8+4ZPJ084ImyGAHeKb1RRuGmi+Hb
+NTYeVtSIFaUD4pXk8FRzN0ssb4tF9QrxVV9jBMON/tFFjBLkaqnsiYCf1J5vN0ZI
+3WDIORqQegOYInrOA7hKOdzMlU5Xeup0SWSY0lzCLdQM3jLtfn1ajb9/YjeRVRzP
+Ty4P/rEvib2y0BAEfphNPSy3o/chdY9G/ra2YZTiL3DjZVv6a+6JuQgG3QfwEZDC
+oMWzIgoNmJr1zZE4qru0/k0zVwFGU/QGBQap0lhqSioyoWmvhaLw9XHg8RfvORNo
+ZjrSMkBBu1uGodiBsZnRS99nd+Gm9bug/3N8pEv7OuNCQycpDjFtdtwW5czjAtqE
+fTxJHUKM7536+K1iiCsGS9ZH0RM0J7tdIDqiPwXRaKGuRRXAhm7thpOR+UqwuPrF
+P1KAKKq1y9GeseOn4lW1awZ7KZxjiOVqR/4cZQJRNlMDOE+WZF//+12AQOrG6sSj
+bgxqO/NbgwxqNSMyNdLAKByCAMxOESEDiQ7zXa5AJH9qX90yWBEtRHG4AU041cf7
+vl8QU7Ki3ZmInYUfT9yhgzQTWNxtX6FxnU8DPC1LAE7j39iRzLdi6EIaTFZIKdig
+nMvZKg4Sp8CJZI+6bVmh7aQBqaEVoRiOMFBzLCq3YcZnNyYlcFDV7PtnAHiVriRq
+mdt9W+u5BO1Zm0Zhm4v4CcaWB5PRQy4VTGYtEt0YB8zgzFYPjJB6afNhUCx+FxJo
+T8F3wjEuYN9Q+s2WlvRO+Oy66YrNPvMJUiw1gtH07vdSxiKF4czmUutt47Y2uCz5
+JGYoqbH3svCINLMZQsaVjeoYR114XxmY++3sqI4FfB+Hpm9J8UyLUx9JEvP51ajH
+Daou9jmRGBWYehWm1Mv7WbrVbV4EInQvNu82xr/koAKGx9XLMpOeb4IESu7mBgbc
+7kuouXbEPKDjNeq+odL2lYoev95XWPuu1m83+Y6yYTvjP9fYB84Ms93Ckwd95zlX
+l/bg0DLFi8IyZzSd6ETHTU7a+gTssXXvX0Pw2Xc77qbszL7OBUhudTwsa5omQNJc
+J3v+Zbp+5WrAgjbtVqEfBSwHADHIBRuSZOQE4hEPyDsE9cCITaj/1GR3yziboNpN
+yjnY83I2uc/fnU4Gq/R54HDY6tcA86PwkvHx1UXkCb+1/QvRZv0uJSG8mQU78AlG
+xaceIAuBOuD3Ie/P5zfwD87nKwj6LueBDhwKV/nwoxQlmwcnXjIGPgTqK3/PbTev
+l4AerZ9tLpcoWkQaUpn/iJqKD/YzZYkQsiHDYktLLrQVFC7UFD5pTMsxDk1+oskS
+5SoWnMBR6f4UDg9J2Mq6PPEHPv4q0XiaP+1OV4nFAGxb1itxL7stp/lN9C6ovSjj
+a/RuvJqpg7QEjjqXtEdE/7ClqokOi3Kko3xXAZnlBzKBwpXxMi/GheHbkPAb+jUx
+JUesoltp1bDS/830MkRBtDRLXLrXT6a9TYA9Z6a4J4ma7QE7eXlk7uIEvkTyA3Sq
+63QMu/111Sbf68ontG1AXAZ/aAYNuqec4eQGRN0WydrznA1/fhkczwbiJmcUR/b+
+LdrTJH7Tnq5/YGYNj9pLog==