Skip to content

Commit a23f964

Browse files
ErotemicErotemic
committed
wip
1 parent 77487af commit a23f964

File tree

11 files changed

+319
-33
lines changed

11 files changed

+319
-33
lines changed

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,6 @@
44
[submodule "tpl/YOLOX"]
55
path = tpl/YOLOX
66
url = https://github.com/Megvii-BaseDetection/YOLOX.git
7+
[submodule "tpl/poop_models"]
8+
path = tpl/poop_models
9+
url = git@github.com:Erotemic/poop_models.git

‎README.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,9 @@ registration via the SIFT+RANSAC algorithm.
395395
+-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
396396
| 2025-07-04 | 9790 | ~3394 | 2444 | 3406 | bafybeidfxayyacq4jbbhjcxbxumqlspmsmkj52nq2ns4vfew5udgysocoy |
397397
+-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
398-
| 2025-08-19 | 10143 | ~3511 | 2517 | 3523 | bafybeigzy526fesd6hfgflorwymum66lixzzgob6rktiv7epvsyyt6e4me |
398+
| 2025-08-19 | 10143 | ~3511 | 2517 | 3523 | bafybeigzy526fesd6hfgflorwymum66lixzzgob6rktiv7epvsyyt6e4me |
399+
+-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
400+
| 2025-10-15 | 10650 | ~3680 | 2632 | 3695 | bafybeia2ccfh6qbk6nqk2sjty3zntg7gj6cayvdkej3ibvdaekq2zig3qa |
399401
+-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
400402

401403

@@ -588,6 +590,7 @@ Current IPFS addresses for each top-level asset group are:
588590
bafybeidvaphwcib2qezdcey4cj3a2r7r7oxskl56yaccgdi75pdou4ggmm - shitspotter_dvc/assets/poop-2025-03-08-T224918
589591
bafybeieyl6yzi6cyz3minjyvmz53ydbpxmljxs5gytv6cwu6ci7tmwyvjq - shitspotter_dvc/assets/poop-2025-04-20-T172113
590592
bafybeicz7kxvmxojmu33pskfrkglosz3tndgsmpzz6cmmcxynaau5xzfeu - shitspotter_dvc/assets/poop-2025-07-03-T230656
593+
bafybeiapgrjibq7ldtuyukiskn7fi6fedjy5cntjq5rouqnlqkb5dq6224 - shitspotter_dvc/assets/poop-2025-10-14-T222355
591594
592595
593596

‎dev/journal.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -496,3 +496,11 @@ This gives us quite a nice extension to our test set.
496496

497497

498498
Testing on this data gives us an AP of 0.7731623658373592.
499+
500+
# 2025-10-17
501+
502+
503+
The mkorzunowicz/poopdetector MAUI app works with YOLOX onnx models but runs at around 3 frames per second. The app will detect poop close up, and false positives are less common than they used to be. But the small object detection has an AP of 0.18ish which is nowhere near good enough. But with more data, better training, better networks, we should be able to do a bit better. Not sure about if we can do better than onnx on the pixel5 hardware. 3 FPS is good enough, but it would be nice to improve.
504+
505+
506+
Training on my new bigger dataset gave me the best YOLOX results I've seen so far. I've got AP around 0.7 (previous best at maybe 0.6ish) and my AP for small objects got to over 0.3 (previous was around 0.15).

‎dev/poc/setup_yolox_training.sh‎

Lines changed: 282 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,8 @@ class Exp(Exp_Base):
5454
# --- model scale (keep as yolox-s) ---
5555
self.depth = 0.33
5656
self.width = 0.50
57-
self.exp_name = os.path.splitext(os.path.basename(__file__))[0]
57+
#self.exp_name = os.path.splitext(os.path.basename(__file__))[0]
58+
self.exp_name = 'shitspotter-custom'
5859
5960
# --- dataset + classes ---
6061
#self.num_classes = <YOUR_NUM_CLASSES> # e.g., 3
@@ -71,8 +72,8 @@ class Exp(Exp_Base):
7172
self.val_dir = vali_fpath.bundle_dpath
7273
7374
# --- training knobs (tweak as needed) ---
74-
#self.input_size = (640, 640)
75-
#self.test_size = (640, 640)
75+
self.input_size = (640, 640)
76+
self.test_size = (640, 640)
7677
#self.max_epoch = 100
7778
#self.eval_interval = 5
7879
#self.enable_mixup = True
@@ -82,12 +83,289 @@ class Exp(Exp_Base):
8283
self.data_num_workers = 8
8384
self.eval_interval = 1
8485
86+
self.enable_mixup = False # big speedup, minor accuracy hit on small datasets
87+
self.mosaic_prob = 0.0 # or 0.0 for max speed
88+
self.mixup_prob = 0.0
89+
self.random_size = (18, 22) # narrower multi-scale range (around 608–736 px)
90+
self.eval_interval = 10 # evaluate less often
91+
8592
if 0:
8693
self = Exp()
94+
datset= self.get_dataset()
8795
8896
" > exps/custom/yolox_s_custom.py
8997

9098

99+
100+
curl -L https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.pth -O
101+
102+
we pyenv3.11.9
103+
python tools/train.py -f exps/custom/yolox_s_custom.py -d 2 -b 16 --fp16 -o
104+
105+
106+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
107+
mkdir -p exps/custom
108+
echo "
109+
# exps/custom/yolox_s_custom.py
110+
from yolox.exp import Exp as Exp_Base
111+
import os
112+
113+
class Exp(Exp_Base):
114+
def __init__(self):
115+
super().__init__()
116+
117+
import kwcoco
118+
train_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/train_imgs8763_e58dbbb2-poop-only.json')
119+
vali_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/vali_imgs1258_63f16fdc-poop-only.json')
120+
121+
# --- model scale (keep as yolox-s) ---
122+
self.depth = 0.33
123+
self.width = 0.50
124+
self.exp_name = 'shitspotter-custom-v2'
125+
126+
# --- dataset + classes ---
127+
# If your json is literally named 'vali.json', put that here. Otherwise use 'val.json'.
128+
# The way they use data dir is broken. Try and work around it
129+
#self.data_dir = train_fpath.bundle_dpath
130+
131+
self.train_ann = train_fpath.fpath
132+
self.val_ann = vali_fpath.fpath
133+
134+
self.train_dir = train_fpath.bundle_dpath
135+
self.val_dir = vali_fpath.bundle_dpath
136+
137+
# --- training knobs (tweak as needed) ---
138+
self.input_size = (640, 640)
139+
self.test_size = (640, 640)
140+
141+
self.num_classes = 1
142+
self.max_epoch = 300
143+
self.data_num_workers = 12
144+
145+
self.enable_mixup = True # big speedup, minor accuracy hit on small datasets
146+
self.mosaic_prob = 0.5 # or 0.0 for max speed
147+
self.mixup_prob = 0.5
148+
#self.random_size = (18, 22) # narrower multi-scale range (around 608–736 px)
149+
self.eval_interval = 5 # evaluate less often
150+
151+
if 0:
152+
self = Exp()
153+
datset= self.get_dataset()
154+
155+
" > exps/custom/yolox_s_custom_v2.py
156+
python tools/train.py -f exps/custom/yolox_s_custom_v2.py -d 2 -b 16 --fp16 -o -c yolox_s.pth
157+
158+
159+
#### EXPORT TO ONNX
160+
we pyenv3.11.9
161+
pip install onnxruntime
162+
163+
we pyenv3.11.9
164+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
165+
ls YOLOX_outputs/shitspotter-custom-v2/last_epoch_ckpt.pth
166+
python tools/export_onnx.py \
167+
--output-name shitspotter-custom-v2.onnx \
168+
--exp_file exps/custom/yolox_s_custom_v2.py \
169+
--ckpt YOLOX_outputs/shitspotter-custom-v2/last_epoch_ckpt.pth
170+
171+
172+
# --- Train V3
173+
174+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
175+
mkdir -p exps/custom
176+
echo "
177+
# exps/custom/yolox_s_custom.py
178+
from yolox.exp import Exp as Exp_Base
179+
import os
180+
181+
class Exp(Exp_Base):
182+
def __init__(self):
183+
super().__init__()
184+
185+
import kwcoco
186+
train_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/train_imgs8763_e58dbbb2-poop-only.json')
187+
vali_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/vali_imgs1258_63f16fdc-poop-only.json')
188+
189+
# --- model scale (keep as yolox-s) ---
190+
self.depth = 0.33
191+
self.width = 0.50
192+
self.exp_name = 'shitspotter-custom-v3'
193+
194+
# --- dataset + classes ---
195+
# If your json is literally named 'vali.json', put that here. Otherwise use 'val.json'.
196+
# The way they use data dir is broken. Try and work around it
197+
#self.data_dir = train_fpath.bundle_dpath
198+
199+
self.train_ann = train_fpath.fpath
200+
self.val_ann = vali_fpath.fpath
201+
202+
self.train_dir = train_fpath.bundle_dpath
203+
self.val_dir = vali_fpath.bundle_dpath
204+
205+
# --- training knobs (tweak as needed) ---
206+
self.input_size = (640, 640)
207+
self.test_size = (640, 640)
208+
209+
self.num_classes = 1
210+
self.max_epoch = 300
211+
self.data_num_workers = 8
212+
213+
self.enable_mixup = False # big speedup, minor accuracy hit on small datasets
214+
self.mosaic_prob = 0.0 # or 0.0 for max speed
215+
self.mixup_prob = 0.0
216+
#self.random_size = (18, 22) # narrower multi-scale range (around 608–736 px)
217+
self.eval_interval = 5 # evaluate less often
218+
219+
if 0:
220+
self = Exp()
221+
datset= self.get_dataset()
222+
223+
" > exps/custom/yolox_s_custom_v3.py
224+
we pyenv3.11.9
225+
python tools/train.py -f exps/custom/yolox_s_custom_v3.py -d 2 -b 16 --fp16 -o -c yolox_s.pth
226+
227+
228+
# --- Train V4
229+
230+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
231+
mkdir -p exps/custom
232+
echo "
233+
# exps/custom/yolox_s_custom.py
234+
from yolox.exp import Exp as Exp_Base
235+
import os
236+
237+
class Exp(Exp_Base):
238+
def __init__(self):
239+
super().__init__()
240+
241+
import kwcoco
242+
train_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/train_imgs8763_e58dbbb2-poop-only.json')
243+
vali_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/vali_imgs1258_63f16fdc-poop-only.json')
244+
245+
# --- model scale (keep as yolox-s) ---
246+
self.depth = 0.33
247+
self.width = 0.50
248+
self.exp_name = 'shitspotter-custom-v3'
249+
250+
# --- dataset + classes ---
251+
# If your json is literally named 'vali.json', put that here. Otherwise use 'val.json'.
252+
# The way they use data dir is broken. Try and work around it
253+
#self.data_dir = train_fpath.bundle_dpath
254+
255+
self.train_ann = train_fpath.fpath
256+
self.val_ann = vali_fpath.fpath
257+
258+
self.train_dir = train_fpath.bundle_dpath
259+
self.val_dir = vali_fpath.bundle_dpath
260+
261+
# --- training knobs (tweak as needed) ---
262+
self.input_size = (640, 640)
263+
self.test_size = (640, 640)
264+
265+
self.num_classes = 1
266+
self.max_epoch = 300
267+
self.data_num_workers = 8
268+
269+
self.enable_mixup = True # big speedup, minor accuracy hit on small datasets
270+
self.mosaic_prob = 0.5 # or 0.0 for max speed
271+
self.mixup_prob = 0.5
272+
#self.random_size = (18, 22) # narrower multi-scale range (around 608–736 px)
273+
self.eval_interval = 5 # evaluate less often
274+
275+
if 0:
276+
self = Exp()
277+
datset= self.get_dataset()
278+
279+
" > exps/custom/yolox_s_custom_v4.py
91280
we pyenv3.11.9
92-
python tools/train.py -f exps/custom/yolox_s_custom.py -d 1 -b 16 --fp16 -o
281+
python tools/train.py -f exps/custom/yolox_s_custom_v4.py -d 2 -b 16 --fp16 -o -c yolox_s.pth
282+
283+
284+
#### -----------
285+
286+
DVC_DATA_DPATH=$HOME/data/dvc-repos/shitspotter_dvc
287+
KWCOCO_BUNDLE_DPATH=$DVC_DATA_DPATH
288+
ORIG_TRAIN_FPATH=$KWCOCO_BUNDLE_DPATH/train_imgs9270_f2b4b17d.kwcoco.zip
289+
ORIG_VALI_FPATH=$KWCOCO_BUNDLE_DPATH/vali_imgs1258_fe7f7dfe.kwcoco.zip
290+
TRAIN_FPATH=$KWCOCO_BUNDLE_DPATH/train_imgs9270_f2b4b17d-simple-poop-only.json
291+
VALI_FPATH=$KWCOCO_BUNDLE_DPATH/vali_imgs1258_fe7f7dfe-simple-poop-only.json
93292

293+
echo "TRAIN_FPATH = $TRAIN_FPATH"
294+
echo "VALI_FPATH = $VALI_FPATH"
295+
296+
kwcoco modify_categories --keep poop --src "$ORIG_TRAIN_FPATH" --dst "$TRAIN_FPATH"
297+
kwcoco modify_categories --keep poop --src "$ORIG_VALI_FPATH" --dst "$VALI_FPATH"
298+
kwcoco conform --legacy=True --src "$TRAIN_FPATH" --inplace
299+
kwcoco conform --legacy=True --src "$VALI_FPATH" --inplace
300+
kwcoco reroot --absolute=True --src "$TRAIN_FPATH" --inplace
301+
kwcoco reroot --absolute=True --src "$VALI_FPATH" --inplace
302+
303+
python -m shitspotter.cli.simplify_kwcoco --src "$TRAIN_FPATH" --inplace
304+
python -m shitspotter.cli.simplify_kwcoco --src "$VALI_FPATH" --inplace
305+
306+
kwcoco stats "$ORIG_TRAIN_FPATH" "$TRAIN_FPATH"
307+
kwcoco stats "$ORIG_VALI_FPATH" "$VALI_FPATH"
308+
309+
310+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
311+
mkdir -p exps/custom
312+
echo "
313+
# exps/custom/yolox_s_custom.py
314+
from yolox.exp import Exp as Exp_Base
315+
import os
316+
317+
class Exp(Exp_Base):
318+
def __init__(self):
319+
super().__init__()
320+
321+
import kwcoco
322+
train_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/train_imgs9270_f2b4b17d-simple-poop-only.json')
323+
vali_fpath = kwcoco.CocoDataset('/home/joncrall/data/dvc-repos/shitspotter_dvc/vali_imgs1258_fe7f7dfe-simple-poop-only.json')
324+
325+
# --- model scale (keep as yolox-s) ---
326+
self.depth = 0.33
327+
self.width = 0.50
328+
self.exp_name = 'shitspotter-custom-v5'
329+
330+
# --- dataset + classes ---
331+
# If your json is literally named 'vali.json', put that here. Otherwise use 'val.json'.
332+
# The way they use data dir is broken. Try and work around it
333+
#self.data_dir = train_fpath.bundle_dpath
334+
335+
self.train_ann = train_fpath.fpath
336+
self.val_ann = vali_fpath.fpath
337+
338+
self.train_dir = train_fpath.bundle_dpath
339+
self.val_dir = vali_fpath.bundle_dpath
340+
341+
# --- training knobs (tweak as needed) ---
342+
self.input_size = (640, 640)
343+
self.test_size = (640, 640)
344+
345+
self.num_classes = 1
346+
self.max_epoch = 300
347+
self.data_num_workers = 8
348+
349+
self.enable_mixup = True # big speedup, minor accuracy hit on small datasets
350+
self.mosaic_prob = 0.5 # or 0.0 for max speed
351+
self.mixup_prob = 0.5
352+
#self.random_size = (18, 22) # narrower multi-scale range (around 608–736 px)
353+
self.eval_interval = 5 # evaluate less often
354+
355+
if 0:
356+
self = Exp()
357+
datset= self.get_dataset()
358+
359+
" > exps/custom/yolox_s_custom_v5.py
360+
we pyenv3.11.9
361+
python tools/train.py -f exps/custom/yolox_s_custom_v5.py -d 2 -b 16 --fp16 -o -c yolox_s.pth
362+
363+
## This trained the best YOLOx model so far:
364+
# /home/joncrall/code/shitspotter/tpl/YOLOX/YOLOX_outputs/shitspotter-custom-v5/epoch_115_ckpt.pth
365+
we pyenv3.11.9
366+
cd /home/joncrall/code/shitspotter/tpl/YOLOX
367+
ls YOLOX_outputs/shitspotter-custom-v5/epoch_115_ckpt.pth
368+
python tools/export_onnx.py \
369+
--output-name "$HOME"/code/shitspotter/tpl/poop_models/shitspotter-custom-v5-epoch_115.onnx \
370+
--exp_file exps/custom/yolox_s_custom_v2.py \
371+
--ckpt YOLOX_outputs/shitspotter-custom-v5/epoch_115_ckpt.pth

‎shitspotter/cid_revisions.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,6 @@ ipfs swarm connect /p2p/12D3KooWPyQK2JEXnqK1QxiV9Y7bG3UsUQC5iQvDxn8bV1uqvsbi
4444
bafybeidfxayyacq4jbbhjcxbxumqlspmsmkj52nq2ns4vfew5udgysocoy
4545
bafybeihuhrp6wtle5wuhsgcgf6bp7w4ol4pft7y2pcplylzly7gfag74lm
4646
bafybeigzy526fesd6hfgflorwymum66lixzzgob6rktiv7epvsyyt6e4me
47+
bafybeigzy526fesd6hfgflorwymum66lixzzgob6rktiv7epvsyyt6e4me
48+
49+
bafybeia2ccfh6qbk6nqk2sjty3zntg7gj6cayvdkej3ibvdaekq2zig3qa

0 commit comments

Comments
 (0)