How to improve background removal?

[lonbinder]

HI all -

First to Vlad, whoa, you've made an amazing package in Human, incredible work!! Second, to Vlad or anyone, I'd love your guidance on this. One of my use cases is to remove the background from a camera. We won't control the lighting conditions, so it needs to dynamically adjust and properly clip around the head and shoulders; think of any typical green screen or video chat "remove background" type functionality. If I fidget with the segmentation.ratio by hand I can get it to look mediocre (see the attached image). But you'll notice that (a) there is a ton of waviness around the shoulders and (b) the edge of the video often still shows up (I highlighted a minor example in yellow on this screenshot. This screenshot is literally the best case scenario I have produced, all others look much much worse. It is also very sensitive to lighting conditions in the room.

Any guidance or tips on how I can dramatically improve the background removal and make it dynamic (adjusts automatically to color and lighting changes)?

Right now, this is using rvm.json for the model and ratio typically around .25-.35.

Thank you!!

[vladmandic]

honestly, there is only so much that can be done inside the browser using lightweight models - but that is only available path for client-only realtime app.
server-side, there are much higher quality models available that can work on video, but they are not suitable for either real-time or in-browser execution.

[lonbinder]

Thank you for the quick and thoughtful reply, Vlad! I believe that most of the commercial video chat tools are doing client-side, including GMeet, which runs in the browser. Have you tried mediapipe's segmentation? It seems to perform pretty well. https://codepen.io/mediapipe/pen/wvJyQpq

I just figured I might be doing something wrong.

[vladmandic]

human supports mediapipe's older models. but their newer models are pretty proprietary as they moved away from tfjs backend.