Interactive and realtime visualisation with gradio

[jrabary]

Hi all,

I'm trying to add an interactive and realtime visualisation node in my graph. I end up having the following events handlers in the GUI

    def handle_submit(command):
        if command != "":
            instruction_array = pa.array([command])
            node.send_output("instruction", instruction_array)
        return ""

    def handle_camera_streams():
        current_im = im
        while True:
            evt = node.next(timeout=FPS)
            if evt is not None:
                if evt["type"] == "INPUT":
                    if evt["id"] == "observations":
                        obs = decode_dict(evt["value"])
                        current_im = obs["overhead_cam"]

            yield current_im

The first one is to handle user action on the GUI and the second on to display a video stream on the GUI. The point is, the two handlers can work separatly but once I put them together I got some errors

node.send_output("instruction", instruction_array)
RuntimeError: Already borrowed

I suspect dora doesn't like I use the same node object concurrently.

Any ideas ?

[haixuanTao]

Yeah, so I think that our current implementation does not handle well concurrent use of the node.

Could you try to add lock mutex:

from threading import Thread, Lock

mutex_lock = Lock()

# ...
with mutex_lock:
   node.send_output("instruction", instruction_array)

# ...
with mutex_lock:
   evt = node.next(timeout=FPS)

I'm not exactly sure why this is happening but this should fix the issue.

[haixuanTao]

In the above response, maybe it is necessary to add a mutex to avoid reading and writing on the same variable concurrently

[haixuanTao]

You can try this PR: #753

But, when I tried myself I had some issues with high frequency sending of data but maybe it's okay for your issue.

[jrabary]

For now, I'll go with the global variable workaround. I'll test your PR later. I published a code to reproduce the issue here https://github.com/jrabary/dora-gradio

[haixuanTao]

Ok great! Thanks!

Do you plan to maintain it or do you to push it to dora-rs/dora/node-hub and let the community maintain it?

[jrabary]

It's more a proof of concept right now. Let me improve it a bit to make it more useable for anyone. Then, I'll make a PR on the node-hub if you want.