Retry to connect to down camera

[gniknalu]

I have a camera that has a scheduled reboot. When that happens, this appears in the logs:
Uncaught exception setting up domain camera for component ffmpeg: FFprobe could not connect to stream.

Once that happens, viseron will not use this camera until everything is bounced.

When a camera goes down - after a predefined time frame, is it plausible to retry connecting?

Just a thought.

Thank you for this excellent project!

[roflcoopter]

Hmm it should try to reconnect in the background.
Are you also restarting Viseron at the same time?

[gniknalu]

Hmm it should try to reconnect in the background. Are you also restarting Viseron at the same time?

It doesn't try . . . . I end up restarting Viseron hours later.

I can confirm the camera works. When it does it's "maintenance", it only lasts about 3 minutes and then it's up and functional.

[roflcoopter]

Gonna need some logs when this happens.
I just tried to reboot my camera and Viseron handles it properly as you can see in this log snippet:

[2022-05-27 10:45:18] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 4 times
[2022-05-27 10:45:18] [DEBUG   ] [viseron.states] - Setting state of sensor.porch_object_detector_fps to state: 11.5, attributes {'preprocessor_fps': 730.7, 'inference_fps': 11.6, 'theoretical_max_fps': 11.5}
[2022-05-27 10:45:19] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
************************************ Disconnected here
[2022-05-27 10:45:19] [ERROR   ] [viseron.watchdog.subprocess_watchdog] - Process None has exited, restarting
[2022-05-27 10:45:20] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:45:24] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 5 times
[2022-05-27 10:45:24] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - [tcp @ 0x5614a0e1f200] Connection to tcp://hikvision_porch.ipcam:554?timeout=5000000 failed: Connection timed out
[2022-05-27 10:45:24] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/101/: Connection timed out
[2022-05-27 10:45:25] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:45:29] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 5 times
[2022-05-27 10:45:30] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:45:35] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 6 times
[2022-05-27 10:45:35] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:45:40] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 5 times
[2022-05-27 10:45:40] [ERROR   ] [viseron.components.ffmpeg.camera.porch] - FFmpeg process has exited
[2022-05-27 10:45:40] [DEBUG   ] [viseron.states] - Setting state of binary_sensor.porch_connected to state: off, attributes {}
[2022-05-27 10:45:44] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 4 times
[2022-05-27 10:45:44] [DEBUG   ] [viseron.states] - Setting state of sensor.doorbell_object_detector_fps to state: 11.5, attributes {'preprocessor_fps': 329.6, 'inference_fps': 11.6, 'theoretical_max_fps': 11.5}
[2022-05-27 10:45:45] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:45:45] [ERROR   ] [viseron.components.ffmpeg.camera.porch] - Restarting frame pipe
[2022-05-27 10:45:45] [DEBUG   ] [viseron.components.ffmpeg.stream.porch] - FFmpeg decoder command: ffmpeg_porch -hide_banner -loglevel error -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -use_wallclock_as_timestamps 1 -vsync 0 -stimeout 5000000 -c:v h264_cuvid -rtsp_transport tcp -i rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/102/ -f rawvideo -pix_fmt nv12 pipe:1
[2022-05-27 10:45:45] [DEBUG   ] [viseron.components.ffmpeg.stream.porch] - FFmpeg segments command: ffmpeg_porch -hide_banner -loglevel error -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -use_wallclock_as_timestamps 1 -vsync 0 -stimeout 5000000 -c:v h264_cuvid -rtsp_transport tcp -i rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/101/ -f segment -segment_time 5 -reset_timestamps 1 -strftime 1 -c:v copy /segments/porch/%Y%m%d%H%M%S.mp4
[2022-05-27 10:45:48] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 3 times
[2022-05-27 10:45:48] [DEBUG   ] [viseron.states] - Setting state of sensor.porch_object_detector_fps to state: 11.5, attributes {'preprocessor_fps': 730.7, 'inference_fps': 11.6, 'theoretical_max_fps': 11.5}
[2022-05-27 10:45:50] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 2 times
[2022-05-27 10:45:50] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - [tcp @ 0x56255cbe6200] Connection to tcp://hikvision_porch.ipcam:554?timeout=5000000 failed: Connection timed out
[2022-05-27 10:45:50] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/101/: Connection timed out
[2022-05-27 10:45:50] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - [tcp @ 0x55fd5f78e080] Connection to tcp://hikvision_porch.ipcam:554?timeout=5000000 failed: Connection timed out
[2022-05-27 10:45:50] [ERROR   ] [viseron.components.ffmpeg.stream.porch] - rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/102/: Connection timed out
[2022-05-27 10:45:50] [ERROR   ] [viseron.components.ffmpeg.camera.porch] - FFmpeg process has exited
[2022-05-27 10:45:55] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 5 times
[2022-05-27 10:45:55] [ERROR   ] [viseron.components.ffmpeg.camera.porch] - Restarting frame pipe
[2022-05-27 10:45:55] [DEBUG   ] [viseron.components.ffmpeg.stream.porch] - FFmpeg decoder command: ffmpeg_porch -hide_banner -loglevel error -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -use_wallclock_as_timestamps 1 -vsync 0 -stimeout 5000000 -c:v h264_cuvid -rtsp_transport tcp -i rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/102/ -f rawvideo -pix_fmt nv12 pipe:1
[2022-05-27 10:45:55] [DEBUG   ] [viseron.components.ffmpeg.stream.porch] - FFmpeg segments command: ffmpeg_porch -hide_banner -loglevel error -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -use_wallclock_as_timestamps 1 -vsync 0 -stimeout 5000000 -c:v h264_cuvid -rtsp_transport tcp -i rtsp://*****:*****@hikvision_porch.ipcam:554/Streaming/Channels/101/ -f segment -segment_time 5 -reset_timestamps 1 -strftime 1 -c:v copy /segments/porch/%Y%m%d%H%M%S.mp4
[2022-05-27 10:46:12] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: [], message repeated 17 times
********************************** Reconnected here
[2022-05-27 10:46:12] [DEBUG   ] [viseron.states] - Setting state of binary_sensor.porch_connected to state: on, attributes {}
[2022-05-27 10:46:12] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:13] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:46:13] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:14] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:46:14] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:14] [DEBUG   ] [viseron.states] - Setting state of sensor.doorbell_object_detector_fps to state: 11.5, attributes {'preprocessor_fps': 270.4, 'inference_fps': 11.5, 'theoretical_max_fps': 11.5}
[2022-05-27 10:46:15] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:46:15] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:16] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:46:16] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:17] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []
[2022-05-27 10:46:17] [DEBUG   ] [viseron.components.mog2.motion_detector.porch] - Max motion area: 0
[2022-05-27 10:46:18] [DEBUG   ] [viseron.components.darknet.object_detector.doorbell] - Objects: []

Also, FFprobe is not involved in the reconnection, only at startup. Are you sure you dont have a cron job or something thats restarting Viseron?

[gniknalu]

Yes - cron restarting Viseron nightly. But the camera was rebooting one hour after Viseron restarted.

Been too busy lately to go further with it so I turned off that "feature" of the camera for now. (When it is enabled, the camera was stable. But when disabled, it would randomly stop working after a week or so.)

Hope to get back to getting more data soon. Thank you!