rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20)...

[miles267]

Describe the problem

Am receiving the following 2 error rows in my archiveloop.log when attempting to sync RecentClips to smb share. Upon encountering these errors, the teslausb stops syncing and the car will be permitted to sleep:

SentryClips/2022-05-10_12-26-32/2022-05-10_12-15-50-back.mp4
SentryClips/2022-05-10_12-26-32/2022-05-10_12-15-50-front.mp4
SentryClips/2022-05-10_12-26-32/2022-05-10_12-15-50-left_repeater.mp4
SentryClips/2022-05-10_12-26-32/2022-05-10_12-15-50-right_repeater.mp4
SentryClips/2022-05-10_12-26-32/2022-05-10_12-16-50-back.mp4
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(503) [generator=v3.2.3]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(701) [sender=v3.2.3]
Wed 11 May 09:01:46 CDT 2022: cleaning cam mount
Wed 11 May 09:01:47 CDT 2022: done cleaning cam mount
Wed 11 May 09:01:47 CDT 2022: Trimming free space in /mnt/cam, which has 169 extents
Wed 11 May 09:01:47 CDT 2022: Trim complete, image now has 169 extents
Wed 11 May 09:01:47 CDT 2022: Unmounting /mnt/cam...
Wed 11 May 09:01:47 CDT 2022: Unmounted /mnt/cam.
Wed 11 May 09:01:47 CDT 2022: Finished archiving.

Device

Raspberry Pi Zero 2 W

Car Model

Model 3

USB connection

Glove box

Logs

teslausb-headless-setup.log
diagnostics.txt
archiveloop (1).log

Additional information

No response

[marcone]

This is because of:

Wed 11 May 09:01:45 CDT 2022: connection dead, killing archive-clips

Basically what happens is that copying the files saturates the network connection to the point that it crowds out the watchdog thread that checks whether the archive server is still reachable, and so that watchdog thread ends up killing the copy process.

[miles267]

@marcone Are there any known fixes for this? Since this only just started despite no network or Rpi config changes, should I reimage the teslausb or should I be switching to a Pi4 then in this case?

It seems to suddenly be happening consistently and my Ri Pi0 2 has near perfect connectivity, transferring at it's full 72 Mbps. It's as if the connectivity is almost too good. I've never seen this before on my network. Everything is the same. Access point, connection, RPi 0 configuration, etc.

[marcone]

Are there any known fixes for this?

You could try adding something like --bwlimit=4000 to the rsync command in archive-clips.sh, to make rsync limit its throughput.

It seems to suddenly be happening consistently and my Ri Pi0 2 has near perfect connectivity, transferring at it's full 72 Mbps

How did you arrive at 72 Mbps?

Wed 11 May 09:01:46 CDT 2022: Error during archiving. Archived 276 event files in 20m51s
...
2022/05/11 09:01:45 [13893] sent 6477293485 bytes received 8807 bytes total size 7667622692

means it sent a little under 5 MB/s, or about 39 Mbps. There will be some additional overhead on top of that, but I'd be surprised if that overhead amounts to 33 Mbps.

[miles267]

Are there any known fixes for this?

You could try adding something like --bwlimit=4000 to the rsync command in archive-clips.sh, to make rsync limit its throughput.

It seems to suddenly be happening consistently and my Ri Pi0 2 has near perfect connectivity, transferring at it's full 72 Mbps

How did you arrive at 72 Mbps?

Wed 11 May 09:01:46 CDT 2022: Error during archiving. Archived 276 event files in 20m51s
...
2022/05/11 09:01:45 [13893] sent 6477293485 bytes received 8807 bytes total size 7667622692

means it sent a little under 5 MB/s, or about 39 Mbps. There will be some additional overhead on top of that, but I'd be surprised if that overhead amounts to 33 Mbps.

Thanks. I'll try that. Unifi registers the Tx and Rx rates for the teslausb RPi Zero 2 at 72 Mbps. I suppose that is the max capability of the wifi NIC itself. As for newer iPhones that same value is 866 Mbps.

[marcone]

Unifi registers the Tx and Rx rates for the teslausb RPi Zero 2 at 72 Mbps.

That's the link speed for a 20 MHz 1x1 channel, not the actual throughput. The 39 Mbps actual throughput I mentioned above is probably about as good as you can expect in a real world environment with the PiZeroW-2.
This is why I suggested using --bwlimit=4000, as it'll stay about 20% under what you're currently achieving. That does mean it won't be optimally using the network connection though. Ideally, the connection checker should be updated to use a different method, like looking at actual throughput through the network interface.

[miles267]

If it would be possible to submit a request to have the connection checker upgraded in a future release, that would be much appreciated.

In the meantime, I've tried to place the --bwlimit=4000 in archive-clips.sh without success. Does it go here?

while [ -n "${1+x}" ]
do
if ! (rsync -avhRL --remove-source-files --temp-dir="$rsynctmp" --no-perms --omit-dir-times --stats
--bwlimit=4000
--log-file=/tmp/archive-rsync-cmd.log --ignore-missing-args
--files-from="$2" "$1/" "$ARCHIVE_MOUNT" &> /tmp/rsynclog || [[ "$?" = "24" ]] )
then
cat /tmp/archive-rsync-cmd.log /tmp/rsynclog > /tmp/archive-error.log

[marcone]

Yes, you can put it there. Don't forget to end the line with a \

[miles267]

I've since re-imaged this SD card completely with the latest teslausb to determine whether perhaps a recent update (i.e. apt-get update) has caused this. It's a peculiar issue that seems to have occurred recently without otherwise there being any other changes on my network. Will confirm back.

[miles267]

@marcone providing more feedback as I troubleshoot. I've since reimaged same card to get it onto teslausb tested, base image. issue still occurring. Have added the --bwlimit=4000 only in one location of the archive-clips.sh as shown below. Is that the correct method? I wasn't sure whether needed it on the --log-file and --files-from rows also? I can then play around with the limit value.

while [ -n "${1+x}" ]
do
  if ! (rsync -avhRL --bwlimit=4000 --remove-source-files --temp-dir="$rsynctmp" --no-perms --omit-dir-times --stats \
        --log-file=/tmp/archive-rsync-cmd.log --ignore-missing-args \
        --files-from="$2" "$1/" "$ARCHIVE_MOUNT" &> /tmp/rsynclog || [[ "$?" = "24" ]] )
  then
    cat /tmp/archive-rsync-cmd.log /tmp/rsynclog > /tmp/archive-error.log
    exit 1

[marcone]

I wasn't sure whether needed it on the --log-file and --files-from rows also?

No, those are all still part of the same command. Everything between the parentheses is a single command line.

[miles267]

Thanks again. Despite different trial/error attempts, I'm finding myself suddenly needing to further restrict --bwlimit=4000 to 3000 and potentially lower. On my Unifi network I've already done a series of other things and it appears it could be connectivity related. However once the SIGTERM occurs and teslausb disconnects from my network, it never attempts to reconnect unless it loses power and needs to reboot/reconnect. Is there any way to have it reconnect periodically if dropped?

So far I've created a specific 2.4 ghz SSID entirely for my teslausb with legacy support enabled (802.11b), boosted the signal level on the nearby AP which is dedicated to the Pi, tried different 2.4 ghz channels (non-overlapping) and removed all interference. It seems to improve the duration of the RPi Zero 2's connection, but eventually it seems to hit a "connection dead" before all of the teslausb clips have archived. The SMB server is Windows Server 2022.

[miles267]

@marcone after further troubleshooting, it appears my data transfer reliability back to smb file share on Windows Server 2022 server improved when I implemented a --bwlimit=2000. I do get occasional sync errors that require a sync be re-triggered from the webUI, but that's been the exemption.

Still odd that I've never had to use this --bwlimit parameter before and these issues only started recently.

[marcone]

I wonder if using a different queuing algorithm on the network interface would help. It looks like Raspberry Pi OS uses "pfifo_fast" by default, whereas other modern linux distributions have apparently switched to "fq_codel".
You could try adding tc qdisc add dev wlan0 root fq_codel || true to the teslausb config file and see if that helps. (reboot afterwards, then run ip link show to check that it's using fq_codel)

[miles267]

Thanks @marcone. I will try out adding that row to the teslausb_setup_cariables.conf and re-running the upgrade (followed by reboot).

UPDATE: after doing that, ip link show returns this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DORMANT group default qlen 1000
link/ether e4:5f:01:51:48:57 brd ff:ff:ff:ff:ff:ff

UPDATE2: this did not resolve the issue. It will go back to transferring at full speed again, but after a few minutes it starts returning repeated errors:

[Mon May 23 17:59:05 2022] brcmfmac: brcmf_sdio_hdparse: HW header checksum error
[Mon May 23 17:59:05 2022] brcmfmac: brcmf_sdio_rxfail: terminate frame

UPDATE3: have reverted back to pfifo_fast judging by forum posts I've run across this is likely an issue with the broadcom wifi drivers for the RPi Zero 2 W that must've come down with an image update. For now, I'm using the --bwlimit=1500 parameter until a fix/update comes in future.

UPDATE4 (last one): replaced the RPi Zero 2 W with my RPi 4 and haven't had the issue on same network. It would appear there's some sort of issue with the wifi drivers for the Zero 2 W. I've shelved that device for the time being.

[miles267]

Upon further troubleshooting, it appears Raspberry Pi Zero, Zero 2 W and 4 devices are incompatible with Unifi access points with VHT80 channels enabled. Even if a particular SSID is configured only as 2.4 ghz (recognizing the Zero and Zero 2 W don't support 5 ghz band).

Am testing further but since changing all APs under Unifi from VHT80 to VHT40 (for 5 ghz), all Raspberry Pi transfers have been stable without any timeouts or disconnects or any need to configure a --bwlimit value.