Sometimes no audio after receiver switchover in Voter

[pe1chl]

One of our repeaters now has 14 configured receivers, of which 10 are permanently active.
Normally it works well and the Voter selects the best receiver for any incoming station.
But really sometimes (maybe once every couple of days), when the Voter switches over to another receiver the audio is not passed on. The output of the repeater is a blank carrier.
This can go on for a minute or so (while the station keeps transmitting and comes in over the same receiver), but when it switches over to another receiver or stops transmitting, the audio is back.
It appears to be some race condition or other timing glitch, e.g. it could be that it is triggered by someone keying up very shortly, or the squelch signal of a receiver toggling very briefly just at the moment of the switchover.
We cannot reproduce it, and it is by far not happening as often as the earlier problem we had in the Voter. (which was fixed)

[pa1okz]

This issue gets more and more urgent as we continuously increase our number of receivers. Yesterday I noted the problem three times (which is excessive for one day, but the repeater is in use almost the whole day currently due to another new large transmitter with a lot of coverage). To add to Rob's correct description: The problem seems to happen particularly in cases where a weak signal comes in at one or other receiver and switches relatively often. Especially when there either are signal drops (below sq level) causing short interruptions -or- when there are doubles (someone keys by accident shortly during another transmission.

Together with the RemoteTrx crashes (but that is another topic), this issue is rather frustrating for users since it happens that the repeater sometimes transmits silence up to three minutes if someone by coincidence makes a long transmission...

[sm0svx]

Have you tried changing the voter parameters? It would be interesting to see if any of the voting parameters can be changed to lower the failure rate. I don't suggest it as a solution but it may help in finding the problem and also maybe lower the impact of the problem until it's fixed.

[pe1chl]

Fwiw, our current Voter parameters are:

[Voter]
TYPE=Voter
RECEIVERS=Rx_HvsE,Rx_HvsW,Rx_Lpk,Rx_Asd,Rx_Zze,Rx_Apd,Rx_Gv,Rx_Nsw,Rx_Gs,Rx_Bo,Rx_Smd,Rx_Lw,Rx_Porto,Rx_AA,Rx_ETE
VOTING_DELAY=0
BUFFER_LENGTH=0
REVOTE_INTERVAL=100
HYSTERESIS=20
RX_SWITCH_DELAY=250
SQL_CLOSE_REVOTE_DELAY=200

The frequency of occurrence is a bit low to experiment with parameters and observe immediate effect. What parameter(s) would you suggest experimenting with?

[sm0svx]

All parameters that cause more fast switching back and forth would be of interest I guess. The REVOTE_INTERVAL is a candidate, RX_SWITCH_DELAY is another. Increasing HYSTERESIS should stop "ping-ponging" between multiple receivers, if that is what's causing the problem.

Also changing parameters to make the problem worse is an interesting test. Using this approach will get faster feed-back but more annoyed users.

[pe1chl]

Unfortunately even when it has happened it is difficult to find any evidence in the logs and events.
When I heard it happening myself I studied the log and also the Voter event output that we keep, but I found nothing that was different from all the normal switchovers.
PC7X has built a very nice site here: http://pc7x.net/repeaters/#/map/google/pi2nos that allows to scroll back in history and hear the audio and see the active receiver at that time, so there is no shortage of information. But that has not yet brought us closer to the problem other than that we sort of know that it happens when multiple receivers open at the same time and then one or more receivers immediately close again. It rectifies itself as soon as another receiver switch is made.

[sm0svx]

Well, some times, with harder problems, one has to throw logic thinking out the window and just resort to "dummy testing", like "this change cannot possibly affect this problem but I'll test it anyway". That line of thinking has helped me a lot of times to solve "impossible" problems.

(And yes, the page you linked to is very cool and I guess also very useful to debug some problems.)

[pe1chl]

Yes it is impressive what he made, especially considering it is a voluntary effort by an outsider after we made available the event source and a JSON file describing the repeater system (which was then expanded to include items that he needs).
You can fetch the JSON file at http://pi2nos.ampr.org/repeater-info.json which describes all the locations of the transmitters and receivers and all the other info he uses to draw that map for this repeater. Replacing pi2nos by one of our other installations (pi3utr, pi6ten) works as well.
It is in wide use. You can see it is now open by 30 users and at peak times during a special event this approached 150.

[gicardous]

Hi,
seems similar to my problem here:
svxlink with simplex and repeaterlogic; simplexlogic linked to local repeater, repeater logic used for three remotetrx-es: one on remote town, one locally used as crossband to talk on repeater around the house.
The issue appeared totally random an I noticed that using that crossband link. Logs as follows:

29.09.2015 19:30:01: Voter: The squelch is OPEN (Rx1=64.09) <- this was on radio, beginning of a qso
29.09.2015 19:30:01: STx1: Turning the transmitter ON
29.09.2015 19:30:01: Remote_Tx3: The transmitter is ON
29.09.2015 19:30:01: Remote_Tx2: The transmitter is ON
..................
Rx1 never went close until I restarted that other town remotetrx:
..................
29.09.2015 19:45:40: Remote_Rx2: Disconnected from remote receiver 5.15.82.178:5210: Connection closed by remote peer
29.09.2015 19:45:40: Remote_Rx2: Disconnected from remote receiver 5.15.82.178:5210: Connection closed by remote peer
29.09.2015 19:45:40: Remote_Tx2: Disconnected from remote transmitter at 5.15.82.178:5210: Connection closed by remote peer
29.09.2015 19:45:40: Voter: The squelch is CLOSED (Rx1=66.6316)
29.09.2015 19:45:41: STx1: Turning the transmitter OFF
29.09.2015 19:45:41: Remote_Tx2: The transmitter is OFF
29.09.2015 19:45:41: Remote_Tx3: The transmitter is OFF
29.09.2015 19:46:00: Remote_Rx2: Disconnected from remote receiver 5.15.82.178:5210: Connection refused
29.09.2015 19:46:00: Remote_Rx2: Disconnected from remote receiver 5.15.82.178:5210: Connection refused
29.09.2015 19:46:00: Remote_Tx2: Disconnected from remote transmitter at 5.15.82.178:5210: Connection refused
29.09.2015 19:46:20: 5.15.82.178:5210: RemoteTrx protocol version 2.4
29.09.2015 19:46:20: Remote_Rx2: Connected to remote receiver at 5.15.82.178:5210
29.09.2015 19:46:20: Remote_Rx2: Requesting CODEC "OPUS"
29.09.2015 19:46:20: Remote_Rx2: Connected to remote receiver at 5.15.82.178:5210
29.09.2015 19:46:20: Remote_Rx2: Requesting CODEC "OPUS"
29.09.2015 19:46:20: Remote_Tx2: Connected to remote transmitter at 5.15.82.178:5210
29.09.2015 19:46:20: Remote_Tx2: Requesting CODEC "OPUS"

That remotetrx is a raspberry pi with no radio, only a speaker and a mike, ptt is on a gpio pin which was not triggered. Voter attached to SimplexLogic.
Voter config:
[Voter]
TYPE=Voter
RECEIVERS=Remote_Rx2,Remote_Rx3,Rx1
VOTING_DELAY=50
....................................
On both sides runs svxlink/remotetrx from latest git.

[pe1chl]

Well, that is not completely the same I think.
What we observe is that the receiver opens and closes correctly, only the audio is not passed.
This rectifies itself automatically when the station stops transmitting and another one takes over.
(i.e. the Voter makes a new decision)
There is no need to restart anything, but the whole transmission from one station was lost and the others hear only a blank carrier on the output.

[gicardous]

I observed the same behavior here, on remotetrx which is linked to svxlink on same machine. As for the other issue, something came out of log:
Sat Oct 3 09:23:45 2015: Rx1: SetMuteState(CONTENT)
Sat Oct 3 09:23:45 2015: Tx1: Turning the transmitter ON
Sat Oct 3 09:23:45 2015: *** ERROR: snd_pcm_prepare failed (unrecoverable error): No such device
Sat Oct 3 09:27:01 2015:
Sat Oct 3 09:27:01 2015: SIGTERM received. Shutting down application...
Sat Oct 3 09:27:01 2015: Tx1: Turning the transmitter OFF
Sat Oct 3 09:27:04 2015: RemoteTrx v1.1.99.23 (Sep 29 2015) Copyright (C) 2003-2015 Tobias Blomberg / SM0SVX
Remotetrx is restarted every half hour to prevent that stuck.
After restart everything went normal.

[pe1chl]

Ok... that is another issue I think. What we face is a problem related to network problems (slow network, disconnect) that occasionally happens because we use WiFi links and VPN over internet connections shared with other users.
What I see above is related to the soundcard. Typically you get "in use" errors that occur because other applications or overzealous "sound servers" like PulseAudio have opened the card inbetween. Maybe there are other such issues that cause "No such device" errors.
It could also be a bug in your sound driver that fails in some small percentage of open calls. What type of sound card is this?

There is another open issue #96 (keep the soundcard open all the time instead of opening it when transmit starts) which when implemented would maybe work around your problem.

[gicardous]

Ok, I will check that too.
Meanwhile it happened again:
Sat Oct 3 13:09:17 2015: *** ERROR: snd_pcm_prepare failed (unrecoverable error): No such device
Sat Oct 3 13:24:17 2015: *** ERROR: Transmitter Tx1 have been active for too long. Turning it off...
Sondcard is a cheap chinese usb adapter:
Bus 001 Device 076: ID 0d8c:000c C-Media Electronics, Inc. Audio Adapter
I don't remember well that hardware configuration, I think maybe that soundcard is in an usb hub. I have no pulseaudio, only alsa and that system is used only for svxlink and aprsx, with no soundcard for aprs, only one tnc connected through a usb/rs232 adapter. I will change the configuration and use then the builtin raspberrypi soundcard for audio out to see what's happen.
But in my opinion, if possible, it would be a good thing to turn off the transmitter if this error happens.

[sm0svx]

@gicardous, you should report your problem as a new issue since it's different from the original one. Your sound device problem is probably triggered by USB communication problems. That is obviously not something we can blame SvxLink for but of course it would be good if SvxLink could handle it in a better way. But as I said, please report it as a new issue.

[sm0svx]

I guess this problem persist even after the recent RemoteTrx fixes?

[pe1chl]

I'm not sure, I have not heard it for a while but I'll keep monitoring. It requires some (not completely known) conditions to occur.

[pa1okz]

@sm0svx: Yes, the issue still persists - I've heard it at least three times today.

I might have some more direction towards this issue: The voter seems to mute the recieved audio in particular when the repeater transmits internal sounds (such as an ID, Echolink connect or disconnect message or similar) while during that event a voter switch to another RX is initiated as well, or even when the squelch opens right at the point where this sound is starting.

It's a bit hard to reproduce since the system is live, maybe it helps to simulate it on the scratchboard. Actually, this looks a bit similar to issue 158 - but maybe it's not related...

[sm0svx]

That is a very interesting observation. I'll see if I can reproduce that in the simulator.

[pa1okz]

We haven't noticed the issue any longer lately, or to be more precise, since the last version is installed.

I guess this one can be closed; no idea what happened that solved the issue. If it shows up again, I can re-open it again...

[sm0svx]

That is good news! It would be good though to find out what changed. I'm not sure but I think we have seen this as well on one of our systems (@sm3sgp?).

If anyone else still have this problem, please comment here.

[sm3sgp]

I have not heard this problem for a long time.