extremely slow param download with multiple vehicles #2784
Comments
|
Possible to share a log? |
|
I've setup the same arrangement and radios at home and reproduced the problem. The logs below are for MissionPlanner and mavproxy both running on the same win10 machine. MAVProxy takes about 30s to get all the plane and rover parameters (one vehicle is plane, one is rover), MissionPlanner takes 10 minutes just to get the plane params. The rover params had started to download (slowly) when I disconnected. |
|
the setup is:
|
|
im just going though this at the moment and |
thanks to Michael Oborne for spotting this: ArduPilot/MissionPlanner#2784
you're right, thanks! |
thanks to Michael Oborne for spotting this: ArduPilot/MissionPlanner#2784
|
@meee1 any update on this? It is also very slow when there is a mavlink gimbal attached (reported by a partner) |
|
The mavftp issue should be resolved, and was likerly due to large packets on a RFD. I think a retest would be good at the moment |
|
@meee1 it is not fixed. I've also now tested with UDP over ethernet and it is just as bad, so it isn't a radio issue.
after mission planner connects it sees both boards, and then if you go to the config screen for the 2nd cube you see the parameter download slowly crawling along: |
|
using "watch PARAM_VALUE" in mavproxy with the above setup seems to show MissionPlanner downloading 2 parameters per second |
|
MP by default only pulls params on the primary, the initial log showed mavftp packets missing from the MAV, causing the large delays this new issue your seeing i will look into further |
ConnectionControl: on change mav check if full param list needed #2784
|
Hi Michael and Andrew, I'm the developer of Ultimate LRS and made a lot of testing of the telemetry between Ardupilot and Mission Planner over the last few years. I noticed something that can be related to this issue : if the link has some lost packets, very quickly Mission Planner will after some time go into a "request one by one" parameters mode. In a perfect world, this would just require the time to transfer every missing parameter one by one. But in practice there's a catch : Ardupilot has no way to know it has to stop sending all parameters (the initial mavlink request) : so in parallel to sending the requested parameters one by one, it continues to send its full list of parameters. And this has an impact on Mission Planner, because when it requests parameters one by one, it expects to receive exactly that parameter ID, and not another one. If it receives another parameter, which happens because Ardupilot continues to send its full list, it's just dropped, and the "correct" parameter is requested again 3 times. It's slowing down because the full list of parameters is received but all of its messages are dropped. And the one by one messages are requested several times before being received. It can also happen that if they are not received in a timely manner, will be requested again and again. It can be seen as a competition between the full list that continues to be sent, and the one by one parameters requested : Ardupilot will try to serve both, but we're never certain if the next parameter will be from the full list or from the one by one request. MissionPlanner/ExtLibs/ArduPilot/Mavlink/MAVLinkInterface.cs Lines 2217 to 2224 in e2a83b3 This can create a death spiral and become very slow : In addition, if a correct parameter (the exact index waited for) isn't received for 4 seconds, it will request all parameters again, up to 3 times : MissionPlanner/ExtLibs/ArduPilot/Mavlink/MAVLinkInterface.cs Lines 1992 to 2004 in e8d6b2f So in the worst case, we receive several times the full list of parameters "in parallel", which are getting completely dropped out, and many one by one parameters as well. As we don't get the exact parameters waited for, the parameters are requested one by one several times, with a lot of "already received" messages. My suggestion to solve this would be for Mission Planner to accept and consume any parameter received, even if not exactly the one expected. This will avoid entering a dead loop like above.
MissionPlanner/ExtLibs/ArduPilot/Mavlink/MAVLinkInterface.cs Lines 1972 to 1974 in e2a83b3 I don't think this issue has a high impact, but maybe in some cases it could trigger a new "request all parameters".
My suggestion here would be that Ardupilot stops sending a first list of parameters (requested with a "request all parameters" message), before sending a new list, rather than sending the two or more lists "simultaneously". In addition I think the mavlink protocol should include a message to stop a previous request of sending all parameters, and maybe a message to request a specific list of parameters in one message. (This said, probably most future boards will probably use MavFTP to transfer the parameters)
However asking for 10 parameters every 600 ms is a bit too fast : Ardupilot will send only 10 parameters per second (if I remember well). Ardupilot will answer all parameters requests, even if there are several requests for the same parameter. So I think this contributes to the slowness : there's a strong chance of requesting several times the same parameters one by one just because Arduplane can't send them as fast as requested. Imagine 50 parameters are missing, and requested one by one. |
|
what is the max speed we are talking about here? ie baud? ie what's a relisting data rate? looking at your analysis -request all parameters is still ongoing and -request parameter 10 one by one yes, after the first 3 attempts at getting the entire param list, and not completing at least 75% of them -receive parameter 54 from the request all parameters : dropped incorrect all packets are accepted, but only those we need are added to the param list, as we already have it. ive stopped here, as it just repeats so my questions are |
|
I'm normally using a 19200 bauds link without flow control, so the typical throughput is 1920 Bps, but for testing purposes I used an FTDI adapter directly connected to the FC (Mateksys F765 wing). MavFTP is deactivated. The SR?_PARAM is at 10 Hz by default, and I tried different other rates but it seems that Ardupilot doesn't take it into account (tested on AP 4.1.3). The parameters are sent at about 25 Hz even if the SR?_PARAM are at 5Hz :
(We can see PARAM_VALUE messages received at 25Hz, while the SR?_PARAM are at 5 Hz) I rechecked and you're right all incoming parameters are accepted. I make some testing by stopping the flow during parameters transmission (removing the serial wire to the FTDI adapter), but with the latest version everything runs smoothly and missing parameters are correctly collected one by one at the end. The only thing to notice is that if running at 19200 bauds, there's something with the get one by one parameters, maybe because the parameter transmission is slower. Ten parameters are requested, but about half of them are already received. This doesn't happen at faster rates such as 115200 bauds, only at 19200. MP 1 3 77 @ 19200 bauds - provoked short interruption during parameters transmission.txt INFO MissionPlanner.MAVLinkInterface - Get param 1 by 1 - got 1129 of 1165 This isn't blocking, but slows down the end of the parameters transmission. To me it's like the one by one requested parameters are already sent by Ardupilot, but not yet received by Mission Planner, so the same parameters are requested again. Maybe are they in some serial buffer ? I also made some checks with longer than 4 seconds break of the serial connection, to provoke a full parameters request. The parameters requested one by one start at param 1048 MP 1 3 77 @ 115200 bauds - provoked long interruption during parameters transmission.txt INFO MissionPlanner.MAVLinkInterface - Get param 1 by 1 - got 1048 of 1165 Again it's not blocking, but slowing down as all parameters are sent multiple times. This last case requires a 4 second interruption in the transmission. A suggestion would be to detect if a full parameters transmission is ongoing, and not send additional request for parameters while it's ongoing (as it's impossible to ask Ardupilot to stop a full parameters transmission). But these are just small improvements, on the latest Mission Planner version I can't create a situation where the parameters transmission would take more than about 1 minute. |
Uh oh!
There was an error while loading. Please reload this page.
testing with two vehicles (a ship and a quadplane) on a single RFD900x, the parameter download can take more than 10 minutes. The same link with mavproxy takes only a few seconds. The user tells me that with 6 vehicles param download takes over an hour.
Watching the download, MissionPlanner quickly falls back to single PARAM_VALUE at a time. I found that if during the param download I forwarded the link to mavproxy and then mavproxy did a parameter fetch (not ftp) then MissionPlanner then got the parameters quickly as it saw the parameters fetched by mavproxy.
The text was updated successfully, but these errors were encountered: