Scatter plot relative to a given location

[nseidle]

Hi,

I realize I am asking for the moon but, could we get a point on the scatter plot that shows a known/fixed location?

Why: When qualifying an RTK engine it is helpful to view the output on a scatter plot in relation to a known fixed point. It is common for the exact location of an antenna to be known (usually by a 24 hour collection and then processed by a service like CSRS). We have multiple antennas affixed to our building, all with known locations down to ~2mm. Watching how an RTK engine fixes (or fails to fix) around the actual known location is very helpful. It's a really fun way to watch solar flares (RTK Fix degradation)!

It could look something like this:

Above, the red dot represents the "known" location and the green dots are the NMEA fix locations. I don't know the best way to get the LLA or ECEF coordinates for the known location from the user, but any method should be sufficient.

This feature would be dependent on the ability to zoom the scatter plot a lot more.

To my knowledge, no other GNSS software has this ability. We are currently creating these plots manually, hence the ask.

[semuadmin]

Hi @nseidle

So re #160 and #161 - extending the zoom range should be fairly trivial - I'll put that on the backlog.

Adding a known fixed location would involve a bit more work, not least to add the necessary HP lat/lon data entry fields. As you're presumably aware, the scatterplot is currently dynamically centered on an averaged position, so the known position marker would move around a bit; an alternative might be to fix the plot centre to the known location and add the scatter plots in relation to that, but that would involve a more extensive rewrite of the core plot routine. Any thoughts?

Bear in mind that, at mm resolutions any plot based on a standard WGS84 coordinate may be somewhat spurious - you're getting into the realm of tectonic plate datums etc.

[gdt]

I too think this would be useful. (I am not part of SparkFun.)

I lean to making 0 be the reference location. The reason is that if you believe that is the true answer (perhaps because you did a 24-48h static solution, maybe even with better equipment), then you want the axes to be error relative to correct, rather than showing a dot which represents the error of the average. Or at least that's what I want. In general I am not comfortable with the broad trend in the GPS processing world (gpsd does this too) of treating the average of an observation set as the correct answer. Sometimes that's all you have, but I think it's important to be clear on that vs believed correct.

As for the WGS84 comment:

• RTK output is often in a plate-fixed datum like NAD83(2011) epoch 2010.0, because people who run RTNs often want to use such coordinate systems. At least in my state, that's how the DOT operates theirs.
• Even if station coordinates have a 3 cm/y velocity, data gathered over 24h and a known location input by the user (which could be from a transform to today's epoch) should not have significant motion errors.

Essentially, I'd argue to try to do accuracy as well as possible not worrying that a user could do something else wrong; to me the point is to do as well as possible for the case when the user gets it right.

[semuadmin]

E.G. fix-centered plot with extended zoom (down to 0.01m) and fixed reference LLA could look something like this ...

(might need some minor tweaks to the planar() and haversine() functions in pynmeagps)

[gdt]

I would want to be able to get a sense for what fraction of observations are within e.g 10cm of the known position. (Again this is just me and I don't claim everybody else would want that.)

[semuadmin]

I would want to be able to get a sense for what fraction of observations are within e.g 10cm of the known position. (Again this is just me and I don't claim everybody else would want that.)

TBH that's sounding more like a numpy / matplotlib kinda solution, but can look into it. Would you be looking for real time stats?

[gdt]

sorry, I didn't mean actually doing that math. You are quite right to get the real answer. I just meant that looking at your plot with the green dot, the rings are drawn with respect to 0 (of course they are) and when this is average, the visual impression is closeness to 0. If one is trying to assess closeness to correct instead of to average, then it's nice to have rings around correct. (I don't mean to be demanding, but rather just explaining my thoughts in the hopes it is net helpful.)

[semuadmin]

sorry, I didn't mean actually doing that math. You are quite right to get the real answer. I just meant that looking at your plot with the green dot, the rings are drawn with respect to 0 (of course they are) and when this is average, the visual impression is closeness to 0. If one is trying to assess closeness to correct instead of to average, then it's nice to have rings around correct.

Understood and makes perfect sense - I've edited the demo above to reflect what I think you'd be after. What I could do fairly easily is centre the plot on either a fixed LLA coordinate (if provided and valid) or the cumulative average position (if no fixed coordinate is entered)

[semuadmin]

In hand with milestone of end November.

If you'd like to evaluate a provisional RC-1.4.25, be my guest...

python3 -m pip install git+https://github.com/semuconsulting/pygpsclient@RC-1.4.25

FYI: this may benefit from some minor tweaks to the bearing(), planar() and haversine() functions in pynmeagps to squeeze a bit more resolution from the calculations. There are of course dedicated geospatial libraries which provide enhanced "Great Circle" distance and bearing estimations, but I've assumed that a simple "Planar" approximation is more appropriate at the mm end of the resolution scale (I'll happily stand corrected on this).

[nseidle]

an alternative might be to fix the plot centre to the known location and add the scatter plots in relation to that, but that would involve a more extensive rewrite of the core plot routine. Any thoughts?

I would prefer to have a movable center (drag via mouse down). Why: The "known" position can be a meter or more off from the output because of exactly what you bring up - reference frames. If the view is centered on the "known", and if all I can do is zoom in/out on that location, then I will never be able to inspect the collection of fixes.

Using your RC (thanks for the install instructions):

Above, I can see my fixed/"known" (it's me just changing a few lon decimals to mimic an alternate reference frame) and the cluster of RTK Fix positions, but zooming in I lose the cluster because I can't drag the frame around.

Here comes the scope creep: I'm really expecting to be able to click and drag in the scatter plot, as well as zoom with the mouse wheel. But what you've got is already very helpful!

If that sort of UI is too much change, then I would prefer the setup you have now - lock the graph to the "known" so we can watch the fixes relative to the known.

[gdt]

That two-choice option (with one only if fixed provided) sounds exactly right.

As for nseidle's concern about using the wrong reference frame, I would say that the current proposal enables detecting that. Once you know you have a frame error (if you are comparing coordinates, they just have to be in the same frame, or the results are undefined and nonsensical), then the next step is to fix the frame error, starting with really understanding what you are doing.

I have found (not using PyGPSCilent) that comparing a "known position" obtained from OPUS to RTK fixes, all of it in EPSG:6319 results in very tight clustering. If I used the known coordinates in ITRFnnnn instead and compared them to EPSG:6319 values, they'd appear a meter of so off. But my only answer to that is "don't compare data in different frames; it doesn't make sense".

This does mean that a lot of people trying to use RTK, including those using a survey-in feature, who are unclear on any of the following are going to have problems.

• WGS84 is an ensemble and one member of that is low quality, and therefore talking about coordinates in "WGS84" to better than a meter or two does not make sense.
• There is no way to obtain accurate positions in WGS84, unless you are the US DoD (which almost all of us are not!).
• many RTK networks are not in ITRF, but instead are in a national/regional plate-fixed datum. Except when the plate-fixed datum is not quite fixed (like NAD83)
• When using a dynamic datum (eg. ITRFnnnn), one probably needs to handle plate motion, once you have data from multiple years and when trying to play at the cm level.
• Positions from a GPS receiver might be in the current WGS84 realization, without differential, and might be in some semi-recent ITRF, with WAAS (but good luck finding documentation). They are never in an ensemble.
• It is unrealistic to determine positions from imagery to the level of assessing RTK accuracy. If you are really lucky you might be able to get 15 cm ish out of imagery, but that takes an imagery provider that publishes a serious level of metadata and operates in the plate-fixed frame, with ground control. My state's GIS department does that, but random TMS does not. Even my state's GIS dept, when asked how they transform the NAD83 imagery to "WGS84" for TMS in 3857, did not have an answer.
• proj does not transform correctly between an NAD83 realization and "WGS84". It deals much better with ITRF2014 or some other specific frame that is substantially equivalent to recent WGS84 realizations

This is how it is. It would be good to help people, but tools accomodating such confusion is in my view not a good plan. None of the above belongs in the PyGPSClient visualization IMHO; that should expect that all data is in the same frame.

[semuadmin]

what he said...

(I might actually reference that contribution in my RTK-TIPS as I fear some casual users have unrealistic expectations)

[nseidle]

What I can do as a short term compromise is make the behaviour elective i.e. have a user-selectable 'centering mode' spinbox with options 'average' and 'fixed reference' (with the latter only being available if valid fixed reference coordinates have been provided) - how does that sound?

Sounds perfect. Thanks!

[semuadmin]

Provisionally done - try installing RC-1.4.5 again for latest updates.

[semuadmin]

Merged into v1.4.6, now available on PyPi

[semuadmin]

@nseidle @gdt

FYI I'm working up an internal rewrite of the scatterplot widget to improve plot precision at higher zooms and add further functionality:

1. Plot zooms down to 1 cm radius, and plot should remain precise at that level.
2. Plot now displays both average and standard deviation of position (the latter assumes the Python statistics package is available - it's standard on most platforms).
3. User can set center point to Average or Fixed.
4. In addition to mouse wheel zooming, user can now right-click anywhere in the plot to set fixed reference coordinates to that position (this is as close as I can get to a 'click and drag' facility with reasonable endeavours).
5. Double-click clears the plot, as before.
6. Improved font scaling when full screen.

[nseidle]

Looks amazing! Let me know when there's an RC.

[semuadmin]

@nseidle @gdt For your scattering pleasure - bouquets and brickbats welcome...

#164

python3 -m pip install git+https://github.com/semuconsulting/pygpsclient@RC-1.4.27

At risk of stating the obvious, for maximum precision I'm guessing you'll be using NAV-HPPOSLLH messages, in which case the full 10dp precision will be preserved in the plot (at least as far as pixel resolution allows). NAV-PVT typically only gives 7-8dp and ECEF and NMEA messages (GGA etc) will all involve some degree of rounding from the original ecef or dms format.

[nseidle]

(... googles brickbats. Oh, I must use that in a sentence today!)

Fantastic! I am superbly happy.

Above is a RTK Torch under test connected to a helical style antenna. The green dot is the known antenna location (1.1mm error on lat, 1.0mm on lon according to the CSRS). The orange is a ~2.5mm grouping of RTK Fixed positions based on RTCM generated by our base station with a baseline of ~10 meters.

Now we can test RTK engines and antenna types and qualitatively, edging on quantitively, compare performance. We gather data over a series of days to see repeat performance.

Neato bandito! Thanks @semuadmin.

[gdt]

I am installing the code right now. @nseidle You say "known" and then you say CSRS. I suspect you are not in Canada, and that you are using a PPP solution. I am skeptical of such an accuracy claim of PPP. What I would be going is gathering 24h of raw observations, converting to RINEX (at 30s intervals probably) and then processing with OPUS against your nearest CORS stations. This is going to get you the position of the (average over your measurements somehow) APC, not the ARP. Then you can use whatever RTK sources and see how that goes. Being careful about NAD83(2011) epoch 2010.0 vs ITRF2020 current epoch or whatever. You may want to get known coordinates with a calibrated antenna, that you can put into OPUS.

Of course, you said "local base", so that brings up "how do you know those coordinates".

You have really great clustering. I would want to run that for 48h.

[semuadmin]

You have really great clustering. I would want to run that for 48h.

FYI doing some final performance testing to see what the latency is like with large point clusters, but I suspect it may struggle to handle 48 hours worth of continuous in-memory data (i.e. around 180,000 points) in real-time. What it does currently is truncate the data to 5000 points (arbitrary value) by randomly removing individual points (leaving the first and last intact), but I can consider other options. You could, of course, collect the data in segments.

For very long surveys, post-processing of logged data is probably the way to go.

[gdt]

The good news is that this seems to work for me and the first-time user installation/startup was remarkably smooth. I've got a few random comments, many of which are not about scatter plot. I am guessing issues for things that I sort into bug or feature request, and discussion for things that are complicated? After trying harder to read help of course.

[semuadmin]

suggest ask in Discussion in the first instance

[semuadmin]

Delivered in version 1.4.27 now available on PyPi