Harp 32-bit protocol thoughts (and a tangent about Harp protocol extensibility)

[PathogenDavid]

(#134 unintentionally nerd sniped me into writing out my thoughts on possible improvements to Harp. This was originally going to be a comment there but I realized that'd cause it to disappear into oblivion when that issue gets closed so I'm rambling here instead.)

If/when we do ratify the 32bit protocol we should take the opportunity to address some areas that could stand to be improved.

I don't expect any of this to be addressed or considered anytime soon, maybe not even ever. Just wanted to get some of these things out of my head.

I have a longer list somewhere, but here's some possible improvements that come to mind (with some added commentary on the 32-bit spec since I was looking at it while writing):

• Addition of frame synchronization (right now it's basically impossible to recover from the UART FIFO overrun or any communication glitches.)
  • See Ethernet frames for an example. (Just an example, they're trying to solve problems we don't have so ours would not look exactly like this.)
• Replace the checksum with something more robust.
  • A CRC variant would be a good choice. CRC-32 is the most well-known but the algorithm generalizes to arbitrary bit lengths. There's no "common standard" 16-bit CRC but CRC-16-CCITT comes close. (CRCs can be calculated easily using small lookup tables. More effort than simple addition but much more robust.)
  • On modern CPUs, xxHash is more performant than CRC and is more robust. Not sure if that's true for microcontrollers though. Could be interesting to try. (Author confirms that using only the upper/lower 16-bits of the hash is acceptable.)
• Re-evaluate having the optional header fields, they make it so you can't parse Harp messages without a state machine.
• Re-evaluate the way the Harp timestamp is encoded.
  • In both protocols, the seconds field doesn't overflow for 136.2 years. This is pretty excessive.
  • The MSB in the microseconds field of the 8-bit protocol could've been reserved. (You only need 15 bits to represent a multiple 32 microseconds totaling a second.)
  • The 32-bit protocol is wasting two bits here. (You only need 30 bits of nanoseconds to total a second.)
  • Instead we could combine the port and timestamp. 8 bits of port and 56 bits of timestamp in nanoseconds.
    • This avoids wasting a full 32 bits on the port (which doesn't feel like it warrants expanding) and makes eliminating optional fields slightly less silly.
    • The separate seconds + microseconds of the old protocol is a leaky abstraction of how the ATxmega core handles timekeeping. This isn't totally out there for an underpowered MCU, but the 32-bit protocol isn't meant for those in the first place.
    • (In contrast, the Pico core does timekeeping in a 64bit microsecond counter (which reflects how the timer hardware works on the RP2040), so it's actually having to go out of its way to accommodate this scheme.)

Thoughts on the 32-bit protocol specifically

• I get why little endian is used for the Harp message overview, but I feel like big endian makes more sense notation-wise. (We aren't accessing registers in memory, we're reading/writing streams of bits.)
• The counter should become before the checksum (and be included in it)
  • Also maybe this should be called the sequence number, I feel like that's more common in protocol lingo.
• I like the general idea of the Flag32 flag in MessageType (making the protocol compatible), but I feel like generally devices will either speak Harp8 or Harp32, but this implementation requires both (and requires resolving weirdness like when a Harp32 devices receives a Harp8 message that would require a Harp32 message to reply to.)
  • It also interacts poorly with frame synchronization (which is usually implemented as a prefix to avoid dropping the first packet upon resync.)
  • It might just be easier to use the device.yml to know which protocol to speak, but we could also negotiate the protocol (see below.)

Protocol negotiation

If using device.yml is undesirable (IE: if an existing Harp devices is upgraded from Harp8 to Harp32), Harp clients negotiate could be mandated to negotiate with the device upon initial connection. Rough implementation would look like this:

1. Client sends a Harp8 message with the following:
   • MessageType = Read
   • Address = R_PROTOCOL_CAPABILITIES
   • PayloadType = u8[], no timestamp
   • Payload = a Harp32 message with the same fields (but no payload)
2. Harp8 devices will ignore the payload since this is a read message (ATxmega core ignores it completely. Pico core indirectly exposes it to application read handlers, which should be fine. The Harp8 spec should be amended to explicitly require this.)
   • They will reply with their R_POROTOCOL_CAPABILITIES as described below.
   • (Older Harp8 devices will respond with an error because they don't support this register.)
3. Harp32 devices will either:
   • A) Be special-cased to recognize the Harp8 header sequence of this register specifically when they're expecting framing or
   • B) Interpret the header as bus de-synchronization and re-synchronize at the Harp32 dummy message contained within the Harp8 message's payload. (TODO: How to handle the Harp8 CRC in this case? Maybe unavoidable special-case or just let it resync again lol.)
   • In either case, the device responds with its R_POROTOCOL_CAPABILITIES
4. The client uses the response to determine A) if it's capable of speaking to the device and B) how to speak to it

R_POROTOCOL_CAPABILITIES replies with a list of lists of capability IDs (variable-length register proposal intensifies](https://github.com/KhronosGroup/glTF/tree/5de957b8b0a13c147c90d4ff569250440931872f/specification/2.0#specifying-extensions)).

A list is terminated either by the end of the payload or a 0x00.

The first list contains any required capabilities (IE: clients should cease communication if they don't recognize anything here.)

The second list contains any optional capabilities (IE: clients can ignore anything they don't understand and may opt in to using ones that they do.)

The separation is important to allow clients to know if they're actually capable of talking to the device. Utilizing optional capabilities is implementation-defined for each capability. The separation also allows a capability to be either (such as a device which was retrofitted to use Harp32 but still supports Harp8 in order to support old clients.)

(This is inspired by the glTF extension mechanism.)

So if you imagine we have the following capabilities defined:

• 0x01 - Harp32
• 0x02 - Harp8Framing (A theoretical Harp8 revision that includes framing)
• 0x03 - Harp8Crc8 (A theoretical Harp8 extension that replaces the checksum with CRC-8.)
• 0x04 - Harp8Nanoseconds (A theoretical Harp8 extension that uses a 64-bit nanosecond timestamps.)

A Harp32 device would respond with [0x01] to indicate it requires Harp32 and provides no optional capabilities.

A Harp8 device might respond with [0x02, 0x00, 0x03] to indicate that it requires Harp8 framing and optionally supports the Harp8 CRC-8 extension. (So a client must use Harp8 framing, but only has to opt-in to using CRC-8 instead of sum complement if knows how.)

A different Harp8 device might respond with [0x02, 0x03, 0x04] to indicate that it requires Harp8 framing, the CRC-8 extension, and uses nanosecond timestamps.

A legacy Harp8 device would respond with an error, which shall be interpreted as an empty payload for the purposes of the protocol negotiation.

One thing important to highlight is that the definition of a sublist always terminates on 0x00. This means that [0x02, 0x00, 0x03, 0x00, 0x42] contains three lists, and the purpose of the third list is undefined (and must be ignored.)