Production Firmware-Update Utility Requirements

[Poofjunior]

Some of the combined efforts from me and @PathogenDavid had me thinking I should capture the requirements for a firmware-update utility from the Allen-Institute side.

Motivation

We currently have ~99 Allen Harp Devices deployed across ~30 rigs and a few testbenches. Each of these devices is connect to a rig PC which runs the experiment. To update firmware, we need physical access to the Harp device such that we can press the BOOTSEL button and upload the firmware manually from a PC connected to the device.

We would like some form of automation to ensure that all rigs are running up-to-date firmware and to deploy new firmware across all rigs without requiring physical access to the device.

Background

We have made light efforts to try to streamline this process by:

• attaching reset functionality to unused bits in the Pico's Core Harp Registers to put a Harp device in a state where it can accept firmware initiated from a software command.
• standardizing publishing binary .uf2 files in firmware releases on the web (example 1, example 2)

but we are still missing capabilities in a few places to automate firmware updates to all Harp devices.

User Story

From the perspective of someone maintaining a collection of Production rigs, we'd like to inspect the version of the firmware running on any Harp device on any rig and possibly update specific device instances or all devices of a specific type. We'd also like the ability to periodically schedule an update such that any rig can self-update its Harp devices when new firmware is released online.

Ultimately, the number of Harp devices in production at AIND is set to grow to ~130 devices next year (at least), so we'd like a streamlined way to push fresh updates while minimizing/eliminating continuous labor to the people who maintain them.

Requirements

We'd like an automation solution (utility/script/application/etc.) with the following features:

• identify all (Allen and Harp-Tech) Harp devices and their types connected to a PC
  • Nice-to-Have: identify Harp devices and their types without needing to open the serial port and send Harp-Protocol transactions.
• be cross-platform compatible (Windows & Linux)
• support multiple deployment options:
  • be able to be deployed as a background service that can execute automatically (think: Systemd or Windows Service)
  • be able to be run manually as a utility/script/application
• support deploying firmware to devices that derive from pico and atxmega cores.
  • Really, anyone building a device that derives from one of these two cores and should be discoverable and flashable by this utility. And if their firmware is uploaded on a release page, the firmware should be downloadable from online as well.
• Update firmware across all Harp devices on a single rig either:
  • according to a specification (config?)
  • according to what is actually connected to the PC (some form of automated discovery)
• Update firmware:
  • by accessing the releases for the web
  • by being explicitly passed a firmware binary from the user.
• does not rely on physical access to the device once to update the firmware once the firmware has been released to production
  • it is OK if Harp-compatible firmware needs to be uploaded to the device for the very first time.

Assumptions

• All AIND Harp devices can be updated manually once to use a modern version of the Pico Core.
• Firmware running on Harp devices in production is functional in that it supports Harp Protocol communication to core registers. Additionally, firmware on Allen Harp devices populates usb descriptors with information hinting device type.
  • Some bugs that would lock up the device are acceptable, but, in the worst case, a powercycle can put the device in a state where it can communicate to core registers.
• Firmware published as a Release on Github is functional in that it supports Harp Protocol communication to core registers.

Examples

Here are some example usages

harp-cli --get_devices 

harp-cli --get_devices --core=pico

harp-cli --get_devices --type=lickety-split

harp-cli --get_firmware_releases --type=lickety-split

harp-cli --upload_firmware --by_id=<serial_number_here> --firmware=latest

harp-cli --upload_firmware --type=lickety-split --firmware=latest

harp-cli --upload_firmware --core=pico --firmware=latest

Bonus:

harp-cli --get_firmware_binary --by_id=<serial_number_here>  # pull uf2 off of a device directly.

More examples here!

References

• WhoAmI Repository with a yaml of all registered devices across institutes.

[bruno-f-cruz]

Not sure where else to leave this, so I’ll just continue the discussion here.

Updating Firmware in an Experiment-Driven Fashion

Given the recent push in harp-tech toward unified releases of firmware, interface, and Bonsai packages, we're in a great position to "lock" the firmware versions needed to run specific experiments. Here's what I have in mind:

• An "experiment" run via a Bonsai workflow that interacts with devices always runs within an environment (even if it’s the global one). This environment is currently fully defined by a single bonsai.config file.
• With the move toward unified releases, a given device’s Bonsai package will now include both a device.yml and the corresponding device firmware binary.
• The Bonsai.Harp core package already implements most of the firmware update logic we need.
• For the sake of argument, let’s say there are two main drivers for updating a device:
  • New features: in this case, both the software interface and firmware should be updated in lockstep.
  • Bug fixes: firmware updates may fix bugs without requiring any changes to the software interface.

The idea is simple: we treat the bonsai.config not just as the specification of the software dependencies required to run an experiment, but also as the definition of the expected hardware state for all connected Harp boards. This approach has several advantages:

• Interface and firmware stay in sync. This is huge—everything from task behavior to logging is guaranteed to be correct and self-consistent.
• We can automate firmware updates. The harp.toolkit CLI could parse the bonsai.config (since the C# package already depends on Bonsai.Core) and implement an “update all” routine that loops through all connected boards and updates any that don’t match the expected version.
• Hardware state becomes version-controlled. Environments (i.e., bonsai.config) are typically committed to Git and tracked via diffs. This allows experimenters to track firmware changes alongside Bonsai package versions. Even in cases where firmware updates do not necessitate the interface updates, at least users have the option to be explicit and track in their diffs such updates.
• Bonsai already prompts users to update packages. If new firmware is available, users could update both the package and firmware simultaneously and benefit from the consistency described above.

---

Open Questions

What about Python or other software?

The current harp.toolkit already works as a standalone CLI, so this proposal would be strictly better. Even for experiments that don’t use Bonsai, users could maintain a bonsai.config whose sole purpose is to define Harp dependencies. The harp.toolkit CLI could then resolve and apply the correct hardware state.

This approach generalizes to any software stack, since harp.toolkit is cross-platform and CLI-based.

Finding a .NET Package from a Given Board

I'm not sure this is a real problem. If software is driving the firmware version, how do we determine the correspondence between an interface and a board? A natural solution is using WhoAmI. A simple algorithm to update all boards could be:

Loop over all devices and read their R_WhoAmI and R_Version registers

For each device, find the assembly that matches the WhoAmI (assuming uniqueness)

Compare the device’s R_Version with the version specified in the corresponding interface / device.yml

If they differ, perform the update; otherwise, skip

We can support dry-run options and no-downgrade flags for safety and control.