settings: Best way to implement an EEPROM-backed, CAN-accessible shared data model in Zephyr?

[towen]

Hi,

I am designing a system that involves quite a lot of data sharing between different threads, ISRs and modules.
The idea is to replace global variables with a flexible data store which enables loose-coupling between modules, for ease of maintenance and ease of testing.

My requirements are:

• Needs to be memory-efficient, zero-copy if possible.
• Must be hierarchical i.e. items can be directories. We should not have to store the full path for each item if possible.
• Must be introspectable - I would like to implement shell commands to display all of the items in the data store and configure their values
  • All data items need to be queryable via CAN as well as the shell, so data items need a unique numeric ID as well as a string key
    • The unique ID does not need to be kept consistent between firmware versions, but it could be handy
  • Key names should be const (in ROM) to minimise RAM use
• Must be thread-safe i.e. directories and other non-atomic types will have mutexes attached
• Must be able to store floats as well as arrays-of-floats and 2D matrices
• It doesn't matter if retrieving a reference to a data item (publish or subscribe) is slow, but getting or setting its value must be as fast as possible
  • It would be useful to know if a data item has been updated by another thread (i.e. publish/subscribe model) but not critically important
• Some of the data items or directories (not all) need to be saved back to EEPROM on shutdown and loaded on boot
  • Data items should be stored with their key-names in EEPROM, so that order of items in EEPROM is not important and adding new items doesn't break the format
  • Size in EEPROM is not a major issue, we have 1Mb (128kB) of EEPROM, and 128kB of RAM..
  • Data will be loaded ONCE from EEPROM on start-up, and then committed back to EEPROM on shutdown, or on command.
  • It must be possible to change and use data values for sharing data between threads, without touching the EEPROM.
• Dynamic allocation is OK on start-up only, so malloc is OK until just before the run state, and free will never be needed.
• I don't YET need userspace support, but it would be useful for the future if the data could live in a shared memory area.

I have considered the settings subsystem and CANopenNode, but it seems to be overcomplicated, and it does not work with EEPROM. Also, if I'm not mistaken, there is no way to use the values in RAM without writing them out to non-volatile storage, which is slow.

One option is for me to implement an EEPROM backend for the settings, if the settings subsystem fits the rest of the above requirements.

Another option would be to implement it manually. I have some prototype code working using linked lists or even a lightweight hashmap library, but it feels like I am reinventing the wheel.

Any advice and suggestions please?
Thanks
Tom

Related: #38841

[martinjaeger]

Hi @towen,

We've had a very similar problem a few years ago and didn't find any existing method that fulfilled our needs, so we developed an easy to understand protocol called ThingSet, which allows to manipulate and report device data using JSON (default) or CBOR (for extremely low memory requirements like for CAN or LoRaWAN).

There is a C/C++ ThingSet library which can be included as a Zephyr module. The data objects which should be exposed via ThingSet are configured in the application, see this example for our MPPT charge controller firmware.

Currently, passing the data to different interfaces like UART (for shell-like access), CAN, Bluetooth and LoRa is still part of the application, but we're working on an integration into above mentioned library, so that each application would only require a few lines of code to make the whole data access layer work.

All data is allocated statically. You can use different types of non-volatile storage (e.g. Zephyr eeprom or nvs API, depending on the MCU, example here) to retrieve stored data at start-up. This is basically the same process as sending out data over the wire. The specified data is serialized to CBOR for storage of data and de-serialized at start-up to initialize the associated variables.

Looking through your requirements, these are the aspects which are not (yet) completely fulfilled out of the box:

• No 2D matrices can be stored, only 1D arrays (with arbitrary variable type including float). But as in C arrays and matrices result in the same memory layout anyway, it should be easy to work around this.
• The library itself is not thread-aware, i.e. it assumes that the application takes care that data is accessed only in an atomic fashion. This can be easily achieved with on-board Zephyr kernel functions.
• Storing and retrieving control values very quickly could possibly be improved, depending on what "as fast as possible" really means. Currently the array of data objects has to be iterated after reception of a message to find the pointer to the actual data in memory. For many data objects, this may take a few cycles.

Feel free to ask any further questions you may have.

Martin

[towen]

Very interesting, thanks!
When you say all data is allocated statically, how does the system represent local copies of data that is located on remote devices?
What happens if a remote device (connected via CAN) or the EEPROM has an additional item of data that you weren't aware of?

When I said "fast access" and "zero copy" I meant so that the publishing thread of a data item, after it has initialised itself and retrieved a pointer to where its data will reside in memory, it can simply write to it like a "const pointer" to memory. For CAN messages it would be different, because we want to re-use the memory.

[martinjaeger]

You would have to reserve sufficient space beforehand, so e.g. if you want to store a string with arbitrary content, new data can only be as long as the amount of reserved memory in the buffer for that string.

Similar for arrays. You could reserve space for e.g. a list of devices currently connected to the CAN bus, but the maximum amount of devices would have to be fixed in advance.

For the EEPROM: If there is an unknown data item, it is ignored. This allows to add / remove data items between different firmware versions without complicated data migration as long as you don't re-use the data object IDs that have been used before.

Dealing with completely unknown data is currently not possible. The "maximum" set of data items and their semantics is hard-coded in the firmware. You can request or update subsets of this data, but you can't add entirely new data. Which would be the use case of adding completely new data to the "database" in an embedded context? How would the firmware know to make use of that data item if it doesn't know its purpose and format?

Regarding fast access / zero copy: The ThingSet database essentially just points to data in RAM used elsewhere in the firmware. So you could update this data in which ever way you want (e.g. through DMA) and use it from different contexts. Internal firmware update rates can be faster than the comms update rate, as long as you make sure that reading/writing data is atomic (at least for data which is not read/written in a single clock cycle).