How to marshal data from MicroPython to C/C++

[gentijo]

Hi @JanStaschulat I am looking for a way to store ,msg files in the Python file space then create C code that can send / receive messages. Is there existing code that can parse .msg files to produce a run time message parser / builder ?

[Gregory-Baker]

Hi @gentijo. I think integrating micropython with ros is an awesome idea. Sadly I'm no expert in micro-ros or micropython (trying to learn more). How is the integration going? Are there any major difficulties that you've found so far?

[JanStaschulat]

@gentijo Sorry for the late response. I have no experience in conversion from micro-python to C/C++.
CC @fujitatomoya, @pablogs9 do you have any experience with such conversions?

[pablogs9]

Not on my side... Sorry

[fujitatomoya]

@gentijo has been already working on that, see https://github.com/gentijo/MicroROS-RTIDL

reference: https://discourse.ros.org/t/request-for-comment-on-device-msg-compiler-for-micropython/32010

[gentijo]

Hi All, I will copy this to the README in the other project, but wanted to followup here first.

Status of Micropython integration:
Integration is in place and working, MP runs as its own freertos thread, you can interact with it either from a serial port using Thoony (Python IDE) or RShell a python based command line shell that allows you to upload code or data then interactively execute from the command line (REPL Mode). I also have webREPL working where you can do the same over Websockets, which Thoony supports. I also have basic examples of how to call a C function from MP and how to call a Python method from C.

Now I am at a point where I am tackling deeper integration with MicroROS, specifically how to effectively Publish and Subscribe to ROS Messages, Micropython is involved but the design is more MicroROS and how to support .msg files to transfer data from MicroROS to a different language.

My first approach was to create specific functions in C, i.e. mp_uros_Twist_toRos and toPython, where those two functions would #include Twist.h (produced by the IDL compiler) and would have knowledge on how to transfer the data between the two domains. I quickly discarded that approach because it would either require that all the supported ROS Types would need to have that data transfer stub created, or create the C functions as needed but that would require someone to write the data transfer code, then recompile a binary image of the code to support that message and upload the image to the device. I viewed this as being out of scope for the avg python developer.

My second approach would be to create a series of functions in the C code that knew how to build a message based on the native types, string, int, float. The on the Python side create a package that would know convert a message type (based only on the flattened version) consisting of just the native types. i.e A Twist message contains two Vector3 objects, which in themselves have 3 floats. The python module would need to know how to call the low level functions to build a message consisting of 6 floats. I did not like this approach because the Python modules would probably need to be hand coded and would be missing out on having that integration use the .msg file defining the message structure. This could be error prone and be affected by drift in msg structures.

My third approach was to go for a fully automated approach consisting of the first two approaches. Which would be, to read in the .msg files and produce what I am calling a .dti file (data transfer instructions) which will define the list of the flattened primitive types that would be used by a generic DTA (Data transfer agent implemented in the C code domain) and it would also create the Python modules to represent the types in Python as native Python objects. It took a while to remove the cobwebs from my brain on how to write Flex/Bison grammar but I now have this as a working model, where the compiler and DTA could be included as as part of the base image on the device, all you would need to do is push the .msg files to the Python device and on boot, Micropython would create all the necessary Python modules and DTI definition in an in-memory data structure. So the compiler is only invoked if there are new .msg files in a specific directory on the device. This approach seemed to be the best for someone who does not have to time to understand all the underlying mechanics. The POC I create allows for people to see that process in action, demos can be more informative than just words.

I had two asks around this topic.

1. Am I simply missing something that already exists in the ROS platform to double check to see if I am needlessly recreating the wheel

2. I am starting to think hosting the compilation on the device adds unnecessary complexity and footprint. I am thinking of porting the POC back to pure Python, used the Lark grammar that already exists and can properly track the ROS versions. Where this portable program could be standalone or integrated with RShell / Thonny to compile the .msg files on the users development computer and automatically upload the artifacts produced to the device running this platform. The on-device requirements would only require the DTA and a function to read a .dti file (really just a CSV file) and build the in-memory version of it.

Since the implementation uses Jinja templates to produce output, the same implementation could be used to support a Javascript implementation with just a change in templates. I wanted something that could be utilized for the MicroROS platform in general and not be specific to the Micropython integration.

For ref:
RShell -> https://github.com/dhylands/rshell
Thonny -> https://thonny.org/

-John