Abstract target mcu to adafruit_blinka like interface

[Teslafly]

My use case for blinka is talking a program that ran on a raspberry pi using the blinka library

I am replacing the blinka powered raspberry pi gpio with a belay powered mcu.
(need gpio, i2c, onewire, and addressable led support)

I was thinking it would be nice if the built in interfaces of the mcu (gpio, i2c, spi, serial, etc), could be abstracted in belay to blinka drop in compatible objects

So you could replace:

# raspberry pi
import board
import digitalio

led = digitalio.DigitalInOut(board.D18)
led.direction = digitalio.Direction.OUTPUT

led.value = True

With:

# belay micropython mcu
import belay

device = belay.Device("/dev/ttyUSB0")

led = device.digitalio.DigitalInOut(device.board.D18)
led.direction = device.digitalio.Direction.OUTPUT

led.value = True

Basically mirroring the "device", "board", and "busio" interfaces from micropython into belay native objects.

I am willing to make the implementation, but was wondering if you had any suggestions about the direction to go.
I am unsure if trying to do this in the belay syntactical sugar way would work well. Or If a hardcoded version of the "device", "board", and "busio" classes that set themselves up on connection would be the easiest option. Subclassing could sort of do this if you built up a full class structure.

As far as I can tell, it's not possible to "mirror/link" an object from the mcu running micropython into the full computer side python environment.

It would be nice to do something like

device = belay.Device("/dev/ttyUSB0")
i2c = device.getreference("busio.I2C(board.SCL, board.SDA)")

And then use "i2c" as a native object that I can pass to device drivers and such. With tab autocompletion when using something like ptpython.
Which means that it would need to be aware of the full underlying tree structure of the object.

Being able to reference any object in that complete of a way seems like it might be difficult to me. So I think only having the "device", "board", and "busio" objects linked would be the most useful and easier to implement.

Thoughts?

[BrianPugh]

What we currently have (ProxyObject) is partially the way there.

Where it starts to break down is when the input/outputs are also objects. For it to truly work the way you intend, we would need to be able to pass references of objects back and forth between CPython and MicroPython. I'll think about it for a little bit!

[BrianPugh]

So I started implementing this, but quickly ran into a problem.

My approach is to also return an integer identifier for all object returns. If the repr of the created object cannot be parsed by ast, then fallback to creating a ProxyObject that references the object by its identifier.

This, however, would make it super easy to leak memory. Consider the example:

import belay

device = belay.Device("/dev/tty.usbmodem12401")
device("import neopixel")

neopixel = device.proxy("neopixel")
Pin = device.proxy("Pin")
pixel = neopixel.NeoPixel(Pin(16), 1)

The Pin(16) object would create the object on device, but it would also have to store it to a "secret" object (e.g. __belay_secret_object_0 = Pin(16)). In the next call, we would send a command like:

pixel = neopixel.NeoPixel(resolve("__belay_secret_object_0"), 1)

Because the __belay_secret_object_0 exists, this will work. However, consider the following:

some_proxy_object = device("object()")  # a ProxyObject handle for the result of the expression
device("object()")  # we don't use the return value, but now this object "permamently" exists in micropython memory
device("object()") # same as above

We basically need to manually implement a garbage collector on the python side. Requires more thought...

[Teslafly]

Thanks for considering this and pointing out the proxyobject. The GC problem is part of why I figured grabbing arbitrary nested objects would be somewhat hard.

If I'm understanding this right, you need to connect the deletion of an object in the upper level python to it's deletion in the micropython context. Could you add a custom __del__ to shared objects that deletes the object in the micropyhton context when the upper level side reference is destroyed?

In the device("object()") context, the object would be immediately destroyed after the init call. This would propagate into the micropython context.
The issue I think would be in nested objects? I believe when del is called, it calls the __del__ function of the top level class, executes it's code, then calls __del__ on all the object's subobjects that don't have a remaining reference as part of GC.

The main issue I can see with just using __del__ of the top level context, is when you pull out a reference for a sub object and then delete the original context.

However, you definitely have a much better understanding of python and micropython internals than me.

[BrianPugh]

So, that was the exact initial thought I had! Yesterday it seemed like an unworkable idea, but today it seems not too bad? It all might sort of "just work." My initial work is in this branch. I'm going to be pretty busy the next few days, but I'll try and get back to it next week!

[BrianPugh]

@Teslafly I just had a bit of time to work on this, checkout #183 and give it a try. I still need to fix all the unit tests and stuff, but I got my example program working. Let me know what you think!

[BrianPugh]

also going to tag @kc64 if you could give #183, since you were the original requester of proxy objects. Basically if a micropython expression returns a non-standard-ast-object, a ProxyObject will be created on the CPython side that controls the remote object quite well.

[Teslafly]

Gave it a quick try, and need to troubleshoot imports in my virtual-environment after installing belay from local branch using poetry.

with from ptpython.repl import embed

I get

Traceback (most recent call last):
  File "C:\work\setup\Micropython RP2350\blinkatest\bkinka_bridge_mcp23017.py", line 11, in <module>
    from ptpython.repl import embed
  File "C:\work\.venv\Lib\site-packages\ptpython\__init__.py", line 3, in <module>
    from .repl import embed
  File "C:\work\.venv\Lib\site-packages\ptpython\repl.py", line 35, in <module>
    from prompt_toolkit.formatted_text import OneStyleAndTextTuple
ImportError: cannot import name 'OneStyleAndTextTuple' from 'prompt_toolkit.formatted_text' (C:\work\.venv\Lib\site-packages\prompt_toolkit\formatted_text\__init__.py). Did you mean: 'StyleAndTextTuples'?

I wouldn't put too much stock into that though. It's very possible I messed something up in my environment.
I will be able to troubleshoot more tomorrow.

This is the full script I used, targeting a circuitpython RP2350 board connected to a MCP23017 gpio expander:

# local mcp32017 driver using remote belay bus device

# setup
# pip3 install adafruit-circuitpython-mcp230xx


import time
import belay

from ptpython.repl import embed

from adafruit_mcp230xx.mcp23017 import MCP23017

device = belay.Device("COM7")

neopixel = device.proxy("import neopixel")  # import statements are now also detected, and ProxyObjects are created.
Pin = device.proxy("from machine import Pin")
pixels = neopixel.NeoPixel(Pin(7), 15)


for i in pixels:
    pixels[i] = (42, 0, 0)
pixels.write()


board = device.proxy("import board")
busio = device.proxy("import busio")
digitalio = device.proxy("import digitalio")

i2c = busio.I2C(board.SCL, board.SDA)
mcp = MCP23017(i2c)  # MCP23017

pin0 = mcp.get_pin(0)
pin1 = mcp.get_pin(1)

# Setup pin0 as an output that's at a high logic level.
pin0.switch_to_output(value=True)

pin1.direction = digitalio.Direction.INPUT
pin1.pull = digitalio.Pull.UP

pin0.value = True
time.sleep(0.5)
pin0.value = False

# drop to ptpython shell w shared history for debugging
embed(
    globals(),
    locals(),
    vi_mode=False,
    history_filename="cache/ptpython_local_dev_history.txt",
)

[Teslafly]

Here is some debugging where I avoid the ptpython conflicting package.
This is a RP2350 running circuit python with the suggested belay adjustments.

>>> import belay
>>> belay.__version__
'0.0.0'

# pip3 install adafruit-circuitpython-mcp230xx
>>> from adafruit_mcp230xx.mcp23017 import MCP23017

>>> device = belay.Device("COM7")

>>> board = device.proxy("import board")
>>> busio = device.proxy("import busio")
>>> digitalio = device.proxy("import digitalio")
>>> i2c = busio.I2C(board.SCL, board.SDA)
>>> i2c 
<belay.proxy_object.ProxyObject object at 0x000001CD6CB460F0>

>>> dir(i2c)
Traceback (most recent call last):
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 97, in __getitem__
    return device(expression, proxy=True, delete=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'dict_view' object isn't subscriptable


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 99, in __getitem__
    _promote_exception(e)
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 24, in _promote_exception
    raise TypeError from e
TypeError

>>> mcp = MCP23017(i2c)
Traceback (most recent call last):
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 115, in __len__
    return device(expression)  # Do not proxy; we want the integer value.
           ^^^^^^^^^^^^^^^^^^
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'bool' has no len()


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\.venv\Lib\site-packages\adafruit_mcp230xx\mcp23017.py", line 67, in __init__
    super().__init__(i2c, address)
  File "C:\work\setup\Micropython RP2350\.venv\Lib\site-packages\adafruit_mcp230xx\mcp23xxx.py", line 39, in __init__
    self._device = i2c_device.I2CDevice(bus_device, address)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\setup\Micropython RP2350\.venv\Lib\site-packages\adafruit_bus_device\i2c_device.py", line 62, in __init__
    self.__probe_for_device()
  File "C:\work\setup\Micropython RP2350\.venv\Lib\site-packages\adafruit_bus_device\i2c_device.py", line 165, in __probe_for_device
    while not self.i2c.try_lock():
              ^^^^^^^^^^^^^^^^^^^
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 117, in __len__
    _promote_exception(e)
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 24, in _promote_exception
    raise TypeError from e
TypeError



>>> board.LED
<belay.proxy_object.ProxyObject object at 0x000001CD6FD2AFF0>
>>> dir(board.LED) 
Traceback (most recent call last):
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 97, in __getitem__
    return device(expression, proxy=True, delete=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'dict_view' object isn't subscriptable


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 99, in __getitem__
    _promote_exception(e)
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 24, in _promote_exception
    raise TypeError from e
TypeError


>>> ledobj = digitalio.DigitalInOut(board.LED) 
>>> ledobj.direction = digitalio.Direction.OUTPUT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 90, in __setattr__
    device(f"{target_name}.{name} = {value!r}")
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax

>>> ledobj.direction = digitalio.Direction.OUTPUT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 90, in __setattr__
    device(f"{target_name}.{name} = {value!r}")
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax

>>> outdir =  digitalio.Direction.OUTPUT
>>> ledobj.direction = outdir
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\work\setup\Micropython RP2350\belay\belay\proxy_object.py", line 90, in __setattr__
    device(f"{target_name}.{name} = {value!r}")
  File "C:\work\setup\Micropython RP2350\belay\belay\device.py", line 384, in __call__
    self._board.exec(cmd, data_consumer=data_consumer)
  File "C:\work\setup\Micropython RP2350\belay\belay\pyboard.py", line 605, in exec
    raise PyboardException(ret_err.decode())
belay.pyboard.PyboardException:

Traceback (most recent call last):
  File "<stdin>", line 1
SyntaxError: invalid syntax

dir() on an object generated this way doesn't work.

Importing micropython side libraries to local variables, and then feeding them into one another seems to always end up with a syntax error.

[BrianPugh]

I'll try and investigate some of these this weekend! I'll have to make my test cases a little more difficult.