Welcome to InterProcessPyObjects Discussions!

[FI-Mihej]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[JonathanParkes]

Hey mate, love your work. this project seems to be exactly what I'm looking for.

Would be interested in a simple, minimal example, especially one not using Async.

The idea I'm working towards is to have two distinct processes (launched from different python shells) using the same named shared memory.

In a nutshell, recreating this example from the python docs: https://docs.python.org/3/library/multiprocessing.shared_memory.html
(second example, NumPy array). Instead of a NumPy array, using a user-defined object (like 'SomeSharedObject' class you have in your example).

hope this makes sense ❤️

[JonathanParkes]

Thanks @FI-Mihej , I'll take a look and give it a shot 😄 .
Also, I'm looking forward to the multi-process sharing that you're working on. It will definitely be helpful for some one-to-many tasks I can think of!
Take it easy 👍

[JonathanParkes]

Hi @FI-Mihej , thanks again for the example. I gave it a shot and it all seems to be working, however, there's a slight issue:

Sender - done.
Receiver - done.
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 239, in main
cache[rtype].remove(name)
KeyError: '/test_shared_memory'

Notes:

• I renamed the shared_memory_name as it was getting a 'too long error'
• no other modifications to the file were made.
• I'm running Python 3.12.3
• I'm on a MacBook Pro, M3 Pro laptop.

Interestingly, the main loop doesn't actually crash, it just shows the err.

On a side note, how's the several-processes work coming along?

thanks in advance 😄

[FI-Mihej]

Hi @JonathanParkes , try 'test_shared_memory' instead. SharedMemory's code adds a leading '/' just before unlinking automatically (on Posix platforms). I'll review and maybe improve naming schema a bit later.

"On a side note, how's the several-processes work coming along?"
It works fine. Same as a week ago: I was not able to continue work on code refactoring these days. :) I plan to release it during the upcoming week. :)

[JonathanParkes]

Cheers for the update mate. I did some digging and I think the err is a red herring. I forced '/' not to be used and delved into the unlinking messages looking for where the '/' was added just to check.

The issue seems to be when using the context manager on the consumer side. Removing the context manager and instead explicitly calling the functions as follows:

receiver: SharedMemory = SharedMemory(
        name=shared_memory_name, create=False, size=200 * 1024 * 1024
    )
    receiver.init_consumer()
    with wait_my_turn_when_has_messages(receiver, time_limit=5):
        data = receiver.take_message()
        print(data)
    receiver.close_consumer()
    # receiver.close(). # this line, when uncommented, causes the KeyError in the resource tracker.

removing receiver.close() from the call makes the program appear to run fine. The resource tracker doesn't complain. When it's included, however, the err pops up.

TL;DR, my question is, on the receiver side, can 'receiver.close()' be ignored? Is it safe to? (no sections of memory left un-allocated, etc). I'm assuming the memory is cleaned up by the producer context manager, which it seems to be.

[FI-Mihej]

Hi @JonathanParkes , unfortunately I was much more busy than expected.

"my question is, on the receiver side, can 'receiver.close()' be ignored? Is it safe to?"

It would be better to Not ignore it. The error text from the ResourceManager (... KeyError: '/test_shared_memory ...) could be safely ignored on the other hand.

I've already made a few changes in an appropriate code. Additionally I'll check an appropriate CPython source code of different versions and different patches and test result on macOS in a few days. Python's ResourceManager's behavior highly depends on the target OS and could be different across different Python interpreter versions.