Connection state mismatch edge-case

[mzealey]

I've not been able to figure out the root cause of this edge case (stompman 2.1.0/python 3.13) but it seems to occasionally be able to get into a broken connection state where most if not all attempts at starting a txn end up with the following traceback:

      async with self.client.begin() as txn:
                 ~~~~~~~~~~~~~~~~~^^
    File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
      return await anext(self.gen)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/client.py", line 139, in begin
      async with Transaction(
                 ~~~~~~~~~~~^
          _connection_manager=self._connection_manager, _active_transactions=self._active_transactions
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      ) as transaction:
      ^
    File "/usr/local/lib/python3.13/site-packages/stompman/transaction.py", line 21, in __aenter__
      await self._connection_manager.write_frame_reconnecting(BeginFrame(headers={"transaction": self.id}))
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_manager.py", line 192, in write_frame_reconnecting
      connection_state = await self._get_active_connection_state()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_manager.py", line 166, in _get_active_connection_state
      connection_result = await self._connect_to_any_server()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_manager.py", line 143, in _connect_to_any_server
      connection_result = await lifespan.enter()
                          ^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_lifespan.py", line 86, in enter
      connection_result = await self._establish_connection()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_lifespan.py", line 82, in _establish_connection
      self.set_heartbeat_interval(server_heartbeat)
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.13/site-packages/stompman/connection_manager.py", line 86, in _restart_heartbeat_tasks
      self._send_heartbeat_task = self._task_group.create_task(
                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
          self._send_heartbeats_forever(server_heartbeat.want_to_receive_interval_ms)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      )
      ^
    File "/usr/local/lib/python3.13/asyncio/taskgroups.py", line 191, in create_task
      raise RuntimeError(f"TaskGroup {self!r} is finished")
  RuntimeError: TaskGroup <TaskGroup entered> is finished

The activemq/artemis service reports that there are connections that just keep timing out along the lines of:

2025-07-24 11:52:37,818 WARN  [org.apache.activemq.artemis.core.server] AMQ222067: Connection failure has been detected:  [code=REMOTE_DISCONNECT]
2025-07-24 11:52:38,439 WARN  [org.apache.activemq.artemis.core.protocol.stomp] AMQ332069: Sent ERROR frame to STOMP client 10.141.93.122:42308:
2025-07-24 11:52:38,439 WARN  [org.apache.activemq.artemis.core.server] AMQ222067: Connection failure has been detected:  [code=REMOTE_DISCONNECT]
2025-07-24 11:52:42,949 WARN  [org.apache.activemq.artemis.core.protocol.stomp] AMQ332069: Sent ERROR frame to STOMP client 10.141.93.122:42332: AMQ229014: Did not receive data from 10.141.93.122:42332 within the 2000ms connection TTL. The connection will now be closed.
2025-07-24 11:52:42,949 WARN  [org.apache.activemq.artemis.core.server] AMQ222067: Connection failure has been detected: AMQ229014: Did not receive data from 10.141.93.122:42332 within the 2000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

This was on a service with 2-3 identical containers running. Only 1 got into this state and only restarting it fixed the issue.

[vrslev]

Hi!

The traceback indicates you're operating on a client whose connection has already been terminated, likely due to the timeouts. This is why the underlying TaskGroup is in a finished state (it got there likely because Client.__aexit__ was called).

To handle this, you can implement a health check. Periodically call Client.is_alive() and shut down the application if it returns False. This behavior is similar to the FastStream integration. You can also configure connection retry parameters when initializing the Client to make it more resilient to transient failures.

[mzealey]

Thanks for the pointers. I'll run some more tests and see. A couple of things that might be nice in the lib either way:

1. Unlimited retries - if this is a background service or similar I just want the service to keep running no matter what happens to one or more of the servers it's connected to - just keep trying to retry (and I can add healthchecks for our own monitoring if we care if messages get backlogged in memory because of this)
2. I'm running this with a separate async task doing the connection (async with stompman.Client(): ...) and subscriptions, which cause that background task to basically run forever, but messages are added from within fastapi via http requests so we have lots of different async tasks interacting. I would have expected that if there was a disconnect the async with would end, but from what I can see of the logging we have when this happens, this hasn't been the case in this situation, perhaps because it thinks the subscriptions are still active? So we seem to be in a bit of a half state where creating txns is failing with a client not connected error, but the Client itself is not leaving the context manager?

[vrslev]

1. That's a potential improvement, I guess. We would need to implement some kind of smart enough backoff. Also, it may be desirable for some to use circuit breaker. Perhaps, we should allow to specify connect retry strategy.
2. Half-conected state is concerning. I couldn't reproduce the issue: it just raises an exception and exits. It would be nice to see minimal reproducible example.

[mzealey]

OK after reviewing my code in more detail I think the half-state issue is my fault - I was saving the client reference outside of the context manager so I believe it was hitting that stale when disconnected because I had forgotten to None it after context manager. So I guess we can close this ticket - sorry about that.

But I still think that the infinite retry case would be quite useful for services, then if configured thusly there's no need for the client side to handle any connection-related complexity just assume there will always be a connection or future connection

[vrslev]

Agreed, that can be useful in some cases. But potentially hard to understand: for example, when non of the servers were available ever

[vrslev]

I’m not sure how to implement simultaneously make sure users know that servers are unavailable and add infinite retries

[mzealey]

A few thoughts:

1. I'm not suggesting to change the current behavior defaults, merely support a connect_retry_attempts=-1 type behaviour if client wants it
2. You could have a flag (on by default) that issues a logger.warning or info or something on every connection attempt?
3. Otherwise, as you said previously the user can always call Client.is_alive() if they want to know the state of the conns, which presumably a client doing (1) should be strongly encouraged to do

[vrslev]

1. Yes, but currently there’s a handmade incremental backoff. If we reconnect forever, attempts will be to rare.

Also, I don’t like -1 as special value, perhaps we should just add a sentinel, or suggest using math.inf

2. Logging can also be useful for other cases. I don’t yet have a good concept for logging in libs