Getting org.freedesktop.DBus.Error.LimitsExceeded error after hours of running

[dowens-pp]

Greetings. I recently started getting the org.freedesktop.DBus.LimitsExceeded error in my applications logs after hours of running, effectively halting all dbus communication. Prior to this, everything dbus related in my application seemed to be working exactly as expected. The only semi-recent change related to my sdbus-c++ code was to upgrade to 2.0.0 (I'm using Yocto to build for my ARM target). E.g., replacing processPendingRequest() with processPendingEvent().

I looked into increasing dbus' max_replies_per_connection limit to see if that makes a difference but feel like that would only delay the problem if replies really aren't getting processed. Other than tweaking dbus settings, I'm not really sure how to go about tracking down this issue. Any recommendations?

Thanks!

[sangelovic]

@dowens-pp

Hi, I've never seen such an error. Here is a series of some thoughts of mine and questions:

• Are you using enterEventLoop() or enterEventLoopAsync()? I guess these methods thrown the exception in your case?
• What message is inside that exception? I.e., what string does what() return? If the answer to the question in the above bullet point is yes, and the exception message starts with "Failed to process bus requests", then we know that processPendingEvent() is the source.
• Internally, processPendingEvent() calls sd_bus_process() in sd-bus here and if that function returns ENOBUFS, then sdbus-c++ translates this on the next line into the org.freedesktop.DBus.LimitsExceeded exception and throws it with that message.
• This could mean that buffers inside underlying sd-bus implementation (implemented and maintaned by libsystemd) get full. The question is then why? sd-bus has two buffers, an inbound and outbound one. Is it the inbound (i.e. the incoming messages are piling up, unprocessed) or outbound one (the outgoing messages are not really sent out).

To be continued...

[dowens-pp]

Thanks for your reply. I'm actually using libevent to manage my event loop in conjunction with getEventLoopPollData(). I have narrowed the method generating the exception down to getPropery() or callMethod() - sorry, still adding debug and watching logs since the problem is difficult to reproduce. The full exception message is:

[org.freedesktop.DBus.Error.LimitsExceeded] The maximum number of pending replies per connection has been reached

This matches what I see using busctl monitor to watch dbus directly.

[sangelovic]

OK, this looks like an error issued by the D-Bus broker. And OK, it's not inside processPendingEvent(). The exception is thrown by a D-Bus proxy object in the same application where the problematic server (which does not reply apparently) is located? Do they share the same D-Bus connection? And you're calling some other D-Bus server here, right? Or is the getPropery() or callMethod() taking place in another process and with them you are calling that problematic server?

I'm thinking hard about what could be happening... And it appears to me as if your application stopped at some point processing the incoming method calls, so it stopped providing replies to the callers. Can you confirm that? Did the incoming method calls reach the callbacks in your application? I suppose not. Could it be that your event loop does not wake the event loop thread up upon incoming D-Bus messages (method calls)? Could it be that it stalls/freezes inside poll() or similar, while the incoming messages get stacked in the inbound queue? Can you verify (by debugging e.g. -- when you stop the debugger, the thread will sleep inside poll() while your callbacks have not been invoked as they should)...?

[sangelovic]

You can try to artificially increase the D-Bus traffic, i.e. put that server under much higher stress. Maybe you will reproduce the issue much faster...

[sangelovic]

I guess when an event comes from libevent, you call IConnection::processPendingEvent() once and then go to poll again? If so, when IConnection::processPendingEvent() returns true, you may try to call it again and again in a loop until it returns false. And try to reproduce with this. Let me know then if it works without issues anymore.