Semaphore overflow

[khromenokroman]

Operating system:
Debian12

@elBoberido Good afternoon, I ran into this problem, when working for more than 2 hours for a long time, I get the following error

июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/semaphore_interface.cpp:56 { iox::expected<void, iox::SemaphoreError> iox::detail::SemaphoreInterface<SemaphoreChild>::post() [with SemaphoreChild = iox::UnnamedSemaphore] -> iox_sem_post }  :::  [ 75 ]  Value too large for defined data type
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Error]: Semaphore overflow. The maximum value of 2147483647 would be exceeded.
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/condition_notifier.cpp:44 [Fatal Error] [POPO__CONDITION_NOTIFIER_SEMAPHORE_CORRUPT_IN_NOTIFY (code = 78)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro audit[2993412]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993412 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro audit[2993466]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993466 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro audit[2993654]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993654 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected<void, iox::MutexLockError> iox::mutex::lock() -> iox_pthread_mutex_lock }  :::  [ 130 ]  Owner died
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent state again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resources were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LOCKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2993340]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993340 comm="ccproc.3" exe="/usr/bin/context-core" sig=6 res=1
июл 20 16:56:22 yadro audit[2995005]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2995005 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/condition_notifier.cpp:44 [PANIC]
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected<void, iox::MutexLockError> iox::mutex::lock() -> iox_pthread_mutex_lock }  :::  [ 130 ]  Owner died
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent state again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resources were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LOCKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2994885]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2994885 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected<void, iox::MutexLockError> iox
::mutex::lock() -> iox_pthread_mutex_lock }  :::  [ 130 ]  Owner died
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent s
tate again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resourc
es were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LO
CKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2994223]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2994223 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected<void, iox::MutexLockError>
 iox::mutex::lock() -> iox_pthread_mutex_lock }  :::  [ 130 ]  Owner died
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consiste
nt state again with Mutex::make_consistent()
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the res
ources were cleaned up by RouDi due to an unresponsive application.

I'm sending the event like this:

static thread_local auto current_tid = gettid();
    auto try_res = publishers.find(current_tid);
    if (try_res == publishers.end()) {
        try {
            auto [fst, snd] = publishers.emplace(std::piecewise_construct, std::forward_as_tuple(current_tid),
                                                 std::forward_as_tuple(iox::capro::ServiceDescription{
                                                     "SES", {TruncateToCapacity_t(), context.empty() ? "_all_" : context.c_str()}, "Event"}));
            try_res = fst;
        } catch (std::exception const& ex) {
            sesc_logger.StdException(::boost::core::demangle(typeid(ex).name()), ex.what());
            return StatusPublish::ERROR;
        } catch (...) {
            return StatusPublish::ERROR;
        }
    }

    try_res->second.publish(event_buffer);

I get the event like this:

SesConverter::SesConverter(::logs::app& logger, std::string const& module, std::shared_ptr<telemetry::IStatistic> statistics,
                           std::shared_ptr<tools::Processor> processor)
    : m_iox_runtime{iox::runtime::PoshRuntime::initRuntime(
          {TruncateToCapacity_t{},
           module == "_all_" ? sextfunc::Cef::APP_NAME : ::std::string(sextfunc::Cef::APP_NAME).append("_").append(module).c_str()})},
      m_statistics{std::move(statistics)},
      m_processor{std::move(processor)},
      m_logger{logger},
      m_iox_listener{::std::make_unique<::iox::popo::Listener>()},
      m_ses_subscriber{new iox::popo::UntypedSubscriber(iox::capro::ServiceDescription{"SES", {TruncateToCapacity_t(), module.c_str()}, "Event"},
                                                        iox::popo::SubscriberOptions{.queueCapacity = 100000, .historyRequest = 10})} {
    m_iox_listener->attachEvent(*m_ses_subscriber, iox::popo::SubscriberEvent::DATA_RECEIVED, iox::popo::createNotificationCallback(ses_event, *this))
        .or_else([](auto) {
            std::cerr << "unable to attach subscriber_ses" << std::endl;
            std::exit(EXIT_FAILURE);
        });
}

... other code

while (subscriber->take().and_then([subscriber, ses_converter](auto& sample) {
        ::ses::des::DeserBuffer deser_buffer(sample);
        [[maybe_unused]] auto const event_id = deser_buffer.get<uint32_t>();
        switch_event(event_id, *ses_converter, deser_buffer);
        subscriber->release(sample);
    })) {
    }

I really want to understand what needs to be done so that there is no problem, I need your help :))

[khromenokroman]

@elBoberido this problem was discovered experimentally, I conducted several experiments
when the publisher sends and the subscriber does not have time to process, gaps appear and semaphores begin to grow.