assertion failure in CAHitNtupletAlpakaPhase2@alpaka:hltPhase2PixelTracksSoA

[dan131riley]

We're getting assertion failures in many of the 2??34.0 series workflows. Prime suspect is #48921.

cmsRun: /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/src/HeterogeneousCore/AlpakaInterface/interface/OneToManyAssoc.h:83: void cms::alpakatools::OneToManyAssocBase<I, ONES, SIZE>::count(const TAcc&, I) [with TAcc = alpaka::AccCpuSerial<std::integral_constant<long unsigned int, 2>, unsigned int>; I = unsigned int; int ONES = -1; int SIZE = -1]: Assertion `b < static_cast<uint32_t>(nOnes())' failed.


A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thu Oct 30 02:50:33 CET 2025
Thread 7 (Thread 0x14d54c9c7700 (LWP 1065946) "cmsRun"):
#0  0x000014d59f731178 in nanosleep () from /lib64/libc.so.6
#1  0x000014d59f73107e in sleep () from /lib64/libc.so.6
#2  0x000014d599ad61ec in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d59f66247b in mprotect () from /lib64/libc.so.6
#5  0x000014d56292025d in dnnl::impl::cpu::x64::jit_generator::create_kernel() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#6  0x000014d562a09032 in dnnl::impl::cpu::x64::brgemm_inner_product_fwd_t<(dnnl::impl::cpu::x64::cpu_isa_t)71>::init(dnnl_engine*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#7  0x000014d562937520 in dnnl::impl::primitive_t::init(dnnl_engine*, bool, dnnl::impl::cache_blob_t const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#8  0x000014d562a2a47d in dnnl_status_t dnnl::impl::primitive_t::create_primitive_common<dnnl::impl::cpu::x64::brgemm_inner_product_fwd_t<(dnnl::impl::cpu::x64::cpu_isa_t)71>, dnnl::impl::cpu::x64::brgemm_inner_product_fwd_t<(dnnl::impl::cpu::x64::cpu_isa_t)71>::pd_t>(std::pair<std::shared_ptr<dnnl::impl::primitive_t>, bool>&, dnnl::impl::cpu::x64::brgemm_inner_product_fwd_t<(dnnl::impl::cpu::x64::cpu_isa_t)71>::pd_t const*, dnnl_engine*, bool, dnnl::impl::cache_blob_t const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#9  0x000014d5629032d4 in dnnl_primitive_desc::create_primitive_iface(std::pair<dnnl_primitive*, bool>&, dnnl::impl::cache_blob_t const&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#10 0x000014d56290acc5 in dnnl::impl::primitive_create(dnnl_primitive**, dnnl_primitive_desc const*, dnnl::impl::cache_blob_t const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#11 0x000014d56290b815 in dnnl_primitive_create () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#12 0x000014d55faebbe3 in tensorflow::MklDnnMatMulFwdPrimitive<float, float, float, float, float>::Setup(tensorflow::MklDnnMatMulFwdParams const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#13 0x000014d55faec9e7 in tensorflow::MklDnnMatMulFwdPrimitive<float, float, float, float, float>::MklDnnMatMulFwdPrimitive(tensorflow::MklDnnMatMulFwdParams const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#14 0x000014d55faee420 in tensorflow::MklDnnMatMulFwdPrimitiveFactory<float, float, float, float, float>::Get(tensorflow::MklDnnMatMulFwdParams const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#15 0x000014d55faf3ddb in tensorflow::MklFusedMatMulOp<Eigen::ThreadPoolDevice, float, true>::Compute(tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#16 0x000014d5573aafcb in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#17 0x000014d55749aa72 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#18 0x000014d55749bb56 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#19 0x000014d56817282c in tsl::thread::ThreadPool::Schedule(std::function<void ()>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#20 0x000014d56828596d in std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#2}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#21 0x000014d557480945 in void tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::RunTask<std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)> >(std::_Bind<void (tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::*(tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>*, tensorflow::SimplePropagatorState::TaggedNode, long))(tensorflow::SimplePropagatorState::TaggedNode, long)>&&, int) [clone .constprop.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#22 0x000014d557487bd3 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#23 0x000014d55748f5d5 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::NodeDone(tsl::Status const&, absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::NodeExecStatsInterface*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#24 0x000014d557499f2c in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ProcessInline(tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*, long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#25 0x000014d55749bb56 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#26 0x000014d56817282c in tsl::thread::ThreadPool::Schedule(std::function<void ()>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#27 0x000014d56828596d in std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&)::{lambda(std::function<void ()>)#2}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#28 0x000014d557487ad3 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::ScheduleReady(absl::lts_20220623::InlinedVector<tensorflow::SimplePropagatorState::TaggedNode, 8ul, std::allocator<tensorflow::SimplePropagatorState::TaggedNode> >*, tensorflow::SimplePropagatorState::TaggedNodeReadyQueue*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#29 0x000014d55748af96 in tensorflow::(anonymous namespace)::ExecutorImpl::RunAsync(tensorflow::Executor::Args const&, std::function<void (tsl::Status const&)>) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_framework.so.2
#30 0x000014d568298d3a in tensorflow::DirectSession::RunInternal(long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#31 0x000014d56829b9da in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tsl::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/external/el8_amd64_gcc13/lib/libtensorflow_cc.so.2
#32 0x000014d56e6dc890 in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tsl::thread::ThreadPoolOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libPhysicsToolsTensorFlow.so
#33 0x000014d56e6dc95f in tensorflow::run(tensorflow::Session*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tsl::thread::ThreadPoolInterface*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libPhysicsToolsTensorFlow.so
#34 0x000014d4beace4c6 in emtf::phase2::algo::ParameterAssignmentLayer::apply(bool const&, std::vector<emtf::phase2::track_t, std::allocator<emtf::phase2::track_t> >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#35 0x000014d4beb304fa in emtf::phase2::SectorProcessor::buildTracks(std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > const&, std::array<emtf::phase2::segment_t, 230ul> const&, bool const&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#36 0x000014d4beb31a32 in emtf::phase2::SectorProcessor::process(std::vector<l1t::phase2::EMTFHit, std::allocator<l1t::phase2::EMTFHit> >&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&, std::vector<l1t::phase2::EMTFInput, std::allocator<l1t::phase2::EMTFInput> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#37 0x000014d4beb33225 in emtf::phase2::TrackFinder::process(edm::Event const&, edm::EventSetup const&, std::vector<l1t::phase2::EMTFHit, std::allocator<l1t::phase2::EMTFHit> >&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&, std::vector<l1t::phase2::EMTFInput, std::allocator<l1t::phase2::EMTFInput> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#38 0x000014d473c309a0 in L1TMuonEndCapPhase2TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginL1TriggerL1TMuonEndCapPhase2Plugins.so

Thread 6 (Thread 0x14d54bfc6700 (LWP 1065947) "cmsRun"):
#0  0x000014d59f731178 in nanosleep () from /lib64/libc.so.6
#1  0x000014d59f73107e in sleep () from /lib64/libc.so.6
#2  0x000014d599ad61ec in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d54e8081ea in trklet::MatchEngineUnit::processPipeline() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerTrackFindingTracklet.so
#5  0x000014d54e810fd6 in trklet::MatchProcessor::execute(unsigned int, double) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerTrackFindingTracklet.so
#6  0x000014d54e8237bf in trklet::Sector::executeMP() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerTrackFindingTracklet.so
#7  0x000014d54e89b84a in trklet::TrackletEventProcessor::event(trklet::SLHCEvent&, std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >&, std::vector<std::vector<trklet::StubStreamData, std::allocator<trklet::StubStreamData> >, std::allocator<std::vector<trklet::StubStreamData, std::allocator<trklet::StubStreamData> > > >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerTrackFindingTracklet.so
#8  0x000014d54e975019 in L1FPGATrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginTrackFindingTrackletPlugins.so

Thread 5 (Thread 0x14d54adff700 (LWP 1065948) "cmsRun"):
#0  0x000014d59f75bbb1 in poll () from /lib64/libc.so.6
#1  0x000014d599ad82ee in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so
#2  0x000014d599ad84f3 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d59f67752f in raise () from /lib64/libc.so.6
#5  0x000014d59f64ae65 in abort () from /lib64/libc.so.6
#6  0x000014d59f64ad39 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
#7  0x000014d59f66fe86 in __assert_fail () from /lib64/libc.so.6
#8  0x000014d53520689f in alpaka::TaskKernelCpuSerial<std::integral_constant<unsigned long, 2ul>, unsigned int, alpaka_serial_sync::caPixelDoublets::GetDoubletsFromHisto<pixelTopology::Phase2>, unsigned int const&, alpaka_serial_sync::CACell<pixelTopology::Phase2>*, unsigned int*, reco::TrackingHitsLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, reco::CAGraphLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, reco::CALayersLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, unsigned int*, cms::alpakatools::HistoContainer<short, 256u, -1, 16u, unsigned int, 28u>*, cms::alpakatools::OneToManyAssocRandomAccess<unsigned int, -1, -1>*, caStructures::AlgoParams const&>::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/lib/el8_amd64_gcc13/pluginRecoTrackerPixelSeedingPortableSerialSync.so
#9  0x000014d5351e84f7 in alpaka_serial_sync::CAHitNtupletGeneratorKernels<pixelTopology::Phase2>::buildDoublets(reco::TrackingHitsLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, reco::CAGraphLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, reco::CALayersLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true> const&, unsigned int, alpaka::QueueGenericThreadsBlocking<alpaka::DevCpu>&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/lib/el8_amd64_gcc13/pluginRecoTrackerPixelSeedingPortableSerialSync.so
#10 0x000014d5351dcea2 in alpaka_serial_sync::CAHitNtupletGenerator<pixelTopology::Phase2>::makeTuplesAsync(reco::TrackingRecHitHost const&, PortableHostMultiCollection<reco::CALayersLayout<128ul, false>, reco::CAGraphLayout<128ul, false>, reco::CAModulesLayout<128ul, false> > const&, float, unsigned int, unsigned int, alpaka::QueueGenericThreadsBlocking<alpaka::DevCpu>&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/lib/el8_amd64_gcc13/pluginRecoTrackerPixelSeedingPortableSerialSync.so
#11 0x000014d5351dd299 in alpaka_serial_sync::CAHitNtupletAlpaka<pixelTopology::Phase2>::produce(alpaka_serial_sync::device::Event&, alpaka_serial_sync::device::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/lib/el8_amd64_gcc13/pluginRecoTrackerPixelSeedingPortableSerialSync.so
#12 0x000014d5351d7943 in alpaka_serial_sync::stream::EDProducer<edm::GlobalCache<reco::CAGeometryParams>, edm::RunCache<cms::alpakatools::MoveToDeviceCache<alpaka::DevCpu, PortableHostMultiCollection<reco::CALayersLayout<128ul, false>, reco::CAGraphLayout<128ul, false>, reco::CAModulesLayout<128ul, false> > > > >::produce(edm::Event&, edm::EventSetup const&) [clone .lto_priv.0] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_0_X_2025-10-29-2300/lib/el8_amd64_gcc13/pluginRecoTrackerPixelSeedingPortableSerialSync.so
#13 0x000014d5a065cb72 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so
#14 0x000014d5a0643da7 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so
#15 0x000014d5a05ce902 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so
#16 0x000014d5a05cee17 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so
#17 0x000014d5a089e451 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libFWCoreConcurrency.so
#18 0x000014d59fdd7263 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=<optimized out>, waiter=..., this=0x14d59dd81f80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#19 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x14d59dd81f80) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#20 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/arena.cpp:215
#21 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/thread_dispatcher_client.h:41
#22 tbb::detail::r1::thread_dispatcher::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/thread_dispatcher.cpp:195
#23 0x000014d59fdcf1c1 in tbb::detail::r1::rml::private_worker::run (this=0x14d59b166000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/private_server.cpp:271
#24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x14d59b166000) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc13/external/tbb/v2022.0.0-ede4a930df4abec602e96e3f3ad15914/tbb-v2022.0.0/src/tbb/private_server.cpp:221
#25 0x000014d59ebe81ca in start_thread () from /lib64/libpthread.so.0
#26 0x000014d59f6628d3 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x14d59fc5f580 (LWP 1065926) "cmsRun"):
#0  0x000014d59f731178 in nanosleep () from /lib64/libc.so.6
#1  0x000014d59f73107e in sleep () from /lib64/libc.so.6
#2  0x000014d599ad61ec in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014d4beac86b3 in emtf::phase2::algo::PatternMatchingLayer::apply(std::vector<std::array<ap_uint<288>, 8ul>, std::allocator<std::array<ap_uint<288>, 8ul> > > const&, bool const&, std::vector<std::array<emtf::phase2::road_t, 288ul>, std::allocator<std::array<emtf::phase2::road_t, 288ul> > >&) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#5  0x000014d4beb30452 in emtf::phase2::SectorProcessor::buildTracks(std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > > const&, std::array<emtf::phase2::segment_t, 230ul> const&, bool const&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#6  0x000014d4beb31a32 in emtf::phase2::SectorProcessor::process(std::vector<l1t::phase2::EMTFHit, std::allocator<l1t::phase2::EMTFHit> >&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&, std::vector<l1t::phase2::EMTFInput, std::allocator<l1t::phase2::EMTFInput> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#7  0x000014d4beb33225 in emtf::phase2::TrackFinder::process(edm::Event const&, edm::EventSetup const&, std::vector<l1t::phase2::EMTFHit, std::allocator<l1t::phase2::EMTFHit> >&, std::vector<l1t::phase2::EMTFTrack, std::allocator<l1t::phase2::EMTFTrack> >&, std::vector<l1t::phase2::EMTFInput, std::allocator<l1t::phase2::EMTFInput> >&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/libL1TriggerL1TMuonEndCapPhase2.so
#8  0x000014d473c309a0 in L1TMuonEndCapPhase2TrackProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02913/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-10-29-1100/lib/el8_amd64_gcc13/pluginL1TriggerL1TMuonEndCapPhase2Plugins.so

Current Modules:

Module: CAHitNtupletAlpakaPhase2@alpaka:hltPhase2PixelTracksSoA (crashed)
Module: L1FPGATrackProducer:l1tTTTracksFromExtendedTrackletEmulation
Module: L1TMuonEndCapPhase2TrackProducer:simEmtfDigisPhase2
Module: L1TMuonEndCapPhase2TrackProducer:simEmtfDigisPhase2

A fatal system signal has occurred: abort signal
timeout: the monitored command dumped core

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @dan131riley.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign RecoTracker/PixelSeeding

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

@cms-sw/tracking-pog-l2 @Parsifal-2045

[makortel]

A question regarding building the 16_0_0_pre2 is that are the affected phase2 workflows important enough for the RelVals and release validation that we should consider reverting #48921 (unless there is a quick fix)? @cms-sw/orp-l2 @cms-sw/pdmv-l2

[Parsifal-2045]

Hi, thanks for bringing this up. I've been investigating a bit to figure out exactly what is causing the assertion to fail. As far as I can tell (please correct me if I missed anything) all the workflows that fail because of the CA are on older geometries (i.e. pre D110). Digging more into the issue, it looks like at this point in time we cannot natively support any geometry other than D110 due to how the number of active modules in BPIX1 (named offsetBPIX2 in the code) is calculated. This is computed dynamically at the cluster level

cmssw/RecoLocalTracker/SiPixelClusterizer/plugins/alpaka/SiPixelRawToClusterKernel.dev.cc

Lines 638 to 655 in 66d8bfc

	constexpr int startBPIX2 = TrackerTraits::layerStart[1];
	// element startBPIX2 hold the number of clusters until BPIX2
	const auto bpix2ClusterStart = cms::alpakatools::make_device_view(
	queue, clusters_d->const_view().clusModuleStart().data() + startBPIX2, 1u);
	auto nModules_Clusters_h_1 = cms::alpakatools::make_host_view(nModules_Clusters_h.data() + 1, 1u);
	alpaka::memcpy(queue, nModules_Clusters_h_1, clusModuleStartLastElement);
	auto nModules_Clusters_h_2 = cms::alpakatools::make_host_view(nModules_Clusters_h.data() + 2, 1u);
	alpaka::memcpy(queue, nModules_Clusters_h_2, bpix2ClusterStart);
	#ifdef GPU_DEBUG
	alpaka::wait(queue);
	std::cout << "SiPixelClusterizerAlpaka results:" << std::endl
	<< " > no. of digis: " << nDigis << std::endl
	<< " > no. of active modules: " << nModules_Clusters_h[0] << std::endl
	<< " > no. of clusters: " << nModules_Clusters_h[1] << std::endl
	<< " > bpix2 offset: " << nModules_Clusters_h[2] << std::endl;
	#endif

however the starting point for this computation is a hard-coded constant:

cmssw/Geometry/CommonTopologies/interface/SimplePixelTopology.h

Lines 453 to 456 in 66d8bfc

	HOST_DEVICE_CONSTANT uint32_t layerStart[nLayersPix + 1] = {
	0, 216, 432, 612, // Barrel
	864, 972, 1080, 1188, 1296, 1404, 1512, 1620, 1728, 1904, 2080, 2256, // Fp
	2432, 2540, 2648, 2756, 2864, 2972, 3080, 3188, 3296, 3472, 3648, 3824, nModulesPix}; // Np

This was indeed updated a while ago to support D110 geometry c9b9516, which in turn broke support for all previous geometries in the CA, where this value is actively used and guarded by the assert that fails here

cmssw/RecoTracker/PixelSeeding/plugins/alpaka/CAPixelDoubletsAlgos.h

Line 431 in 66d8bfc

outerHitHisto->count(acc, oi - hh.offsetBPIX2());

(the failure is actually related to oi being smaller than the wrong offset, making the unsigned difference underflow)
While I have been trying to implement a fix to allow the CA to work with older geometries, the issue seems to be more complex than expected, and a very-short term fix seems unlikely at this point. We could however implement a simple fix just changing the default pixelTracking algorithm back to the legacy one, keeping all the other infrastructure under a procModifier, while we investigate a more permanent fix (or decide if it instead makes more sense to drop these older geometries). Any input from other experts would be greatly appreciated

[mandrenguyen]

From the ORP side, a workaround is acceptable if it can be delivered ~ today. We are behind schedule for pre2 and we shouldn't push that too much.