[PPC] Segfault via cling::IncrementalExecutor::runStaticInitializersOnce #1126
Description
What
Taken from cms-sw#41658
Many workflows failed in CMSSW_13_2_X_2023-05-11-2300 el8_ppc64le_gcc11 with
Fri May 12 01:48:41 CEST 2023 Thread 5 (Thread 0x10086aa09210 (LWP 127950) "cmsRun"): #2 0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so #3 <signal handler called> #4 0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0 #5 0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865 #6 std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #7 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #8 0x0000100000f49198 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #9 0x0000100000f5484c in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #10 0x0000100001aec050 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #11 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #12 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so #13 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #14 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #15 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #16 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so <snip> #37 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #38 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #39 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #40 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #41 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #42 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #43 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #44 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #45 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so #46 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so Thread 4 (Thread 0x100866609210 (LWP 127949) "cmsRun"): #2 0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so #3 <signal handler called> #4 0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0 #5 0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865 #6 std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #7 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #8 0x0000100000f46bb8 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::ReadLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #9 0x0000100000f547cc in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::ReadLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #10 0x0000100001afbc08 in TClass::GetClass(std::type_info const&, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #11 0x0000100000a89d90 in edm::TypeWithDict::TypeWithDict(std::type_info const&, long) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so #12 0x0000100000a8bf7c in edm::TypeWithDict::TypeWithDict(std::type_info const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so #13 0x000010086ee52ce4 in StringCutObjectSelector<reco::Muon, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #14 0x000010086ee4ee90 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #15 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #16 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #17 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #18 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #19 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so #20 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so Thread 3 (Thread 0x100865bf9210 (LWP 127948) "cmsRun"): #3 0x0000100009cb5e9c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so #4 <signal handler called> #5 0x00001008c3642b40 in ?? () #6 0x00001008c3641bcc in ?? () #7 0x00001008c36434b0 in ?? () #8 0x00001008c36432fc in ?? () #9 0x00001008c363bfe8 in ?? () #10 0x00001008c363be90 in ?? () #11 0x00001008c363d7c4 in ?? () #12 0x00001008c363e8c0 in ?? () #13 0x00001008c363e820 in ?? () #14 0x00001008c3630188 in ?? () #15 0x0000100814722ac8 in cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #16 0x000010081468810c in cling::Interpreter::executeTransaction(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #17 0x000010081473a154 in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #18 0x000010081473b930 in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #19 0x00001008146837b0 in cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #20 0x000010081457937c in ExecAutoParse(char const*, bool, cling::Interpreter*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #21 0x0000100814589b98 in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #22 0x000010081458f934 in TCling::AutoParse(char const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so #23 0x0000100001aec1d0 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #24 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #25 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so #26 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #27 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #28 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #29 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so <snip> #50 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #51 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #52 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #53 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #54 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #55 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #56 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #57 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #58 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so #59 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so Thread 1 (Thread 0x10000277d0b0 (LWP 127467) "cmsRun"): #2 0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so #3 <signal handler called> #4 0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0 #5 0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865 #6 std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155 #7 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41 #8 0x0000100000f49198 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #9 0x0000100000f5484c in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so #10 0x0000100001aec050 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #11 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so #12 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so #13 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #14 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #15 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #16 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so <snip> #37 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #38 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so #39 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #40 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #41 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #42 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so #43 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #44 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so #45 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so #46 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.soCurrent Modules:
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef (crashed)
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef
...
To quote Andrea Valenzuela
I have been looking further into this issue and it seems it is caused by cms-sw#41590. If I use the latest IBs and revert the changes made in this PR, relval workflows run fine. I see @thesps is the original author of the PR. Could you please have a look at it?
In any case, I am not sure still why it is only failing on the ppc64le architecture.
Recipe to recreate
TBD
@thesps could I ask you to take a look at this? This issue seems to go away when your PR is reverted.