[outerloop] System.Runtime.Tests crash with exit code 137 in linux debug coreclr #98964
Description
Found in a PR that fixed an unrelated failure affecting all outerloop runs, hiding other failures. #97135
Error Blob
{
"ErrorMessage": "",
"BuildRetry": false,
"ErrorPattern": "Killed.+System\\.Runtime\\.Tests\\.runtimeconfig\\.json.+OuterLoop",
"ExcludeConsoleLog": false
}Reproduction Steps
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195
- Queues:
linux_musl-x64 Debug CoreCLR_Debuglinux-x64 Debug CoreCLR_Debug
- Job results:
- https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=592eed89-78f5-5202-9da5-ad17df3bef4a&t=4feb010c-08b1-5201-8e2c-97e3caf24223
- https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=df56ac33-e9fe-5eb2-5cdb-3303d7aad000&t=f62247f8-d051-5cb6-d5f0-06f532f9a005
- Log files:
- https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-6fc0e798b0d84fd99e/System.Runtime.Tests/1/console.c8ffe024.log?helixlogtype=result
- https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-ca234b0b7f3e48249a/System.Runtime.Tests/1/console.40eef85c.log?helixlogtype=result
- Output:
========================== End custom configuration settings ===============================
----- start Mon Feb 26 23:56:03 UTC 2024 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Tests (found 29 of 9308 test cases)
Starting: System.Runtime.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 180: 21 Killed "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Mon Feb 26 23:56:28 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
----- start =============== XUnitLogChecker Output =====================================================
dmesg: klogctl: Operation not permitted
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================
+ export '_commandExitCode=137'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 13931354 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhMTBlYjk0MS0xNzM2LTRjNzYtOWFiZC1iZDBlZGYxNjg0ODgiLCJzaWQiOiIxZTViN2VlMC1iNmQyLTQzNjgtODZjOS0zOTQ5MGIxM2FlODAiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzU4MDE5NSIsImpvYnJlZiI6IjMxNTZhMmQwLTExYWItNGExMC04NTA0LWY3Y2UxM2ZiMTk2OTo1OTJlZWQ4OS03OGY1LTUyMDItOWRhNS1hZDE3ZGYzYmVmNGEiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvNTgwMTk1Iiwib3JjaGlkIjoiMzE1NmEyZDAtMTFhYi00YTEwLTg1MDQtZjdjZTEzZmIxOTY5LmJ1aWxkLmJ1aWxkX2xpbnV4X211c2xfeDY0X2RlYnVnX2NvcmVjbHJfZGVidWcuX19kZWZhdWx0IiwicmVwb0lkcyI6IiIsImlzcyI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb20iLCJhdWQiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tfHZzbzo2ZmNjOTJlNS03M2E3LTRmODgtOGQxMy1kOTA0NWI0NWZiMjciLCJuYmYiOjE3MDg5ODg5MzgsImV4cCI6MTcwOTAwMDkzOH0.rjlcWYrlHLfuHwUtUIiDkhhUDwy8cnQZ3rCBJjmd8FuCN0BDFzjz0jEMQEbQ9WZ7LHqdWN40vrD1MHH2p9a2tC9JYY7ca4b2w8Yy05wlbG-_cjF_ksT3jfhgz4EOLBiSmE7-bW_m1b3gtKHo-rdFDyMt_vciLiIB_C3543QhcPGJJdph5ok1SjBVKbl1q9wzw4Tut1lLyo8-ToLRotVPvv2Fahz3TSp--EDbdhY2SYtNrPyGVW7t9F8o3TZdG5iyrtyTnCAlr2Qdu-cOUJhyXM_4vhztXvLdgDMuDgGINPpA7sDXoqo_7LB9RP1iNc35uIEM-430j7jeG9MDrZzEZQ
2024-02-26T23:56:35.053Z INFO run.py run(48) main Beginning reading of test results.
2024-02-26T23:56:35.054Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/e' for test results files
2024-02-26T23:56:35.054Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/uploads' for test results files
2024-02-26T23:56:35.062Z WARNING run.py __init__(55) read_results No results file found in any of the following formats: xunit, junit, trx
2024-02-26T23:56:35.063Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2024-02-26T23:56:35.067Z INFO run.py packing_test_reporter(33) report_results Packed 1541 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 580195 -workitem System.Runtime.Tests -jobid 6fc0e798-b0d8-4fd9-9e43-a36cfb9441c4 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
+ exit 137
+ export '_commandExitCode=137'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137
[END EXECUTION]
Exit Code:137
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 2 | 14 | 61 |
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195
Error message validated: [Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/27/2024 11:09:08 PM UTC