The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2760073	dotnet-roslyn	Build	Log
2760004	dotnet-deployment-tools	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2759910	dotnet-runtime	Run performance job script	Log
2759937	dotnet-roslyn	Test	Log
1107761	dotnet/roslyn	Run Integration Tests	Log
2759921	dotnet-roslyn	Test	Log
1107665	dotnet/roslyn	Run Integration Tests	Log
1107537	dotnet/runtime	Build product	Log	dotnet/runtime#118128
1107278	dotnet/runtime	Send to Helix	Log
2759482	dotnet-dotnet	Build (centOSStreamContainer)	Log
2759614	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1107306	dotnet/runtime	Build product	Log	dotnet/runtime#118007
2759336	dotnet-aspnetcore	🔒 Publish SBOM manifest	Log
1106911	dotnet/runtime	Send tests to Helix (Unix)	Log
1106915	dotnet/runtime	Send to Helix	Log
2759322	dotnet-orleans	Test	Log
2759222	dotnet-dotnet	Build (centOSStreamContainer)	Log
1106808	dotnet/runtime	Send tests to Helix (Unix)	Log
2759136	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
1106678	dotnet/runtime	Send tests to Helix (Unix)	Log
2758814	dotnet-runtime	Run performance job script	Log
1106673	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117800
1106556	dotnet/runtime	AOT compile CoreCLR tests	Log
1106249	dotnet/runtime	Send tests to Helix (Unix)	Log
2758408	dotnet-orleans	Test	Log
1106290	dotnet/runtime	Send to Helix	Log
2758346	dotnet-dotnet	Build (centOSStreamContainer)	Log
2758356	dotnet-runtime	Run performance job script	Log
1106297	dotnet/runtime	Send to Helix	Log
2758340	dotnet-sdk	Run Tests in Helix and non Helix in parallel	Log	#51890
2758325	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2758291	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2758278	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2758245	dotnet-runtime	Run performance job script	Log
1106105	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#49932
1106058	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#118037
1106037	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#117996
2758127	dotnet-dotnet	Build (centOSStreamContainer)	Log
2758110	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
1106024	dotnet/roslyn	Run Integration Tests	Log
2758103	dotnet-dotnet	Checkout dotnet-dotnet@fix-signing-val-again to s	Log
2757854	dotnet-xharness	🛡 Guardian: TSAUpload	Log
2757822	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1105084	dotnet/runtime	Send tests to Helix (Unix)	Log
1105147	dotnet/runtime	Send to Helix	Log
1105222	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118043
2757514	dotnet-performance	Send job to Helix (Windows)	Log
1105151	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118029
1105212	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49847
2757492	dotnet-performance	Send job to Helix (Unix)	Log
1105023	dotnet/roslyn	Run Integration Tests	Log
2757331	dotnet-dotnet	Build (centOSStreamContainer)	Log
1104953	dotnet/runtime	Send to Helix	Log	dotnet/runtime#108655
1104947	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118030
1104939	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117939
2757072	dotnet-runtime	Run performance job script	Log
2757286	dotnet-orleans	Test	Log
1104886	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117991
1104956	dotnet/runtime	Send job to Helix (Unix)	Log
1104910	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79555
1104853	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117890
1104811	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118027
1103625	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117868
1103985	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117996
2756711	dotnet-dotnet	Build (centOSStreamContainer)	Log
1104368	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117997
1104748	dotnet/runtime	Send to Helix	Log	dotnet/runtime#106099
1104724	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117993
1104487	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117980
1104139	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117258
1104709	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115826
1104705	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118020
1104672	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117985
1104575	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118015
2757006	dotnet-aspnetcore	SDL Binary Analysis Checks (1ES PT)	Log
1104541	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117978
1104526	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117277
1104486	dotnet/runtime	Send tests to Helix (Unix)	Log
1104483	dotnet/runtime	Send tests to Helix (Windows)	Log
2757026	dotnet-dotnet	Build (azurelinuxArmMuslCrossContainer)	Log
2756870	dotnet-diagnostics	Pre-job: Download secrets: EngKeyVault	Log
1104316	dotnet/aspnetcore	Build x64	Log
1104248	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117845
1103973	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117949
1104295	dotnet/roslyn	Run Integration Tests	Log
1103564	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117898
1104168	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117966
2756808	dotnet-runtime	Run performance job script	Log
1102433	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117894
1104136	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118003
1104143	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117995
1103861	dotnet/runtime	Send to Helix	Log
1104158	dotnet/runtime	AOT compile CoreCLR tests	Log
1104127	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117987
1104180	dotnet/runtime	Send job to Helix (Unix)	Log
1104097	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117738
2756783	dotnet-aspire	🟣Restore	Log
2756737	dotnet-orleans	Test	Log
1104067	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117991
1104041	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118002
Displaying 100 of 1393 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
14	283	1393

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez