The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2678012	dotnet-sdk	Run AoT Tests	Log	#48611
1003241	dotnet/runtime	Send to Helix	Log
2678133	dotnet-sdk	Run AoT Tests	Log	#48610
1003497	dotnet/sdk	Run Tests in Helix	Log
1002969	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113596
2678110	dotnet-sdk	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log	#48610
1002962	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113595
1003303	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#47818
993357	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#47819
990830	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#47827
2677925	dotnet-sdk	Run AoT Tests	Log	#48626
1002951	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113561
1003249	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#48005
1002939	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113516
2678052	dotnet-sdk	Run TestBuild Tests	Log	#48610
2678053	dotnet-sdk	Run AoT Tests	Log	#48918
2678013	dotnet-sdk	Run dotnet-format on dotnet/msbuild MSBuild.sln	Log	#48610
2678026	dotnet-sdk	Run AoT Tests in Helix	Log	#48906
2678025	dotnet-sdk	Run AoT Tests in Helix	Log	#48907
2678024	dotnet-sdk	Run AoT Tests in Helix	Log	#48905
1003313	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#47645
1002929	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113461
1002918	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113515
1003175	dotnet/runtime	Send to Helix	Log
1003203	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#48117
1003301	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#47884
1002911	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113481
2677991	dotnet-sdk	Run AoT Tests in Helix	Log	#48905
1002908	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113475
2677992	dotnet-sdk	Run Tests in Helix	Log	#48907
2677990	dotnet-sdk	Run dotnet new Integration Tests	Log	#48906
1003150	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#48115
2677988	dotnet-efcore	Send job to helix	Log	#48910
1002933	dotnet/runtime	Send to Helix	Log
1002922	dotnet/runtime	Send to Helix	Log
2677830	dotnet-emsdk	Validate	Log
2677924	dotnet-sdk	Run TestBuild Tests	Log	#48610
2677917	dotnet-sdk	Run Tests in Helix	Log	#48905
2677919	dotnet-sdk	Run Tests in Helix	Log	#48907
2677918	dotnet-sdk	Run Tests in Helix	Log	#48906
1003123	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114080
2677920	dotnet-performance	AzureCLI	Log
1002979	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
2677859	dotnet-performance	Send job to Helix (Unix)	Log
1002846	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114162
2677842	dotnet-dotnet	Build	Log
1003143	dotnet/roslyn	Run Integration Tests	Log
1003106	dotnet/runtime	AOT compile CoreCLR tests	Log
1002834	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#110818
1002848	dotnet/runtime	Send to Helix	Log
1002814	dotnet/runtime	Send to Helix	Log
2677683	dotnet-runtime	Send job to Helix (Unix)	Log
2677689	dotnet-maui	Add build to workload sets channel	Log
2677680	dotnet-maui	Add build to workload sets channel	Log
2677645	dotnet-maui	Add build to workload sets channel	Log
2677593	dotnet-winforms	🛡 Guardian: TSAUpload	Log
1001409	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
2677513	dotnet-winforms	🛡 Guardian: CredScan (Binary)	Log
2677422	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2677475	dotnet-runtime	Send job to Helix (Unix)	Log
2677448	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2677442	dotnet-runtime	Initialize containers	Log
2677441	dotnet-runtime	Pre-job: Download secrets: HelixProdKV	Log
2677440	dotnet-runtime	Component Detection (auto-injected by policy)	Log
2677430	dotnet-runtime	Agent Pool Validation	Log
1002176	dotnet/runtime	AOT compile CoreCLR tests	Log
2677326	dotnet-dotnet	Build (centOSStreamContainer)	Log
2677228	dotnet-runtime	Send job to Helix (Unix)	Log
1001857	dotnet/aspire	Build	Log
2677030	dotnet-dotnet	Build	Log
1001715	dotnet/aspire	Run helix tests	Log
2677011	dotnet-runtime	Run performance job script	Log
1001653	dotnet/aspire	Run helix tests	Log
2676998	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log
1001556	dotnet/aspire	Run non-helix tests	Log
1001479	dotnet/runtime	AOT compile CoreCLR tests	Log
2676882	dotnet-dotnet	Build (centOSStreamContainer)	Log
2674217	dotnet-runtime	Build product	Log
2676873	dotnet-dotnet	Validate Hosted Pool Information (1ES PT)	Log
2676655	dotnet-dotnet	Build	Log
1001158	dotnet/runtime	Publish	Log	dotnet/runtime#114110
2676791	dotnet-runtime	Publish AOT Mono Artifacts	Log
2676756	dotnet-runtime	Download Build Artifact LinuxMonoAOTx64	Log
1001264	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#76906
1001274	dotnet/aspire	Build	Log
1001268	dotnet/aspire	Build	Log
1001204	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#113916
2676700	dotnet-runtime	Send job to Helix (Unix)	Log
1001126	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#114105
2676724	dotnet-dotnet-docker	Validate Hosted Pool Information (1ES PT)	Log
1001050	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#114080
2676654	dotnet-dotnet	Checkout dotnet-dotnet@sign-and-harden-pkgs to s	Log
1000655	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
1000894	dotnet/roslyn	Test	Log	dotnet/roslyn#77942
2676570	dotnet-performance	Send job to Helix (Windows)	Log
2676502	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2676525	dotnet-performance	AzureCLI	Log
2676496	dotnet-runtime	Send job to Helix (Unix)	Log
2676492	dotnet-roslyn	Build	Log
1000700	dotnet/aspire	Build	Log
Displaying 100 of 1033 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
67	207	1033

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)