Skip to content

Add micro-benchmark validation #29695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Mar 12, 2021
Merged

Add micro-benchmark validation #29695

merged 13 commits into from
Mar 12, 2021

Conversation

BrennanConroy
Copy link
Member

@BrennanConroy BrennanConroy commented Jan 27, 2021

Fixes #27607
Fixes #23025

Open question:
Should projects have to opt-in for validation? If not, we need to align on a folder/project name structure so we don't try to validate non-micro benchmark projects.

This is currently turned off because BenchmarkDotNet hasn't released a package that works with net6.0 yet. Using nightlies I can verify this locally.

On failure:

  Running benchmark validation for:
  Microsoft.AspNetCore.SignalR.Microbenchmarks

  'HubConnectionReceiveBenchmark.ReceiveAsync: Dry(Toolchain=InProcessNoEmitToolchain, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=1) [ArgumentCount=0, MessageCount=1, Protocol=json], 0 runs' failed, reason: 'AllMeasurements'
E:\ASPNETCORE\aspnetcore\eng\targets\CSharp.Common.targets(103,5): error MSB3073: The command "dotnet E:\ASPNETCORE\aspnetcore\artifacts\bin\Microsoft.AspNetCore.SignalR.Microbenchmarks\Debug\net6.0\Microsoft.AspNetCore.SignalR.Microbenchmarks.dll -f * --validate-fast" exited with code 1. [E:\ASPNETCORE\aspnetcore\src\SignalR\perf\Microbenchmarks\Microsoft.AspNetCore.SignalR.Microbenchmarks.csproj]

Build FAILED.

@BrennanConroy BrennanConroy requested review from halter73 and a team January 27, 2021 19:08
@halter73
Copy link
Member

I'd rather fix the folder structure than make validation opt-in. BenchmarkDotNet hasn't release a package in a while now. Can we use the nightly package feed for our microbenchmarks?

@BrennanConroy
Copy link
Member Author

BenchmarkDotNet hasn't release a package in a while now.

Yeah, I asked and they should be releasing an update in a week or two

@Pilchie Pilchie added the area-signalr Includes: SignalR clients and servers label Jan 27, 2021
@BrennanConroy BrennanConroy added area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework and removed area-signalr Includes: SignalR clients and servers labels Jan 28, 2021
@BrennanConroy
Copy link
Member Author

I like the current format for SignalR and Middleware:
C:\aspnetcore\src\SignalR\perf\Microbenchmarks\Microsoft.AspNetCore.SignalR.Microbenchmarks.csproj
C:\aspnetcore\src\Middleware\perf\Microbenchmarks\Microsoft.AspNetCore.WebSockets.Microbenchmarks.csproj

Which would make Kestrel:
C:\aspnetcore\src\Servers\Kestrel\perf\Kestrel.Performance\Microsoft.AspNetCore.Server.Kestrel.Performance.csproj -> C:\aspnetcore\src\Servers\Kestrel\perf\Microbenchmarks\Microsoft.AspNetCore.Server.Kestrel.Microbenchmarks.csproj

@halter73
Copy link
Member

Sounds good to me.

@sebastienros We're planning to move and rename the Microsoft.AspNetCore.Server.Kestrel.Performance microbenchmarks. This is just a heads up in case you need to react.

@sebastienros
Copy link
Member

I will update our config files when it's merged, and maybe get them running in the perf CI.

@BrennanConroy BrennanConroy marked this pull request as ready for review March 10, 2021 02:54
@BrennanConroy
Copy link
Member Author

Any :shipit: 's 😃

Copy link
Member

@javiercn javiercn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@BrennanConroy BrennanConroy merged commit 340d161 into main Mar 12, 2021
@BrennanConroy BrennanConroy deleted the brecon/microbench branch March 12, 2021 05:28
@BrennanConroy BrennanConroy added this to the 6.0-preview3 milestone Mar 12, 2021
dougbu pushed a commit to dougbu/razor-compiler that referenced this pull request Nov 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework
Projects
None yet
7 participants