Max working set regression on INTEL Linux for Plaintext and JSON Platform

[sebastienros]

[sebastienros]

Using Plaintext as this has the most stable max working set measurement while showing the same disgression.

First disgression found in April:

Investigations:

crank --scenario plaintext --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --application.framework net8.0 --no-metadata --no-measurements --profile aspnet-citrine-win  --application.aspNetCoreVersion 8.0.0-preview.4.23211.1 --application.runtimeVersion 8.0.0-preview.4.23211.7 --application.sdkVersion 8.0.100-preview.4.23211.7 --application.source.branchOrCommit main#b0da25608eb6a8d3094e6ba431a92124b974bce4  --iterations 5 --repeat application

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 91                                   |
| Max Cores usage (%)     | 2,540                                |
| Max Working Set (MB)    | 59                                   |
| Max Private Memory (MB) | 80                                   |
| Build Time (ms)         | 5,164                                |
| Start Time (ms)         | 406                                  |
| Published Size (KB)     | 115,341                              |
| Symbols Size (KB)       | 51                                   |
| .NET Core SDK Version   | 8.0.100-preview.4.23211.7            |
| ASP.NET Core Version    | 8.0.0-preview.4.23211.1+0f65d69374b7 |
| .NET Runtime Version    | 8.0.0-preview.4.23211.7+476bce345600 |

crank --scenario plaintext --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --application.framework net8.0 --no-metadata --no-measurements --profile aspnet-citrine-win  --application.aspNetCoreVersion 8.0.0-preview.4.23213.2 --application.runtimeVersion 8.0.0-preview.4.23212.4 --application.sdkVersion 8.0.100-preview.4.23211.7 --application.source.branchOrCommit main#b0da25608eb6a8d3094e6ba431a92124b974bce4  --iterations 5 --repeat application

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 93                                   |
| Max Cores usage (%)     | 2,593                                |
| Max Working Set (MB)    | 61                                   |
| Max Private Memory (MB) | 80                                   |
| Build Time (ms)         | 6,363                                |
| Start Time (ms)         | 459                                  |
| Published Size (KB)     | 115,480                              |
| Symbols Size (KB)       | 51                                   |
| .NET Core SDK Version   | 8.0.100-preview.4.23211.7            |
| ASP.NET Core Version    | 8.0.0-preview.4.23213.2+68ae6b0d8aa2 |
| .NET Runtime Version    | 8.0.0-preview.4.23212.4+5117e1f78d2f |

----

Only changing ASP.NET (regression still visible)

crank --scenario plaintext --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --application.framework net8.0 --no-metadata --no-measurements --profile aspnet-citrine-win  --application.aspNetCoreVersion 8.0.0-preview.4.23213.2 --application.runtimeVersion 8.0.0-preview.4.23211.7 --application.sdkVersion 8.0.100-preview.4.23211.7 --application.source.branchOrCommit main#b0da25608eb6a8d3094e6ba431a92124b974bce4  --iterations 5 --repeat application

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 92                                   |
| Max Cores usage (%)     | 2,589                                |
| Max Working Set (MB)    | 61                                   |
| Max Private Memory (MB) | 80                                   |
| Build Time (ms)         | 3,799                                |
| Start Time (ms)         | 437                                  |
| Published Size (KB)     | 115,469                              |
| Symbols Size (KB)       | 51                                   |
| .NET Core SDK Version   | 8.0.100-preview.4.23211.7            |
| ASP.NET Core Version    | 8.0.0-preview.4.23213.2+68ae6b0d8aa2 |
| .NET Runtime Version    | 8.0.0-preview.4.23211.7+476bce345600 |

Only changing runtime (no visible regression)

crank --scenario plaintext --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --application.framework net8.0 --no-metadata --no-measurements --profile aspnet-citrine-win  --application.aspNetCoreVersion 8.0.0-preview.4.23211.1 --application.runtimeVersion 8.0.0-preview.4.23212.4 --application.sdkVersion 8.0.100-preview.4.23211.7 --application.source.branchOrCommit main#b0da25608eb6a8d3094e6ba431a92124b974bce4  --iterations 5 --repeat application

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 92                                   |
| Max Cores usage (%)     | 2,589                                |
| Max Working Set (MB)    | 59                                   |
| Max Private Memory (MB) | 79                                   |
| Build Time (ms)         | 3,792                                |
| Start Time (ms)         | 430                                  |
| Published Size (KB)     | 115,352                              |
| Symbols Size (KB)       | 51                                   |
| .NET Core SDK Version   | 8.0.100-preview.4.23211.7            |
| ASP.NET Core Version    | 8.0.0-preview.4.23211.1+0f65d69374b7 |
| .NET Runtime Version    | 8.0.0-preview.4.23212.4+5117e1f78d2f |

Identified to be when Metrics were added, but this should be fixed by dotnet/aspnetcore#50565. Among the probable reason is some TagList creation on each request

[sebastienros]

Second visible regression:

Isolated with runtime changes between these runs:

8.0.0-preview.5.23269.1

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 92                                   |
| Max Cores usage (%)     | 2,558                                |
| Max Working Set (MB)    | 61                                   |
| Max Private Memory (MB) | 80                                   |
| Build Time (ms)         | 4,480                                |
| Start Time (ms)         | 376                                  |
| Published Size (KB)     | 116,018                              |
| Symbols Size (KB)       | 53                                   |
| .NET Core SDK Version   | 8.0.100-preview.5.23268.8            |
| ASP.NET Core Version    | 8.0.0-preview.5.23268.9+4be2661fe8ed |
| .NET Runtime Version    | 8.0.0-preview.5.23269.1+89678442a862 |

| Requests/sec            | 5,232,756           |

8.0.0-preview.5.23269.4 (regression)

| application             |                                      |
| ----------------------- | ------------------------------------ |
| Max CPU Usage (%)       | 93                                   |
| Max Cores usage (%)     | 2,596                                |
| Max Working Set (MB)    | 63                                   |
| Max Private Memory (MB) | 81                                   |
| Build Time (ms)         | 5,604                                |
| Start Time (ms)         | 379                                  |
| Published Size (KB)     | 116,010                              |
| Symbols Size (KB)       | 53                                   |
| .NET Core SDK Version   | 8.0.100-preview.5.23268.8            |
| ASP.NET Core Version    | 8.0.0-preview.5.23268.9+4be2661fe8ed |
| .NET Runtime Version    | 8.0.0-preview.5.23269.4+e108a1a43c7d |

| Requests/sec            | 5,873,205           |

The changeset contains Enable Tiered PGO by default which is most probably the reason of an increased max working as the RPS is going from 5.2M to 5.8M RPS.