Tests which use deployers do not work on Helix

[natemcmaster]

The current design of test deployers requires the source code for the test projects to exist on the machine running tests AND for the code to be buildable at test-time. This creates two problems:

1. In distributed testing on Helix, the source code is not present, only test binaries. Tests which rely on deployers are currently disabled
2. Simply copying the source code into a distributed test environment is insufficient because the projects are not currently written to build in a clean, vanilla "dotnet build" context.

Potential solutions

Two options:

1. build test assets along with the test project and copy the test assets as binaries into the test output. Deployers run on binaries (dotnet foo.dll) instead of on source (dotnet publish && dotnet run)
2. change the test asset code to be buildable in a 'vanilla' build context - don't build locally, only during test time. Deployers work the same, but test code should not build in VS or during build.cmd with the rest of the repo -- the code only builds during test execution.

Option 1 is preferable because it is easier to ensure the code can compile when we change APIs, and in many cases we're not especially interested in testing the behavior of the SDK.

Option 2 has some marginal benefit in that it reduces the up-front build time of the repo and makes the test payload smaller.

cc @Tratcher @anurse

[natemcmaster]

@anurse - thoughts on this one? Is this something Build Team or K Core should do?

[analogrelay]

I think think belongs mostly on the KCore side. We need to update all our tests to build the site during build and just run them during tests.

@Tratcher @jkotalik I think you've both done some looking at that in the past, do you know where we stand there? Which server functional tests remain un-Helixable?

[jkotalik]

From what I can tell, besides IIS tests, all functional test with deployers do not run on helix. From my knowledge so far, I think all of them can be put on helix, it's just that no one has committed the work to fix them.

[analogrelay]

Ok, cool. I think that work probably lives with us then. Swapping to servers area. I'll put it in p8 because I do think we should try to get this done. The fact that it doesn't affect customer code does mean it'll probably have a lowish priority for now though.

[jkotalik]

@anurse this seems like a good time for me to overhaul the functional tests deployments. #9693 and re-enabling music store tests are tangentially related.

[ericstj]

@anurse is this only for the hosting Shutdown tests or are you using this to track other tests in aspnetcore? /cc @maryamariyan

[Tratcher]

@ericstj this tracks the deployers used in the server functional tests.
https://github.com/dotnet/aspnetcore/tree/master/src/Servers/test/FunctionalTests

At one point we made a simplified copy of those deployers for the hosting shutdown tests in the extensions repo.

[ericstj]

OK, we'll make a separate issue to track the extensions project.

[jkotalik]

@JunTaoLuo you were looking into deployers for grpc tests, right? Is that tracked in a separate issue?

Besides that, there are a few server tests and music store tests that could be ported to helix, but they aren't critical tests.