Spread Windows CI load

[xavfernandez]

Follow-up of #6767

Current status:

8 interpreters (2.7, 3.5, 3.6 & 3.7 on x64 & x86) are tested.
Tests are split into 2 categories "unit" or "integration".

interpreter	unit	integration
2.7 x86	pz	p
2.7 x64	pz	z
3.5 x86	pz
3.5 x64	pz	z
3.6 x86	pz
3.6 x64	p	p
3.7 x86	z
3.7 x64	z

p means tested on AppVeyor, z means tested on Azure (and pz on both)

Azure

It takes:

• ~ 2/3 min for unit tests
• ~ 33 min for unit + integration tests

and currently performs 8 unit + 2 integration in ~ 43 min

Appveyor

It takes:

• ~ 2 min for unit tests
• ~ 10 / 20 min for unit + integration tests (with x64 much faster than x86)

and currently performs 6 unit + 2 integration in ~ 36 min

Suggestion

interpreter	unit	integration
2.7 x86	z	z
2.7 x64	p	p
3.5 x86	z
3.5 x64	p	p
3.6 x86	z
3.6 x64	p	p
3.7 x86	z
3.7 x64	z	z

To remove all overlap and prefer x64 over x86 on Appveyor since it seems significantly faster (while appearing the same on Azure).
(Edited: this was a false impression, the difference came from python 2.7 vs 3.6 instead)

Azure would now be testing 5 unit + 2 inte (from 8 +2)
Appveyor would now be testing 3 unit + 3 inte (from 6 + 2)

Edited:

• moved 3.7 integration to Azure since Appveyor is picky and fails with ERROR: virtualenv is not compatible with this system or executable

[webknjaz]

That's a great idea! I just want to point out that Travis CI also supports Windows so maybe it makes sense to also put 1-2 jobs there?

[pradyunsg]

My opinion: Travis's Windows support is experimental so I'd prefer to avoid it. 🤷🏻‍♂️

[webknjaz]

Which may mean that testing in another weird env can reveal more corner cases.

[pradyunsg]

One of the consequences of #6790 is that AppVeyor should now be "required" for CI. It's still the slowest CI we have, and it only has 1 runner for all PRs.

I propose dropping the 3.6 integration tests from AppVeyor and moving it to Azure Pipelines.

[webknjaz]

Fair enough

[ichard26]

Is this issue still relevant? Windows CI is split across four GHA jobs and are definitely not the slowest jobs today (MacOS jobs are, example).

[webknjaz]

No, AppVeyor is no longer in use, neither is Azure Pipelines. And I don't think they're coming back since Pradyun mentioned getting the org into some plan with a lot more GHA compute for free in the winter.