Refactoring of testing infrastructure

[ByteB4rb1e]

The testing infrastructure for pipenv is serving its purpose well, but as the contributor base diversifies across different platforms and environments, certain limitations are surfacing. This proposal aims to refator the test suite to improve cross-platform compatibility, reduce maintenance overhead, and align with pypa conventions. By adopting tools like tox, modularizing workflows, and ensuring local executability, it tries to address critical pain points while enhancing accessibility for contributors, including those using unconventional environments like MSYS2.
My proposal is designed to uphold pipenv's commitment to a seamless developer experience while minimizing disruption for existing contributors.

The initiative for that came through PR #6373, where I struggled to set up the testing infrastructure within my development environment, which is MSYS2, an environment consisting of a POSIX middleware and a collection of common UNIX and GNU tools running natively under Windows.

I am a TUI user and am currently using MSYS2 as my daily driver, moving away from macOS at work and Ubuntu in private. The reason for that is because the majority of my colleagues and customers use Windows and I'd like to be able to support them better. Also, Windows in conjunction with MSYS2 has a few very unique value propositions. It enables me to stay within a POSIX environment, while using UNIX tooling on Windows natively and having access to Windows facilities such as the Windows Forms API. This allows me to rapidly integrate TUI applications for GUI users and makes me a better, less opinionated collaborator.

The first approach is to move (light) testing facilities like Linting and Vendoring away from pre-commit to tox (pypa) and just wrap it by pre-commit for CI and contributors that are unfamiliar with the tox interface. This allows for contributors like me to use their native development environments without virtualization. This is necessary due to the fact, that the author of pre-commit is unwilling to support environments such as MSYS2 out of personal preference, since they consider Windows to be the "wrong platform" and POSIX-emulation on Windows to be "the wrong convention". If there are any concerns for supporting the MSYS2, or other unconventional environments, I'd be happy to address them directly.

nox would be an alternative to tox, but I suggest tox over nox, as it uses static configuration (compared to nox using dynamic configuration) which hinders the introduction of side-effects.

In addition to migrating light testing facilities away from pre-commit, I suggest adapting the current CI (heavy) testing workflow to also be executable locally, without introducing additional maintenance overhead. This could be achieved through an integration with a python executable management tools like pythonbrew and pythonz, or pyenv, so that all tests (except for the matrix, multi-platform facilities) can be executed locally and therefore reduce the overall build minutes of the CI, in addition to giving contributors a bigger toolset for validating their code integrations.

I haven't yet grasped the development philosophy of pipenv, so I'm not able to make a recommndation yet on which of the previously mentioned tools for Python executable management aligns best with the pipenv methodology. Since tox allows for great modularization, the introduction of this sort of tooling is optional though, so wouldn't introduce any complexities for contributors unfamiliar with testing setups for multiple Python executable versions.

All of this would keep the testing environment contained within the Python and pypa ecosystem, in addition to removing the dependency of GitHub workflows (except for matrix multi-platform tests) in regards to testing.

Outline of steps I am proposing and will refine if the overall proposal is of interest to the project:

• resolve all open issues in regards to testing infrastructure
• migrate (light) testing orchestration from pre-commit to tox
• wrap tox by pre-commit
• migrate (heavy) testing from CI to tox, without breaking multi-platform testing
• wrap tox by CI

Kind regards

[matteius]

Hey @ByteB4rb1e thank you for taking the time to write this up and for helping with some things in the project recently. I am not sure I understand the motivation for adopting tox -- we are using pytest. pre-commit is the linting hooks, not the testing framework. Have you checked how the github CI sets up the pypiserver which hosts the local artifacts for testing non pypi installs? Id suggesting giving a read through: https://github.com/pypa/pipenv/blob/main/.github/workflows/ci.yaml

[ByteB4rb1e]

Hi. Please excuse the delay in response.

I am not sure I understand the motivation for adopting tox

Over the years many different concepts for testing have been introduced throughout the different ecosystems, this includes not only the Python ecosystem. It's hard for me to find a common wording which is explicit about what I mean and understandable by most peers. Pardon. I try to be as generic as possible when talking about testing and am primarily always referring to the historic origin of testing in the sense of static code analysis, which I generically refer to as light testing and anything that came after that, which is done against the runtime environment, as hard testing. It's @ByteB4rb1e speak... Sorry. So linting, to me, is also a kind of (light) test, which probably is because of my UNIX perspective. Though the PyPa fellow pip project uses nox (not tox), I'm speculating they have a similar viewpoint, or at least they're (mis)using nox for executing their linting.

Have you checked how the github CI sets up the pypiserver which hosts the local artifacts for testing non pypi installs?

Yes, I've looked at the CI configuration and that's where the initiative for this discussion arose from. I ran into a couple of issue while locally working with the pipenv testing environment.

The pre-commit project is intolerant towards MinGW environments (POSIX on Windows). I've provided the same fix, as for pipenv, it was rejected immediately, with a reasoning of Windows being the wrong platform, and MinGW using the wrong (POSIX) convention. The rationale feels limiting, especially as pre-commit is increasingly adopted by many projects.. Because of the lack of (technical) tolerance, I'm forced to switch to a WSL distribution just to execute pre-commit. Decoupling whatever pre-commit is wrapping would allow for more developers on more platforms to participate in development through their native development environments. I'm not suggesting to ditch pre-commit altogether, just to decouple.

The dependencies required by the test environment referenced in the CI configuration were difficult for me to set up locally and I had to manually override a few pytest configurations for them to work. Introducing an automation tool, explicitly for wrapping the set up and tear down of dependencies could reduce the effort of setting up a test environment locally and encouraging contributors to adopt shift-left testing, this could result in fewer roundtrips of approving CI executions by maintainers, as well as reduce the CI execution time, and therefore cost. I’m unaware of the specific GitHub plan in use, but reducing CI execution time could still lead to measurable efficiency improvements.

tox and nox could serve as such test wrappers in a Pythonic manner. tox is my personal preference as it uses static configuration and therefore hinders the introduction of side-effects. nox uses dynamic configurations and it can be misused for other tasks quite easily. It's, nonetheless, of the same quality as tox. Both serve the same purpose though of providing a decoupled wrapper for testing environments.

If there are any specific aspects that require further clarification or alignment with the project’s philosophy, I’d be more than happy to iterate on this further.

I'm also going to reread the Getting started guide to see if the proposal I'm raising is really founded or would only introduce unnecessary complexity.

[ByteB4rb1e]

I've slept over it and I don't want to set a precedence for these types of discussions. Slowly familiarizing myself with the pipenv development, I think it's better suited for contributions through resolving already open issues, rather than conceptualizing marginal improvements.

I'll try to contribute whenever I need to take a step back from my own endeavors.

Cheers!