feat: Support extras in stubtest_third_party.py #8467

kkirsche · 2022-08-02T14:17:53Z

This pull request adds a new key to the METADATA.toml file, under the tool.stubtest key, named extras_dependencies. This key is to allow the use of extras when running stubtest in upstream tests.

For example, the current schema would change from:

version = ...
requires = ...
extra_description = ...
obsolete_since = ...
no_longer_updated = ...

[tool.stubtest]
skip = ...
apt_dependencies = ...
ignore_missing_stubs = ...

to:

version = ...
requires = ...
extra_description = ...
obsolete_since = ...
no_longer_updated = ...

[tool.stubtest]
skip = ...
apt_dependencies = ...
extras_dependencies = ["sock"]; # the list of names of extras to install.
ignore_missing_stubs = ...

The reason for this is that when a stub, such as urllib3, which provides contributor functionality based on other libraries, such as PySock, may have a complex version dependency. In those cases, rather than trying to ensure that the version specification is manually or automatically detected as updated, this bypasses the issue by ensuring that we're using the version specification defined by the upstream library, tracking it as it changes.

AlexWaygood · 2022-08-02T14:22:39Z

Thanks!

Do any of our stubs packages that currently have requirements-stubtest.txt files specify extra dependencies at runtime? If so, can you update the stubs for those packages so that they use this new key, instead of the requirements-stubtest.txt file, as part of this PR? That way we can see whether stubtest still works correctly on those packages with this change.

kkirsche · 2022-08-02T15:05:46Z

Thanks!

Do any of our stubs packages that currently have requirements-stubtest.txt files specify extra dependencies at runtime? If so, can you update the stubs for those packages so that they use this new key, instead of the requirements-stubtest.txt file, as part of this PR? That way we can see whether stubtest still works correctly on those packages with this change.

I've updated all the stubs that currently use extras. Depending on the merge order between this and #8457, I'll merge and update whichever merge request gets merged second with any required changes

hauntsaninja

extras_dependencies is a little confusing to me, since it's not clear whether it's a list of names of extras or a list of names of dependencies. Maybe extras or extras_to_install?

hauntsaninja · 2022-08-02T18:38:44Z

tests/stubtest_third_party.py

-        dist_req = f"{dist.name}=={dist_version}"
+        assert isinstance(extras_dependencies, list)
+        dist_extras = ", ".join(extras_dependencies)
+        dist_req = f"{dist.name}[{dist_extras}]=={dist_version}"


Huh, didn't realise this, but pip install pkg[] does work

Yep, as long as you conform to the structure, the behavior will work. It's only if you break the calling structure that it fails to execute :)

stubs/html5lib/METADATA.toml

kkirsche · 2022-08-02T18:53:24Z

extras_dependencies is a little confusing to me, since it's not clear whether it's a list of names of extras or a list of names of dependencies. Maybe extras or extras_to_install?

I'm perfectly happy with those, but will through out a few others in case they help:

requires_extras
stubs_use_extras
tests_require_extras
stubs_require_extras

Just let me know the preference, and I can proceed with that

* master: Add `urllib3.contrib.socks`; improve `urllib3.connectionpool` (#8457) Improve `multiprocessing.context` module (#8466) Always return `True` from `xml.dom.minidom.Node.__bool__` (#8480) Adds missing `__dir__` definitions (#8479) Various pyvmomi improvements (#8469) Bump pytype version. (#8478) Remove `Enum.__hash__`, improve `Enum.__reduce_ex__` (#8477) refactor: prefer f-strings to other format/concatentation styles (#8474) Remove redundant `__str__` methods (#8475) Add some missing stubs in datetimerange (#8470) tkinter: get rid of unnecessary Incomplete (#8471) Improve urllib3.util.url annotations (#8460) Fix `complex` constructor (#8473) [pre-commit.ci] pre-commit autoupdate (#8461) Use typeshed.Incomplete and object instead of Any in tkinter stubs (#8458) `io.IOBase`: correct metaclass (#8468)

kkirsche · 2022-08-04T14:29:30Z

Updated:

Merged latest master to pick up the changes from Add urllib3.contrib.socks; improve urllib3.connectionpool #8457
Updated Add urllib3.contrib.socks; improve urllib3.connectionpool #8457 to install extras via this feature.
Updated extras_dependencies to extras
Clarification is required (either from discussion or trial and error about test failures) from the team about feat: Support extras in stubtest_third_party.py #8467 (comment) as lxml (as I understand it) would not be installed with the extras.

kkirsche · 2022-08-04T14:54:13Z

#8439 will, if merged, also be a candidate for this via https://github.com/psf/requests/blob/main/setup.py#L125

EDIT: fix the links

* master: (51 commits) Update pytype_test to be compatible with the latest pytype version. (#8551) Normalise use of `Never` vs `NoReturn` (#8549) Add stubs for tree-sitter-languages (#8548) Add SAFE_RESTARTABLE strategy to ldap3 Connection client_strategy parameter (#8547) Add stubs for tree_sitter (#8533) Additional return types for psycopg2 connections (#8528) fpdf2: fix for 2.5.6 changes (#8546) cryptography: Rename parameter of serialize_key_and_certificates (#8543) Add `_threading_local.local.__delattr__` back to the stub (#8526) Fix `UDPServer` to correctly inherit from `TCPServer` (#8542) Upgrade pyright (#8541) Add "Naming convention" to `test_cases/README.md` (#8521) Improve `.keys()`, `.values()`, `.items()` methods for `TypedDict`s (#8532) setuptools: fix stubtest (#8540) Annotate Model in SQLAlchemy (#8535) Unpin stubtest from 3.10.5 (#8523) [stubsabot] Bump setuptools to 64.0.* (#8534) Add `@type_check_only` to two fictional classes in `builtins.pyi` (#8531) Enable flake8-pyi's Y034 check in `stdlib/typing.pyi` (#8530) setuptools: delete _distutils.command.check.HAS_DOCUTILS (#8529) ...

kkirsche · 2022-08-18T12:35:40Z

Merge request updated with the extras for requests since #8485 was merged.

This is using the socks extra seen here:
https://github.com/psf/requests/blob/main/setup.py#L125

github-actions · 2022-08-18T20:52:57Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

hauntsaninja

Thanks for updating, sorry for the delay here! :-)

kkirsche · 2022-08-19T00:03:22Z

Thanks for updating, sorry for the delay here! :-)

No worries, life happens 🙂 thanks for your time

feat: Support extras in stubtest_third_party.py

7842708

kkirsche mentioned this pull request Aug 2, 2022

Add urllib3.contrib.socks; improve urllib3.connectionpool #8457

Merged

fix: use extras_dependencies key in packages that allow for it

bfe141b

This comment has been minimized.

Sign in to view

hauntsaninja reviewed Aug 2, 2022

View reviewed changes

Kevin Kirsche added 3 commits August 4, 2022 10:23

refactor: rename extras_dependencies to extras

54a39b2

refactor: use extras to install urllib3 pysocks

3a7f481

This comment has been minimized.

Sign in to view

This was referenced Aug 4, 2022

refactor: Add missing types related to requests #8439

Closed

refactor: use re-exports in requests rather than assignments #8485

Merged

JelleZijlstra approved these changes Aug 13, 2022

View reviewed changes

Kevin Kirsche added 2 commits August 18, 2022 08:31

fix: use extras for requests rather than requirements-stubtest.txt

06ea8d6

This comment has been minimized.

Sign in to view

update: html5lib use all extras per hauntsaninja feedback

4cf8bf8

hauntsaninja approved these changes Aug 18, 2022

View reviewed changes

hauntsaninja merged commit 8c51fab into python:master Aug 18, 2022

kkirsche deleted the feat/stubtest-extras branch September 2, 2022 13:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support extras in stubtest_third_party.py #8467

feat: Support extras in stubtest_third_party.py #8467

kkirsche commented Aug 2, 2022 •

edited

Loading

AlexWaygood commented Aug 2, 2022 •

edited

Loading

This comment has been minimized.

kkirsche commented Aug 2, 2022

hauntsaninja left a comment

hauntsaninja Aug 2, 2022

kkirsche Aug 2, 2022

kkirsche commented Aug 2, 2022

kkirsche commented Aug 4, 2022

This comment has been minimized.

kkirsche commented Aug 4, 2022 •

edited

Loading

kkirsche commented Aug 18, 2022

This comment has been minimized.

github-actions bot commented Aug 18, 2022

hauntsaninja left a comment •

edited

Loading

kkirsche commented Aug 19, 2022

feat: Support extras in stubtest_third_party.py #8467

feat: Support extras in stubtest_third_party.py #8467

Conversation

kkirsche commented Aug 2, 2022 • edited Loading

AlexWaygood commented Aug 2, 2022 • edited Loading

This comment has been minimized.

kkirsche commented Aug 2, 2022

hauntsaninja left a comment

Choose a reason for hiding this comment

hauntsaninja Aug 2, 2022

Choose a reason for hiding this comment

kkirsche Aug 2, 2022

Choose a reason for hiding this comment

kkirsche commented Aug 2, 2022

kkirsche commented Aug 4, 2022

This comment has been minimized.

kkirsche commented Aug 4, 2022 • edited Loading

kkirsche commented Aug 18, 2022

This comment has been minimized.

github-actions bot commented Aug 18, 2022

hauntsaninja left a comment • edited Loading

Choose a reason for hiding this comment

kkirsche commented Aug 19, 2022

kkirsche commented Aug 2, 2022 •

edited

Loading

AlexWaygood commented Aug 2, 2022 •

edited

Loading

kkirsche commented Aug 4, 2022 •

edited

Loading

hauntsaninja left a comment •

edited

Loading