|
| 1 | +.. _tools: |
| 2 | + |
| 3 | +******************************************** |
| 4 | +Testing and Ensuring Type Annotation Quality |
| 5 | +******************************************** |
| 6 | + |
| 7 | +Testing Annotation Accuracy |
| 8 | +=========================== |
| 9 | + |
| 10 | +When creating a package with type annotations, authors may want to validate |
| 11 | +that the annotations they publish meet their expectations. |
| 12 | +This is especially important for library authors, for whom the published |
| 13 | +annotations are part of the public interface to their package. |
| 14 | + |
| 15 | +There are several approaches to this problem, and this document will show |
| 16 | +a few of them. |
| 17 | + |
| 18 | +.. note:: |
| 19 | + |
| 20 | + For simplicity, we will assume that type-checking is done with ``mypy``. |
| 21 | + Many of these strategies can be applied to other type-checkers as well. |
| 22 | + |
| 23 | +Testing Using ``mypy --warn-unused-ignores`` |
| 24 | +-------------------------------------------- |
| 25 | + |
| 26 | +Clever use of ``--warn-unused-ignores`` can be used to check that certain |
| 27 | +expressions are or are not well-typed. |
| 28 | + |
| 29 | +The idea is to write normal python files which contain valid expressions along |
| 30 | +with invalid expressions annotated with ``type: ignore`` comments. When |
| 31 | +``mypy --warn-unused-ignores`` is run on these files, it should pass. |
| 32 | +A directory of test files, ``typing_tests/``, can be maintained. |
| 33 | + |
| 34 | +This strategy does not offer strong guarantees about the types under test, but |
| 35 | +it requires no additional tooling. |
| 36 | + |
| 37 | +If the following file is under test |
| 38 | + |
| 39 | +.. code-block:: python |
| 40 | +
|
| 41 | + # foo.py |
| 42 | + def bar(x: int) -> str: |
| 43 | + return str(x) |
| 44 | +
|
| 45 | +Then the following file tests ``foo.py``: |
| 46 | + |
| 47 | +.. code-block:: python |
| 48 | +
|
| 49 | + bar(42) |
| 50 | + bar("42") # type: ignore [arg-type] |
| 51 | + bar(y=42) # type: ignore [call-arg] |
| 52 | + r1: str = bar(42) |
| 53 | + r2: int = bar(42) # type: ignore [assignment] |
| 54 | +
|
| 55 | +Checking ``reveal_type`` output from ``mypy.api.run`` |
| 56 | +----------------------------------------------------- |
| 57 | + |
| 58 | +``mypy`` provides a subpackage named ``api`` for invoking ``mypy`` from a |
| 59 | +python process. In combination with ``reveal_type``, this can be used to write |
| 60 | +a function which gets the ``reveal_type`` output from an expression. Once |
| 61 | +that's obtained, tests can assert strings and regular expression matches |
| 62 | +against it. |
| 63 | + |
| 64 | +This approach requires writing a set of helpers to provide a good testing |
| 65 | +experience, and it runs mypy once per test case (which can be slow). |
| 66 | +However, it builds only on ``mypy`` and the test framework of your choice. |
| 67 | + |
| 68 | +The following example could be integrated into a testsuite written in |
| 69 | +any framework: |
| 70 | + |
| 71 | +.. code-block:: python |
| 72 | +
|
| 73 | + import re |
| 74 | + from mypy import api |
| 75 | +
|
| 76 | + def get_reveal_type_output(filename): |
| 77 | + result = api.run([filename]) |
| 78 | + stdout = result[0] |
| 79 | + match = re.search(r'note: Revealed type is "([^"]+)"', stdout) |
| 80 | + assert match is not None |
| 81 | + return match.group(1) |
| 82 | +
|
| 83 | +
|
| 84 | +For example, we can use the above to provide a ``run_reveal_type`` pytest |
| 85 | +fixture which generates a temporary file and uses it as the input to |
| 86 | +``get_reveal_type_output``: |
| 87 | + |
| 88 | +.. code-block:: python |
| 89 | +
|
| 90 | + import os |
| 91 | + import pytest |
| 92 | +
|
| 93 | + @pytest.fixture |
| 94 | + def _in_tmp_path(tmp_path): |
| 95 | + cur = os.getcwd() |
| 96 | + try: |
| 97 | + os.chdir(tmp_path) |
| 98 | + yield |
| 99 | + finally: |
| 100 | + os.chdir(cur) |
| 101 | +
|
| 102 | + @pytest.fixture |
| 103 | + def run_reveal_type(tmp_path, _in_tmp_path): |
| 104 | + content_path = tmp_path / "reveal_type_test.py" |
| 105 | +
|
| 106 | + def func(code_snippet, *, preamble = ""): |
| 107 | + content_path.write_text(preamble + f"reveal_type({code_snippet})") |
| 108 | + return get_reveal_type_output("reveal_type_test.py") |
| 109 | +
|
| 110 | + return func |
| 111 | +
|
| 112 | +
|
| 113 | +For more details, see `the documentation on mypy.api |
| 114 | +<https://mypy.readthedocs.io/en/stable/extending_mypy.html#integrating-mypy-into-another-python-application>`_. |
| 115 | + |
| 116 | +pytest-mypy-plugins |
| 117 | +------------------- |
| 118 | + |
| 119 | +`pytest-mypy-plugins <https://github.com/typeddjango/pytest-mypy-plugins>`_ is |
| 120 | +a plugin for ``pytest`` which defines typing test cases as YAML data. |
| 121 | +The test cases are run through ``mypy`` and the output of ``reveal_type`` can |
| 122 | +be asserted. |
| 123 | + |
| 124 | +This project supports complex typing arrangements like ``pytest`` parametrized |
| 125 | +tests and per-test ``mypy`` configuration. It requires that you are using |
| 126 | +``pytest`` to run your tests, and runs ``mypy`` in a subprocess per test case. |
| 127 | + |
| 128 | +This is an example of a parametrized test with ``pytest-mypy-plugins``: |
| 129 | + |
| 130 | +.. code-block:: yaml |
| 131 | +
|
| 132 | + - case: with_params |
| 133 | + parametrized: |
| 134 | + - val: 1 |
| 135 | + rt: builtins.int |
| 136 | + - val: 1.0 |
| 137 | + rt: builtins.float |
| 138 | + main: | |
| 139 | + reveal_type({[ val }}) # N: Revealed type is '{{ rt }}' |
| 140 | +
|
| 141 | +Improving Type Completeness |
| 142 | +=========================== |
| 143 | + |
| 144 | +One of the goals of many libraries is to ensure that they are "fully type |
| 145 | +annotated", meaning that they provide complete and accurate type annotations |
| 146 | +for all functions, classes, and objects. Having full annotations is referred to |
| 147 | +as "type completeness" or "type coverage". |
| 148 | + |
| 149 | +Here are some tips for increasing the type completeness score for your |
| 150 | +library: |
| 151 | + |
| 152 | +- Make type completeness an output of your testing process. Several type |
| 153 | + checkers have options for generating useful output, warnings, or even |
| 154 | + reports. |
| 155 | +- If your package includes tests or sample code, consider removing them |
| 156 | + from the distribution. If there is good reason to include them, |
| 157 | + consider placing them in a directory that begins with an underscore |
| 158 | + so they are not considered part of your library’s interface. |
| 159 | +- If your package includes submodules that are meant to be |
| 160 | + implementation details, rename those files to begin with an |
| 161 | + underscore. |
| 162 | +- If a symbol is not intended to be part of the library’s interface and |
| 163 | + is considered an implementation detail, rename it such that it begins |
| 164 | + with an underscore. It will then be considered private and excluded |
| 165 | + from the type completeness check. |
| 166 | +- If your package exposes types from other libraries, work with the |
| 167 | + maintainers of these other libraries to achieve type completeness. |
| 168 | + |
| 169 | +.. warning:: |
| 170 | + |
| 171 | + The ways in which different type checkers evaluate and help you achieve |
| 172 | + better type coverage may differ. Some of the above recommendations may or |
| 173 | + may not be helpful to you, depending on which type checking tools you use. |
| 174 | + |
| 175 | +``mypy`` disallow options |
| 176 | +------------------------- |
| 177 | + |
| 178 | +``mypy`` offers several options which can detect untyped code. |
| 179 | +More details can be found in `the mypy documentation on these options |
| 180 | +<https://mypy.readthedocs.io/en/latest/command_line.html#untyped-definitions-and-calls>`_. |
| 181 | + |
| 182 | +Some basic usages which make ``mypy`` error on untyped data are:: |
| 183 | + |
| 184 | + mypy --disallow-untyped-defs |
| 185 | + mypy --disallow-incomplete-defs |
| 186 | + |
| 187 | +``pyright`` type verification |
| 188 | +----------------------------- |
| 189 | + |
| 190 | +pyright has a special command line flag, ``--verifytypes``, for verifying |
| 191 | +type completeness. You can learn more about it from |
| 192 | +`the pyright documentation on verifying type completeness |
| 193 | +<https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness>`_. |
| 194 | + |
| 195 | +``mypy`` reports |
| 196 | +---------------- |
| 197 | + |
| 198 | +``mypy`` offers several options options for generating reports on its analysis. |
| 199 | +See `the mypy documentation on report generation |
| 200 | +<https://mypy.readthedocs.io/en/stable/command_line.html#report-generation>`_ for details. |
0 commit comments