Protocols (#3132)

This is an implementation of protocols proposed in PEP 544. This PR adds 
support for:

* generic protocols (including inference for protocols)
* recursive protocols
* special support for Type[] and isinstance()
* structural subtyping for Callable and Tuple
* other things

Author: ilevkivskyi

docs/source/class_basics.rst

@@ -151,7 +151,174 @@ concrete. As with normal overrides, a dynamically typed method can
 implement a statically typed abstract method defined in an abstract
 base class.
 
+.. _protocol-types:
+
+Protocols and structural subtyping
+**********************************
+
+.. note::
+
+   The support for structural subtyping is still experimental. Some features
+   might be not yet implemented, mypy could pass unsafe code or reject
+   working code.
+
+There are two main type systems with respect to subtyping: nominal subtyping
+and structural subtyping. The *nominal* subtyping is based on class hierarchy,
+so that if class ``D`` inherits from class ``C``, then it is a subtype
+of ``C``. This type system is primarily used in mypy since it allows
+to produce clear and concise error messages, and since Python provides native
+``isinstance()`` checks based on class hierarchy. The *structural* subtyping
+however has its own advantages. In this system class ``D`` is a subtype
+of class ``C`` if the former has all attributes of the latter with
+compatible types.
+
+This type system is a static equivalent of duck typing, well known by Python
+programmers. Mypy provides an opt-in support for structural subtyping via
+protocol classes described in this section.
+See `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ for
+specification of protocols and structural subtyping in Python.
+
+User defined protocols
+**********************
+
+To define a protocol class, one must inherit the special
+``typing_extensions.Protocol`` class:
+
+.. code-block:: python
+
+   from typing import Iterable
+   from typing_extensions import Protocol
+
+   class SupportsClose(Protocol):
+       def close(self) -> None:
+          ...
+
+   class Resource:  # Note, this class does not have 'SupportsClose' base.
+       # some methods
+       def close(self) -> None:
+          self.resource.release()
+
+   def close_all(things: Iterable[SupportsClose]) -> None:
+       for thing in things:
+           thing.close()
+
+   close_all([Resource(), open('some/file')])  # This passes type check
+
+.. note::
+
+   The ``Protocol`` base class is currently provided in ``typing_extensions``
+   package. When structural subtyping is mature and
+   `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ is accepted,
+   ``Protocol`` will be included in the ``typing`` module. As well, several
+   types such as ``typing.Sized``, ``typing.Iterable`` etc. will be made
+   protocols.
+
+Defining subprotocols
+*********************
+
+Subprotocols are also supported. Existing protocols can be extended
+and merged using multiple inheritance. For example:
+
+.. code-block:: python
+
+   # continuing from previous example
+
+   class SupportsRead(Protocol):
+       def read(self, amount: int) -> bytes: ...
+
+   class TaggedReadableResource(SupportsClose, SupportsRead, Protocol):
+       label: str
+
+   class AdvancedResource(Resource):
+       def __init__(self, label: str) -> None:
+           self.label = label
+       def read(self, amount: int) -> bytes:
+           # some implementation
+           ...
+
+   resource = None  # type: TaggedReadableResource
+
+   # some code
+
+   resource = AdvancedResource('handle with care')  # OK
+
+Note that inheriting from existing protocols does not automatically turn
+a subclass into a protocol, it just creates a usual (non-protocol) ABC that
+implements given protocols. The ``typing_extensions.Protocol`` base must always
+be explicitly present:
+
+.. code-block:: python
+
+   class NewProtocol(SupportsClose):  # This is NOT a protocol
+       new_attr: int
+
+   class Concrete:
+      new_attr = None  # type: int
+      def close(self) -> None:
+          ...
+   # Below is an error, since nominal subtyping is used by default
+   x = Concrete()  # type: NewProtocol  # Error!
+
+.. note::
+
+   The `PEP 526 <https://www.python.org/dev/peps/pep-0526/>`_ variable
+   annotations can be used to declare protocol attributes. However, protocols
+   are also supported on Python 2.7 and Python 3.3+ with the help of type
+   comments and properties, see
+   `backwards compatibility in PEP 544 <https://www.python.org/dev/peps/pep-0544/#backwards-compatibility>`_.
+
+Recursive protocols
+*******************
+
+Protocols can be recursive and mutually recursive. This could be useful for
+declaring abstract recursive collections such as trees and linked lists:
+
+.. code-block:: python
+
+   from typing import TypeVar, Optional
+   from typing_extensions import Protocol
+
+   class TreeLike(Protocol):
+       value: int
+       @property
+       def left(self) -> Optional['TreeLike']: ...
+       @property
+       def right(self) -> Optional['TreeLike']: ...
+
+   class SimpleTree:
+       def __init__(self, value: int) -> None:
+           self.value = value
+           self.left: Optional['SimpleTree'] = None
+           self.right: Optional['SimpleTree'] = None
+
+   root = SimpleTree(0)  # type: TreeLike  # OK
+
+Using ``isinstance()`` with protocols
+*************************************
+
+To use a protocol class with ``isinstance()``, one needs to decorate it with
+a special ``typing_extensions.runtime`` decorator. It will add support for
+basic runtime structural checks:
+
+.. code-block:: python
+
+   from typing_extensions import Protocol, runtime
+
+   @runtime
+   class Portable(Protocol):
+       handles: int
+
+   class Mug:
+       def __init__(self) -> None:
+           self.handles = 1
+
+   mug = Mug()
+   if isinstance(mug, Portable):
+      use(mug.handles)  # Works statically and at runtime.
+
 .. note::
+   ``isinstance()`` is with protocols not completely safe at runtime.
+   For example, signatures of methods are not checked. The runtime
+   implementation only checks the presence of all protocol members
+   in object's MRO.
 
-   There are also plans to support more Python-style "duck typing" in
-   the type system. The details are still open.

docs/source/common_issues.rst

@@ -226,6 +226,48 @@ Possible strategies in such situations are:
          return x[0]
      f_good(new_lst) # OK
 
+Covariant subtyping of mutable protocol members is rejected
+-----------------------------------------------------------
+
+Mypy rejects this because this is potentially unsafe.
+Consider this example:
+
+.. code-block:: python
+
+   from typing_extensions import Protocol
+
+   class P(Protocol):
+       x: float
+
+   def fun(arg: P) -> None:
+       arg.x = 3.14
+
+   class C:
+       x = 42
+   c = C()
+   fun(c)  # This is not safe
+   c.x << 5  # Since this will fail!
+
+To work around this problem consider whether "mutating" is actually part
+of a protocol. If not, then one can use a ``@property`` in
+the protocol definition:
+
+.. code-block:: python
+
+   from typing_extensions import Protocol
+
+   class P(Protocol):
+       @property
+       def x(self) -> float:
+          pass
+
+   def fun(arg: P) -> None:
+       ...
+
+   class C:
+       x = 42
+   fun(C())  # OK
+
 Declaring a supertype as variable type
 --------------------------------------
 

docs/source/faq.rst

@@ -101,35 +101,38 @@ Is mypy free?
 Yes. Mypy is free software, and it can also be used for commercial and
 proprietary projects. Mypy is available under the MIT license.
 
-Why not use structural subtyping?
-*********************************
+Can I use structural subtyping?
+*******************************
 
-Mypy primarily uses `nominal subtyping
-<https://en.wikipedia.org/wiki/Nominative_type_system>`_ instead of
+Mypy provides support for both `nominal subtyping
+<https://en.wikipedia.org/wiki/Nominative_type_system>`_ and
 `structural subtyping
-<https://en.wikipedia.org/wiki/Structural_type_system>`_. Some argue
-that structural subtyping is better suited for languages with duck
-typing such as Python.
-
-Here are some reasons why mypy uses nominal subtyping:
+<https://en.wikipedia.org/wiki/Structural_type_system>`_.
+Support for structural subtyping is considered experimental.
+Some argue that structural subtyping is better suited for languages with duck
+typing such as Python. Mypy however primarily uses nominal subtyping,
+leaving structural subtyping opt-in. Here are some reasons why:
 
 1. It is easy to generate short and informative error messages when
    using a nominal type system. This is especially important when
    using type inference.
 
-2. Python supports basically nominal isinstance tests and they are
-   widely used in programs. It is not clear how to support isinstance
-   in a purely structural type system while remaining compatible with
-   Python idioms.
+2. Python provides built-in support for nominal ``isinstance()`` tests and
+   they are widely used in programs. Only limited support for structural
+   ``isinstance()`` exists for ABCs in ``collections.abc`` and ``typing``
+   standard library modules.
 
 3. Many programmers are already familiar with nominal subtyping and it
    has been successfully used in languages such as Java, C++ and
    C#. Only few languages use structural subtyping.
 
-However, structural subtyping can also be useful. Structural subtyping
-is a likely feature to be added to mypy in the future, even though we
-expect that most mypy programs will still primarily use nominal
-subtyping.
+However, structural subtyping can also be useful. For example, a "public API"
+may be more flexible if it is typed with protocols. Also, using protocol types
+removes the necessity to explicitly declare implementations of ABCs.
+As a rule of thumb, we recommend using nominal classes where possible, and
+protocols where necessary. For more details about protocol types and structural
+subtyping see :ref:`protocol-types` and
+`PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_.
 
 I like Python and I have no need for static typing
 **************************************************

docs/source/generics.rst

@@ -1,6 +1,8 @@
 Generics
 ========
 
+.. _generic-classes:
+
 Defining generic classes
 ************************
 
@@ -489,6 +491,73 @@ restrict the valid values for the type parameter in the same way.
 A type variable may not have both a value restriction (see
 :ref:`type-variable-value-restriction`) and an upper bound.
 
+Generic protocols
+*****************
+
+Generic protocols (see :ref:`protocol-types`) are also supported, generic
+protocols mostly follow the normal rules for generic classes, the main
+difference is that mypy checks that declared variance of type variables is
+compatible with the class definition. Examples:
+
+.. code-block:: python
+
+   from typing import TypeVar
+   from typing_extensions import Protocol
+
+   T = TypeVar('T')
+
+   class Box(Protocol[T]):
+       content: T
+
+   def do_stuff(one: Box[str], other: Box[bytes]) -> None:
+       ...
+
+   class StringWrapper:
+       def __init__(self, content: str) -> None:
+           self.content = content
+
+   class BytesWrapper:
+       def __init__(self, content: bytes) -> None:
+           self.content = content
+
+   do_stuff(StringWrapper('one'), BytesWrapper(b'other'))  # OK
+
+   x = None  # type: Box[float]
+   y = None  # type: Box[int]
+   x = y  # Error, since the protocol 'Box' is invariant.
+
+   class AnotherBox(Protocol[T]):  # Error, covariant type variable expected
+       def content(self) -> T:
+           ...
+
+   T_co = TypeVar('T_co', covariant=True)
+   class AnotherBox(Protocol[T_co]):  # OK
+       def content(self) -> T_co:
+           ...
+
+   ax = None  # type: AnotherBox[float]
+   ay = None  # type: AnotherBox[int]
+   ax = ay  # OK for covariant protocols
+
+See :ref:`variance-of-generics` above for more details on variance.
+Generic protocols can be recursive, for example:
+
+.. code-block:: python
+
+   T = TypeVar('T')
+   class Linked(Protocol[T]):
+       val: T
+       def next(self) -> 'Linked[T]': ...
+
+   class L:
+       val: int
+       def next(self) -> 'L': ...
+
+   def last(seq: Linked[T]) -> T:
+       ...
+
+   result = last(L())  # The inferred type of 'result' is 'int'
+
 .. _declaring-decorators:
 
 Declaring decorators