Protocols

docs/source/class_basics.rst

@@ -151,7 +151,101 @@ concrete. As with normal overrides, a dynamically typed method can
 implement a statically typed abstract method defined in an abstract
 base class.
 
+.. _protocol-types:
+
+Protocols and structural subtyping
+**********************************
+
+Mypy provides support for structural subtyping and protocol classes.
+To define a protocol class, one must inherit the special ``typing.Protocol``
+class:
+
+.. code-block:: python
+
+   from typing import Protocol
+
+   class SupportsClose(Protocol):
+       def close(self) -> None:
+          ...
+
+   class UnrelatedClass:
+       # some methods
+       def close(self) -> None:
+          self.resource.release()
+
+   def close_all(things: Sequence[SupportsClose]) -> None:
+       for thing in things:
+           thing.close()
+
+   close_all([UnrelatedClass(), open('some/file')])  # This passes type check
+
+Subprotocols are also supported. Inheriting from an existing protocol does
+not automatically turn a subclass into a protocol, it just creates a usual
+ABC. The ``typing.Protocol`` base must always be explicitly present.
+Generic and recursive protocols are also supported:
+
+.. code-block:: python
+
+   from typing import Protocol, TypeVar
+
+   T = TypeVar('T')
+   class Linked(Protocol[T]):
+       val: T
+       next: 'Linked[T]'
+
+   class L:
+       val: int
+       next: 'L'
+
+   def last(seq: Linked[T]) -> T:
+       ...
+
+   result = last(L())  # The inferred type of 'result' is 'int'
+
+See :ref:`generic-classes` for more details on generic classes.
+The standard ABCs in ``typing`` module are protocols, so that the following
+class will be considered a subtype of ``typing.Sized`` and
+``typing.Iterable[int]``:
+
+.. code-block:: python
+
+   from typing import Iterator, Iterable
+
+   class Bucket:
+       ...
+       def __len__(self) -> int:
+           return 22
+       def __iter__(self) -> Iterator[int]:
+           yield 22
+
+   def collect(items: Iterable[int]) -> int: ...
+   result: int = collect(Bucket())  # Passes type check
+
+To use a protocol class with ``isinstance()``, one needs to decorate it with
+a special ``typing.runtime`` decorator. It will add support for basic runtime
+structural checks:
+
+.. code-block:: python
+
+   from typing import Protocol, runtime
+
+   @runtime
+   class Portable(Protocol):
+       handles: int
+
+   class Mug:
+       def __init__(self) -> None:
+           self.handles = 1
+
+   mug = Mug()
+   if isinstance(mug, Portable):
+      use(mug.handles)  # Works statically and at runtime.
+
+See `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ for
+specification of structural subtyping in Python.
+
 .. note::
 
-   There are also plans to support more Python-style "duck typing" in
-   the type system. The details are still open.
+   The support for structural subtyping is still experimental. Some features
+   might be not yet implemented, mypy could pass unsafe code or reject
+   working code.

docs/source/faq.rst

@@ -101,35 +101,38 @@ Is mypy free?
 Yes. Mypy is free software, and it can also be used for commercial and
 proprietary projects. Mypy is available under the MIT license.
 
-Why not use structural subtyping?
-*********************************
+Can I use structural subtyping?
+*******************************
 
-Mypy primarily uses `nominal subtyping
-<https://en.wikipedia.org/wiki/Nominative_type_system>`_ instead of
+Mypy provides support for both `nominal subtyping
+<https://en.wikipedia.org/wiki/Nominative_type_system>`_ and
 `structural subtyping
 <https://en.wikipedia.org/wiki/Structural_type_system>`_. Some argue
 that structural subtyping is better suited for languages with duck
-typing such as Python.
-
-Here are some reasons why mypy uses nominal subtyping:
+typing such as Python. Mypy however primarily uses nominal subtyping,
+leaving structural subtyping opt-in. Here are some reasons why:
 
 1. It is easy to generate short and informative error messages when
    using a nominal type system. This is especially important when
    using type inference.
 
-2. Python supports basically nominal isinstance tests and they are
-   widely used in programs. It is not clear how to support isinstance
-   in a purely structural type system while remaining compatible with
-   Python idioms.
+2. Python provides built-in support for nominal ``isinstance()`` tests and
+   they are widely used in programs. Only limited support for structural
+   ``isinstance()`` exists for ABCs in ``collections.abc`` and ``typing``
+   standard library modules.
 
 3. Many programmers are already familiar with nominal subtyping and it
    has been successfully used in languages such as Java, C++ and
    C#. Only few languages use structural subtyping.
 
-However, structural subtyping can also be useful. Structural subtyping
-is a likely feature to be added to mypy in the future, even though we
-expect that most mypy programs will still primarily use nominal
-subtyping.
+However, structural subtyping can also be useful. For example, a "public API"
+will be more flexible and convenient for users if it is typed with protocols.
+Also, using protocol types removes the necessity to explicitly declare
+implementations of ABCs. Finally, protocol types may feel more natural for
+Python programmers. As a rule of thumb, one could prefer protocols for
+function argument types and normal classes for return types. For more details
+about protocol types and structural subtyping see :ref:`protocol-types` and
+`PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_.
 
 I like Python and I have no need for static typing
 **************************************************

docs/source/generics.rst

@@ -1,6 +1,8 @@
 Generics
 ========
 
+.. _generic-classes:
+
 Defining generic classes
 ************************
 

mypy/checker.py

@@ -43,7 +43,7 @@
 from mypy.subtypes import (
     is_subtype, is_equivalent, is_proper_subtype, is_more_precise,
     restrict_subtype_away, is_subtype_ignoring_tvars, is_callable_subtype,
-    unify_generic_callable,
+    unify_generic_callable, get_missing_members
 )
 from mypy.maptype import map_instance_to_supertype
 from mypy.typevars import fill_typevars, has_no_typevars
@@ -1216,14 +1216,16 @@ def check_assignment(self, lvalue: Lvalue, rvalue: Expression, infer_lvalue_type
                 else:
                     rvalue_type = self.check_simple_assignment(lvalue_type, rvalue, lvalue)
 
-                # Special case: only non-abstract classes can be assigned to variables
-                # with explicit type Type[A].
+                # Special case: only non-abstract non-protocol classes can be assigned to
+                # variables with explicit type Type[A], where A is protocol or abstract.
                 if (isinstance(rvalue_type, CallableType) and rvalue_type.is_type_obj() and
-                        rvalue_type.type_object().is_abstract and
+                        (rvalue_type.type_object().is_abstract or
+                         rvalue_type.type_object().is_protocol) and
                         isinstance(lvalue_type, TypeType) and
                         isinstance(lvalue_type.item, Instance) and
-                        lvalue_type.item.type.is_abstract):
-                    self.fail("Can only assign non-abstract classes"
+                        (lvalue_type.item.type.is_abstract or
+                         lvalue_type.item.type.is_protocol)):
+                    self.fail("Can only assign concrete classes"
                               " to a variable of type '{}'".format(lvalue_type), rvalue)
                     return
                 if rvalue_type and infer_lvalue_type:
@@ -2289,6 +2291,14 @@ def check_subtype(self, subtype: Type, supertype: Type, context: Context,
             if extra_info:
                 msg += ' (' + ', '.join(extra_info) + ')'
             self.fail(msg, context)
+            if (isinstance(supertype, Instance) and supertype.type.is_protocol and
+                    isinstance(subtype, Instance)):
+                missing = get_missing_members(subtype, supertype)
+                if missing:
+                    self.note("'{}' missing following '{}' protocol members:"
+                              .format(subtype.type.fullname(), supertype.type.fullname()),
+                              context)
+                    self.note(', '.join(missing), context)
             return False
 
     def contains_none(self, t: Type) -> bool:

mypy/checkexpr.py

@@ -35,7 +35,7 @@
 from mypy import join
 from mypy.meet import narrow_declared_type
 from mypy.maptype import map_instance_to_supertype
-from mypy.subtypes import is_subtype, is_equivalent
+from mypy.subtypes import is_subtype, is_equivalent, get_missing_members
 from mypy import applytype
 from mypy import erasetype
 from mypy.checkmember import analyze_member_access, type_object_type, bind_self
@@ -186,6 +186,15 @@ def visit_call_expr(self, e: CallExpr, allow_none_return: bool = False) -> Type:
                 and callee_type.implicit):
             return self.msg.untyped_function_call(callee_type, e)
         ret_type = self.check_call_expr_with_callee_type(callee_type, e)
+        if isinstance(e.callee, RefExpr) and e.callee.fullname in ('builtins.isinstance',
+                                                                   'builtins.issubclass'):
+            for expr in mypy.checker.flatten(e.args[1]):
+                tp = self.chk.type_map[expr]
+                if (isinstance(tp, CallableType) and tp.is_type_obj() and
+                        tp.type_object().is_protocol and
+                        not tp.type_object().runtime_protocol):
+                    self.chk.fail('Only @runtime protocols can be used with'
+                                  ' instance and class checks', e)
         if isinstance(ret_type, UninhabitedType):
             self.chk.binder.unreachable()
         if not allow_none_return and isinstance(ret_type, NoneTyp):
@@ -363,6 +372,11 @@ def check_call(self, callee: Type, args: List[Expression],
                 self.msg.cannot_instantiate_abstract_class(
                     callee.type_object().name(), type.abstract_attributes,
                     context)
+            elif (callee.is_type_obj() and callee.type_object().is_protocol
+                  # Exceptions for Type[...] and classmethod first argument
+                  and not callee.from_type_type and not callee.is_classmethod_class):
+                self.chk.fail('Cannot instantiate protocol class "{}"'
+                              .format(callee.type_object().fullname()), context)
 
             formal_to_actual = map_actuals_to_formals(
                 arg_kinds, arg_names,
@@ -845,19 +859,29 @@ def check_arg(self, caller_type: Type, original_caller_type: Type,
         """Check the type of a single argument in a call."""
         if isinstance(caller_type, DeletedType):
             messages.deleted_as_rvalue(caller_type, context)
-        # Only non-abstract class can be given where Type[...] is expected...
+        # Only non-abstract non-protocol class can be given where Type[...] is expected...
         elif (isinstance(caller_type, CallableType) and isinstance(callee_type, TypeType) and
-              caller_type.is_type_obj() and caller_type.type_object().is_abstract and
-              isinstance(callee_type.item, Instance) and callee_type.item.type.is_abstract and
+              caller_type.is_type_obj() and
+              (caller_type.type_object().is_abstract or caller_type.type_object().is_protocol) and
+              isinstance(callee_type.item, Instance) and
+              (callee_type.item.type.is_abstract or callee_type.item.type.is_protocol) and
               # ...except for classmethod first argument
               not caller_type.is_classmethod_class):
-            messages.fail("Only non-abstract class can be given where '{}' is expected"
+            messages.fail("Only concrete class can be given where '{}' is expected"
                           .format(callee_type), context)
         elif not is_subtype(caller_type, callee_type):
             if self.chk.should_suppress_optional_error([caller_type, callee_type]):
                 return
             messages.incompatible_argument(n, m, callee, original_caller_type,
                                            caller_kind, context)
+            if (isinstance(original_caller_type, Instance) and
+                    isinstance(callee_type, Instance) and callee_type.type.is_protocol):
+                missing = get_missing_members(original_caller_type, callee_type)
+                if missing:
+                    messages.note("'{}' missing following '{}' protocol members:"
+                                  .format(original_caller_type.type.fullname(),
+                                          callee_type.type.fullname()), context)
+                    messages.note(', '.join(missing), context)
 
     def overload_call_target(self, arg_types: List[Type], arg_kinds: List[int],
                              arg_names: List[str],
@@ -2137,8 +2161,9 @@ def check_awaitable_expr(self, t: Type, ctx: Context, msg: str) -> Type:
 
         Also used by `async for` and `async with`.
         """
-        if not self.chk.check_subtype(t, self.named_type('typing.Awaitable'), ctx,
-                                      msg, 'actual type', 'expected type'):
+        if not self.chk.check_subtype(t, self.chk.named_generic_type('typing.Awaitable',
+                                                                     [AnyType()]),
+                                      ctx, msg, 'actual type', 'expected type'):
             return AnyType()
         else:
             method = self.analyze_external_member_access('__await__', t, ctx)
@@ -2478,6 +2503,8 @@ def overload_arg_similarity(actual: Type, formal: Type) -> int:
             # subtyping algorithm if type promotions are possible (e.g., int vs. float).
             if formal.type in actual.type.mro:
                 return 2
+            elif formal.type.is_protocol and is_subtype(actual, erasetype.erase_type(formal)):
+                return 2
             elif actual.type._promote and is_subtype(actual, formal):
                 return 1
             else:

mypy/constraints.py

@@ -314,6 +314,8 @@ def visit_instance(self, template: Instance) -> List[Constraint]:
             actual = actual.fallback
         if isinstance(actual, Instance):
             instance = actual
+            # We always try nominal inference if possible,
+            # it is much faster than the structural one.
             if (self.direction == SUBTYPE_OF and
                     template.type.has_base(instance.type.fullname())):
                 mapped = map_instance_to_supertype(template, instance.type)
@@ -337,6 +339,29 @@ def visit_instance(self, template: Instance) -> List[Constraint]:
                     res.extend(infer_constraints(
                         template.args[j], mapped.args[j], neg_op(self.direction)))
                 return res
+            if (template.type.is_protocol and self.direction == SUPERTYPE_OF and
+                    # We avoid infinite recursion for structural subtypes by checking
+                    # whether this type already appeared in the inference chain.
+                    not any(is_same_type(template, t) for t in template.type.inferring) and
+                    mypy.subtypes.is_subtype(instance, erase_typevars(template))):
+                template.type.inferring.append(template)
+                for member in template.type.protocol_members:
+                    inst = mypy.subtypes.find_member(member, instance, instance)
+                    temp = mypy.subtypes.find_member(member, template, template)
+                    res.extend(infer_constraints(temp, inst, self.direction))
+                template.type.inferring.pop()
+                return res
+            elif (instance.type.is_protocol and self.direction == SUBTYPE_OF and
+                  # We avoid infinite recursion for structural subtypes also here.
+                  not any(is_same_type(instance, i) for i in instance.type.inferring) and
+                  mypy.subtypes.is_subtype(erase_typevars(template), instance)):
+                instance.type.inferring.append(instance)
+                for member in instance.type.protocol_members:
+                    inst = mypy.subtypes.find_member(member, instance, instance)
+                    temp = mypy.subtypes.find_member(member, template, template)
+                    res.extend(infer_constraints(temp, inst, self.direction))
+                instance.type.inferring.pop()
+                return res
         if isinstance(actual, AnyType):
             # IDEA: Include both ways, i.e. add negation as well?
             return self.infer_against_any(template.args)
@@ -350,6 +375,11 @@ def visit_instance(self, template: Instance) -> List[Constraint]:
                 cb = infer_constraints(template.args[0], item, SUPERTYPE_OF)
                 res.extend(cb)
             return res
+        elif isinstance(actual, TupleType) and template.type.is_protocol:
+            if mypy.subtypes.is_subtype(actual.fallback, erase_typevars(template)):
+                res.extend(infer_constraints(template, actual.fallback, self.direction))
+                return res
+            return []
         else:
             return []
 
@@ -380,6 +410,14 @@ def visit_callable_type(self, template: CallableType) -> List[Constraint]:
             return self.infer_against_overloaded(self.actual, template)
         elif isinstance(self.actual, TypeType):
             return infer_constraints(template.ret_type, self.actual.item, self.direction)
+        elif isinstance(self.actual, Instance):
+            # Instances with __call__ method defined are considered structural
+            # subtypes of Callable with a compatible signature.
+            call = mypy.subtypes.find_member('__call__', self.actual, self.actual)
+            if call:
+                return infer_constraints(template, call, self.direction)
+            else:
+                return []
         else:
             return []
 

mypy/fastparse.py

@@ -517,7 +517,7 @@ def visit_Assign(self, n: ast3.Assign) -> AssignmentStmt:
     @with_line
     def visit_AnnAssign(self, n: ast3.AnnAssign) -> AssignmentStmt:
         if n.value is None:  # always allow 'x: int'
-            rvalue = TempNode(AnyType())  # type: Expression
+            rvalue = TempNode(AnyType(), no_rhs=True)  # type: Expression
         else:
             rvalue = self.visit(n.value)
         typ = TypeConverter(self.errors, line=n.lineno).visit(n.annotation)