Cannot extend built-in Protocols

[FuegoFro]

In typing.py, a number of built-in protocols extend from Generic, not Protocol (eg Iterable, Container, Sized, etc). However, at runtime the Procotol class enforces that all bases for a class also inherit from Protocol. Thus the following code

from typing import Iterable, TypeVar
from typing_extensions import Protocol

T = TypeVar('T')

class Foo(Iterable[T], Protocol[T]):
    pass

causes the error

TypeError: Protocols can only inherit from other protocols, got typing.Iterable

This is further confused by the fact that, in recent versions of typeshed, typing.pyi has these types like Iterable extending Protocol and not Generic (opposite of runtime, but also seemingly the more "correct" version).

[lihu-zhong]

Hitting the same issue -- is there a known workaround, or do I just rip out type hints from anything that requires structural subtyping until this is fixed?

[gvanrossum]

To both reporters: Exactly which versions of Python, typing (unless in the stdib), and typing_extensions did you use?

[JelleZijlstra]

Also, a workaround (haven't tested) is to do something like this:

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    class Foo(Iterable[T], Protocol[T]):
        pass
else:
    class Foo:
        def __iter__(self): ...

(The second branch can probably also still inherit from Iterable.)

[lihu-zhong]

$ pipenv graph
typing-extensions==3.6.5

$ pipenv run python
Python 3.6.5 (default, Apr  4 2018, 15:01:18) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Iterable, TypeVar
>>> from typing_extensions import Protocol
>>> 
>>> T = TypeVar('T')
>>> 
>>> class Foo(Iterable[T], Protocol[T]):
...     pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lihu/.local/share/virtualenvs/Documents-8_-tGN99/lib/python3.6/site-packages/typing_extensions.py", line 852, in __init__
    ' protocols, got %r' % base)
TypeError: Protocols can only inherit from other protocols, got typing.Iterable
>>> 

Thanks for your responses, and all the work you do here!

[gvanrossum]

Yeah, I guess Jelle's workaround is what you need here.

[ilevkivskyi]

I wanted to work on this last weekend, but didn't have time. I think the real fix would be to just allow subclassing collections.Iterable etc. I could imagine it will be a common use case to extend a "built-in" protocol. I will tae care of this as soon as I will have some free time.

[gvanrossum]

Is the plan just to make exceptions for a handful of builtin Protocols like Iterable, or will there be some indicator to say "subclassing is fine"?

[ilevkivskyi]

I wanted to write that this is now solved in newly released typing/typing_extensions 3.7.4, but there is a little problem. I discovered that my fix for extending builtin protocols doesn't always work. Namely, subclassing Protocol together with collections[.abc].Iterable seems to work on all Python versions, but subclassing Protocol together with typing.Iterable only works on Python 3.7+ (because of PEP 560).

[srittau]

If this is still a problem, it should be reported to bugs.python.org.

[leycec]

Necrobump. Although this issue remains unresolved, a real-world solution finally arises. Courtesy beartype's typing compatibility API, PEP 544-compliant protocols subclassing beartype.typing.Protocol (rather than either typing.Protocol or typing_extensions.Protocol) transparently support subclassing from non-protocol superclasses:

# This is fine.
>>> from beartype.typing import Iterable, Protocol, TypeVar
>>> T = TypeVar('T')
>>> class Foo(Iterable[T], Protocol[T]): pass

# This is fine, too.
>>> import collections
>>> issubclass(Foo, collections.abc.Iterable)
True
>>> isinstance(('bar',), Foo)
True

# Actual structural subtyping or it didn't happen.
>>> class Zed(Iterable[T], Protocol[T]):
...     def zeds_dead_baby(self): pass
>>> class WhosZed(tuple):
...     def zeds_dead_baby(self): pass
>>> issubclass(WhosZed, Zed)
True
>>> isinstance(WhosZed("Whose motorcycle is this?"), Zed)
True
>>> isinstance(("It's a chopper, baby."), Zed)
False

As the principal maintainer of @beartype,^...hi! I'm as surprised as you are. 😮