Handling signature overlaps with overloaded definitions and Protocols

[zero323]

Let's imagine that we'd like to provide more precise annotations for operator.itemgetter. Currently typeshed annotations look like this

@overload
def itemgetter(item: Any) -> Callable[[Any], Any]: ...
@overload
def itemgetter(*items: Any) -> Callable[[Any], Tuple[Any, ...]]: ...

That clearly not very useful. One approach would be to provide a Protocol like this:

from typing_extensions import Protocol
from typing import Any, Generic, TypeVar

T = TypeVar("T", contravariant=True)
U = TypeVar("U", covariant=True)

class HasGetItem(Protocol, Generic[T, U]):
    def __getitem__(self, item: T) -> U: ...

which in isolation works pretty well

class Foo(HasGetItem[int, str]): ...

foo = Foo()
foo[41]
foo["41"]  # E: Invalid index type "str" for "Foo"; expected type "int"

foo[42].strip()
foo[42] + 1  # E: Unsupported operand types for + ("str" and "int")

However, when plugged into itemgetter annotations (mypy 0.770+dev. 7055725)

from typing import overload, Callable, Tuple

@overload
def itemgetter(item: T) -> Callable[[HasGetItem[T, U]], U]: ...
@overload
def itemgetter(*items: T) -> Callable[[HasGetItem[T, U]], Tuple[U, ...]]: ...

the whole thing fails with

...: error: Overloaded function signatures 1 and 2 overlap with incompatible return types

This is relatively generic pattern in functional interfaces and it would be great if there was some way to express it.

Related to #4020

example.txt

[msullivan]

This one can be fixed pretty easily be rewriting it so that the definitions don't overlap:

@overload
def itemgetter(item: T) -> Callable[[HasGetItem[T, U]], U]: ...
@overload
def itemgetter(item: T, __item2: T, *items: T) -> Callable[[HasGetItem[T, U]], Tuple[U, ...]]: ...

Also, when it is hard to do this, you can often just # type: ignore the error about overlaps and have the right behavior work.

[zero323]

That's an interesting workaround and I appreciate your time, but the problem is more general, isn't it?

If we'd change signature, to let's say this one

from typing import overload, Callable, List, Tuple

@overload
def itemgetter(item: T) -> Callable[[HasGetItem[T, U]], U]: ...
@overload
def itemgetter(items: List[T]) -> Callable[[HasGetItem[T, U]], Tuple[U, ...]]: ...

we're back to square one, aren't we?

As of #type: ignore[misc], effectively hides the problem, but it doesn't resolve it, and from my limited experience, ignores have ugly tendency to escalate to other related components.

But I guess the real question is if that, for whatever reason, is the expected behavior?

(And, just thinking out loud, if this won't-fix, it might be useful to have more specific error code, than misc).

[JukkaL]

@zero323 Your example has overlapping signatures, so requiring # type: ignore[misc] seems reasonable to me. Some potentially useful overloaded signatures can't be represented without making them overlapping.

In your example, consider what happens if we call itemgetter with a value of type Sequence[int]. Now the return type is ambiguous -- either of the overload items could be taken at runtime, depending on whether the runtime value is list or tuple, for example. Mypy could assume that both items could be taken and infer a union return type, but this would often be not what the user expects.

[zero323]

Thank you @JukkaL.