mypy doesn't understand set operations on optional #13192

Activity

1. Palpatineli

   added

   bugmypy got something wrongmypy got something wrong

   on Jul 19, 2022
2. AlexWaygood commented on Jul 20, 2022

   

   AlexWaygood

   on Jul 20, 2022

   Last edited by AlexWaygood

   Member

   More actions

   The typeshed stub for the __sub__ method of builtins.set is currently:

   from typing import MutableSet, Generic, AbstractSet, TypeVar
   
   _T = TypeVar("_T")
   
   class set(MutableSet[_T], Generic[_T]):
       def __sub__(self, __s: AbstractSet[_T | None]) -> set[_T]: ...

   I guess we could special-case None in the stub:

   from typing import MutableSet, Generic, AbstractSet, TypeVar, overload
   
   _S = TypeVar("_S")
   _T = TypeVar("_T")
   
   class set(MutableSet[_T], Generic[_T]):
       @overload
       def __sub__(self: set[_S | None], __s: AbstractSet[None]) -> set[_S]: ...
       @overload
       def __sub__(self, __s: AbstractSet[_T | None]) -> set[_T]: ...

   But I'm not sure this kind of thing is common enough to be worth special-casing here.

   Such a change would have to happen over at the typeshed repo rather than the mypy repo, since mypy vendors its stdlib stubs from typeshed.
3. AlexWaygood

   added

   feature

   and removed

   bugmypy got something wrongmypy got something wrong

   on Jul 20, 2022
4. tungol commented on Jul 22, 2022

   

   tungol

   on Jul 22, 2022

   Contributor

   More actions

   Shouldn't the special casing should really be for singleton types in general, similar to the changes made here? #13117
5. TeamSpen210 commented on Jul 22, 2022

   

   TeamSpen210

   on Jul 22, 2022

   Contributor

   More actions

   Well that one is different since it's about is and if - core syntax, unlike set operations here which are regular magic method operations.

   Implementing the other singleton objects too would be tricky, since you'd need to account for all possible combinations of NotImplemented, None, Ellipsis, True and False. Bools might not really work since {False, True} is just going to have a type of set[bool], it's not going to track whether or not both values are present which you'd need in order to remove from the union.

   👍React with 👍1Reacted by Alex Waygood
6. AlexWaygood commented on Jul 23, 2022

   

   AlexWaygood

   on Jul 23, 2022

   Member

   More actions

   I agree with @TeamSpen210. There's no generalised way in the type system at the moment of indicating that a type is a singleton, and I don't think there would be enough use cases to make it worth the added complexity of creating a generalised way of doing that. The changes made in #13117 are highly specific to EllipsisType; it's the opposite of "special casing for singleton types in general".

   I'm closing this as I don't think there's anything actionable for mypy here. If we did want to do something in this area, the change would have to be made over at typeshed.
7. AlexWaygood

   closed this as not plannedon Jul 23, 2022
8. finite-state-machine commented on Feb 11, 2024

   

   finite-state-machine

   on Feb 11, 2024

   Last edited by finite-state-machine

   More actions

   Is there a suggested workaround, short of resorting to a cast()?

   I've tried the kitchen sink without success (mypy-play.net):

   from __future__ import annotations
   from typing import *
   
   base_chars: Set[Optional[str]]
   chars = frozenset(base_chars - {None})
   assert None not in chars
   assert all(chars)
   assert all(c is not None for c in chars)
   assert all(isinstance(c, str) for c in chars)
   reveal_type(chars)   # FrozenSet[Union[str, None]]

   _{[edited to use frozenset() rather than set()]}
9. erictraut commented on Feb 11, 2024

   

   erictraut

   on Feb 11, 2024

   More actions

   Here's a straightforward workaround:

   def func(base_chars: set[str | None]):
       chars = frozenset(x for x in base_chars if x is not None)
       reveal_type(chars) # frozenset[str]

   🎉React with 🎉1Reacted by Finite State Machine
10. finite-state-machine commented on Feb 11, 2024

    

    finite-state-machine

    on Feb 11, 2024

    Last edited by finite-state-machine

    More actions

    Thanks, @erictraut! (I hadn't considered a comprehension because it wasn't verifying the type of the set, but that works 99% of the time. Where zero-copy/performance really matters, there's always cast().)

    If this issue is ever reconsidered, I'd like to offer a thought: perhaps an every() function would help make set narrowing more type-agnostic and relatively affordable to implement:

    Te = TypeVar('Te', bound=bool|Enum|Ellipsis|None)
    
    def every(type_: Type[Te], /) -> CompleteFrozenSet[Te]:
        '''returns every member of an enumerable type
        
        Args:
            type_: 'None', 'bool', 'Ellipsis', or an 'Enum' type
        Returns:
            a frozen set containing every value of the given type
        '''
        # magic happens
        return ...
        
    
    class CompleteFrozenSet(FrozenSet[T]):
        '''(empty body)'''
        
    
    # (I'm not sure if this would work as written:)
    class Set(...):
        ...
        @overload
        def __isub__(self: Set[A|B], other: CompleteFrozenSet[B]) -> Set[A]: ...
        ....

    I suppose an def __every__(self: Type[Te]) -> CompleteFrozenSet[Te]: ... magic method would be the natural extension to other types.

    I do realize this probably isn't a priority.