os.open doesn't accept Union[bytes, Text]

@JukkaL

The following code:

from typing import Text, Union
import os

def test(path):
    # type: (Union[bytes, Text]) -> None
    f = os.open(path, os.O_RDONLY)

Passes with --py2 but fails in python3 as follows:

error: Type argument 1 of "open" has incompatible value "Union[bytes, str]"

@JukkaL

FWIW if we change os.open() to take Union[str, bytes] the error goes away, but I'm sort of thinking that passing that union to something expecting AnyStr should work...

Yes, Union[str, bytes] should probably be sometimes valid when AnyStr is expected (but it isn't right now). Example:

def f(x: AnyStr) -> AnyStr:
    return x

def g(x: Union[str, bytes]) -> None:
    f(x)  # should be fine, return type Union[str, bytes]

However, List[Union[str, bytes]] should not be compatible with List[AnyStr] because the latter is uniform but the prior could have a mix of str and bytes. Also, this shouldn't be valid, following similar reasoning:

def f2(x: AnyStr, y: AnyStr) -> AnyStr:
    return x + y

def g2(x: Union[str, bytes], y: Union[str, bytes]) -> None:
    f2(x, y)  # error, could be a mix of str and bytes

Here's how we could support these (and other things not involving type variables, such as overloads):

If caller has arguments with union types, expand all the combinations of the union types into non-union types and test the call with each of these expansions. The call is valid if all combinations type check okay.

The call to f2 in the second example would be checked 4 times:

f2(<str>, <str>)
f2(<str>, <bytes>)
f2(<bytes>, <str>)
f2(<bytes>, <bytes>)

The obvious problem with this is that this could be really slow if there are many union types in a call. An implementation could plausibly be clever and only expand those unions where it seems like a useful thing to do.

@euresti

I think the right fix to @euresti's original issue is that the type of os.open's parameter should not be AnyStr -- it should be Union[bytes, str]. In general, AnyStr doesn't provide any additional value when it appears only once in a type signature, and it introduces complexity because its semantics are somewhat complicated.

It's possible we should also handle the interaction of AnyStr with Union differently in general. I think that ought to be a separate issue if we want to pursue it further.

@JukkaL

Closing in favor of the typeshed issue -- @JukkaL , I haven't fully absorbed your comment, but if you think there's more to discuss please do make an issue for it.

Thanks for reporting this, David!

@matthiaskramm

This issue wasn't actually closed, and I don't think it should be just yet. I still think that a function with exactly one AnyStr parameter (not occurring in a container) should be supported -- as @matthiaskramm mentioned it is a pretty convenient notation. That said I agree we should just fix os.open() -- and all other os functions that have just one string argument and that don't return a string.

I don't want to ban using AnyStr just once in a function signature, but I think the semantics as currently written in PEP 484 are pretty complicated (details in that typeshed thread). Do you think what we're currently doing differs from those semantics? Or is there another fix you think we can make here?

Increasing priority to normal since issues with union types are frequently causing trouble for users. This may also affect code that uses --strict-optional.

Adds support for basic union math with overloads

Currently the original example passes in both Python 2 and Python 3 modes. So I am closing this since we have another issue for tracking union maths.

Navigation Menu

Repository navigation

os.open doesn't accept Union[bytes, Text] #1943

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

os.open doesn't accept Union[bytes, Text] #1943

Description

Activity

gvanrossum commented on Jul 26, 2016

JukkaL commented on Jul 27, 2016

gnprice commented on Aug 4, 2016

gnprice commented on Aug 4, 2016

gvanrossum commented on Aug 4, 2016

gnprice commented on Aug 11, 2016

JukkaL commented on Jan 18, 2017

ilevkivskyi commented on May 17, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions