Skip to content

Refine how overload selection handles *args, **kwargs, and Any - #5166

Merged
ilevkivskyi merged 4 commits into
python:masterfrom
Michael0x2a:overloads-refine-handling-of-starargs-and-any
Jun 13, 2018
Merged

ilevkivskyi merged 4 commits into
python:masterfrom
Michael0x2a:overloads-refine-handling-of-starargs-and-any

Conversation

@Michael0x2a

Copy link
Copy Markdown
Collaborator

This pull request implements the changes discussed in #5124.

Specifically...

  1. When two overload alternatives match due to Any, we return the last matching return type if it's a supertype of all of the previous ones. If it's not a supertype, we give up and return 'Any' as before.

  2. If a user calls an overload with a starred expression, we try matching alternatives with a starred arg or kwarg first, even if those alternatives do not appear first in the list. If none of the starred alternatives are a valid match, we fall back to checking the other remaining alternatives in order.

This pull request implements the changes discussed in
python#5124.

Specifically...

1. When two overload alternatives match due to Any, we return the last
   matching return type if it's a supertype of all of the previous ones.
   If it's not a supertype, we give up and return 'Any' as before.

2. If a user calls an overload with a starred expression, we try
   matching alternatives with a starred arg or kwarg first, even if
   those alternatives do not appear first in the list. If none of the
   starred alternatives are a valid match, we fall back to checking the
   other remaining alternatives in order.
@Michael0x2a

Copy link
Copy Markdown
Collaborator Author

This pull request should be orthogonal to my other pending one -- the two can be reviewed independently.

@ilevkivskyi ilevkivskyi self-assigned this Jun 7, 2018
@ilevkivskyi
ilevkivskyi self-requested a review June 7, 2018 10:10

@ilevkivskyi ilevkivskyi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks and sorry for a delay! Just few minor comments.

Comment thread test-data/unit/check-overloading.test Outdated
@@ -1313,15 +1313,15 @@ def f(x): pass

a: Any
# Any causes ambiguity

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is now less clear, I would rather remove it or clarify.

Comment thread test-data/unit/check-overloading.test Outdated
@@ -1365,7 +1365,7 @@ def f(x): pass

a: Any
# TODO: We could infer 'int' here

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this TODO looks interesting. Would it be hard to fix?

@overload
def foo(x: int, y: int) -> B: ...
@overload
def foo(x: int, y: int, z: int, *args: int) -> C: ...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a test with overload like this:

@overload
def f(x: int) -> Tuple[int]: ...
@overload
def f(x: int, y: int) -> Tuple[int, int]: ...
@overload
def f(*xs: int) -> Tuple[int, ...]: ...

And check how it various calls with and without star.

Specific changes made:

1. Modify the code that checks for ambiguity due to Any to handle the
   case where an argument maps to multiple formals.

2. Add extra test cases.

3. Clarify an out-of-date comment.

@ilevkivskyi ilevkivskyi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Just one minor suggestion.

Also looking at the tests, it seems to me we don't have enough tests for overload calls where overload definition had errors (that were type ignored). Could you please add this to the TODO list if you agree?


reveal_type(f(*[])) # E: Revealed type is 'builtins.tuple[builtins.int]'
reveal_type(f(*[i])) # E: Revealed type is 'builtins.tuple[builtins.int]'
reveal_type(f(*[i, i])) # E: Revealed type is 'builtins.tuple[builtins.int]'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if there is a tuple instead of list? Ideally f(*(i, i)) should infer Tuple[int, int]. If this is the case, then I would add this call to this test.

@Michael0x2a

Michael0x2a commented Jun 10, 2018 •

Copy link
Copy Markdown
Collaborator Author

@ilevkivskyi -- fyi I need to run to a thing; I'll check the tuple thing once I get back.

@Michael0x2a

Copy link
Copy Markdown
Collaborator Author

@ilevkivskyi -- ok, I think this is ready for another round of review.

I discovered that f(*(int, int)) was indeed not being inferred to Tuple[int, int]. I decided to fix this by just disabling the "let's move the alternative to the front" thing if the starred arg we're passing in is a type that has a "shape".

Also, I ran into an interesting edge case in the very last test I added (testOverloadVarargsAndKwargsSelection) -- do you have any thoughts about how we should handle that case?

@ilevkivskyi ilevkivskyi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, this is now ready to land. Thanks!

reveal_type(f(*a)) # E: Revealed type is '__main__.B'
reveal_type(f(*b)) # E: Revealed type is 'Any'

# TODO: Should this be 'Any' instead?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am OK with the current behaviour. This is exactly one of those cases where the rule "when in doubt -- take first" applies.

@ilevkivskyi
ilevkivskyi merged commit 0f76dfc into python:master Jun 13, 2018
@Michael0x2a
Michael0x2a deleted the overloads-refine-handling-of-starargs-and-any branch July 9, 2018 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants