What if some versions of a library have inline stubs? #52

Activity

1. matthiaskramm commented on Jan 22, 2016

   

   matthiaskramm

   on Jan 22, 2016

   Contributor

   More actions

   I feel the convention of preferring pyi over .py shouldn't apply for files in typeshed.
   So the priority, when checking a file called ./foo.py, would be something like:

   1. ./foo.pyi
   2. ./foo.py
   3. go look in typeshed
2. gvanrossum commented on Jan 22, 2016

   

   gvanrossum

   on Jan 22, 2016

   MemberAuthor

   More actions

   The actual algorithm in mypy is this
   https://github.com/JukkaL/mypy/blob/master/mypy/build.py#L942:

   def find_module(id: str, lib_path: List[str]) -> str:
       """Return the path of the module source file, or None if not found."""
       for pathitem in lib_path:
           for extension in PYTHON_EXTENSIONS:
               comp = id.split('.')
               path = os.path.join(pathitem, os.sep.join(comp[:-1]), comp[-1]
   + extension)
               if not os.path.isfile(path):
                   path = os.path.join(pathitem, os.sep.join(comp),
   '__init__{}'.format(extension))
               if os.path.isfile(path) and verify_module(id, path):
                   return path
       return None
   

   So I think you can fix it by passing a carefully-crafted path where the
   tornado package appears before typeshed. (Note that $MYPYPATH always comes
   before the typeshed-derived paths in lib_path -- see build() and
   default_lib_path() in the same file.)
3. gvanrossum commented on Jan 26, 2016

   

   gvanrossum

   on Jan 26, 2016

   MemberAuthor

   More actions

   My conclusion is that this is a non-problem -- it can be solved by setting MYPYPATH.
4. gvanrossum

   closed this as completedon Jan 26, 2016
5. bdarnell commented on Jan 27, 2016

   

   bdarnell

   on Jan 27, 2016

   Contributor

   More actions

   But shouldn't python's sys.path take precedence over MYPYPATH? A requirement to set MYPYPATH is surprising - mypy will mostly work without it, except for this issue.
6. gvanrossum commented on Jan 27, 2016

   

   gvanrossum

   on Jan 27, 2016

   MemberAuthor

   More actions

   Well, sys.path points to the stdlib of the Python version running mypy. That's not necessarily the same as the app you're checking. Same for PYTHONPATH.

   I'm not sure why you think mypy (mostly) works without it. It just uses the current directory and the root of the package tree containing the module(s) you're checking into account (it crawls up until it finds a directory without __init__.py). And it knows the structure of typeshed, and appends the relevant directories underneath it (depending on the Python version you're checking) at the end of its search path.

   (Even if it were to somehow use PYTHONPATH, it would still need something to adjust the search path, because it won't adjust sys.path based on dynamic manipulations thereof by the program being checked.)
7. gvanrossum

   reopened this on Jan 27, 2016
8. gvanrossum

   closed this as completedon Jan 27, 2016
9. bdarnell commented on Jan 27, 2016

   

   bdarnell

   on Jan 27, 2016

   Contributor

   More actions

   I'm not sure why you think mypy (mostly) works without it.

   Um, because I've never set MYPYPATH and it seems to mostly work (or at least it's not failing in ways that are visibly linked to path settings). This may be because the presence of stubs in typeshed is masking things that should be manifesting as path errors (e.g. I'm running mypy tornado/escape.py and mistakenly expecting the same sys.path semantics as python -m tornado.escape, which is not quite right. It looks like this is getting fixed by the "walk up to directory without __init__.py" rule instead of typeshed, though).

   However, I can no longer reproduce what I thought I was seeing in python/mypy#1136. The Tornado files in my working directory always seem to take precedence over typeshed (which may be a problem in its own right but is exactly what I want in this case). There must have been some difference in my working directory or invocation; this is still subtle.

   Regardless, I don't think the find_module() approach of taking the first file on the search path that exists is viable. How is a user of Tornado 5.0 (with inline annotations) to know that they must add its path to MYPYPATH, while a user of Tornado 4.0 (with no annotations) must keep it off of MYPYPATH to avoid masking the typeshed versions? The approach that makes the most sense is to always put your dependencies on mypy's path, and let mypy skip over things that it either cannot handle or contain no usable annotations.
10. gvanrossum commented on Jan 27, 2016

    

    gvanrossum

    on Jan 27, 2016 via email

    MemberAuthor

    More actions

    How is a user of Tornado 5.0 (with inline annotations) to know ...

    Through documentation. I agree that the current approach to the search path in mypy is a prototype and we should update it based on feedback. For me, so far, it has worked without a hitch. But our use case at Dropbox is somewhat unusual. Would you submit an issue to the mypy tracker describing the path search algorithm you think would be an improvement (and why)?
11. hswong3i

    added a commit that references this issue on May 25, 2025

    Prepare release 4.3.0 (python#52)

    9c0759a