Dynamic PEP 484 type checking without code changes?

Are there tools that do dynamic type checking for code annotated according to PEP 484, without requiring code changes?

I know of three projects that can check types dynamically based on PEP 484 type annotations, but require code changes (usually annotations) to trigger the checks; two of them have feature requests for checking without code changes:

enforce
https://github.com/RussBaz/enforce
Run on any function automatically RussBaz/enforce#1
typeguard
https://github.com/agronholm/typeguard
Check all called functions at runtime agronholm/typeguard#1
typecheck-decorator
https://github.com/prechelt/typecheck-decorator

Thanks for the links to other systems -- I'm glad the PEP 484 notation is catching on.

I'm not sure what you're asking for though -- what do you mean by "without requiring code changes"? Do you mean something that wraps all (annotated) functions in an implicit decorator that does the dynamic type checking, like requested in the issues you link to? (Both by the same user -- a friend of yours? :-)

I don't think I would like that on my code -- I'd much rather use a static checker like mypy or an IDE with integrated checking like PyCharm. Those systems catch errors like this one without even running the code:

def f(a: int) -> int:
    return a+1
f('')  # That's an error

If you just run that code it'll also detect the error -- on the a+1 expression though, not on the call. :-)

Or are you asking for a system that doesn't even require annotations? I'm not even sure how that would work.

def f(a):
    return a+1
f('')  # Same error

I know of a system that can detect that through static analysis (https://github.com/google/pytype) but I wouldn't know how one would even write a system that would detect that the string passed in to f() is wrong before the call evaluates a+1.

So please give some examples of what you'd like to see!

@The-Compiler

Indeed, I am looking for something that checks all code that has PEP 484 annotations automatically during execution, without requiring adding decorators or any other code change (typeguard can check a function also by calling typeguard.check_argument_types() from within the function body). This is also what @The-Compiler is asking for in the issues I linked (no, not my friend).

Yep, I just happen to really want this as well, but not having the time to implement it so far 😉

Is there a reason to keep this issue open here? I don't see an action item for PEP 484 or typing.py.

The reason I asked the question here in the first place is because I saw issue #200 and thought this type of questions are appropriate. If this is not the case, feel free to close it.

It's okay to ask. I don't know if you'll get more answers if we leave it open, but I'll just label it as a question for now.

@mihaic

@mihaic @The-Compiler There is a PEP https://www.python.org/dev/peps/pep-0511/ on registering of AST transformers. If that PEP is accepted, then it will be easy implement "automatic" run-time type checking: write an AST transformer that appends the desired decorator to all ClassDef and FunctionDef decorator lists, and register it with sys.set_code_transformers() at import time. Then only one change of code will be needed: import the type checker at first line of your main module.

Something that you might also try is using sys.settrace to intercept call/return events and do run-time type checking there. It may slow down your code, but I'm assuming this is for debugging and the checking itself will make calls much slower anyway.

One problem you might find (which has stopped me from writing a proof of concept with the time I have) is that it's not trivial to get to the annotations from the frame object (which has a reference to the code object but not the function object which is where the annotations are stored), you might be able to recover it from the filename+name if it's something global-ish.

Actually I think a debugging context (pdb tools, or sentry-like tools) would be really great places where to apply run time type checking; when there's a crash you could type-check the whole stack, and if there's a type error farther from the top of stack, it's a more probable cause of the error than the actual exception or latest stack frame (which probably is about a wrongly typed value that got carried through several calls)

sys.gettrace

There's someone at Dropbox who has a prototype that AFAIK works like that, but it's advertised as a hack, and I haven't dared look at it yet. Given your description it shouldn't be too hard for someone thus inclined to create a cleaner version.

type-check the whole stack

In the cases where it's hardest to find the culprit, the problem typically isn't on the stack but somewhere in a global (or at least shared) data structure. If you find a None in a list that's expected to be List[int], how do you find who put it there? The culprit is likely long gone from the call stack.

It's true that many times the problem is in a datastructure, but for me the debugging path of type-related crashes tends to be:

I get an exception ("NoneType has no attribute foo", or some kind of TypeError)
Here I start going up the functions in the traceback to see where the invalid value was generated
I find the culprit which may be a value returned from a function (computed) or a data structure (stored)
If it's computed I know which function to fix. If it's stored, I have to trace possible sources of the data as you mentioned
I figure out who wrote the wrong value

And at that point I know which code to fix. Type checking the stack would save me most of the work between (1) and (2). To get from (2) to (3) may require different kind of effort. but some of it is saved. In this situation I also know where I'd get more benefit adding annotations (the attribute/variable that got the wrong value) if I'm annotating new code. So it's not a full solution but it's better than nothing.

I'm adding this here just to document the discussion (given that there's no mailing list), in any case I don't think any changes to "typing" are required for this.

I gave up on my experimental implementation after noticing that the settrace hook is reset to None when the hook raises an exception. If there is a workaround, I would like to learn about it. Link here.

@agronholm

@agronholm added support for runtime type checking without code changes to Typeguard:
agronholm/typeguard@69ce354

I have just used it successfully in Pytest with a fixture like this:

from warnings import filterwarnings
from typeguard import TypeChecker, TypeWarning

@pytest.fixture(scope="session", autouse=True)
def start_typechecker():
    filterwarnings("error", category=TypeWarning)
    checker = TypeChecker("brainiak")
    checker.start()

Typeguard is not perfect itself, but this feature satisfies my request. As far as I am concerned, this issue is solved.

Happy to close this, since there is no action for the typing.py/PEP-484 core team here anyway.

Navigation Menu

Repository navigation

Dynamic PEP 484 type checking without code changes? #310

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Dynamic PEP 484 type checking without code changes? #310

Description

Activity

gvanrossum commented on Oct 28, 2016

mihaic commented on Oct 28, 2016

The-Compiler commented on Oct 28, 2016

gvanrossum commented on Oct 28, 2016

mihaic commented on Oct 28, 2016

gvanrossum commented on Oct 28, 2016

ilevkivskyi commented on Oct 30, 2016

dmoisset commented on Oct 31, 2016

gvanrossum commented on Oct 31, 2016

dmoisset commented on Nov 1, 2016

agronholm commented on Dec 10, 2016

mihaic commented on Apr 11, 2017

gvanrossum commented on Apr 11, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions