Repository navigation
DeprecationWarnings should be visible by default in the interactive REPL #68482
Description
| BPO | 24294 |
|---|---|
| Nosy | @terryjreedy, @ncoghlan, @vstinner, @rbtcollins, @ezio-melotti, @bitdancer, @njsmith, @asmeurer, @takluyver, @vadmium, @JimJJewett, @serhiy-storchaka, @The-Compiler, @jayvdb, @Carreau |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-10-18.12:46:53.493>
created_at = <Date 2015-05-27.08:06:30.608>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'DeprecationWarnings should be visible by default in the interactive REPL'
updated_at = <Date 2018-10-18.12:46:53.491>
user = 'https://github.com/njsmith'bugs.python.org fields:
activity = <Date 2018-10-18.12:46:53.491>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-10-18.12:46:53.493>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-05-27.08:06:30.608>
creator = 'njs'
dependencies = []
files = ['39612', '42500']
hgrepos = []
issue_num = 24294
keywords = ['patch']
message_count = 39.0
messages = ['244147', '244148', '244162', '244166', '244240', '244242', '244245', '244276', '244279', '244286', '244450', '244794', '244852', '245805', '245994', '248789', '248893', '248895', '248899', '248901', '248902', '248903', '248907', '249075', '249076', '249108', '249113', '249117', '249125', '249190', '249196', '257304', '261709', '263621', '306137', '306162', '306181', '306182', '327470']
nosy_count = 15.0
nosy_names = ['terry.reedy', 'ncoghlan', 'vstinner', 'rbcollins', 'ezio.melotti', 'r.david.murray', 'njs', 'Aaron.Meurer', 'takluyver', 'martin.panter', 'Jim.Jewett', 'serhiy.storchaka', 'The Compiler', 'jayvdb', 'mbussonn']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24294'
versions = ['Python 3.7']
Activity
njsmith commentedon May 27, 2015
DeprecationWarning and PendingDeprecationWarning are invisible by default. The rationale for this is that they are only useful to people who are writing code, so they should not be shown to end-users who are merely running code.
If someone is typing stuff into the interactive REPL, though, and the code they type uses some deprecated functionality, then we should actually show them this warning. We know that the author is sitting right there. And they're probably going to take whatever they tested interactively and move it into a more permanent form and ugh.
This problem is particularly acute for packages that are primarily used through the interactive prompt, like numpy. I am tired of getting bugs like this: numpy/numpy#5919
The solution is simple: when entering interactive mode, the REPL should do something like:
njsmith commentedon May 27, 2015
I also filed a similar bug with ipython: ipython/ipython#8478
vadmium commentedon May 27, 2015
I have learnt to run the interactive interpeter (and also most of my own scripts) with the -b -Wall options. But having these switched on automatically may not be a bad thing.
serhiy-storchaka commentedon May 27, 2015
See discussion on Python-Ideas [1].
[1] http://comments.gmane.org/gmane.comp.python.ideas/32191
ncoghlan commentedon May 28, 2015
+1 for showing DeprecationWarning by default (as these features may be going away in the next X.Y release of Python)
-0 for showing PendingDeprecationWarning by default (as these won't be going away until X.Y+1 at the earliest)
njsmith commentedon May 28, 2015
*cough* You know, there's more to life than Python-X.Y.tar.gz :-). Not that I know how PendingDeprecationWarning is used in the wild. I've been thinking maybe we (numpy) should start using it for stuff that we want to discourage people from using (we know it was a bad idea or intrinsically broken or whatever), but don't yet have a full replacement to offer. It would help if there were some official guidance on what these things mean -- I can't find anything written down anywhere that even documents what you just said about how CPython proper uses them, so I imagine people have come up with all kinds of interpretations.
Anyway, my logic would be: when I am trying something out at the REPL, usually it is because I want to check how it works, to figure out what I should be doing in the real module code that I'm writing in the next window. When doing this, I definitely appreciate knowing that it will be removed and is being replaced -- it won't necessarily stop me from putting it into my code, but at least it's information that I can take into account.
I don't care a huge amount either way, though; if we decided to hide PendingDeprecationWarnings by default while showing DeprecationWarnings then numpy would just define a NumPyPendingDeprecationWarning subclass of DeprecationWarning and use that instead of PendingDeprecationWarning and all would work out fine :-).
ncoghlan commentedon May 28, 2015
The difference between the two used to be clearer: prior to Python 2.7, PendingDeprecationWarning was hidden by default (and thus mainly only visible to folks testing with -Wall), while DeprecationWarning was visible by default.
We blurred the line between the two thoroughly when DeprecationWarning also became hidden by default, giving the status quo:
Test frameworks: both visible by default
Interactive REPL: both hidden by default
Non-interactive execution: both hidden by default
Splitting them in the interactive REPL case would restore a meaningful behavioural difference that can help pragmatically guide decisions as to which is more appropriate to use:
Test frameworks: both visible by default
Interactive REPL: DW visible by default, PendingDW hidden by default
Non-interactive execution: both hidden by default
The current unanswerable "How do my users interpret the difference between DW and PendingDW?" question would be replaced by the much simpler "Do I want folks using the interactive REPL to see this warning or not?"
njsmith commentedon May 28, 2015
Okay, that sounds reasonable to me.
terryjreedy commentedon May 28, 2015
For Idle, the addition could be made in current versions. Idle compiles user code in the idle process and ships it to the user process for execution. In particular, idlelib.run.Executive.runcode, line 351, is
exec(code, self.locals)
Am I to presume that a) the warnings that you want to display will escape the exec call and that b) they are currently being caught at the module level and c) that the filterwarnings line or lines would do what you want if added to run.main, such as at line 104? Do warnings from python code and user code get treated the same?
Please post test cases to enter by hand, both at the prompt and in the editor, to get system warnings for 2.7 and 3.4 or 3.5. Also, please copy the positive and negative examples you posted to python-ideas.
njsmith commentedon May 28, 2015
There isn't really any magic in how warnings work. Basically someone calls warnings.warn(...), which is a regular Python function, and it gathers up some information about the specific warning message and calling context, checks a global variable ("warnings.filters") to decide how this message in this context should be handled, and then either does nothing, prints something to stderr, or raises an exception.
There are lots of warnings that are printed by default, and I'm sure IDLE is handling them fine already. It's just (Pending)DeprecationWarnings in particular that have an entry stuck into warnings.filters saying "please ignore these" (unless it gets overridden by something else).
So you just need to make sure that a filter is added to warnings.filters that says to treat DeprecationWarnings generated by the __main__ module using the "default" action. ("default" is the name of a specific way of handling warnings; for most types of warnings, the default handler is the one named "default", but for DeprecationWarning, the default handler is the one named "ignore". Obvious, right?)
So you just need to make sure to run the following line of code somewhere in the user process:
Adjust as necessary if (a) you want to apply similar handling to PendingDeprecationWarning, (b) your user namespace has some value of __name__ that is different from "__main__".
Then to test, you can just type
at the prompt, and it should be printed.
Note that because the warnings module tries to suppress duplicate warnings (which is good), and it has a bug where it can't tell the difference between different lines of code at the REPL (this is bad -- see ipython/ipython#6611 , and there should probably be a python bug too but I haven't gotten around to filing it), then the *second* time you run that line of code in the same REPL, nothing will be printed. So the moral is just, when testing this, make sure you use a different warning message each time, or you'll get very confused.
njsmith commentedon May 30, 2015
Recording this here so it doesn't get lost: Marc-Andre Lemberg suggested on python-ideas that for the builtin REPL, this should be enabled iff sys.stdin.isatty(). (I guess he is worried about 'cat script.py | python'.) I'm not really sure whether this falls on the useful or confusing side of the line myself.
Carreau commentedon Jun 3, 2015
I gave that a shot.
Doing it "cleanly" in C as the warning module is initialized much earlier.
Though I'm not super used to CPython internals.
Doing just before the repl by using
PyRun_SimpleStringmake the patch relatively small.njsmith commentedon Jun 5, 2015
For whatever it's worth as a non-core-developer, the patch looks good to me.
14 remaining items
bitdancer commentedon Aug 25, 2015
I'm not sure that is acceptable. "Just changing it to a file" could break an application's structure that is depending on being able to use stdin to run scripts. As one example, vim scripts can embed python code...now, what VIM does behind the scenes with that I don't know, and it might be able to write it out to a file or put somewhere in memory for the embedded interpreter to run, but the point is that here is a an application whose API is embedded scripts that get fed into python, not independent files that python gets called on. I'm guessing there are other applications out there that *would* be affected by this, even if vim itself isn't (and I don't know if it is or not).
Carreau commentedon Aug 25, 2015
# unittest
(Pending)DeprecationWarning are already enabled if no -W flags are given.
# doctest
We enable the DW in REPL only if originate from
__main__, this seem to be painful to do in doctest, as each doctest execute in a module that have the name of the file it is executed in. Should I enable globally regardless of where the DW comes from, or explore more ?Should the doctest fail if such a warning is raised, or actually just print it to the output, letting the doctest pass ?
ncoghlan commentedon Aug 26, 2015
The only way for this change to *break* something is if:
I think we're far more likely to break something else messing about with TTY detection, than we are keeping things as simple as we can and saying that if folks are running code as if they're a human rather than like a computer, then they're going to get the same deprecation warnings we want a human to see. *If* we get significant bug reports about that during the 3.6 alpha/beta cycle, *then* we can potentially consider limiting it to cases with an actual TTY. I just don't want us to borrow trouble and make this unnecessarily hard to test in the process.
bitdancer commentedon Aug 26, 2015
Matthias: just print it and let the doctest pass. That's consistent with how unittest works (unless warnings have been turned into errors, in which case it should fail, but I don't think you have to do anything to make that happen).
Nick: OK. It is true that we aren't going to break anything if we get this wrong, just mildly annoy people :)
ezio-melotti commentedon Jan 1, 2016
A while ago I proposed to document our deprecation process and the use of PDW/DW in a PEP:
https://mail.python.org/pipermail/python-dev/2011-October/114199.html
There I suggest that we (CPython) stop using PDW but leave it around for other projects to use as they see fit. I'm not sure CPython should do anything special to distinguish PDW from DW.
rbtcollins commentedon Mar 14, 2016
@mbussonn - I don't see an updated non-tty-checking patch from you?
Carreau commentedon Apr 17, 2016
Sorry for the delay, trying to get back on this.
Please find attached a new patch that does not check whether is is a tty.
Still struggling a bit with HG (looking fwd to migration to GH)
njsmith commentedon Nov 13, 2017
See also PEP-565.
vstinner commentedon Nov 13, 2017
If you consider that the REPL is designed for developers, I would also suggest to show ResourceWarning by default.
I'm not sure of that since I like to write crappy code of REPL (and I hope that nobody logs my keyboard)! Example:
I may be annoyed by a ResourceWarning warning here, since it's a oneliner written to be only run once. I know that my code is crappy, but I also know that it works well and it's much shorter to write than "with open(..) as fp: fp.read()" :-)
Yet another approach: I proposed to add a "developer mode", -X dev option:
https://mail.python.org/pipermail/python-dev/2017-November/150514.html
ncoghlan commentedon Nov 14, 2017
I don't think anybody consistently does proper resource management in the REPL, so the trade-offs involved there are quite different from those for deprecation warnings.
Assuming PEP-565 gets accepted, we'll end up with once-per-session warnings for use of deprecated APIs (due to the REPL's perpetually reset line counter, as per bpo-1539925), and that seems like a good level of notification to me (not too noisy, not so unobtrusive as to never be seen at all)
asmeurer commentedon Nov 14, 2017
If it's of any interest to this discussion, for SymPy (for some time) we have used a custom subclass of DeprecationWarning that we enable by default https://github.com/sympy/sympy/blob/master/sympy/utilities/exceptions.py. I don't know if there are major libraries that do something similar.
Our reasoning is that we really do want everybody to see the warnings. Obviously direct users of SymPy (both interactive users and library developers) need to see them so they can fix their code. But also if library X uses a deprecated behavior and a user of library X sees a deprecation warning for SymPy inside of library X, that incentivises them to bug the library X developers to fix the behavior (or PR it). The whole point of warnings as we see it is to be as loud as possible while still keeping things working, to avoid the situation where things stop working (when the deprecated behavior is removed).
And +<however many points I'm allowed to have> to Nathaniel's point that DeprecationWarnings are about more than just the standard library. Tons of libraries use the built in warnings, and the default warnings behavior makes no distinction between warnings coming from the standard library and warnings coming from other places.
serhiy-storchaka commentedon Oct 10, 2018
After implementing PEP-565, are there reasons to keep this issue open?