We should clear interactive namespace's __warningregistry__ between executing each cell

A very very common source of confusion when using python interactively is that warnings disappear. Specifically, they show up the first time they might appear, but then never again after that...:

In [14]: np.zeros(10) / 0
/usr/bin/ipython3:1: RuntimeWarning: invalid value encountered in true_divide
  #!/usr/bin/env python3
Out[14]: array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan])

In [15]: 0 / np.zeros(5)
Out[15]: array([ nan,  nan,  nan,  nan,  nan])

This is because by default, the warnings module attempts to only show each warning once per "source"; the idea is that if you write something like

for i in xrange(10000): 
    np.log(0)  # **

then you really only need to hear about the problem on line ** once, not 10,000 times.

But this interacts badly with interactive usage, because the warnings module has a bug: it thinks that the first line of one interactive cell is the same line as the first line of another interactive cell, so if np.zeros(10) / 0 has raised a warning then it thinks 0 / np.zeros(5) has as well. Because they are the same line. Oops.

Specifically, the way the warnings module keeps track of these things is with a really weird hack: whenever it issues a warning, it reaches into the execution environment of the code issuing the warning, and creates a global variable called __warningregistry__. For example:

In [1]: __warningregistry__
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-41b71685b3a2> in <module>()
----> 1 __warningregistry__

NameError: name '__warningregistry__' is not defined

In [2]: np.zeros(10) / 0
/home/njs/.user-python2.7-64bit-3/bin/ipython:1: RuntimeWarning: invalid value encountered in divide
  #!/home/njs/.user-python2.7-64bit-3/bin/python
Out[2]: array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan])

In [3]: __warningregistry__
Out[3]: {('invalid value encountered in divide', RuntimeWarning, 1): True}

This __warningregistry__ value says that line 1 of the code in this namespace has already issued the given warning, so if you ever see another piece of code whose line number is 1, and which is executing in this namespace, try to issue this warning, then that warning should not be displayed.

However, in the interactive case, we know that every time we execute a new cell, the line numbers are effectively reset -- line 1 of this cell is totally different from line 1 of the previous cell. Ipython should take care of informing the warnings module of this fact.

This can be done very simply: just delete the __warningregistry__ variable from the interactive namespace after each time a cell finishes executing. This will cause the warnings module to treat each new block of code as a new block of code.

You can do this by registering a post_execute hook. I don't agree that it should be the default behavior.

What possible advantage is there to making this user-configurable? In
circumstance would I ever want valid warnings to be randomly suppressed
based on unrelated code that I just happen to have run at another time?

On Sat, Oct 4, 2014 at 9:53 PM, Min RK [email protected] wrote:

You can do this by registering a post_execute hook. I don't agree that it
should be the default behavior.

—
Reply to this email directly or view it on GitHub
#6611 (comment).

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

(Also, it's new users who are constantly bitten by this -- we average
something like 1 post per week on the numpy list where this is a
contributing factor in the confusion. Telling them that they ought to have
known to install a post_execute hook is not going to really help ;-).)

On Sat, Oct 4, 2014 at 9:56 PM, Nathaniel Smith [email protected] wrote:

What possible advantage is there to making this user-configurable? In
circumstance would I ever want valid warnings to be randomly suppressed
based on unrelated code that I just happen to have run at another time?

On Sat, Oct 4, 2014 at 9:53 PM, Min RK [email protected] wrote:

You can do this by registering a post_execute hook. I don't agree that it
should be the default behavior.

—
Reply to this email directly or view it on GitHub
#6611 (comment).

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

The vast majority of warnings that I see are the result of code I don't control. I don't want to see those warnings every time I use somebody else's code that I can't fix. There's a reason Python only shows warnings the first time they happen, and I don't think we should break that because the result would be super annoying.

The warnings that you don't control will not be affected. Only the warnings caused directly by the code you typed in would be affected by Nathaniel's suggestion.

[~/scratch]
|1> import warnings

[~/scratch]
|2> warnings.warn('bar')
/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/ipython:1: UserWarning: bar
  #!/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/python

[~/scratch]
|3> warnings.warn('bar')
/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/ipython:1: UserWarning: bar
  #!/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/python

[~/scratch]
|4> !cat foo.py
import warnings

def f():
    warnings.warn('foo')

[~/scratch]
|5> import foo

[~/scratch]
|6> foo.f()
foo.py:4: UserWarning: foo
  warnings.warn('foo')

[~/scratch]
|7> foo.f()

[~/scratch]
|8> foo.f()

[~/scratch]
|9> !cat /Users/rkern/.ipython/extensions/clear_warnings.py

class WarningClearer(object):
    def __init__(self, ip):
        self.ip = ip

    def clear_warnings(self):
        self.ip.user_ns.pop('__warningregistry__', None)


def load_ipython_extension(ip):
    wc = WarningClearer(ip)
    ip.events.register('post_execute', wc.clear_warnings)

Thanks for the explanation, I didn't realize that there are many warnings registries. In that case, I think this proposal seems reasonable.

Yes, sorry for not making that clear -- each execution environment has its
own warningregistry, which only tracks the warnings that warnings.py
thinks were directly caused by code running in that environment.

On Sat, Oct 4, 2014 at 11:07 PM, Min RK [email protected] wrote:

Thanks for the explanation, I didn't realize that there are many warnings
registries. In that case, I think this proposal seems reasonable.

—
Reply to this email directly or view it on GitHub
#6611 (comment).

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

Reset the interactive namespace __warningregistry__ before executing …

Navigation Menu

Repository navigation

We should clear interactive namespace's warningregistry between executing each cell #6611

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

We should clear interactive namespace's __warningregistry__ between executing each cell #6611

Description

Activity

minrk commented on Oct 4, 2014

njsmith commented on Oct 4, 2014

njsmith commented on Oct 4, 2014

minrk commented on Oct 4, 2014

rkern commented on Oct 4, 2014

rkern commented on Oct 4, 2014

minrk commented on Oct 4, 2014

njsmith commented on Oct 4, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

We should clear interactive namespace's warningregistry between executing each cell #6611