Accidentally abstract classes

We've had several issues where mypy produces incorrect "Cannot instantiate abstract class" errors because of faulty typeshed stubs (e.g., #1379, #1160, #351, #1475). To find more bugs like those, I hacked mypy (JelleZijlstra/mypy@1fa9c65) to error on any class with abstract methods. Here are the results for 3.6, excluding private classes and those clearly intended to be abstract:

stdlib/3/collections/__init__.pyi:62: error: Class collections.UserDict has abstract attributes ['__delitem__', '__getitem__', '__iter__', '__len__', '__setitem__']
stdlib/3/collections/__init__.pyi:63: error: Class collections.UserList has abstract attributes ['__delitem__', '__getitem__', '__len__', '__setitem__', 'insert']
stdlib/3/collections/__init__.pyi:64: error: Class collections.UserString has abstract attributes ['__getitem__', '__len__']
third_party/2and3/requests/structures.pyi:5: error: Class requests.structures.CaseInsensitiveDict has abstract attributes ['__delitem__', '__getitem__', '__iter__', '__len__', '__setitem__']
third_party/2and3/pytz/lazy.pyi:6: error: Class pytz.lazy.LazyDict has abstract attributes ['__getitem__', '__iter__', '__len__']
stdlib/2and3/codecs.pyi:205: error: Class codecs.StreamReaderWriter has abstract attributes ['close', 'fileno', 'flush', 'isatty', 'readable', 'seekable', 'tell', 'truncate', 'writable']
stdlib/2and3/codecs.pyi:221: error: Class codecs.StreamRecoder has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
stdlib/2and3/bz2.pyi:14: error: Class bz2.BZ2File has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
stdlib/3/urllib/response.pyi:6: error: Class urllib.response.addinfourl has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
stdlib/3/http/cookies.pyi:21: error: Class http.cookies.BaseCookie has abstract attributes ['__delitem__', '__getitem__', '__iter__', '__len__']
stdlib/3/http/cookies.pyi:31: error: Class http.cookies.SimpleCookie has abstract attributes ['__delitem__', '__getitem__', '__iter__', '__len__']
stdlib/3.4/tracemalloc.pyi:64: error: Class tracemalloc.Traceback has abstract attributes ['__getitem__', '__len__']
stdlib/3.4/enum.pyi:7: error: Class enum.EnumMeta has abstract attributes ['__len__']
stdlib/2and3/distutils/cmd.pyi:7: error: Class distutils.cmd.Command has abstract attributes ['finalize_options', 'initialize_options', 'run']
stdlib/3/urllib/error.pyi:8: error: Class urllib.error.HTTPError has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
stdlib/3/io.pyi:260: error: Class io.IncrementalNewlineDecoder has abstract attributes ['decode']
stdlib/3/encodings/utf_8.pyi:3: error: Class encodings.utf_8.IncrementalEncoder has abstract attributes ['encode']
stdlib/3/encodings/utf_8.pyi:5: error: Class encodings.utf_8.IncrementalDecoder has abstract attributes ['_buffer_decode']
stdlib/2and3/distutils/command/build_py.pyi:7: error: Class distutils.command.build_py.build_py has abstract attributes ['finalize_options', 'initialize_options', 'run']
stdlib/2and3/distutils/command/build_py.pyi:8: error: Class distutils.command.build_py.build_py_2to3 has abstract attributes ['finalize_options', 'initialize_options', 'run']
stdlib/2and3/distutils/command/bdist_msi.pyi:5: error: Class distutils.command.bdist_msi.bdist_msi has abstract attributes ['finalize_options', 'initialize_options', 'run']
stdlib/3.4/asyncio/events.pyi:207: error: Class asyncio.events.BaseDefaultEventLoopPolicy has abstract attributes ['get_child_watcher', 'set_child_watcher']
stdlib/2and3/asynchat.pyi:12: error: Class asynchat.async_chat has abstract attributes ['collect_incoming_data', 'found_terminator']
third_party/2and3/requests/packages/urllib3/response.pyi:25: error: Class requests.packages.urllib3.response.HTTPResponse has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'isatty', 'readline', 'readlines', 'seek', 'seekable', 'truncate', 'writable', 'write', 'writelines']
third_party/2and3/requests/cookies.pyi:44: error: Class requests.cookies.RequestsCookieJar has abstract attributes ['__len__']

I believe most of these aren't intended to be abstract.

Wow, great catch, and thanks for endeavoring to fix these.

can we get this in mypy for real?

maybe with --strict-abstract and fixed with a decorator in mypy_extensions:

@mypy_extensions.abstract
class Foo(Iterable):
    pass

"error: Class io.IncrementalNewlineDecode must implement base members ['decode'] or be declared abstract"

Issue: python/mypy#3705

Let's keep this discussion in #3705.

make io.IncrementalNewlineDecoder concrete

make io.IncrementalNewlineDecoder concrete (#1489)

Are there plans to finish cleaning this up? We just ran into the SimpleCookie one today.

I haven't looked at this in a while, sorry. Perhaps we can spend some time on this in the sprints.

If people have ideas for automated ways to detect these classes, that would also be helpful, since otherwise it's easy to have new instances of this bug slip in.

improve codecs stubs

make BZ2File instantiable

make http.cookies.{Base,Simple}Cookie instantiable

make EnumMeta instantiable

I just regenerated the list by rebasing my mypy hack and making the heuristic a bit better (it's at https://github.com/JelleZijlstra/mypy/tree/noabstract). Here's a new list, excluding classes that clearly looked abstract, are fixed in an open PR, contain methods explicitly marked as abstract, or are private:

stdlib/3/shlex.pyi:14: error: Class shlex.shlex has abstract attributes ['__next__']
stdlib/3/urllib/response.pyi:6: error: Class urllib.response.addinfourl has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
stdlib/3/multiprocessing/pool.pyi:18: error: Class multiprocessing.pool.IMapIterator has abstract attributes ['__iter__']
stdlib/3.3/lzma.pyi:70: error: Class lzma.LZMAFile has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'flush', 'isatty', 'readlines', 'truncate', 'writelines']
stdlib/3/urllib/error.pyi:8: error: Class urllib.error.HTTPError has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'close', 'fileno', 'flush', 'isatty', 'read', 'readable', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines']
third_party/2and3/requests/packages/urllib3/response.pyi:25: error: Class requests.packages.urllib3.response.HTTPResponse has abstract attributes ['__enter__', '__exit__', '__iter__', '__next__', 'isatty', 'readline', 'readlines', 'seek', 'seekable', 'truncate', 'writable', 'write', 'writelines']