Repository navigation
stdlib stubs are unnecessarily strict with file-like objects #4212
Description
Problem
Currently the IO situation is less than ideal. Not only are IO[str]/TextIO and IO[bytes]/BinaryIO a bit confusing (interchangeable in most cases), but the use of IO through stdlib is inconsistent and doing things like passing an object with a write() method to json.dump() does not work.
This is because the IO object, while describing the actual objects returned by open() perfectly, is not suitable to represent the "file-like object" interface. This interface is well known, documented prominently in the standard library's documentation (glossary: "file object" and "file-like object") and a testament to duck-typing; however it's not compatible with how typeshed is currently written (for the most part).
Proposal
I propose to introduce Protocols (not abstract classes) to be used for parameters where a "file object" is expected, allowing one to correctly type their file-like objects without having to inherit one of the abstract base classes. Furthermore, I think we should have two protocols representing files that can be read from or written to.
This work can be done incrementally, and I am willing to spend time doing this if there is no veto to this ticket.
Pros
This would allow a file-like object to be passed to json.dump(), zipfile.ZipFile, and others (like it already can to csv.write()).
Using Protocols of this small scale would allow objects that already conform to be used in interfaces expecting file-like object, without having to implement too many methods (or explicitly inherit from the base class, as is required now). This should lower the effort of bringing libraries to the typing world. Using two separate protocols is similar to how most languages do this, off the top of my head:
- Rust (
io::Readandio::Writetraits). - Java (
InputStreamwithread(),OutputStreamwithwrite()andflush()) - C++ (
istream,ostream)
It is interesting to note that the protocols I describe already exist in typeshed. Not wanting to put IO where the documentation called for file-like object, protocols have already been introduced:
- for
shutil.copyfileobj():_Readerand_Writerprotocols - for
csv.writer():_csv._Writerprotocol
Introducing those protocols would also allow us to remove some of the IO[str]/TextIO complexity: while TextIO and BinaryIO are still needed for the native file objects (they have additional methods compared to IO), the protocols used for function parameters everywhere can be only Read[str] and Read[bytes].
Cons
This is a sizeable change, and people are likely to use both the base class and those protocols for some time. However code using the base class should not break when passed to functions expecting the protocol.
Another caveat is that this might give a false sense of security: libraries in the wild do their own check to determine if an object conform to the interface, and for example pandas will not accept to write on a file object that does implement __iter__. Therefore objects conforming to the protocol might still not be accepted by (IMHO buggy) libraries, while inheriting the base class would make their objects look more like file objects (maybe too much, since it gives everything the attributes of both a readable and a writable file!).
Draft
Unfortunately this is where the typeshed-bikeshed starts, but this is my proposal:
AnyStr = typing.TypeVar('AnyStr', str, bytes) # typing.AnyStr
class WriteIO(Protocol[AnyStr]):
def write(self, s: AnyStr) -> int: ...
def flush(self) -> None: ...
def close(self) -> None: ...
def __enter__(self) -> 'WriteIO[AnyStr]': ...
def __exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType]) -> Optional[bool]: ...
class ReadIO(Protocol[AnyStr]):
def read(self, size: typing.Optional[int] = None) -> AnyStr: ...
def close(self) -> None: ...
def __enter__(self) -> 'WriteIO[AnyStr]': ...
def __exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType]) -> Optional[bool]: ...
def __iter__(self) -> Iterator[AnyStr]: ...Additional protocols can be added to provide seek()/tell() (similar to Rust's io::Seek trait)
Activity
JelleZijlstra commentedon Jun 10, 2020
We're already moving in this direction, thanks mostly to @srittau's efforts. However, our approach has been to create ad-hoc protocols for individual cases, instead of one-size-fits-all protocols like you propose. The trouble is that while in theory the file-like object concept may be "well known", in practice it is ill-defined, and there are lots of variations in exactly what methods are expected to exist on file-like objects.
remram44 commentedon Jun 10, 2020
I unfortunately experienced that first-hand with
pandas, however:shutil._Writerand_csv._Writerare identical)pandasand tell them that their attribute-checking is unusual, for exampleAlso adding methods on top of those core protocols is easy (e.g. extend the protocol) so there is probably no need to redefine those from scratch every place they are needed...
In any case I am glad to hear that an effort is going on. So long as I can get my custom writers to type-check when I pass them to
json.dump()and similar, I can move forward with typing. Is that effort documented or tracked somewhere? Would PRs fixing individual module stubs likejsonbe accepted?JelleZijlstra commentedon Jun 10, 2020
Yes, such PRs would be accepted. There's no centralized tracking, but we recently (#4161) added a
_typeshedpackage for internal types, so that would be a good place for IO protocols.srittau commentedon Jun 10, 2020
One of the goals of using ad-hoc protocols for now is to determine which protocols are needed in practice and then move those to
_typeshed.remram44 commentedon Jun 10, 2020
Once we converge on a set of protocols, wouldn't it be better to expose them publicly? Them being protocols, I can duplicate them in my code, but that still feels wrong.
JelleZijlstra commentedon Jun 10, 2020
Maybe once we make typeshed modular we can also generate
.pycode for_typeshedand publish it as a package.ramalho commentedon Jun 14, 2020
@remram44 this is a great proposal. I've been teaching Python for 20+ years and "a file-like object" has always been one of the best examples of the informal protocols that appear often in the standard library.
I think we should seek inspiration with Go, which popularized static duck typing several years before PEP 544. Their philosophy is that interfaces should be narrow, often just a single method, and it works very well for them. Take a look at:
io.Reader,io.Closeretc.And combinations like:
io.ReadCloser,io.ReadSeeker,io.ReadWriteSeekeretc.I also like very much their naming convention of turning a verb into a noun by affixing "er".
@srittau started a discussion about naming in #4174.
srittau commentedon Jun 14, 2020
Some previous discussion in python/typing#564.
srittau commentedon May 16, 2021
I'll just copy my comment from python/typing#213 here:
srittau commentedon Aug 11, 2021
I don't think there's anything immediately actionable here for typeshed. We are already moving in the directory of small ad-hoc protocols in typeshed that can be used for this purpose. Larger protocols only make sense if they are currently used.
twoertwein commentedon Nov 17, 2021
Pandas 1.4 will use protocols for its IO functions pandas-dev/pandas#43951
This will be slightly relaxed in 1.4.