CGIHTTPServer fix for Windows

@gvanrossum

BPO	427345
Nosy	@gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2003-01-08.18:57:25.000>
created_at = <Date 2001-05-25.17:37:32.000>
labels = ['library']
title = 'CGIHTTPServer fix for Windows'
updated_at = <Date 2003-01-08.18:57:25.000>
user = 'https://bugs.python.org/kasplat'

bugs.python.org fields:

activity = <Date 2003-01-08.18:57:25.000>
actor = 'holdenweb'
assignee = 'holdenweb'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2001-05-25.17:37:32.000>
creator = 'kasplat'
dependencies = []
files = []
hgrepos = []
issue_num = 427345
keywords = []
message_count = 14.0
messages = ['4865', '4866', '4867', '4868', '4869', '4870', '4871', '4872', '4873', '4874', '4875', '4876', '4877', '4878']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'nobody', 'holdenweb', 'justinshaw', 'kasplat', 'oztourer', 'kyrrigle']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue427345'
versions = []

CGIHTTPServer.py in the Python 2.1 library needs two
changes to the popen2 section in order to support
binary data and to avoid buffering problems under
Windows. The complete code block is shown at the end
of this message. The two changed lines are:
cmdline = "%s -u %s" % (interp, cmdline)
fi, fo = os.popen2(cmdline, 'b')

I've tested this under Windows 2000 Pro and binary
file uploads now work, however more extensive tests
should probably be done.

It appears there is another socket-related problem
and/or end-of-line conversion error that occurs when
using Internet Explorer 5.x and BASEHTTPServer with
CGIHTTPServer on the same machine. I'm still
investigating that issue.

Finally, I have done a small rewrite of
CGIHTTPServer.py to simplify subclassing it to support
running CGIs in any or all directories and using other
file extensions such as .cgi. The maintainer for the
current file should email me about the changes to see
whether they want to update the official library file.
I didn't post it here since it needs more tweaking

ka
---
CGIHTTPServer.p fixes...
elif self.have_popen2:
# Windows -- use popen2 to create a subprocess
import shutil
os.environ.update(env)
cmdline = scriptfile
if self.is_python(scriptfile):
interp = sys.executable
if interp.lower().endswith("w.exe"):
# On Windows, use python.exe, not
python.exe
interp = interp[:-5] = interp[-4:]
cmdline = "%s -u %s" % (interp, cmdline)
if '=' not in query and '"' not in query:
cmdline = '%s "%s"' % (cmdline, query)
self.log_error("command: %s", cmdline)
try:
nbytes = int(length)
except:
nbytes = 0
fi, fo = os.popen2(cmdline, 'b')
if self.command.lower() == "post" and nbytes > 0:
data = self.rfile.read(nbytes)
fi.write(data)
fi.close()
shutil.copyfileobj(fo, self.wfile)
sts = fo.close()
if sts:
self.log_error("CGI script exit status %#x",
sts)
else:
self.log_error("CGI script exited OK")

Logged In: NO

I want to elaborate on the second-to-last
paragraph in this report. CGIHTTPServer
in 2.1 and 2.0 is apparently broken for
POST requests (only) to CGI scripts, when
running locally on Windows with an IE client.

The details: there is a problem with
the combination of a CGIHTTPServer and
Intenernet Explorer both running locally
on Windows (with server name "localhost").
In this setup, POST requests to CGI scripts
fail, but GET requests to the exact same
script work fine. That is, a form with
method=GET, or a URL with an appended
query string, both work ok.

In POST mode, the CGI script gets proper
data in the input stream, and generates
a correct reply stream (which is in fact
identical to the output generated for
quivalent GET requests). So, somewhere
between CGIHTTPServer and IE, the data
seems to be dropped or munged. The same
thing happens of you force CGIHTTPServer
to use execfile() to launch the script,
instead of os.popen2.

I've also observed that endline conventions
seem to be irrelevant to the problem; using
\n or \r\n makes no difference in both the
input and reply streams. I've also been
told that the problem may not exist in
Netscape clients.

Since CGIHTTPServer is a very nice way to test
CGI scripts (all you need is a standard Python
install), this seems important. It would also
be nice if the directory assumptions in that
module were more customizable, but that's just
a wish.

--Mark Lutz

Logged In: YES
user_id=6380

I agree that this is a laudable goal, but I don't have the
time to spend to work on this. Please upload a patch to the
SF patch manager and assign it to me and I'll look at it.

In the mean time, I've applied the two small suggestions and
will close this bug report.

Logged In: YES
user_id=120658

To further elaborate on the problems with POST on Windows
95: without having reached any real understanding of the
problem I have found a tolerable workaround. By modifying
the default _rbufsize for class _fileobject in socket.py I
can get any POSTs with text length less than _rbufsize to
work. Here is the mod:

<pre>
class _fileobject:
def __init__(self, sock, mode, bufsize):
self._sock = sock
self._mode = mode
if bufsize < 0:
bufsize = 512
# SP 9 Aug 2001: default _rbufsize is too small,
crashing CGIHTTPServer on POST's
# This fix still only allows pages of less than
given buffer size to be updated,
# so the real fix has still to be discovered.
#self._rbufsize = max(1, bufsize)
self._rbufsize = max(16*1024, bufsize)
self._wbufsize = bufsize
self._wbuf = self._rbuf = ""
</pre>

-- StevePike

Logged In: YES
user_id=6380

Reopening -- not that I have time for this. :-(

Note that that patch is in socket.py.

If this really is a work-around, I'd like to understand why.

Logged In: YES
user_id=6380

Load shedding.

Logged In: YES
user_id=247536

this looks like the same issue as bpo-430160. and here's what
I just wrote there...

it appears that IE is sending 2 extra bytes ('\r\n') after
the request data. and if you don't read those two extra
bytes off, the window's socket handling gets messed up.

the result is that a partial response is returned and the
socket closed. IE tries to recover by re-POST'ing (which
is behavior specified in the HTTP/1.1 RFC)... only they
seem to add an embedded NULL the second time through, and
the original socket problem happens again anyway.

Try reading an extra 2 bytes from the rfile before sending
your response and the problem should go away. (you can do
that by 'self.rfile._rbufsize = content_length + 2' inside
your do_POST method before reading the content)

not sure what the real fix for this should be?

Logged In: YES
user_id=135558

I'm experiencing a problem on windows.
When I execute the scripts as
'http://localhost:8080/cgi-bin\\junk.py'
the page seems to load ok but I get the popup message:
Internet Explorer cannot open the Internet site
http://localhost:8080/cgi-bin\\junk.py
The connection with the server ws reset

On Netscape no errors but the download also takes forever.
The slow download seems to be caused by the
FieldStorage() call.

Logged In: YES
user_id=135558

To: Matthew King
Thanks for reposting that solution. That did seem to fix
the problem.

To: Mark Lutz
Using: Win2000 Pro IE5 python2.2.1c locally
The bug shows up as a popup message but the page loads.
When I tried it accross a lan the page wouldn't even
load (blowing a little demo). So I guess it is not ONLY a
local problem.

Thanks for posting.
Justin Shaw

Logged In: NO

hello

Logged In: YES
user_id=228025

This bug is related to another bug

https://sourceforge.net/tracker/?
func=detail&aid=430160&group_id=5470&atid=105470

Bob Denny (author of the WebSite server, formerly sold by
O'Reilly) had this to say about the POST problem:

"This is an old problem, traced back to Netscape and Unix-
based shell CGI scripts. Netscape browsers did this to force
completion of the POST into shell scripts. The cr/lf is not part
of the POST payload, and the CGI script that is dealing with
the POST needs to toss it away. The general solution is, read
Content-Length bytes, then reas anything else that may be in
the buffer and junk it. IE is emulating the behavior of
Netscape."

So, that explains why IE (and possibly other browsers) sticks
a CR/LF on the end of a POST causing a mismatch with the
Content-length header. Now the question is where a fix can
be inserted in the libs?

Looking again at CGIHTTPServer.py, perhaps this problem is
isolated to Windows, in which case the fix might be in the
block of code starting on line 218, and more specifically, line
246

data = self.rfile.read(nbytes)

so at that point, the remaining 2 bytes should be read, but
tossed. I'll investigate. Steve Holden and I have been emailing
back and forth on this issue, so if anyone else has other
suggestions, post here or email one of us.

Thanks,

ka

Logged In: YES
user_id=228025

I have a possible fix, at least when the server is running on
Windows. As suggested previously, the key is to read the
bytes beyond the content-length, but without blocking. Steve
Holden suggested I use select, which seems simpler than
another solution which is to make the socket non-blocking,
then attempting to read 2 bytes of data within a try/except
block.

The solution below just loops with select.select

            if self.command.lower() == "post" and nbytes > 0:
                data = self.rfile.read(nbytes)
                fi.write(data)
                # now throw away data past Content-length
                while select.select([self.rfile._sock], [], [], 0)[0] != 
[]:
                    waste = self.rfile._sock.recv(1)
            fi.close()

This is slightly verbose. The two additional lines go below the
fi.write statement on line 247 of CGIHTTPServer.py, line 248 if
you count the import select statement that needs to be
added at the top of the file.

I have only tested this on Windows since the server code is
Windows-specific. I do not know if a server running on Unix
suffers this same problem. I verified that POST works
correctly from the following browsers on Windows 2000: IE
5.5, Opera 6.0.4, and Netscape 4.7. Only IE sends the extra
CR/LF, Opera and Netscape do not.

ka

Logged In: YES
user_id=6380

Like 430160, I'm assigning this to Steve Holden who may
check in a fix. Good luck!

Logged In: YES
user_id=88157

Should be fixed by patch 654910, committed today.

Navigation Menu

Repository navigation

CGIHTTPServer fix for Windows #34546

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

CGIHTTPServer fix for Windows #34546

Description

Activity

kasplat commented on May 25, 2001

nobody commented on May 29, 2001

gvanrossum commented on Aug 7, 2001

oztourer commented on Aug 9, 2001

gvanrossum commented on Aug 9, 2001

gvanrossum commented on Sep 5, 2001

kyrrigle commented on Mar 28, 2002

justinshaw commented on Mar 29, 2002

justinshaw commented on Mar 31, 2002

nobody commented on Jul 9, 2002

kasplat commented on Aug 3, 2002

kasplat commented on Aug 5, 2002

gvanrossum commented on Aug 5, 2002

holdenweb commented on Jan 8, 2003

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions