Skip to content

Replace urllib2 calls with requests to avoid SNI issues - #3

Merged
mingchen merged 6 commits into
python-cas:masterfrom
jeremylongo:master
Oct 27, 2016
Merged

Replace urllib2 calls with requests to avoid SNI issues#3
mingchen merged 6 commits into
python-cas:masterfrom
jeremylongo:master

Conversation

@jeremylongo

@jeremylongo jeremylongo commented Sep 15, 2016

Copy link
Copy Markdown
Contributor

In our environment, despite the upgrade to python 2.7.11, urllib2 has compatibility issues with haproxy on https, probably because of SNI.
However, requests works flawlessly.
We only have a mama-cas server, so we unfortunately can't test each case.

@nitmir

nitmir commented Sep 16, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor

urllib2 support SNI on my debian jessie, so I guess your problem is elsewhere. See the example below run on python 2.7.9 ran on a webserver with more than 25 vitualhosts, each one with a different X.509 cert.

In [1]: import urllib2

In [2]: r = urllib2.urlopen("https://cas.genua.fr")

In [3]: r.read()
Out[3]: '<!DOCTYPE html>\n<html lang="en">\n    <head>\n        <meta charset="utf-8">\n        <!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge" /><![endif]-->\n        <meta name="viewport" content="width=device-width, initial-scale=1">\n        <title>Central Authentication Service</title>\n        <link href="/static/javascript/bootstrap/css/bootstrap.min.css" rel="stylesheet">\n        <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->\n        <!-- WARNING: Respond.js doesn\'t work if you view the page via file:// -->\n        <!--[if lt IE 9]>\n        <script src="//oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>\n        <script src="//oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>\n        <![endif]-->\n        <link rel="shortcut icon" href="/static/cas_server/favicon.ico" />\n        <link href="/static/cas_server/styles.css" rel="stylesheet">\n    </head>\n    <body>\n      <div id="wrap">\n        <div class="container">\n            \n            <div class="row">\n              <div class="col-lg-12 col-md-12 col-sm-12 col-xs-12">\n                <h1 id="app-name">\n                    \n                    Central Authentication Service</h1>\n              </div>\n            </div>\n            \n            <div class="row">\n            <div class="col-lg-3 col-md-3 col-sm-2 col-xs-12"></div>\n            <div class="col-lg-6 col-md-6 col-sm-8 col-xs-12">\n            \n            \n            \n            \n\n<h2 class="form-signin-heading">Please log in</h2>\n\n\n            \n            \n            \n<form class="form-signin" method="post" id="login_form">\n  <input type=\'hidden\' name=\'csrfmiddlewaretoken\' value=\'MqKrpMLix6OL1wwdYx6YYr1tPOW9wQzr\' />\n  \n\n<input id="id_service" name="service" type="hidden" /><input id="id_lt" name="lt" type="hidden" value="LT-7iooUU81VFLeFkpizKhMFxoNBUwnwpBbo95kd0QlfUNBn4vrBCITUOz1hs4BU" /><input id="id_renew" name="renew" type="hidden" value="False" /><input id="id_gateway" name="gateway" type="hidden" /><input id="id_method" name="method" type="hidden" />\n<div class="form-group"><label class="control-label" for="id_username">username</label><input autofocus="autofocus" class="form-control" id="id_username" name="username" placeholder="username" required="required" type="text" /></div>\n\n<div class="form-group"><label class="control-label" for="id_password">password</label><input class="form-control" id="id_password" name="password" placeholder="password" required="required" type="password" /></div>\n\n<div class="form-group"><div class="checkbox"><label for="id_warn"><input id="id_warn" name="warn" type="checkbox" />Warn me before logging me into other sites.</label></div></div>\n\n\n  \n  <button class="btn btn-primary btn-block btn-lg" type="submit">Login</button>\n  \n</form>\n\n            </div>\n            <div class="col-lg-3 col-md-3 col-sm-2 col-xs-0"></div>\n            </div>\n        </div> <!-- /container -->\n      </div>\n      <div style="clear: both;"></div>\n      \n      <div id="footer">\n          <p><a class="text-muted" href="https://pypi.python.org/pypi/django-cas-server">django-cas-server powered</a></p>\n      </div>\n      \n      <script src="/static/javascript/jquery/jquery.min.js"></script>\n      <script src="/static/javascript/bootstrap/js/bootstrap.min.js"></script>\n      <script src="/static/cas_server/functions.js"></script>\n      <script type="text/javascript">\n\n\n\njQuery(function( $ ){\n    $("#id_warn").click(function(e){\n        if($("#id_warn").is(\':checked\')){\n            createCookie("warn", "on", 10 * 365);\n        } else {\n            eraseCookie("warn");\n        }\n    });\n});\n\n</script>\n      \n    </body>\n</html>\n<!--\nPowered by django-cas-server version 0.7.4\n\nPypi: https://pypi.python.org/pypi/django-cas-server\ngithub: https://github.com/nitmir/django-cas-server\n-->\n'

@nitmir

nitmir commented Sep 16, 2016

Copy link
Copy Markdown
Contributor

I found this, maybe it will resolve your issue:

In order to support SNI in python 2.6/2.7 you need to install additional packages:

  • pyOpenSSL, a Python wrapper module around the OpenSSL library.
  • pyasn1, ASN.1 types and codecs.
  • ndg-httpsclient, enhanced HTTPS support for httplib and urllib2.

The packages are python-openssl, python-pyasn1 and python-ndg-httpsclient on debian like systems.
@evgenyfadeev maybe it would be nice to add it to the README ?

@jeremylongo

jeremylongo commented Sep 20, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@nitmir python-ndg-httpsclient is not available on ubuntu 14.04 lts.
I updated python to 2.7.12 (as SNI was included in 2.7.9) and installed openssl, pyasn1 and ndg-httpsclient using pip .
Thing is, I checked SNI with a special check site, and it worked perfectly.
If I test with your code, it works as well, I get the same results as you.
But if I check my own server, which is behind a HAProxy, SNI is not recognized and I get a 503 error :

>>> import urllib2
>>> r = urllib2.urlopen('https://mysecreturl:443/cas/serviceValidate')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 503: Service Unavailable

However, using requests works perfectly.

@mingchen
mingchen merged commit d7bf775 into python-cas:master Oct 27, 2016
@mingchen

mingchen commented Oct 27, 2016

Copy link
Copy Markdown
Contributor

thank you.

mingchen added a commit that referenced this pull request Aug 3, 2018
@mingchen
Merge pull request #3 from jeremylongo/master
146acc2 
Replace urllib2 calls with requests to avoid SNI issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

3 participants

@jeremylongo @nitmir @mingchen