By Vasudev Ram
While checking out a tool that uses the requests HTTP library for Python, I happened to see that requests itself uses a library called urllib3 internally. (Here is urllib3 on PyPI.)
Since I had requests installed in my Python installation's directory, I searched for filenames like urllib3* in Python's lib/site-packages. Found the module there, in the directory:
requests/packages/urllib3
I also searched the Net and found this article by Kenneth Reitz, creator of the requests library:
Major Progress for Requests
in which he mentions collaborating with the creator of urllib3 to make use of it in requests.
urllib3 seems to have a good set of features, some of which are:
[
Re-use the same socket connection for multiple requests (HTTPConnectionPool and HTTPSConnectionPool) (with optional client-side certificate verification).
File posting (encode_multipart_formdata).
Built-in redirection and retries (optional).
Supports gzip and deflate decoding.
Thread-safe and sanity-safe.
Tested on Python 2.6+ and Python 3.2+, 100% unit test coverage.
Small and easy to understand codebase perfect for extending and building upon. For a more comprehensive solution, have a look at Requests which is also powered by urllib3.
]
So after checking the urllib3 docs a bit, I wrote a small program to test urllib3 by using it to download the home page of my web site, dancingbison.com:
# try_urllib3.py
# A program to try basic usage of the urllib3 Python library.
from requests.packages import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://dancingbison.com/index.html')
print "r.status: ", r.status
print "r.data", r.data
with open("dancingbison_index.html", "w") as out_fil:
out_fil.write(r.data)
It worked, and downloaded the file index.html.
Interestingly, urllib3 itself uses httplib under the hood. So it's turtles at least 3 levels down ... :-)
- xtopdf: programmable PDF creation for business
Vasudev Ram - Python / open source / Linux training and consulting
