Net prog

Network Programming
in Python
Steve Holden
Holden Web
LinuxWorld
January 20, 2004

Steve Holden - LinuxWorld, January 20, 2004
Introductions
• Steve Holden
– Professional instructor
• TCP/IP
• Database
• Security topics
– Consultant
• Over thirty years as a programmer
– Author of Python Web Programming
• New Riders, 2002

Pleased to Meet You …
• In less than 30 seconds:
– Your name
– Your organization
– Your current position
– Your interest in this tutorial
• Now we all know each other

Course Objectives
• Review principles of networking
• Contrast TCP and UDP features
• Show how Python programs access networking
functionality
• Give examples of client and server program
structures
• Demonstrate some Python network libraries
• Give pointers to other network functionality

One-Minute Overview
• Introduction to TCP/IP networking
– Not IPv6
• Though Python 2.3 handles IPv6
• Sockets: servers and clients
• Popular client libraries
• HTTP servers and clients
• What’s in the future?

Network Layering
• Applications talk to each other
– Call transport layer functions
• Transport layer has to ship packets
– Calls network layer
• Network layer talks to next system
– Calls subnetwork layer
• Subnetwork layer frames data for transmission
– Using appropriate physical standards
– Network layer datagrams "hop" from source to
destination through a sequence of routers

Inter-Layer Relationships
• Each layer uses the layer below
– The lower layer adds headers to the data from
the upper layer
– The data from the upper layer can also be a
header on data from the layer above …
PROTOCOL DATA
DATA
Upper layer
Lower layer HDR

The TCP/IP Layering Model
• Simpler than OSI model, with four layers
A
T
N
DATA
DL CRC
Application
Host-to-host
Internetwork
Subnetwork
Socket API

TCP/IP Components
• Just some of the protocols we expect to be
available in a “TCP/IP” environment
Telnet SSH SMTP FTP NFS DNS SNMP
TCP UDP
IP
Ethernet, Token Ring, RS232, IEEE 802.3, HDLC,
Frame Relay, Satellite, Wireless Links, Wet String
Application
Host-to-host
Internetwork
Subnetwork

IP Characteristics
• Datagram-based
– Connectionless
• Unreliable
– Best efforts delivery
– No delivery guarantees
• Logical (32-bit) addresses
– Unrelated to physical addressing
– Leading bits determine network membership

UDP Characteristics
• Also datagram-based
– Connectionless, unreliable, can broadcast
• Applications usually message-based
– No transport-layer retries
– Applications handle (or ignore) errors
• Processes identified by port number
• Services live at specific ports
– Usually below 1024, requiring privilege

TCP Characteristics
• Connection-oriented
– Two endpoints of a virtual circuit
• Reliable
– Application needs no error checking
• Stream-based
– No predefined blocksize
• Processes identified by port numbers
• Services live at specific ports

Client/Server Concepts
• Server opens a specific port
– The one associated with its service
– Then just waits for requests
– Server is the passive opener
• Clients get ephemeral ports
– Guaranteed unique, 1024 or greater
– Uses them to communicate with server
– Client is the active opener

Connectionless Services
socket()
bind()
recvfrom()
sendto()
[blocked]
SERVER
socket()
bind()
recvfrom()
[blocked]
sendto()
CLIENT

Simple Connectionless Server
from socket import socket, AF_INET, SOCK_DGRAM
s = socket(AF_INET, SOCK_DGRAM)
s.bind(('127.0.0.1', 11111))
while 1: # nowadays, "while True"
data, addr = s.recvfrom(1024)
print "Connection from", addr
s.sendto(data.upper(), addr)
• How much easier does it need to be?
Note that the bind() argument is a two-element tuple of address and port number

Simple Connectionless Client
s.bind(('127.0.0.1', 0)) # OS chooses port
print "using", s.getsocketname()
server = ('127.0.0.1', 11111)
s.sendto("MixedCaseString", server)
print "received", data, "from", addr
s.close()
• Relatively easy to understand?

Exercise 1: UDP Client/Server
• Run the sample UDP client and server I have
provided (see 00_README.txt)
– udpserv1.py
– udpcli1.py
• Additional questions:
– How easy is it to change the port number and
address used by the service?
– What happens if you run the client when the
server isn't listening?

Sample Python Module
• Problem: remote debugging
– Need to report errors, print values, etc.
– Log files not always desirable
• Permissions issues
• Remote viewing often difficult
• Maintenance (rotation, etc.) issues
• Solution: messages as UDP datagrams
– e.g. "Mr. Creosote" remote debugger
– http://starship.python.net/crew/jbauer/creosote/

Creosote Output
def spew(msg, host='localhost', port=PORT):
s = socket.socket((socket.AF_INET,
socket.SOCK_DGRAM))
s.bind(('', 0))
while msg:
s.sendto(msg[:BUFSIZE], (host, port))
msg = msg[BUFSIZE:]
• Creates a datagram (UDP) socket
• Sends the message
– In chunks if necessary

Creosote Input
def bucket(port=PORT, logfile=None):
s = socket.socket(socket.AF_INET,
socket.SOCK_DGRAM)
s.bind(('', port))
print 'waiting on port: %s' % port
while 1:
try:
data, addr =
s.recvfrom(BUFSIZE)
print `data`[1:-1]
except socket.error, msg:
print msg
• An infinite loop, printing out received messages

Exercise 2: Mr Creosote Demo
• This module includes both client and server
functionality in a single module
– creosote.py
• Very simple module with no real attemot to
use object-oriented features
• The production code is more complex
– creosoteplus.py
– Defines a bucket listener class
– Instance created when called with no arguments

Connection-Oriented Services
socket()
bind()
listen()
accept()
read()
write()
[blocked]
socket()
connect()
write()
read()
[blocked]
[blocked]
Server Client
When interaction is over, server
loops to accept a new connection

Connection-Oriented Server
from socket import
socket, AF_INET, SOCK_STREAM
s = socket(AF_INET, SOCK_STREAM)
s.bind(('127.0.0.1', 9999))
s.listen(5) # max queued connections
while 1:
sock, addr = s.accept()
# use socket sock to communicate
# with client process
• Client connection creates new socket
– Returned with address by accept()
• Server handles one client at a time

Connection-Oriented Client
s.connect((HOST, PORT))
s.send('Hello, world')
data = s.recv(1024)
s.close()
print 'Received', `data`
• This is a simple example
– Sends message, receives response
– Server receives 0 bytes after close()

Some socket Utility Functions
• htonl(i), htons(i)
– 32-bit or 16-bit integer to network format
• ntohl(i), ntohs(i)
– 32-bit or 16-bit integer to host format
• inet_aton(ipstr), inet_ntoa(packed)
– Convert addresses between regular strings and
4-byte packed strings

Handling Names & Addresses
• getfqdn(host='')
– Get canonical host name for host
• gethostbyaddr(ipaddr)
– Returns (hostname, aliases, addresses)
• Hostname is canonical name
• Aliases is a list of other names
• Addresses is a list of IP address strings
• gethostbyname_ex(hostname)
– Returns same values as gethostbyaddr()

Treating Sockets as Files
• makefile([mode[, bufsize]])
– Creates a file object that references the socket
– Makes it easier to program to handle data
streams
• No need to assemble stream from buffers

Exercise 3: TCP Client/Server
• Run the sample client and server I have
provided
– tcpserv1.py
– tcpcli1.py
– What happens if the client aborts (try entering
CTRL/D as input, for example)?
– Can you run two clients against the same
server?

Summary of Address Families
• socket.AF_UNIX
– Unix named pipe (NOT Windows…)
• socket.AF_INET
– Internet – IP version 4
– The basis of this class
• socket.AF_INET6
– Internet – IP version 6
– Rather more complicated … maybe next year

Summary of Socket Types
• socket.SOCK_STREAM
– TCP, connection-oriented
• socket.SOCK_DGRAM
– UDP, connectionless
• socket.SOCK_RAW
– Gives access to subnetwork layer
• SOCK_RDM, SOCK_SEQPACKET
– Very rarely used

Other socket.* Constants
• The usual suspects
– Most constants from Unix C support
SO_*, MSG_*, IP_* and so on
• Most are rarely needed
– C library documentation should be your guide

Timeout Capabilities
• Originally provided by 3rd-party module
– Now (Python 2.3) integrated with socket
module
• Can set a default for all sockets
– socket.setdefaulttimeout(seconds)
– Argument is float # of seconds
– Or None (indicates no timeout)
• Can set a timeout on an existing socket s
– s.settimeout(seconds)

Server Libraries
• SocketServer module provides basic server
features
• Subclass the TCPServer and UDPServer classes
to serve specific protocols
• Subclass BaseRequestHandler, overriding its
handle() method, to handle requests
• Mix-in classes allow asynchronous handling

Using SocketServer Module
• Server instance created with address and
handler-class as arguments:
SocketServer.UDPServer(myaddr,
MyHandler)
• Each connection/transmission creates a request
handler instance by calling the handler-class*
• Created handler instance handles a message
(UDP) or a complete client session (TCP)
* In Python you instantiate a class by calling it like a function

Writing a handle() Method
• self.request gives client access
– (string, socket) for UDP servers
– Connected socket for TCP servers
• self.client_address is remote address
• self.server is server instance
• TCP servers should handle a complete client
session

Skeleton Handler Examples
• No error checking
• Unsophisticated session handling (TCP)
• Simple tailored clients
– Try telnet with TCP server!
• Demonstrate the power of the Python network
libraries

UDP Upper-Case SocketServer
# udps1.py
import SocketServer
class UCHandler(SocketServer.BaseRequestHandler):
def handle(self):
remote = self.client_address
data, skt = self.request
print data
skt.sendto(data.upper(), remote)
myaddr = ('127.0.0.1', 2345)
myserver = SocketServer.UDPServer(myaddr, UCHandler)
myserver.serve_forever()
•Note: this server never terminates!
Change this function to
alter server's functionality

UDP Upper-Case Client
# udpc1.py
srvaddr = ('127.0.0.1', 2345)
data = raw_input("Send: ")
s.bind(('', 0))
s.sendto(data, srvaddr)
print "Recv:", data
• Client interacts once then terminates
• hangs if no response

TCP Upper-Case SocketServer
# tcps1.py
import SocketServer
class UCHandler(SocketServer.BaseRequestHandler):
def handle(self):
print "Connected:", self.client_address
while 1:
data = self.request.recv(1024)
if data == "rn":
break
print data[:-2]
self.request.send(data.upper())
myaddr = ('127.0.0.1', 2345)
myserver = SocketServer.TCPServer(myaddr, UCHandler)
myserver.serve_forever()
Change this function to
alter server's functionality

TCP Upper-Case Client
# tcpc1.py
from socket import socket, AF_INET, SOCK_STREAM
srvaddr = ('127.0.0.1', 2345)
s.connect(srvaddr)
while 1:
data = raw_input("Send: ")
s.send(data + "rn")
if data == "":
break
data = s.recv(1024)
print data[:-2] # Avoids doubling-up the newline
s.close()

Exercise 4: SocketServer Usage
• Run the TCP and UDP SocketServer-based
servers with the same clients you used before
– SockServUDP.py
– SockServTCP.py
– Is the functionality any different?
– What advantages are there over writing a
"classical" server?
– Can the TCP server accept multiple
connections?

Skeleton Server Limitations (1)
• UDP server adequate for short requests
– If service is extended, other clients must wait
• TCP server cannot handle concurrent sessions
– Transport layer queues max 5 connections
• After that requests are refused
• Solutions?
– Fork a process to handle requests, or
– Start a thread to handle requests

Simple Server Limitations (2)
accept()
read()
write()
[blocked]
[blocked]
Client connection
Forked server process or
thread runs independently
Remote
Client
Process
Server creates a new
thread or forks a new
process to handle
each request

Asynchronous Server Classes
• Use provided asynchronous classes
myserver = SocketServer.TCPServer(
myaddr, UCHandler)
becomes
myserver = SocketServer.ThreadingTCPServer(
myaddr, UCHandler)
or
myserver = SocketServer.ForkingTCPServer(
myaddr, UCHandler)

Implementation Details
• This is the implementation of all four
servers (from SocketServer.py):
class ForkingUDPServer(ForkingMixIn,
UDPServer): pass
class ForkingTCPServer(ForkingMixIn,
TCPServer): pass
class ThreadingUDPServer(ThreadingMixIn,
UDPServer): pass
class ThreadingTCPServer(ThreadingMixIn,
TCPServer): pass
• Uses Python's multiple inheritance
– Overrides process_request() method

More General Asynchrony
• See the asyncore and asynchat modules
• Use non-blocking sockets
• Based on select using an event-driven model
– Events occur at state transitions on underlying
socket
• Set up a listening socket
• Add connected sockets on creation

Exercise 5: Async TCP servers
• Can also be used with UDP, but less often
required (UDP often message-response)
– SockServTCPThread.py
• Very simple to replace threading with forking
– Non-portable, since forking not supported
under Windows (like you care … )

Network Client Libraries
• Python offers a rich variety of network client
code
– Email: smtplib, poplib, imaplib
•rfc822 and email modules handle content
– File transfer: ftplib
– Web: httplib, urllib
• More on these later
– Network news: nntplib
– Telnet: telnetlib

General Client Strategy
• Library usually defines an object class
• Create an instance of the object to interact with
the server
• Call the instance's methods to request
particular interactions

Using smtplib
• s = smtplib.SMTP([host[, port]])
– Create SMTP object with given
connection parameters
• r = s.sendmail(from, to, msg
[, mopts[, ropts]])
– from : sender address
– to : list of recipient addresses
– msg : RFC822-formatted message
(including all necessary headers)
– mopts, ropts : ESMTP option lists

SMTP Example (1)
import smtplib, socket
frad = "sholden@holdenweb.com"
toads = ["bookuser@holdenweb.com",
"nosuchuser@holdenweb.com",
"sholden@holdenweb.com"]
msg = """To: Various recipients
From: Steve Holden <sholden@holdenweb.com>
Hello. This is an RFC822 mail message.
"""

SMTP Example (2)
try:
server = smtplib.SMTP('10.0.0.1')
result = server.sendmail(frad, toads, msg)
server.quit()
if result:
for r in result.keys():
print "Error sending to", r
rt = result[r]
print "Code", rt[0], ":", rt[1]
else:
print "Sent without errors"
except smtplib.SMTPException, arg:
print "Server could not send mail", arg

Using poplib
• p = poplib.POP3(host[, port])
– Creates a POP object with given
connection parameters
• p.user(username)
– Provide username to server
• p.pass_(password)
– Provide password to server
• p.stat()
– Returns (# of msgs, # of bytes)

Using poplib (continued)
• p.retr(msgnum)
– returns (response, linelist, bytecount)
• p.dele(msgnum)
– Marks the given message for deletion
• p.quit()
– Terminate the connection
– Server actions pending deletes and
unlocks the mailbox

poplib Example (1)
import poplib, rfc822, sys, StringIO
SRVR = "mymailserver.com"
USER = "user"
PASS = "password"
try:
p = poplib.POP3(SRVR)
except:
print "Can't contact %s" % (SRVR, )
sys.exit(-1)
try:
print p.user(USER)
print p.pass_(PASS)
except:
print "Authentication failure"
sys.exit(-2)

poplib Example (2)
msglst = p.list()[1]
for m in msglst:
mno, size = m.split()
lines = p.retr(mno)[1]
print "----- Message %s" % (mno, )
file = StringIO.StringIO(
"rn".join(lines))
msg = rfc822.Message(file)
body = file.readlines()
addrs = msg.getaddrlist("to")
for rcpt, addr in addrs:
print "%-15s %s" % (rcpt, addr)
print len(body), "lines in message body"
print "-----"
p.quit()

Using ftplib
• f = ftplib.FTP(host[,user[,passwd[,acct]]])
– Creates an FTP object
• f.dir(directory)
– Send directory listing to standard output
• f.cwd(directory)
– Change to given directory
• f.mkd(directory)
– Create directory on server
• f.pwd()
– Returns current directory on server

Using ftplib (continued)
• retrbinary(command, callback[,
maxblocksize[, rest]])
– Retrieve a file in binary mode
– command - an FTP command
• E.g. "RETR myfile.dat"
– callback - processes each block
– maxblocksize – how much data per block
– rest – restart position

Using ftplib (continued)
• f.retrlines(command[, callback])
– Retrieves a file in text mode
– command - an FTP command
• E.g. "RETR myfile.txt"
– callback - processes each line as an argument
• Default callback prints line to standard output
• f.storlines(command, file)
– Sends content of file line-by-line
• f.storbinary(command, file, blocksize)
– Sends content of file block-by-block

Abbreviated ftplib Example
class Writer:
def __init__(self, file):
self.f = open(file, "w")
def __call__(self, data):
self.f.write(data)
self.f.write('n')
print data
FILENAME = "AutoIndent.py"
writer = Writer(FILENAME)
import ftplib
ftp = ftplib.FTP('127.0.0.1', 'book',
'bookpw')
ftp.retrlines("RETR %s" % FILENAME, writer)

HTTP and HTML Libraries
• Python applications are often web-based
• htmllib, HTMLParser – HTML parsing
• httplib – HTTP protocol client
• urllib, urllib2 – multiprotocol client
• SimpleHTTPServer, CGIHTTPServer –
SocketServer-based servers
• cgi, cgitb – CGI scripting assistance
• Various web samples also available

Using urllib
• f = urllib.urlopen(URL)
– Create file-like object that allows you to
read the identified resource
• urlretrieve(url[, filename[,
reporthook[, data]]])
– Reads the identified resource and store it
as a local file
• See documentation for further details
• This is very convenient for interactive use

Interactive urllib Session
>>> import urllib
>>> f = urllib.urlopen("http://www.python.org/")
>>> page = f.read() # treat as file to get body
>>> len(page)
14790
>>> h = f.info()
>>> h.getheader("Server")
'Apache/1.3.26 (Unix)'
>>> h.getheaders("Date")
['Thu, 29 May 2003 15:07:27 GMT']
>>> h.type
'text/html'
• Useful for testing & quick interactions

Using urllib2
• urllib has limitations - difficult to
– Include authentication
– Handle new protocols/schemes
• Must subclass urllib.FancyURLOpener and
bind an instance to urllib._urlopener
• urllib2 is intended to be more flexible
• The price is added complexity
– Many applications don't need the
complexity

urllib2.Request Class
• Instance can be passed instead of a URL
to the urllib2.urlopen() function
• r = Request(url, data=None, headers={})
– r.add_header(key, value)
• Can only add one header with a given key
– r.set_proxy(host, scheme )
• Sets the request to use a given proxy to
access the given scheme
– r.add_data(data)
• Forces use of POST rather than GET
• Requires http scheme

Serving HTTP
• Several related modules:
– BaseHTTPServer defines
•HTTPServer class
•BaseHTTPRequestHandler class
– SimpleHTTPServer defines
• SimpleHTTPRequestHandler class
– CGIHTTPServer defines
• CGIHTTPRequestHandler class
• All request handlers use the standard
HTTPServer.BaseHTTPRequestHandler

The Simplest Web Server …
import CGIHTTPServer, BaseHTTPServer
httpd = BaseHTTPServer.HTTPServer(('', 8888),
CGIHTTPServer.CGIHTTPRequestHandler)
httpd.serve_forever()
• Uses the basic HTTP server class
• Request handler methods implement the
HTTP PUT/GET/HEAD requests
• Yes, this really works!

Standard CGI Support
• cgi module provides input handling
• Recent (2.2) changes make things easier
– cgitb module traps errors
• Easier to diagnose problems
– Gives complete Python traceback
– Situation previously complicated by
differences in multi-valued form inputs
• Had to check, and program different actions
(string vs list)
• Python is excellent for producing HTML!

The cgi.FieldStorage Class
• Makes web client's input accessible
– Consumes input, so only instantiate once!
– Handles method GET or POST
– Optional argument retains blank values
• f.getfirst(name, default=None)
– Returns first (only) input value with given
name
• f.getlist(name)
– Returns a list of all values with given
name

Error Handling
• Should use for all CGI scripts!
import cgitb; cgitb.enable()
• Traps any errors, producing legible trace

Sample CGI Script
#!/usr/bin/python
import cgi, cgitb; cgitb.enable()
fields = ["subnum", "reviewer", "comments"]
form = cgi.FieldStorage()
vlist = []
for f in fields:
vlist.append("%s=%s" % (f, form.getfirst(f)))
print pgtmpl = """Content-Type: text/html
<html><head><title>Hello!</title></head>
%s
</body></html>
""" % "<br>".join(vlist)

Course Summary
• Reviewed principles of networking
• Contrasted TCP and UDP features
• Shown how Python programs access
networking functionality
• Given examples of client and server program
structures
• Demonstrated some Python network libraries
• Given pointers to other network functionality

Need more training?
Want to offer feedback?
Need help with a Python project?
Got budget?
Get in touch:
steve@holdenweb.com

Net prog

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Net prog (20)

Net prog

Editor's Notes