Marrow: A Meta-Framework for Python 2.6+ and 3.1+

Marrow
Meta–Framework for Python 2.6+ and 3.1+

Alice Zoë Bevan–McGregor

Conﬁguration
YAML-Based Application Conﬁguration

Introspective Scripting
Non-Imperative Command-Line Parsing

Blueprint
Template–Derived Directory Trees
Interactive & Command–Line Interrogation

Streaming Templates
A Python Micro–Language

Server Interface
Modiﬁed Tornado IOLoop and IOStream
Server and Protocol Wrappers

HTTP/1.1 WSGI 2 Server
Highly Performant Pure Python HTTP/1.1 Implementation

Object Wrappers
PEP 444 Request / Response Objects
HTTP Status Code Exception Applications

Middleware / Filters
Compression, Sessions, etc.

Performance & Optimizations
Time for Timeit

Compatibility
Python 2.6+ and 3.1+

Least Developed
(So far.)

1 [server]
2 use = marrow.server.http:HTTPServer
3 host = 127.0.0.1, ::1
4 port = 8080, 8088
5
6 [mapping]
7 / = root
8
9 [app:root]
10 use = marrow.server.http.testing:hello
11 name = ConFoo

1 version: 1
2
3 server:
4 use: marrow.server.http:HTTPServer
5 host: ["127.0.0.1", "::1"]
6 port: [8080, 8088]
7
8 mapping:
9 /: *root
10
11 root: &root
12 use: marrow.server.http.testing:hello
13 name: ConFoo

Direct Object Access
(Entry points are for chumps.)

16 logging:
17 formatters:
18 brief:
19 format: '%(levelname)-8s: %(name)-15s: %(message)s'
20 handlers:
21 console:
22 class: logging.StreamHandler
23 formatter: brief
24 level: INFO
25 stream: ext://sys.stdout
26 loggers:
27 foo:
28 level: ERROR
29 handlers: [console]
30 root:
31 level: DEBUG
32 handlers: [console]

Paste Script
(entry point magic)

Paste Script
(paster <name> […])

Paste Script
(context–aware)

Commonality?
(un–Pythonic…)

Commonality?
(…hideous, hideous, imperative parser objects…)

1 import optparse
2
3 if __name__=="__main__":
4 parser = optparse.OptionParser("usage: %prog [options]
arg1 arg2")
5 parser.add_option("-H", "--host", dest="hostname",
6 default="127.0.0.1", type="string",
7 help="specify hostname to run on")
8 parser.add_option("-p", "--port", dest="portnum",
9 default=80, type="int",
10 help="port number to run on")
11 (options, args) = parser.parse_args()
12 if len(args) != 2:
13 parser.error("incorrect number of arguments")
14 hostname = options.hostname
15 portnum = options.portnum

1 import argparse
2
3 parser = argparse.ArgumentParser(description='Process some integers.')
4 parser.add_argument('integers', metavar='N', type=int, nargs='+',
5 help='an integer for the accumulator')
6 parser.add_argument('--sum', dest='accumulate', action='store_const',
7 const=sum, default=max,
8 help='sum the integers (default: find the max)')
9
10 args = parser.parse_args()
11 print(args.accumulate(args.integers))

1 # encoding: utf-8
2
3 def ultima(required, value=None, name="world",
switch=False, age=18, *args, **kw):
4 print "Hello %s!" % (name, )
5
6
7 if __name__ == '__main__':
8 __import__('marrow.script').script.execute(ultima)

Usage: ultima.py [OPTIONS] [--name=value...] <required> [value...]

OPTIONS may be one or more of:

-a, --age=VAL Override this value.
Default: 18
-h, --help Display this help and exit.
-n, --name=VAL Override this value.
Default: 'world'
-s, --switch Toggle this value.
Default: False
-v, --value=VAL Set this value.

@annotate
Argument Typecasting & Validation Callbacks

1 # encoding: utf-8
2
3 import marrow.script
4
5
6 @marrow.script.describe(
7 host = "specify a hostname to run on",
8 port = "port number to run on"
9 )
10 def serve(arg1, arg2, host="127.0.0.1", port=80):
11 pass
12
13
14 if __name__ == '__main__':
15 marrow.script.execute(serve)

1 class PackageBlueprint(Blueprint):
2 """Create an installable Python package."""
3
4 base = 'marrow.blueprint.package/files'
5 engine = 'mako'
6
7 settings = [
8 Setting(
9 'name',
10 "Project Name",
11 "The name to appear on the Python Package Index, e.g. CluComp.",
12 required=True
13 ),
14 Setting(
15 'package',
16 "Package Name",
17 "The name of the Python package, periods indicating namespaces, e.g. clueless.compiler.",
18 required=True
19 ),
20 # ...
21 ]
22
23 manifest = [
24 # ...
25 File('setup.py'),
26 File('setup.cfg'),
27 Folder('tests', children=[
28 File('.keep', 'keep')
29 ]),
30 package
31 ]

1 def package(settings):
2 def recurse(name):
3 head, _, tail = name.partition('.')
4
5 return [
6 Folder(head, children=[
7 File('__init__.py', 'namespace.py' if tail else 'init.py')
8 ] + (recurse(tail) if tail else []))
9 ]
10
11 return recurse(settings.package)

Marrow: A Meta-Framework for Python 2.6+ and 3.1+

Command–Line Answers
(marrow.script + **kw ;)

1 #!/usr/bin/env python
2 # encoding: utf-8
3
4 from __future__ import unicode_literals
5
6 from marrow.tags.html5 import *
7
8 from master import SITE_NAME, site_header, site_footer
9
10
11 def welcome():
12 return html [
13 head [ title [ 'Welcome!', ' — ', SITE_NAME ] ],
14 flush, # allow the browser to start downloading static resources early
15 body ( class_ = "nav-welcome" ) [
16 site_header,
17 p [
18 'Lorem ipsum dolor sit amet, consectetur adipisicing elit…'
19 ],
20 site_footer
21 ]
22 ]
23
24
25 if __name__ == '__main__':
26 with open('welcome.html', 'w') as fh:
27 for i in welcome().render('utf8'):
28 fh.write(i)

1 login = Form('sign-in', class_="tabbed", action='/users/action:authenticate', children=[
2 HiddenField('referrer'),
3 FieldSet('local', "Local Users", TableLayout, [
4 TextField('identity', "User Name", autofocus=True),
5 PasswordField('password', "Password")
6 ]),
7 FieldSet('yubikey', "Yubikey Users", TableLayout, [
8 TextField('identity', "User Name"),
9 PasswordField('password', "Password"),
10 PasswordField('yubikey', "Yubikey")
11 ]),
12 FieldSet('openid', "OpenID Users", TableLayout, [
13 URLField('url', "OpenID URL")
14 ])
15 ], footer=SubmitFooter('form', "Sign In"))

1 class Tag(Fragment):
2 def __call__(self, data_=None, strip=NoDefault, *args, **kw):
3 self = deepcopy(self)
4
5 self.data = data_
6 if strip is not NoDefault: self.strip = strip
7 self.args.extend(list(args))
8 self.attrs.update(kw)
9
10 return self

12 def __getitem__(self, k):
13 if not k: return self
14
15 self = deepcopy(self)
16
17 if not isinstance(k, (tuple, list)):
18 k = [k]
19
20 for fragment in k:
21 if isinstance(fragment, basestring):
22 self.children.append(escape(fragment))
23 continue
24
25 self.children.append(fragment)
26
27 return self

29 def __unicode__(self):
30 """Return a serialized version of this tree/branch."""
31 return ''.join(self.render('utf8')).decode('utf8')
32
33 def enter(self):
34 if self.strip:
35 raise StopIteration()
36
37 if self.prefix:
38 yield self.prefix
39
40 yield u'<' + self.name + u''.join([attr for attr in quoteattrs(self, self.attrs)]) + u'>'
41
42 def exit(self):
43 if self.simple or self.strip:
44 raise StopIteration()
45
46 yield u'</' + self.name + u'>'

48 def render(self, encoding='ascii'):
49 # ...
50
51 for k, t in self:
52 if k == 'enter':
53 # ...
54 continue
55
56 if k == 'exit':
57 # ...
58 continue
59
60 if k == 'text':
61 # ...
62 continue
63
64 if k == 'flush':
65 yield buf.getvalue()
66 del buf
67 buf = IO()
68
69 yield buf.getvalue()

71 def __iter__(self):
72 yield 'enter', self
73
74 for child in self.children:
75 if isinstance(child, Fragment):
76 for element in child:
77 yield element
78 continue
79
80 if hasattr(child, '__call__'):
81 value = child(self)
82
83 if isinstance(value, basestring):
84 yield 'text', unicode(value)
85 continue
86
87 for element in child(self):
88 yield element
89
90 continue
91
92 yield 'text', unicode(child)
93
94 yield 'exit', self

29 def __unicode__(self):
30 """Return a serialized version of this tree/branch."""
31 return ''.join(self.render('utf8')).decode('utf8')

96 def clear(self):
97 self.children = list()
98 self.args = list()
99 self.attrs = dict()
100
100 def empty(self):
101 self.children = list()

Py3K Tornado + Patches
IOLoop + IOStream

1 # encoding: utf-8
2
3 """An example raw IOLoop/IOStream example.
4
5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël.
6 """
7
8 import errno, functools, socket
9
10 from marrow.util.compat import exception
11 from marrow.io import ioloop, iostream
12
13 def connection_ready(sock, fd, events):
14 while True:
15 connection, address = sock.accept()
16
17 connection.setblocking(0)
18 stream = iostream.IOStream(connection)
19 stream.write(b"HTTP/1.0 200 OKrnContent-Length: 5rnrnPong!rn", stream.close)
20
21 if __name__ == '__main__':
22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
24 sock.setblocking(0)
25 sock.bind(("", 8010))
26 sock.listen(5000)
27
28 io_loop = ioloop.IOLoop.instance()
29 io_loop.set_blocking_log_threshold(2)
30 callback = functools.partial(connection_ready, sock)
31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ)
32
33 try:
34 io_loop.start()
35 except KeyboardInterrupt:
36 io_loop.stop()

1 # encoding: utf-8
2
3 """A simplified version of the raw example."""
4
5
6 from marrow.server.base import Server
7 from marrow.server.protocol import Protocol
8
9
10
11 class HTTPResponse(Protocol):
12 def accept(self, client):
13 client.write(
b"HTTP/1.0 200 OKrnContent-Length: 7rnrnPong!rn",
client.close)
14
15
16 if __name__ == '__main__':
17 Server(None, 8010, HTTPResponse).start()

functools.partial
(Pass the client object to your callbacks.)

Multi–Process
(incl. processor detection)

PEP 444
Is a draft of one possible WSGI 2 solution.

1 def hello(environ):
2 return b'200 OK', [
3 (b'Content-Type', b'text/plain')],
4 (b'Content-Length', b'12')
5 ], b"Hello world!"

An application is any function, method, or instance with a __call__ method. Applications must:
1. Be able to be invoked more than once. If this can not be guaranteed by the application implementation, it must be wrapped in a function
that creates a new instance on each call.
2. Accept a single positional argument which must be an instance of a base Python dictionary containing what is referred to as the WSGI
environment. The contents of this dictionary are fully described in the WSGI Environment section.
3. Return a 3-tuple of (status, headers, body) where:
1.status must contain the HTTP status code and reason phrase of the response. The status code and reason must be present, in that
order, separated by a single space. (See RFC 2616, Section 6.1.1 for more information.)
2.headers must be a standard Python list containing 2-tuples of (name, value) pairs representing the HTTP headers of the response. Each
header name must represent a valid HTTP header field name (as defined by RFC 2616, Section 4.2) without trailing colon or other
punctuation.
3.body must be an iterable representing the HTTP response body.
4.status and the name of each header present in headers must not have leading or trailing whitespace.
5.status, and the contents of headers (both name and value) must not contain control characters including carriage returns or linefeeds.
6.status, headers, and the chunks yielded by the body iterator should be returned as byte strings, though for implementations where
native strings are unicode, native strings may be returned. The server must encode unicode values using ISO-8859-1.
7. The amount of data yielded by the body iterable must not exceed the length specified by the Content-Length response header, if
defined.
8. The body iterable may be a native string, instead of a native string wrapped in a list for the simple case, but this is not recommended.
Additionally, applications and middleware must not alter HTTP 1.1 "hop-by-hop" features or headers, any equivalent features in HTTP 1.0, or
any headers that would affect the persistence of the client's connection to the web server. Applications and middleware may, however,
interrogate the environment for their presence and value. These features are the exclusive province of the server, and a server should consider
it a fatal error for an application to attempt sending them, and raise an error if they are supplied as return values from an application in the
headers structure.

A WSGI 2 server must:
1. Invoke the application callable once for each request it receives from an HTTP client that is directed at the application.
2. Pass a single positional value to the application callable representing the request environment, described in detail in the WSGI Environment
section.
3. Ensure that correct response headers are sent to the client. If the application omits a header required by the HTTP standard (or other relevant
speciﬁcations that are in effect), the server must add it. E.g. the Date and Server headers.
1. The server must not override values with the same name if they are emitted by the application.
4. Raise an exception if the application attempts to set HTTP 1.1 "hop-by-hop" or persistence headers, or equivalent headers in HTTP 1.0, as
described above.
5. Encode unicode data (where returned by the application) using ISO-8859-1.
6. Ensure that line endings within the body are not altered.
7. Transmit body chunks to the client in an unbuffered fashion, completing the transmission of each set of bytes before requesting another one.
(Applications should perform their own buffering.)
8. Call the close() method of the body returned by the application, if present, upon completion of the current request. This should be called
regardless of the termination status of the request. This is to support resource release by the application and is intended to complement PEP
325's generator support, and other common iterables with close() methods.
9. Support the HTTP 1.1 speciﬁcation where such support is made possible by the underlying transport channel, including full URL REQUEST_URI,
pipelining of requests, chunked transfer, and any other HTTP 1.1 features mandated in the relevant RFC.
Additionally,
1. HTTP header names are case-insensitive, so be sure to take that into consideration when examining application-supplied headers.
2. The server may apply HTTP transfer encodings or perform other transformations for the purpose of implementing HTTP features such as
chunked transfer.
3. The server must not attempt to handle byte range requests; the application can optimize this use case far more easily than a server. (For example
an application can generate the correct body length vs. generating the whole body and having the server buffer and slice it.)
4. Servers must not directly use any other attributes of the body iterable returned by the application.

4.5KR/sec
(Single process, single thread.)

C10K
(4 processes, single thread.)

10KR/sec
(4 process, single thread, lower concurrency.)

1 #!/usr/bin/env python
2 # encoding: utf-8
3
4 from __future__ import unicode_literals
5
6 from pprint import pformat
7
8 from marrow.server.http import HTTPServer
9 from marrow.wsgi.objects.decorator import wsgify
10
11
12 @wsgify
13 def hello(request):
14 resp = request.response
15 resp.mime = "text/plain"
16 resp.body = "%rnn%snn%s" % (request, request, pformat(request.__dict__))
17
18
19 if __name__ == '__main__':
20 import logging
21 logging.basicConfig(level=logging.DEBUG)
22
23 HTTPServer(None, 8080, application=hello).start()

24 class Request(object):

28 body = RequestBody('wsgi.input')
29 length = Int('CONTENT_LENGTH', None, rfc='14.13')
30 mime = ContentType('CONTENT_TYPE', None)
31 charset = Charset('CONTENT_TYPE')

102 def __getitem__(self, name):
103 return self.environ[name]
104
105 def __setitem__(self, name, value):
106 self.environ[name] = value
107
108 def __delitem__(self, name):
109 del self.environ[name]

1 class ReaderWriter(object):
2 default = NoDefault
3 rw = True
4
5 def __init__(self, header, default=NoDefault, rw=NoDefault, rfc=None):
6 pass # save arguments
7
8 def __get__(self, obj, cls):
9 try:
10 return obj[self.header]
11
12 except KeyError:
13 pass
14
15 if self.default is not NoDefault:
16 if hasattr(self.default, '__call__'):
17 return self.default(obj)
18
19 return self.default
20
21 raise AttributeError('WSGI environment does not contain %s key.' % (self.header, ))
22
23 def __set__(self, obj, value):
24 if not self.rw:
25 raise AttributeError('%s is a read-only value.' % (self.header, ))
26
27 if value is None:
28 del obj[self.header]
29 return
30
31 obj[self.header] = value
32
33 def __delete__(self, obj):
34 if not self.rw:
35 raise AttributeError('%s is a read-only value.' % (self.header, ))
36
37 del obj[self.header]

38 class CompressionFilter(object):
39 def __init__(self, level=6):
40 self.level = level
41
42 super(CompressionFilter, self).__init__()
43
44 def __call__(self, request, status, headers, body):
45 """Compress, if able, the response.
46
47 This has the side effect that if your application does not declare a content-length, this filter will.
48 """
49
50 # TODO: Remove some of this debug logging; it'll slow things down and isn't really needed.
51
52 if request.get('wsgi.compression', True) is False:
53 log.debug("Bypassing compression at application's request.")
54 return status, headers, body
55
56 if request.get('wsgi.async') and hasattr(body, '__call__'):
57 log.debug("Can not compress async responses, returning original response.")
59
60 if b'gzip' not in request.get('HTTP_ACCEPT_ENCODING', b''):
61 log.debug("Browser support for GZip encoding not found, returning original response.")

s="Content-Type: text/htmlrn"
^

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")
• Python 2.7: 0.631 • Python 3.1: 0.837

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")
• Python 2.7: 0.631 • Python 3.1: 0.837

a,c = s.partition(":")[::2]

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")
• Python 2.7: 0.631 • Python 3.1: 0.837

• Python 2.7: 0.642 • Python 3.1: 0.690

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")
• Python 2.7: 0.631 • Python 3.1: 0.837

• Python 2.7: 0.642 • Python 3.1: 0.690

a,b,c = s.partition(":")

a,b = s.split(":", 1)
• Python 2.7: 0.665 • Python 3.1: 0.909

a,b = s.split(":")
• Python 2.7: 0.631 • Python 3.1: 0.837

• Python 2.7: 0.642 • Python 3.1: 0.690

a,b,c = s.partition(":")
• Python 2.7: 0.407 • Python 3.1: 0.429

"Content-Type: text/htmlrn".upper()

• Python 2.7: 0.479 • Python 3.1: 0.469

• Python 2.7: 0.479 • Python 3.1: 0.469
"CONTENT-TYPE: text/htmlrn".upper()

• Python 2.7: 0.479 • Python 3.1: 0.469
• Python 2.7: 0.417 • Python 3.1: 0.616

• Python 2.7: 0.479 • Python 3.1: 0.469
• Python 2.7: 0.417 • Python 3.1: 0.616

"CONTENT-TYPE: TEXT/HTMLrn".upper()

• Python 2.7: 0.479 • Python 3.1: 0.469
• Python 2.7: 0.417 • Python 3.1: 0.616

• Python 2.7: 0.291 • Python 3.1: 0.407

• Python 2.7: 0.479 • Python 3.1: 0.469
• Python 2.7: 0.417 • Python 3.1: 0.616

• Python 2.7: 0.291 • Python 3.1: 0.407
"Content-Type: text/htmlrn".lower()

• Python 2.7: 0.479 • Python 3.1: 0.469
• Python 2.7: 0.417 • Python 3.1: 0.616

• Python 2.7: 0.291 • Python 3.1: 0.407
"Content-Type: text/htmlrn".lower()
• Python 2.7: 0.319 • Python 3.1: 0.497

", ".join((a, b))
• Python 2.7: 0.405 • Python 3.1: 0.319

", ".join((a, b))
• Python 2.7: 0.405 • Python 3.1: 0.319

a + ", " + b

", ".join((a, b))
• Python 2.7: 0.405 • Python 3.1: 0.319

a + ", " + b
• Python 2.7: 0.257 • Python 3.1: 0.283

a="://"; b="http://www.example.com/"

b.ﬁnd(a)
• Python 2.7: 0.255 • Python 3.1: 0.448

b.ﬁnd(a)
• Python 2.7: 0.255 • Python 3.1: 0.448

a in b

b.ﬁnd(a)
• Python 2.7: 0.255 • Python 3.1: 0.448

a in b
• Python 2.7: 0.104 • Python 3.1: 0.119

a.endswith(".bz")
• Python 2.7: 0.338 • Python 3.1: 0.515

a.endswith(".bz")
• Python 2.7: 0.338 • Python 3.1: 0.515

a[-3:] == ".bz"

a.endswith(".bz")
• Python 2.7: 0.338 • Python 3.1: 0.515

a[-3:] == ".bz"
• Python 2.7: 0.229 • Python 3.1: 0.312

uri.startswith("/")
• Python 2.7: 0.324 • Python 3.1: 0.513

uri.startswith("/")
• Python 2.7: 0.324 • Python 3.1: 0.513

uri[0] == "/"

uri.startswith("/")
• Python 2.7: 0.324 • Python 3.1: 0.513

uri[0] == "/"
• Python 2.7: 0.133 • Python 3.1: 0.146

formatdate
rfc822 vs. email.utils

“foo” vs. b“foo” vs. u“foo”

from __future__ import
unicode_literals

216 def _handle_read(self):
217 try:
218 chunk = self.socket.recv(self.read_chunk_size)
219
220 except socket.error:
221 e = exception().exception
222 if e.args[0] in (errno.EWOULDBLOCK, errno.EAGAIN):
223 return

Chasing Corporate care of Air Review Relevant Specifications
myspace.com/airreview PEP 333

WSGI 1.0
PEP 391

Dict Logging Configuration
Core Developers PEP 444

WSGI 2.0
Alice Bevan-McGregor PEP 3148

Futures
Alex Grönholm PEP 3333

WSGI 1.1

Resources RFC 1945 — HTTP 1.0
HTTP: The Definitive Guide, David RFC 2616 — HTTP 1.1
Gourley & Brian Totty, O’Reilly Press
Porting to Python 3, Lennart Regebro, Get GA to GA for PyCon!
CreateSpace http://pledgie.com/campaigns/14434

Marrow: A Meta-Framework for Python 2.6+ and 3.1+

More Related Content

What's hot (20)

Similar to Marrow: A Meta-Framework for Python 2.6+ and 3.1+ (20)

More from ConFoo (20)

Marrow: A Meta-Framework for Python 2.6+ and 3.1+