Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014

Byterun:
A (C)Python interpreter in Python
Allison Kaptur
!
github.com/akaptur
akaptur.github.io
@akaptur

Byterun
with Ned Batchelder
!
Based on
# pyvm2 by Paul Swartz (z3p)
from http://www.twistedmatrix.com/users/z3p/

Why would you do such a thing
>>> if a or b:
... do_stuff()

Some things we can do
out = ""
for i in range(5):
out = out + str(i)
print(out)

def fn(a, b=17, c="Hello", d=[]):
d.append(99)
print(a, b, c, d)
!
fn(1)
fn(2, 3)
fn(3, c="Bye")
fn(4, d=["What?"])
fn(5, "b", "c")

def verbose(func):
def _wrapper(*args, **kwargs):
return func(*args, **kwargs)
return _wrapper
!
@verbose
def add(x, y):
return x+y
!
add(7, 3)

try:
raise ValueError("oops")
except ValueError as e:
print("Caught: %s" % e)
print("All done")

class NullContext(object):
def __enter__(self):
l.append('i')
return self
!
def __exit__(self, exc_type, exc_val, exc_tb):
l.append('o')
return False
!
l = []
for i in range(3):
with NullContext():
l.append('w')
if i % 2:
break
l.append('z')
l.append('e')
!
l.append('r')
s = ''.join(l)
print("Look: %r" % s)
assert s == "iwzoeiwor"

g = (x*x for x in range(3))
print(list(g))

A problem
h = (y+1 for y in g)
print(list(h))

The Python virtual machine:
!
A bytecode interpreter

Bytecode:
the internal representation of a python
program in the interpreter

Bytecode: it’s bytes!
>>> def mod(a, b):
... ans = a % b
... return ans

>>> def mod(a, b):
... ans = a % b
... return ans
>>> mod.func_code.co_code
Function Code
object
Bytecode

>>> def mod(a, b):
... ans = a % b
... return ans
'|x00x00|x01x00x16}x02x00|x02x00S'

>>> def mod(a, b):
... ans = a % b
... return ans
‘|x00x00|x01x00x16}x02x00|x02x00S'
>>> [ord(b) for b in mod.func_code.co_code]
[124, 0, 0, 124, 1, 0, 22, 125, 2, 0, 124,
2, 0, 83]

dis, a bytecode disassembler
>>> import dis
>>> dis.dis(mod)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 BINARY_MODULO
7 STORE_FAST 2 (ans)
!
3 10 LOAD_FAST 2 (ans)
13 RETURN_VALUE

dis, a bytecode disassembler
>>> import dis
>>> dis.dis(mod)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 BINARY_MODULO
!
13 RETURN_VALUE
Line
Number
Index in
bytecode Instruction
name, for
humans
More bytes, the
argument to each
instruction
Hint about
arguments

whatever
some other thing
something
whatever
some other thing
something
a
b
whatever
some other thing
something
ans
Before
After
BINARY_MODULO
After
LOAD_FAST
Data stack on a frame

def foo():
x = 1
def bar(y):
z = y + 2
return z
return bar(x)
foo() # <--- (1)
!
c
a
l
l
!
s
t ---------------------
a | main (module) Frame | -> blocks: []
c | (oldest) | -> data: [<foo>]
k ---------------------

def foo():
x = 1
def bar(y):
z = y + 2 # <--- (3)
return z
foo() # <--- (1)
!
c
a
l
l ---------------------
s | | -> data: [3]
t ---------------------
k ---------------------

def foo():
x = 1
def bar(y):
z = y + 2 # <--- (3)
return z
foo() # <--- (1)
!
c
a
l
l
!
s
t ---------------------
c | (oldest) | -> data: [3]
k ---------------------

Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014

} /*switch*/
/* Main switch on opcode
*/
READ_TIMESTAMP(inst0);
!
switch (opcode) {

#ifdef CASE_TOO_BIG
default: switch (opcode) {
#endif
/* Turn this on if your compiler chokes on the big switch: */
/* #define CASE_TOO_BIG 1 */

Back to that bytecode
!
>>> dis.dis(mod)
2 0 LOAD_FAST 0 (a)
3 LOAD_FAST 1 (b)
6 BINARY_MODULO
!
13 RETURN_VALUE

case LOAD_FAST:
x = GETLOCAL(oparg);
if (x != NULL) {
Py_INCREF(x);
PUSH(x);
goto fast_next_opcode;
}
format_exc_check_arg(PyExc_UnboundLocalError,
UNBOUNDLOCAL_ERROR_MSG,
PyTuple_GetItem(co->co_varnames, oparg));
break;

case BINARY_MODULO:
w = POP();
v = TOP();
if (PyString_CheckExact(v))
x = PyString_Format(v, w);
else
x = PyNumber_Remainder(v, w);
Py_DECREF(v);
Py_DECREF(w);
SET_TOP(x);
if (x != NULL) continue;
break;

It’s “dynamic”
>>> def mod(a, b):
... ans = a % b
... return ans
>>> mod(15, 4)
3

“Dynamic”
>>> def mod(a, b):
... ans = a % b
... return ans
>>> mod(15, 4)
3
>>> mod(“%s%s”, (“Py”, “Gotham”))

“Dynamic”
>>> def mod(a, b):
... ans = a % b
... return ans
>>> mod(15, 4)
3
>>> mod(“%s%s”, (“Py”, “Gotham”))
PyGotham

“Dynamic”
>>> def mod(a, b):
... ans = a % b
... return ans
>>> mod(15, 4)
3
>>> mod(“%s%s”, (“Py”, “Gotham”))
PyGotham
>>> print “%s%s” % (“Py”, “Gotham”)
PyGotham

>>> class Surprising(object):
… def __mod__(self, other):
… print “Surprise!”
!
>>> s = Surprising()
>>> t = Surprsing()
>>> s % t
Surprise!

“In the general absence of type information, almost
every instruction must be treated as
INVOKE_ARBITRARY_METHOD.”
!
- Russell Power and Alex Rubinsteyn, “How Fast Can
We Make Interpreted Python?”

Back to our problem
h = (y+1 for y in g)
print(list(h))

More
Great blogs
http://tech.blog.aknin.name/category/my-projects/
pythons-innards/ by @aknin
http://eli.thegreenplace.net/ by Eli Bendersky
!
Contribute! Find bugs!
https://github.com/nedbat/byterun
!
Apply to Hacker School!
www.hackerschool.com/apply

Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014 (20)

Recently uploaded (20)

Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014