Metaprogramming in Scala 2.10, Eugene Burmako,

Metaprogramming in Scala 2.10

Eugene Burmako

EPFL, LAMP

28 April 2012

Agenda

Intro

Reﬂection

Universes

Demo

Macros

Summary

Metaprogramming

Metaprogramming is the writing of computer programs
that write or manipulate other programs or themselves as
their data.
—Wikipedia

Compiler

Q: How to enable metaprogramming?
A: Who has more data about a program than a compiler?
Let’s expose the compiler to the programmer.

Reflection

In 2.10 we expose data about programs via reflection API.
The API is spread between scala-library.jar (interfaces),
scala-reflect.jar (implementations) and scala-compiler.jar
(runtime compilation).

Martin knows better

Design of the reﬂection API is explained in great detail in a recent
talk by Martin Odersky:
channel9.msdn.com/Lang-NEXT-2012/Reﬂection-and-Compilers

Hands on

Today we will learn the fundamentals of reﬂection API and learn to
learn more about it via a series of hands-on examples.

Macros

Q: Hey! What about macros?
A: Reflection is at the core of macros, reflection provides macros
with an API, reflection enables macros. Our focus today is
understanding reflection, macros are just a tiny bolt-on.
For more information about macros,
their philosophy and applications, take a look at my Scala Days talk:
http://scalamacros.org/talks/2012-04-18-ScalaDays2012.pdf.

Core data structures

Trees
Symbols
Types
C:ProjectsKepler>scalac -Xshow-phases
phase name id description
---------- -- -----------
parser 1 parse source into ASTs, simple desugaring
namer 2 resolve names, attach symbols to named trees
typer 4 the meat and potatoes: type the trees
pickler 7 serialize symbol tables

I’ll do my best to explain these concepts, but it’s barely possible to
do it better than Paul Phillips. Be absolutely sure to watch the
Inside the Sausage Factory talk (when the video is up).

Trees

Short-lived, mostly immutable, mostly plain case classes.
Apply(Ident("println"), List(Literal(Constant("hi!"))))

Comprehensive list of public trees can be found here:
scala/reﬂect/api/Trees.scala (be sure to take a look at the
”standard pattern match” section in the end of the ﬁle).

Learn to learn

-Xprint:parser (for naked trees)
-Xprint:typer (for typechecked trees)
-Yshow-trees and its cousins
scala.reflect.mirror.showRaw(scala.reflect.mirror.reify(...))
Q: Where do I pull these compiler flags from?
A: scala/tools/nsc/settings/ScalaSettings.scala

-Yshow-trees
// also try -Yshow-trees-stringified
// and -Yshow-trees-compact (or both simultaneously)
>scalac -Xprint:parser -Yshow-trees HelloWorld.scala
[[syntax trees at end of parser]]// Scala source:
HelloWorld.scala
PackageDef(
"<empty>"
ModuleDef(
0
"Test"
Template(
"App" // parents
ValDef(
private
"_"
<tpt>
<empty>
)
...

showRaw
scala> scala.reflect.mirror.reify{object Test {
println("Hello World!")
}}
res0: reflect.mirror.Expr[Unit] = ...

scala> scala.reflect.mirror.showRaw(res0.tree)
res1: String = Block(List(ModuleDef(
Modifiers(),
newTermName("Test"),
Template(List(Ident(newTypeName("Object"))), List(
DefDef(Modifiers(), newTermName("<init>"), List(),
List(List()), TypeTree(),
Block(List(Apply(Select(Super(This(newTypeName("")),
newTypeName("")), newTermName("<init>")),
List())), Literal(Constant(())))),
Apply(Select(Select(This(newTypeName("scala")),
newTermName("Predef")), newTermName("println")),
List(Literal(Constant("Hello World!")))))))),
Literal(Constant(())))

Symbols
Link definitions and references to definitions. Long-lived, mutable.
Declared in scala/reflect/api/Symbols.scala (very much in flux
right now!)
def foo[T: TypeTag](x: Any) = x.asInstanceOf[T]
foo[Long](42)

foo, T, x introduce symbols (T actually produces two different
symbols). DefDef, TypeDef, ValDef - all of those subtype
DefTree.
TypeTag, x, T, foo, Long refer to symbols. They are all
represented by Idents, which subtype RefTree.
Symbols are long-lived. This means that any reference to Int (from
a tree or from a type) will point to the same symbol
instance.

Learn to learn

-Xprint:namer or -Xprint:typer
-uniqid
symbol.kind and -Yshow-symkinds
:type -v
Don’t create them by yourself. Just don’t, leave it to Namer.
In macros most of the time you create naked trees, and Typer
will take care of the rest.

-uniqid and -Yshow-symkinds
>cat Foo.scala
def foo[T: TypeTag](x: Any) = x.asInstanceOf[T]
foo[Long](42)

// there is a mysterious factoid hidden in this printout!
>scalac -Xprint:typer -uniqid -Yshow-symkinds Foo.scala
[[syntax trees at end of typer]]// Scala source: Foo.scala
def foo#8339#METH
[T#8340#TPE >: Nothing#4658#CLS <: Any#4657#CLS]
(x#9529#VAL: Any#4657#CLS)
(implicit evidence$1#9530#VAL:
TypeTag#7861#TPE[T#8341#TPE#SKO])
: T#8340#TPE =
x#9529#VAL.asInstanceOf#6023#METH[T#8341#TPE#SKO];

Test#14#MODC.this.foo#8339#METH[Long#1641#CLS](42)
(scala#29#PK.reflect#2514#PK.‘package‘#3414#PKO
.mirror#3463#GET.TypeTag#10351#MOD.Long#10361#GET)

:type -v

We have just seen how to discover symbols used in trees.
However, symbols are also used in types.
Thanks to Paul (who hacked this during one of Scala Nights)
there’s an easy way to inspect types as well. Corresponding REPL
incantation is shown on one of the next slides.

Types

Immutable, long-lived, sometimes cached case classes declared in
scala/reﬂect/api/Types.scala.
Store the information about the full wealth of the Scala type
system: members, type arguments, higher kinds, path
dependencies, erasures, etc.

Learn to learn

-Xprint:typer
-Xprint-types
:type -v
-explaintypes

-Xprint-types

-Xprint-types is yet another option that modiﬁes tree printing.
Nothing very fancy, let’s move on to something really cool.

:type -v
scala> :type -v def impl[T: c.TypeTag](c: Context) = ???
// Type signature
[T](c: scala.reflect.makro.Context)(implicit evidence$1:
c.TypeTag[T])Nothing

// Internal Type structure
PolyType(
typeParams = List(TypeParam(T))
resultType = MethodType(
params = List(TermSymbol(c: ...))
resultType = MethodType(
params = List(TermSymbol(implicit evidence$1: ...))
resultType = TypeRef(
TypeSymbol(final abstract class Nothing)
)
)
)
)

-explaintypes
>cat Test.scala
class Foo { class Bar; def bar(x: Bar) = ??? }

object Test extends App {
val foo1 = new Foo
val foo2 = new Foo
foo2.bar(new foo1.Bar)
}

// prints explanations of type mismatches
>scalac -explaintypes Test.scala
Test.foo1.Bar <: Test.foo2.Bar?
Test.foo1.type <: Test.foo2.type?
Test.foo1.type = Test.foo2.type?
false
false
false
Test.scala:6: error: type mismatch;
...

Big picture

Trees are created naked by Parser.
Both deﬁnitions and references (expressed as ASTs) get their
symbols ﬁlled in by Namer (tree.symbol).
When creating symbols, Namer also creates their completers,
lazy thunks that know how to populate symbol types
(symbol.info).
Typer inspects trees, uses their symbols to transform trees
and assign types to them (tree.tpe).
Shortly afterwards Pickler kicks in and serializes reachable
symbols along with their types into ScalaSignature
annotations.

Universes

Universes are environments that pack together trees, symbols and
their types.
Compiler (scala.tools.nsc.Global) is a universe.
Reflective mirror (scala.reflect.mirror) is a universe too.
Macro context (scala.reflect.makro.Context) holds a reference
to a universe.

Symbol tables

Universes abstract population of symbol tables.
Compiler loads symbols from pickles using its own *.class
parser.
Reﬂective mirror uses Java reﬂection to load and parse
ScalaSignatures.
Macro context refers to the compiler’s symbol table.

Entry points

Using a universe depends on your scenario.
You can play with compiler’s universe (aka global) in REPL’s
:power mode.
With mirrors you go through the Mirror interface, e.g.
symbolOfInstance or classToType.
In a macro context, you import c.mirror. and can use
imported factories to create trees and types (don’t create
symbols manually, remember?).

Path dependency

An important quirk is that all universe artifacts are path-dependent
on their universe. Note the ”reflect.mirror” prefix in the type of
the result printed below.
scala> scala.reflect.mirror.reify(2.toString)
res0: reflect.mirror.Expr[String] =
Expr[String](2.toString())

When you deal with runtime reflection, you simply import
scala.reflect.mirror. , and enjoy, because typically there is only one
runtime mirror.
However with macros it’s more complicated. To pass artifacts
around (e.g. into helper functions), you need to also carry the
universe with you. Or you can employ the technique outlined in
a discussion at scala-internals.

Inspect members
scala> import scala.reflect.mirror
import scala.reflect.mirror

scala> trait X { def foo: String }
defined trait X

scala> typeTag[X]
res1: TypeTag[X] = ConcreteTypeTag[X]

scala> res1.tpe.members
res2: Iterable[reflect.mirror.Symbol] = List(method
$asInstanceOf, method $isInstanceOf, method sync
hronized, method ##, method !=, method ==, method ne,
method eq, constructor Object, method notifyAl
l, method notify, method clone, method getClass, method
hashCode, method toString, method equals, me
thod wait, method wait, method wait, method finalize,
method asInstanceOf, method isInstanceOf, meth
od !=, method ==, method foo)

Analyze, typecheck and invoke members
val d = new DynamicProxy{ val dynamicProxyTarget = x }
d.noargs
d.noargs()
d.nullary
d.value
d.-
d.$("x")
d.overloaded("overloaded with object")
d.overloaded(1)
val car = new Car
d.typeArgs(car) // inferred
d.default(1, 3)
d.default(1)
d.named(1, c=3, b=2) // works, wow

The proxy uses reflection that is capable of resolving all these calls.
More examples at test/files/run/dynamic-proxy.scala.
Implementation is at scala/reflect/DynamicProxy.scala.

Defeat erasure

scala> def foo[T](x: T) = x.getClass
foo: [T](x: T)Class[_ <: T]

scala> foo(List(1, 2, 3))
res0: Class[_ <: List[Int]] = class
scala.collection.immutable.$colon$colon

scala> def foo[T: TypeTag](x: T) = typeTag[T]
foo: [T](x: T)(implicit evidence$1: TypeTag[T])TypeTag[T]

scala> foo(List(1, 2, 3))
res1: TypeTag[List[Int]] = ConcreteTypeTag[List[Int]]

scala> res1.tpe
res2: reflect.mirror.Type = List[Int]

Compile at runtime

import scala.reflect.mirror._
val tree = Apply(Select(Ident("Macros"),
newTermName("foo")), List(Literal(Constant(42))))
println(Expr(tree).eval)

Eval creates a toolbox under the covers (universe.mkToolBox),
which is a full-fledged compiler.
Toolbox wraps the input AST, sets its phase to Namer (skipping
Parser) and performs the compilation into an in-memory
directory.
After the compilation is finished, toolbox fires up a classloader that
loads and lauches the code.

Macros

In our hackings above, we used the runtime mirror
(scala.reflect.mirror) to reflect against program structure.
We can do absolutely the same during the compile time. The
universe is already there (the compiler itself), the API is there as
well (scala.reflect.api.Universe inside the macro context).
We only need to ask the compiler to call ourselves during the
compilation (currently, our trigger is macro application and the
hook is the macro keyword).
The end.

No, really

That’s it.

Summary

In 2.10 you can have all the information about your program
that the compiler has (well, almost).
This information includes trees, symbols and types. And
annotations. And positions. And more.
You can reﬂect at runtime (scala.reﬂect.mirror) or at
compile-time (macros).

Status

We’re already there.
Recently released 2.10.0-M3 includes reﬂection and macros.

Thanks!

eugene.burmako@epﬂ.ch

Metaprogramming in Scala 2.10, Eugene Burmako,

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Metaprogramming in Scala 2.10, Eugene Burmako, (20)

More from Vasil Remeniuk (20)

Recently uploaded (20)

Metaprogramming in Scala 2.10, Eugene Burmako,