Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Taro L. Saito, Ph.D.
GitHub: @xerial
Arm Treasure Data
Airframe
Lightweight Building Blocks for Scala
Treasure Dataを支える技術: Airframe編
October 17th, 2018
Plazma - TD Tech Talk
1

Airframe
● Lightweight Building Blocks for Scala
● Essential for building any applications
● Used in production for 2+ years
● Based on my code collection since 2009
● Initially written in Java
● Gradually migrated to Scala
● Repackaged into wvlet.airframe in 2016
● For maintainability
● 18 Modules
● Simplifying your daily programming in Scala
2

Airframe
● Named From A Novel By Michael Crichton (1942-2008)
● The author of Jurassic Park
3

About Me: Taro L. Saito (Leo)
● An Engineer with Research Background
● Ph.D., University of Tokyo
● DBMS & Genome Science
● Developing Query Engines in TD
● Living in US for 3+ years
● Bay Area, Silicon Valley
● Active OSS Developer
● airframe
● sqlite-jdbc
■ More than 1000 GitHub stars
● snappy-java
■ Compression library used in
Spark, Parquet
● sbt-sonatype
■ Used in 2000+ Scala projects
● ...
4

Personal Goal of Today
● Collect 200 GitHub Stars
● keyword: Airframe + Scala
● https://github.com/wvlet/airframe
5

Major Goals
● Providing A Standard Toolkit For Building Reliable Services
● Removing Complexities In Application Development
● Providing Simplicity By Design
6

Simplicity By Design
● “Simplicity” by Philippe Dufour
● A clock made by a legendary
watchmaker in Switzerland
● Every part of the clock is built
by himself
● Airframe
● Provides simplicity for
application developers
7

Application Development with Airframe
● Bootstrap
● Parsing command-line options
● Reading configuration files
● Reading databases
● Object - Data Mapping
● Mapping data to objects (object mapping)
● Saving objects to files (serialization)
● Debugging
● Logging
● Collecting metrics
● Monitoring
● Building Services
● Creating service objects using dependency injection (DI)
8

18 Airframe Modules
● Bootstrap
● airframe-config Configuration loader
● airframe-opts Command-line option parser
● Object Serialization
● airframe-codec encoder/decoder SPI + standard codecs
● airframe-msgpack pure-Scala MessagePack implementation
● airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object
● Monitoring & Debugging
● airframe-log Logging
● airframe-metrics Human-readable metrics for time, date, data size, etc
● airframe-jmx Object metrics provider through JMX
● Building Service Objects
● airframe Dependency injection
● airframe-surface Object type inspector
● Misc:
● airframe-control, airframe-jdbc, airframe-json, airframe-http, etc.
9

Configuring Applications (airframe-config)
● Embedding static configurations for all environments into a docker image
● Merging YAML + external configurations + object default parameters
YAML
development:
addr: api-dev.com
production:
addr: api.com
Config Object
case class ServerConfig(
addr: String,
port: Int = 8080,
password: String
)
production:
addr: api.com
Select env:production Credentials and Local
Configurations
Merge
Immutable
Object Default Parameters
(e.g., port = 8080)
Object
Mapping
11

Reading And Saving Query Results
● Can we standardize this pattern?
RDBMS
JDBC
ResultSet
Seq[A]
Object
Mapping
External
Storage
(Cache)
Object
Serialization
Object
Deserialization
(Reload from cache)
12

Problem: Data Mapping is Everywhere
● How many data readers and object mappers do we need?
● How can we simplify this?
YAML
JDBC
ResultSet
YAML Parser +
Object Mapper
Config
Object
Table
Object
Object-Relation
Mapper
JSON
JSON Parser +
Object Mapper
Object
13

Using MessagePack As An Intermediate Data Format
● Why MessagePack?
● Flexible to support conversions from various types of data format
● Compact and efficient compared to JSON
● Easy to create schema-on-read object mapper
JDBC
ResultSet
MessagePack Object
Pack/Unpack
Unpack
Pack
YAML
JSON
14

Airframe Codec: Pack/Unpack Interface
● MessageCodec[A]
● pack: Convert object A into MessagePack
● unpack: Convert MessagePack into object A
Input MessagePack Output
Pack Unpack
PackUnpack
15

Pre-defined Codecs in airframe-codec
● Primitive Codecs
● ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
● FloatCodec, DoubleCodec
● StringCodec
● BooleanCodec
● TimeStampCodec
● Collection Codec
● ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
● OptionCodec
● JsonCodec
● Java-specific Codec
● FileCodec, ZonedDateTimeCodec, ResultSetCodec, etc.
● Adding Custom Codecs
● Implement MessageCodec[X] interface
16

ObjectCodec[A]: Combination of Codecs
● Generate Complex Codecs From The Parameter List of Objects
● MessagePack based serializer/deserializer
class A(
port:Int,
name:String,
timeoutSec:Double
)
Unpack
Pack
IntCodec
StringCodec
DoubleCodec
MessagePack
Array
Map
ObjectCodec[A]
17

Power of Schema-On-Read Codec
● MessagePack values describe their own data types (self-describing)
● How to deserialize the data can be determined based on MessagePack types
Int
Float
Boolean
String
Array
Map
Binary
CSV
MessagePack
JDBC
ResultSet
Column
Scala.Int
parseInt
toInt
0 or 1
IntCodec
Pack Unpack
Error or
Zero.of[Int]
“100”
(string)
100
(bigint)
100
(int)
18

Application 1/5: Treasure Data = MessagePack DBMS
● Fluentd -> MessagePack -> Treasure Data
● Automatically Generating Schema from Data
● Apply schema–on-read for providing table data to Presto/Hive/Spark, etc.
MessagePack
Fluentd
Mobile SDK
Table Schema
IntCodec
StringCodec
Generate Generate
Table Reader
Presto
Hive
Spark
Schema-free Data
19

Application 2/5: Data Transformation
● Airframe Codec Works As A Lightweight Embulk
List[A] MessagePack
Pack
Unpack
TSV
Pack/Unpack
JSON
SQLite3
20

Application 3/5: Taking Snapshots of Workflow Tasks
● Frequently Used for Data Analytic Pipelines
● Save Task Results As MessagePack (binary)
● Save the cost of re-computation
Result: Seq[A] MessagePack Storage
Pack
Save
Unpack
Task
Run
Load
Load
Compute
(e.g., 10 min)
First run
Snapshot
21

Application 4/5: Scala.js RPC
● Scala.js
● Compiling Scala code into JavaScript for Web Browsers
● Model classes can be shared between Scala and Scala.js
● airframe-msgpack
● Added pure-Scala MessagePack implementation for supporting Scala.js
UserInfo MessagePack UserInfo
Pack Unpack
PackUnpack
Scala
Server Side
Scala.js
Client Side
XML RPC
22

Application 5/5: Airframe HTTP Web Service
● Mapping HTTP Responses and Requests to Method Call Argument and Return
Values
● Airframe HTTP: Building Low-Friction Web Services Over Finagle (Medium Blog)
Http
Request
MessagePack
Pack
Request
Handler
Method
Unpack to Function Arguments
Http
Response
MessagePack
Unpack Return value
23

A Challenge: Type Erasure in Java Compiler (javac)
● Java class files (byte code) removes generic type information (type erasure)
class A (data:List[B])
class A
data: List[java.lang.Object]
javac
compiler
Class Parameter Type List ObjectCodec
Generate
???
Type Erasure
24

Airframe Surface
● Reading Type Signatures From ScalaSig
● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files
● Airframe Surface
● A library for inspecting object shapes
class A (data:List[B])
class A
class A
ScalaSig: data:List[B]
javac
scalac
Surface.of[A]
data: List[B]
scala.reflect.runtime.
universe.TypeTag
Type Erasure
25

What is Dependency Injection (DI)?
● Many Confusing Articles
● Inversion of Control Containers and the Dependency Injection pattern. Martin
Fowler (2004)
● StackOverflow, Wikipedia, …
● Many Frameworks
● Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc.
● No framework approaches do exists (Pure-Scala DI)
● Recent Definition:
● Dependency Injection is the process of creating the static, stateless graph of
service objects, where each service is parameterised by its dependencies.
■ What is Dependency Injection? by Adam Warski (2018)
● However, it’s still difficult to understand what is DI
27

Simplifying DI with Airframe
● Airframe Usage
● import wvlet.airframe._
● Simple 3 Step DI
● bind
● design
● build
● To Fully Understand DI …
● Think about what you can simplify
with DI
● Thinking about DI itself doesn’t
make much sense
■ e.g., comparing Guice,
Airframe, etc.
28

3 Things You Can Forget with Airframe DI
● 1. How to Build Service Objects
● 2. How to Manage Resource Lifecycle
● 3. How to Use DI Itself (!!)
Airframe Gives You A Focus On Application Development
29

1: Forget How to Build Service Objects
● When coding A and B
● You can focus on only direct dependencies
● You can forget about indirect dependencies
● Airframe DI builds A, B, and direct/indirect dependencies on your behalf.
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
30
You can forget this part

Replacing Modules For Testing The Service
● In Airframe Design
● You can replace DB and FluentdLogger to In-Memory Impl
● How to build A and B differs, but the same code can be used
A
Memory
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
In-memory
Logger
B
31
Overriding Design for Testing

2: Forget How to Manage Resource Lifecycle
● FILO := First-In Last-Out
● Airframe can add onStart and onShutdown lifecycle hooks when creating
instances
● When closing sessions, onShutdown will be called in the reverse order
● Dependencies forms DAG
● Dependencies will be generated when creating new service objects
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
134
56
7
2
8
Shared
Resource
32

3: Forget How to Use DI
33

Summary: Reducing Code Complexity with Airframe DI
● You can effectively forget about:
● How to build service objects
● How to manage resources in FILO order
● How to use DI itself
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
134
56
7
2
8
34
Implementation Details

Debugging Applications: Airframe Log
● Airframe Log: A Modern Logging Library for Scala (Medium Blog)
● ANSI color, source code location support
36

Airframe Metrics
● Human Readable Data Format (Duration, DataSize, etc.)
● Handy Time Window String Support (Used in TD_INTERVAL)
37

Airframe JMX
● Checking the internal states of remote JVM processes
● JMX clients
● jconsole has JMX metric monitor
● Airframe JMX -> DataDog -> Dashboard
38

Future Work
● Airframe Stream
● Stream Query Processing Engine for MessagePack
● Can be a query engine for various types of data through MessagePack
● Airframe Fluentd
● Metric objects -> MessagePack -> Fluentd
● and more ...
Input MessagePack
Pack
Unpack
Stream SQL MessagePack
Query
Processing
Filter/Aggregation/Join, etc.
40

Current State of Airframe
● Version 0.69 (As of October 2018)
● We already had 35+ releases in 2018
● Automated Release
● Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js
● ‘sbt release’ command took 3 hours
■ Sequential steps:
○ compile -> test -> package -> upload x 18 modules x 4 Scala versions
● Now a new version can be released in 10 minutes on Travis CI
● Blog
● 3 Tips for Maintaining Scala Projects
41

Summary
● Airframe
● Simplicity By Design
● 18 modules for simplifying application
development
● Key Technologies
● MessagePack-based codec
● airframe-surface to inspect object shapes
● Dependency injection (DI)
● Think What Can Be Simplified
● How MessagePack can be used to simplify
data transformation
● Airframe DI: 3 things you can forget
Don’t Forget Adding GitHub Star!
wvlet/airframe
42

Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17

More Related Content

What's hot (20)

Similar to Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17 (20)

More from Taro L. Saito (18)

Recently uploaded (20)

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17