From Trill to Quill: Pushing the Envelope of Functionality and Scale

Badrish Chandramouli @ DEBS 2016

• Real-time
raise alerts
• Real-time with historical
• Correlate
• Offline
• Develop initial monitoring query
• Back-test
• Progressive
Non-temporal analysis
Engine
+ Fabric
Interactive Query Authoring
Real-Time
Dashboard

• Performance
• Fabric & language integration
• Query model
Scenarios
• monitor
telemetry &
raise alerts
• correlate real-
time with logs
• develop initial
monitoring
query
• back-test over
historical logs
• offline analysis
(BI) with early
results

• Performance
• Fabric & language integration
• Query model

Q
1
2
3
2
1
5min Window
snapshots
logical time
Input
T-1
T-2
T-3
Output
Q = COUNT(*)
3
Relational
Model
Tempo-Relational
Model
QQQ Q Q𝜹𝜹𝜹 𝜹 𝜹
Supports broad & rich analytics
scenarios (relational, progressive,
time-based)

• Key enabler: performance +
fabric & language integration +
query model

struct ClickEvent { long ClickTime; long User; long AdId; }
var str = Network.ToStream(e => e.ClickTime, Latency(10secs));
var query =
str.Where(e => e.User % 100 < 5)
.Select(e => { e.AdId })
.GroupApply( e => e.AdId,
s => s.Window(5min).Aggregate(w => w.Count()));
query.Subscribe(e => Console.Write(e)); // write results to console

stream of batches
• More load  larger batches  better throughput
…
𝑜𝑝2
…
…
𝑜𝑝1

class DataBatch {
long[] SyncTime;
...
Bitvector BV;
}
class UserData_Gen : DataBatch {
long[] c_ClickTime;
long[] c_User;
long[] c_AdId;
}
…
𝑜𝑝2
…
…
𝑜𝑝1
timestamp payload columns
bitvector

str.Where(e => e.User % 100<5);
Send(events)
...
Application
Receive(results)
On(Batch b) {
for i = 0 to b.Size {
if !(b.c_User[i]%100 < 5)
set b.bitvector[i]
}
next-operator.On(b)
}
Trill

Func<TState> InitialState();
Func<TState, long, TInput, TState> Accumulate();
Func<TState,long, TInput, TState> Deaccumulate();
Func<TState, TState, TState> Sum();
Func<TState, TState, TState> Difference();
Func<TState, TResult> ComputeResult();
InitialState: () => 0
Accumulate: (oldCount, timestamp, input) => oldCount + 1
Deaccumulate: (oldCount, timestamp, input) => oldCount - 1
Sum: (leftCount, rightCount) => leftCount + rightCount
Difference: (leftCount, rightCount) => leftCount - rightCount
ComputeResult: count => count

session windows,
http://aka.ms/trill

• Increasing interest in real-time processing over
out-of-order streams
0
20
40
60
80
100
Refresh every second

Up to 8X faster

use existing high-perf in-order Trill operators unchanged

Low-latency
Completeness
1 sec, 98%
1 hour, 100%
?
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
10 seconds
Refresh every secondCloud telemetry log

Impatience framework gives us low latency, high
completeness, high throughput, and low memory usage
Latency Completeness
{1 sec} ~ 1 sec 98%
{1 hour} ~ 1 hour 100%
{1 sec}
+ {1 min}
+ {1 hour}
~ 1 sec 100%
{1 sec,
1 min,
1 hour}
~ 1 sec 100%
0
2
4
6
8
10
12
14
Throughput(million/sec)
Throughput
{1sec, 1min, 1hour} {1sec}+{1min}+{1hour}
1
10
100
1000
Memoryusage(MB),logscale
Memory usage
{1sec, 1min, 1hour} {1sec}+{1min}+{1hour}

no overlapping lifetimes
0
20
40
60
80
100

data streams and operations
arrays of numerical values

rich space
temporal logic
• Transfer
ShardedStreamable

shards
• querying
• data movement
• keying
Operation Description
Query Applies unmodified query on each
(keyed) shard
Broadcast Duplicate each shard’s contents on
all shards
Multicast Copy tuples from each input shard
to zero or more specific result
shards
ReShard Load balance across shards
ReDistribute Move tuples so that same key
resides in same result shard
ReKey Changes key associated with each
row in each shard
…
…
…
…

e => e.Count()
Flat re-
distribute
e => e.Count()
e => e.Sum()

e => e.Count()
[ReDist]
Union
[ReDist]
Union
[ReKey] [ReKey]
AGG AGG
[ReDist]
Union
[ReDist]
Union
[ReKey] [ReKey]
[ReDist]
Union
[ReDist]
Union
AGG AGG
[ReDist]
Union
[ReDist]
Union
AGG AGG
AGG AGG
e => e.Sum()

(l,r) => l.Join(r, …)
(l,r) => l.Join(r, …)
Flat re-
distribute
Flat
broadcast
No data
movement

str => str.SlidingWindow(Y).Count()
.Where(c => c > threshold)
(l, r) => l.WhereNotExists(y)
str => str.HoppingWindow(Z).Count()

•
•
•
•
•

Scan (Quill vs. SparkSQL) Time taken & scheduling overhead

Grouped agg with 40M groups Hopping window (Github data)

http://badrish.net/papers/shrink-TR.pdf

https://www.microsoft.com/en-us/research/people/badrishc/
http://aka.ms/streams/

From Trill to Quill: Pushing the Envelope of Functionality and Scale

More Related Content

What's hot (20)

Similar to From Trill to Quill: Pushing the Envelope of Functionality and Scale (20)

Recently uploaded (20)

From Trill to Quill: Pushing the Envelope of Functionality and Scale