Mining event streams with BeepBeep 3

MINING
event streams
with
Sylvain Hallé, Raphaël Khoury
Massiva Roudjane, Eva Terzago
Quen�n Be�, Paul Lesur

$$$$$
$$$$$
$$$$$
x=104 y=12
x=232 y=21
119.5 s 4 1-2
1955-11-12
APPL
MSFT
=123.34
=208.56
GOGL
AMZN
=314.16
=271.82
10432.3
Src
Dst
=1.2.3.4:403
=5.6.7.8:221
Many elements of so�ware systems can be
modelled as pieces of data called events.

A stream (or trace) is a sequence of events.
The rate at which
events are produced
is called the
throughput.
A stored copy of a
stream is called a log.
. . .

”
∑
Event streams can be processed in various
ways. Some examples:
Aggrega�on
Pa�ern
detec�on
Visualiza�on
What is the
average price of
MSFT over 5 days?
Display bandwidth
usage for the
last 24 hours.
Does
ever collide
with ?
” ”
“ “ “

We are interested in two special kinds of
computa�on over streams.
Compu�ng trends over
a stream
Finding out if a stream
deviates from a given trend

We are interested in two special kinds of
computa�on over streams.
Compu�ng trends over
a stream
Finding out if a stream
deviates from a given trend
data mining
monitoring

Event stream query engine developed
based on the previous observations
Aims at borrowing strengths from both
RV and CEP (and beyond)
Key concepts: composability, modularity,
extensibility
Open source, developed in Java
http://liflab.github.io/beepbeep-3
peeB peeB 3

EventsEvents
An event is an element e taken from
some set E, called the event type.
No restriction on the type! In BeepBeep,
events can be any Object.
Booleans Numbers
2
3
4
π
Strings
abc
Functions Sets PlotsTuples
3 8 a
3 8 a
2 6 c
+
⊇?
XML
documents
<a><a><a>
. . .
?

TracesTraces
An event trace (or event stream) is a potentially
infinite sequence of events of a given type:
2 0 6 3
4 9 . . .
Traces are symbolically denoted by:
e = e0 e1 e2 e3 ...
The set of all traces of type T is denoted as:
T*

FunctionsFunctions
A function takes 0 or more events as its
input, and returns 1 or more events.
Functions are first-class objects; they descend
from the class Function
1 : 1
function
2 : 1
function
1 : 2
function
⊇?
3
4
2+5i
₹
2 5
6
0 : 1
function
6

ProcessorsProcessors
A processor takes 0 or more event traces as
its input, and returns 0 or more event traces as
its output
1 : 1 processor
2 : 1 processor
. . . . . .

ProcessorsProcessors
When a processor takes more than one input
trace, the set of events at matching positions
in each trace is called a front.
bacd
3601
b 3
a 6
c 0
d 1
1st event
2nd
3rd
4th
. . .
. . . . . .

Synchronous processingSynchronous processing
Events are processed one front at a time.
+
Buffers collect events until a complete
front can be processed.
⇒

+
⇒
5
3

+
⇒
5
3
+

+
⇒
8

+
⇒
1

+
⇒
6
1

+
⇒
6
1
4

+
⇒
6
1
4
+

+
⇒
6
5

+
⇒
6

Makes a couple of things simpler
Don't care about what event arrived
first or upstream computation time
"Pen and paper" calculation is identical
to the real one
Otherwise, can do a lot with simple
timeouts ⇒ contained asynchrony
Motto: Don't use asychronous processing...

Makes a couple of things simpler
Don't care about what event arrived
first or upstream computation time
"Pen and paper" calculation is identical
to the real one
Otherwise, can do a lot with simple
timeouts ⇒ contained asynchrony
Motto: Don't use asychronous processing...
...unless you really have to

In BeepBeep, all synchronous processors
are descendents of the SingleProcessor
class
Takes care of handling input/output
buffers
Calls (abstract) method process() when
an input front is ready to be consumed
Processor only needs to produce an
output front from this input
Makes it easy to create your own
(more on that later)

A high-level event trace can be produced by
composing ("piping") together one or more
processors from lower-level traces
CompositionComposition

Each processor has its own input/output
buffers

Any output can be connected to any input, as
long as they have the same type

Any output can be connected to any input, as
long as they have the same type
Many types can occur in the same chain

ArchitectureArchitecture
BeepBeep provides only a few built-in
processors and functions
Palette
Set of processors and functions,
centered around a particular
use case
Concretely, a JAR library
defining new Processor and
Function objects

<?
+
<? =?
−
÷ ×
f Σ f
n
{
n
n
Function Cumulative Trim
ForkDecimate Group
WindowSliceFilter
Built-in processors
Built-in
functions
.n
<
<

SemanticsSemantics
Let P be a processor and a
b
c
= a1,a2,...
= b1,b2,...
= c1,c2,... be traces
a,b,c : P[[
n
= e1,e2,...
denotes the n-th output trace of P, given
traces a, b, c as input.

f
FunctionFunction
Applies an n-ary function f to
every front of size n
"Lifts" any function into a
processor
a,b : f[[
n
= f n
(ai,bi)

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output
i
The i-th event

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output
i
The i-th event
f
+
f
<0?
Pairwise sum of
events
Is each event
negative?

CumulativeCumulative
Applies a 2 : 1 function f to
its previous value and the
current event
a : Σf[[ = f(x,a1), f(f(x,a1),a2), ...
+
<Sum of all
events
Have we seen
⊤ so far?
Σ f
x
Σ f Σ f
0 ⊥

TrimTrim
Returns the input trace,
trimmed of its first n events
a : [[ = an+1, an+2, ...
n
n

TrimTrim
Returns the input trace,
trimmed of its first n events
a : [[ = an+1, an+2, ...
n
n
1
f
=?=?
<
Σ f
⊥
Does the same
number
repeat twice in
a row?

GroupGroup
Makes a set of connected
processors behave as a single
processor

GroupGroup
Makes a set of connected
processors behave as a single
processor
1
f
=?=?
<
Σ f
⊥

ForkFork
Replicates its input on each
of its n outputs
a : Ψ[[ = a1, a2, ...
n

DecimateDecimate
Outputs every n-th input
event
a : [[ = a1, an+1, a2n+1, ...n
Ψ
n

FilterFilter
A n : n-1 processor; outputs
the n-1 first components of a
front if its last component is ⊤;
otherwise, discards it
a , a , ..., a : =[[ 2
k1 n
F
i
{a if a =⊤
ε otherwise
k
i
n
i
Powerful mechanism: "anything can
be filtered" (don't care about condition)
Boolean trace does not need to come
from the same source as the inputs being
filtered

FilterFilter
A n : n-1 processor; outputs
the n-1 first components of a
front if its last component is ⊤;
otherwise, discards it
f
<0?
Get only the negative
events of the input

WindowWindow
Returns the output of a
processor P on a sliding
window of width n
a : Υ =[[ P i
be windowed" (don't care about function)
n
{
n
ai, ... ai+n : P[[ *

WindowWindow
window of width n
a : Υ =[[ P i
be windowed" (don't care about function)
n
{
n
ai, ... ai+n : P[[ *
The last event

WindowWindow
window of width n
n
{
3
{+
Σ f
0

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 3 4 2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 3 24

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 243

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
2
243

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
2
243
4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
6
243
2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
6
243
23

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2
3 2 1

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2
9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 243 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 43 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
43

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
7 4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
7 41

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
8 7 4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 81
8 7 4
9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 81 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
31 8 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
31 8 9
The sum of all 3
successive events

SliceSlice
Dispatches an event e to a
distinct instance of processor
P according to the value of
some function f
{a if f(ai)=k
ε otherwise
i
πf
k[a : =[i
P
f[a : =[i U πf
k[a1,...ai : [i[ [: P
k *
+

SliceSlice
Dispatches an event e to a
distinct instance of processor
P according to the value of
some function f
{a if f(ai)=k
ε otherwise
i
πf
k[a : =[i
P
f[a : =[i U πf
k[a1,...ai : [i[ [: P
k *
+
Multiset union

f(x) = x mod 2
πf
0[a : =[subtrace of even numbers
πf
1[a : =[subtrace of odd numbers

f(x) = x mod 2
πf
πf
mod 2

f(x) = x mod 2
πf
πf
mod 2
5

f(x) = x mod 2
πf
πf
mod 2
5
U+

f(x) = x mod 2
πf
πf
mod 2
5
U+
{9}{5}

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}6

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}
6

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}
6
U+

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
U+
{9}{5}

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
{9}{5}1

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
{9}{5}
1

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,5}
6
{9}{5}
5
U+

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,1}
6
{9}{6,5}
5
U+
{5}

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,1}
6
{9}{6,5}
5
U+
{5}
The last odd and even
numbers seen so far

f(x) = x mod 2
πf
πf
mod 2
+
Σ f
0

f(x) = x mod 2
πf
πf
mod 2
+
Σ f
0
The sum of all odd
numbers and all even
numbers seen so far
{9}{6,6} {9}{6,5} {5}

Input/outputInput/output
0 : 1 processors can be used to produce an
event trace out of an external source (i.e.
standard input, a file, etc.)
Ditto for 1 : 0 processors
a . . .b
a . . .b

PalettesPalettes
BeepBeep provides only a few built-in
processors and functions
Palette
Set of processors and functions,
centered around a particular
use case
Concretely, a JAR library
defining new (reusable!)
Processor and Function
objects

Let's put it all
toghether!
i.e. a few examples of
queries from past publications

Use BeepBeep as
a library in your
program

→
Fork f = new Fork(2);
Use BeepBeep as
a library in your
program

→
→→
→
FunctionProcessor sum =
new FunctionProcessor( );
f
Use BeepBeep as
a library in your
program

→
→→
+
→
new FunctionProcessor(Addition.instance);
f
Use BeepBeep as
a library in your
program

→
→→
+
→
CountDecimate decimate = new CountDecimate(n);
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→→
Connector.connect(fork, LEFT, sum, LEFT)
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(fork, RIGHT, decimate, INPUT)
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(decimate, OUTPUT, sum, RIGHT);
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(decimate, OUTPUT, sum, RIGHT);
Pullable p = sum.getOutputPullable(OUTPUT);
while (p.hasNext() != NextStatus.NO) {
Object o = p.next();
...
}
f
n
Use BeepBeep as
a library in your
program

→
→ →
^
n
→
→ →
+
→
→
→
→
→
÷
→ →1 → →
+
Σ
Σ
0
0
The statistical moment of order n

→
→ →
^
n
→
→ →
+
→
→
→
→
→
÷
→ →1 → →
+
Σ
Σ
0
0
The statistical moment of order n
→ →E(x)
n
As a group
processor

→
→
→
→
1
→ →
>
Trigger an alarm when two successive
events

→
→
→
1
→
→
→
→
→
<?
→
→
→
1
→ →
>
events are more than 1

→
→
→
→
1
→ →
→
÷
→
→
→
→
→
<?
→
→
→
1
→
→ →σ
→
>
events are more than 1 standard
deviation

→
→
→
→
→
1
→ →
→
-
→ →
→
÷
→
→
→
→
→
<?
→
→
→
1
→
→ →E(x)
1
→ →σ
→
>
events are more than 1 standard
deviation from the mean

→
⊇?
→ → → A →→
T
??...<a> <a> ...
/a/b
//character/id/text()
→
→

→
→
?
?
T
/a/b
//status/text()
=? Walker
*
/a/b
//status/text()
=? Blocker
*
*
→→ →
/a/b
//character
→
→→
A
/a/b
//id/text()
→
T
??...<a> <a> ...

→
A
/a/b
//character[status=Walker]/id/text()
→ p1
→
A
→ p2
→ →
→
→
/a/b
//character[status=Blocker]/id/text()
→ →
→
3
→
→
→
<?
→
→
→
→
→
→
→
→
f1
f2
→
→ →
→
→
→
→
/a/b
//character[id=p1]/position/x/text()
/a/b
-
|...|
<?
6
>
...
f1
/a/b
/a/b
-
|...|
f2

→
→
*
*
*
*
Create Auction
=?
0
@
Last Price
Days
0:=
3@Max. Days :=Min. Price 2@:=
Days :=
Days 1
+
End of Day
=?
0
@
>
<?
Days
Last Price
2
@:=
Bid
=?
0
@
>
Min. Price
<?
2
@
Last Price
>?
2
@
Bid
=?
0
@
>
Last Price
>?
2
@
Days :=
Days 1
+
End of Day
=?
0
@
Max. Days
→
End of Day
*
1@
*
→
→→
→
*
Sold
=?
0
@
Days
Days
+ | |.÷

→
→
1@
+=
Sanitise
=?
0
@
Clean :=
Derive
=?
0
@
>
∊ ?
Clean
T
??...Derive(a,b)
Clean
2@
+=Clean :=
Clean
1
@
Use
=?
0
@
>
∊ ?
Clean
1
@
T *
*
T
...

is used for most projects
developed at LIF. We follow stringent
backward-compa�bility constraints:
all our code compiles on
1.6
is used for archiving
and version control.
Code is hosted in repositories on
git
.

Mining event streams with BeepBeep 3

All projects have standardized build scripts.All projects have standardized
Automated tests
are run with
using for con�nuous
integra�on.
Test results and coverage are
tracked through a dashboard on

$ git clone
$ ant download-deps
$ ant build
$ ant test
$ ant javadoc
$ ant jar

So�ware is released
under the licences
Libraries can be freely used,
including in commercial so�ware.
Apache
or

Tuples
Basic rela�onal algebra
NetP
Parsing of network packets
MTNP
Manipulate Tables N'Plots
ML
Machine Learning, Clustering,
Data Mining
Let us look at a few pale�es that we
will use...
h�ps://github.com/liﬂab/beepbeep-3-pale�es

3 8 a
3 8 a
2 6 c
Palette Tuples
New event type: tuple
(set of key-value pairs).
Turns an incoming event v into a tuple k=v
Turns the value of column x into the header of
column yx y
k
x y
Keeps only keys x, y and discards the others
Tuple union∪

Palette NetP
New event type: packet.
Parses a binary
blob into a packet
k
Fetches header
k from packet
h�ps://youtu.be/wA0ZIA-6tVE

In a stream of network packets, what is the set of all
source IP addresses?
?
?
f
src
Σ
∅
∪
f

?
?
f
src
Σ
∅
∪
f
converts blob
into packet

?
?
f
src
Σ
∅
∪
f
converts blob
into packet fetches source
address from
packet

?
?
f
src
Σ
∅
∪
f
converts blob
into packet fetches source
address from
packet
aggregates addresses
into a set

?
?
f
src
Σ
∅
∪
f
ApplyFunction to_packet =
new ApplyFunction(PacketParse.instance);
ApplyFunction get_src =
new ApplyFunction(new GetHeader("src"));
Cumulate address_set = new Cumulate(Sets.UNION);
connect(to_packet, get_src, address_set);

{
5m/1m
f
src
dst
Σ
∅
∪
f
#
Over a window of 5 minutes, how many source IP
addresses are associated to each destination address?
?
?
→

In a stream of network packets, what is the cumulative
bandwidth for each incoming IP address?
?
?
→
f
size
Σ
0
+
1
Σ
0
+
f
t
src
f
∪
+

t
0
1
2
3
...
10.10.10.1
3
61
61
108
...
103.1.57.6
-
-
89
89
...
The output is a stream of tables,
each of which looks like this:
�mestamp one column per source IP
cumula�ve
packet size

+
This table stream can be used as
the source of a plot that is dynamically
updated.
. . .
3

<?d
{
β d
δP
n
input
An
arbitrary
stream of
events.

<?d
{
β d
δP
n
trend processor
Computes a "trend" over
a ﬁnite sequence of events

<?d
{
β d
δP
n
sliding window
width
The trend is
computed over the
last n events.

<?d
{
β d
δP
n
reference trend
Used as a basis for
comparison with the current stream

<?d
{
β d
δP
n
distance metric
Computes the "distance" between
the computed trend and the reference

<?d
{
β d
δP
n
distance threshold
Maximum acceptable
distance, according to the
selected metric

<?d
{
β d
δP
n
comparison func�on
Checks if distance is above
threshold

<?d
{
β d
δP
n
output
A stream of
Booleans.
⊤ is emi�ed
whenever the
input stream
deviates "too
much" from P.

<?d
{
β d
δP
n
This pa�ern can be encapsulated into a
generic group processor taking 6 parameters.
Various computa�ons can be achieved by giving
diﬀerent values to these parameters.

β
i
f
#
Calculates the number of dis�nct symbols in
the input stream
iden�ty
func�on
map size
map
symbol → # occurrences

<?d
{
3
4
i
f
#
1
- ≤
Alerts whenever more than 3 dis�nct symbols
were seen in the last 4 events

Group counter = new Group(1, 1);
{
SlicerMap slicer = new SlicerMap(
new IdentityFunction(1), new Passthrough(1));
counter.associateInput(INPUT, slicer, INPUT);
ApplyFunction size = new ApplyFunction(Maps.Size);
connect(slicer, size);
counter.associateOutput(OUTPUT, size, OUTPUT);
counter.addProcessors(slicer, size);
}
TrendDistance<> alarm =
new TrendDistance<HashMap,Number,Number>(
3, 4, slicer, Numbers.Subtraction, 1,
Numbers.IsLessThan);
8 lines of code

β
f
Σ
0
+
Σ
0
+
÷
1
Calculates the running average of the input
stream...
...i.e. the average of all values seen
so far

β
f
Σ
0
+
Σ
0
+
÷
1
n
3 6
P δ
½
d
≤
| |
21
−

β
f
Σ
0
+
Σ
0
+
÷
1
n
3 6
P δ
½
d
≤
| |
21
−
21ABS( − ) Manha�an Distance
of dimension 1

<?d
{
6
3
½
≤
Alerts whenever the running average of the
last 3 events deviates by more than ½ from 6
f
Σ
0
+
Σ
0
+
÷
1
| |
21
−

<?d
{
6
3
½
≤
last 3 events deviates by more than ½ from 6
f
Σ
0
+
Σ
0
+
÷
1
| |
21
−
15 lines of code

β
i
Σ
0
+
1
Calculates the number of occurrences of
each symbol in the input stream
non-normalized probability
density func�on
a
b
c
3
1
5

n
9
P δ
2
d
≤
a
b
c
6
1
2
Reference pa�ern
is a distribu�on
(Manha�an) map distance func�on
a
b
c
6
1
2
a
b
c
3
4
1
= |6−3|+|1−4|+|2−1| = 7

β
i
Σ
0
+
1
n
9
P δ
2
d
≤
a
b
c
6
1
2

<?d
{
9
2
≤
Alerts whenever the distribu�on of the last 9
events is at a Manha�an map distance of more
than 2 from the reference distribu�on
i
Σ
0
+
1
a
b
c
6
1
2

<?d
{
9
2
≤
events is at a Manha�an map distance of more
than 2 from the reference distribu�on
i
Σ
0
+
1
a
b
c
6
1
2
9 lines of code

f
src
dst
Σ
∅
∪
f
#
β
In a stream of network packets, calculates the
number of source addresses associated
to each des�na�on address
10.1.104.32
103.208.1.2
15.61.6.80
...
6
11
24
...
a "distribu�on"

n
10 min
P δ
30
d
≤
10.1.104.32
103.208.1.2
15.61.6.80
...
6
11
24
...

n
10 min
P δ
30
d
≤
10.1.104.32
103.208.1.2
15.61.6.80
...
6
11
24
...
Maximum map distance func�on
a
b
c
6
1
2
a
b
c
3
4
1
= {|6−3|,|1−4|,|2−1|} = 3max

f
src
dst
Σ
∅
∪
f
#
β
n
10 min
P δ
30
d
≤
10.1.104.32
103.208.1.2
15.61.6.80
...
6
11
24
...

<?d
{ 30
≤
Alerts whenever,
in a window of 10 minutes, the number of source
addresses for some des�na�on IP address deviates
by more than 30 from the reference distribu�on
f
src
dst
Σ
∅
∪
f
#
10.1.104.32
103.208.1.2
15.61.6.80
...
6
11
24
...
10m/10m

<?d
{
β d
δP
n
So far, the reference trend has been
a single object.

What if there could be
mul�ple possible trends?
PP
mul�modal trend
<?d
{
β d
δP
n
So far, the reference trend has been
a single object.

Given a ﬁnite set of points P = {p1,p2,...},
deﬁne func�on:
Δ
min
p'∈P
Δ(p,p')(p) =
It is the distance
between p and the
closest point in P.
Δ can be any distance metric: Euclidean distance,
Manha�an distance, etc.
p1
p2
p3

β
f
Σ
0
+
Σ
0
+
÷
1
n
3 6
P δ
½
d
≤
| |
21
−
"Alerts whenever the
running average of the
last 3 events deviates
by more than ½
from 6"

β
f
Σ
0
+
Σ
0
+
÷
1
n
3 6
P δ
½
d
≤
| |
21
−
{6,9}
P δ
| |
21
−
"Alerts whenever the
running average of the
last 3 events deviates
by more than ½
from 6 and 9"

3 4 5 6 7 8 9 10 11 ......
In this case, the func�on , along with
and , deﬁnes two 1-dimensional "balls"
of radius ½.
The alarm is triggered when the running average
does not lie within one of these balls.
P d

3 4 5 6 7 8 9 10 11 ......
of radius ½.
P d
6.1, 6.0, 6.2 6.1
x
✓

3 4 5 6 7 8 9 10 11 ......
of radius ½.
P d
6.1, 6.0, 6.2 6.1
x
✓
7.3, 8.9, 7.5 7.9
x
✗

3 4 5 6 7 8 9 10 11 ......
of radius ½.
P d
6.1, 6.0, 6.2 6.1
x
✓
7.3, 8.9, 7.5 7.9
x
✗
Mul�-modal trends can also be
mul�-dimensional!

Consider streams of symbols a and b.
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1

i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
computes a
map of the number
of occurrences of
a and b
extracts the
map's values
normalizes the vector
casts into DoublePoint

i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
The output is a two-dimensional point
(or vector)
( 0.33, 0.66 )
frac�on
of a's frac�on
of b's

Suppose that the observed streams fall in
two categories:
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
Roughly
70% of a, 30% of b
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
Roughly
30% of a, 70% of b
( 0.7, 0.3 ) ( 0.3, 0.7 )
1 2

Streams can be seen as two-dimensional
points:
Fraction of a's
Fractionofb's
0% 100%
100%
1
2
(0.7,0.3)
(0.3,0.7)

β
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
n P δ
.15
d
≤9 ΔE
(0.3,0.7)
(0.7,0.3)
{
}

<?d
{
9
.15
≤
ΔE
(0.3,0.7)
(0.7,0.3)
{
}
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
events is further than .15 from either category

Fraction of a's
Fractionofb's
.15

Fraction of a's
Fractionofb's
a,a,b,a,a,b,a,b,a ( 0.67, 0.33 )p=
p

Fraction of a's
Fractionofb's
a,a,b,a,a,b,a,b,a ( 0.67, 0.33 )p=
p
d=0.047✓

Fraction of a's
Fractionofb's
b,a,b,a,b,a,b,a,b ( 0.44, 0.56 )p=
p

Fraction of a's
Fractionofb's
b,a,b,a,b,a,b,a,b ( 0.44, 0.56 )p=
p
d=0.2
This stream is "too diﬀerent" from
either category or !1 2
✗

Group vector = new Group(1, 1); {
Group counter = new Group(1, 1); {
Constant one = new Constant(1);
counter.associateInput(INPUT, one, INPUT);
Cumulate sum_one = new Cumulate(new CumulativeFunction<Number>(Numbers.Addition));
connect(one, sum_one);
counter.associateOutput(OUTPUT, sum_one, OUTPUT);
counter.addProcessors(one, sum_one);
}
SlicerMap slicer = new SlicerMap(new IdentityFunction(1), counter);
ApplyFunction to_normalized_vector = new ApplyFunction(
new FunctionTree(DoublePointCast.instance,
new FunctionTree(Normalize.instance,
new FunctionTree(ToValueArray.instance, new StreamVariable(0)))));
connect(slicer, to_normalized_vector);
vector.associateInput(INPUT, slicer, INPUT);
vector.associateOutput(OUTPUT, to_normalized_vector, OUTPUT);
vector.addProcessors(slicer, to_normalized_vector);
}
Multiset pattern = new Multiset();
pattern.add(new DoublePoint(new double[]{0.7, 0.3}))
.add(new DoublePoint(new double[]{0.3, 0.7}));
TrendDistance<> alarm = new TrendDistance<Multiset,Multiset,Number>(pattern, 9, vector,
new FunctionTree(AbsoluteValue.instance, new FunctionTree(
new DistanceToClosest(new EuclideanDistance()), new StreamVariable(0),
new StreamVariable(1))), 0.15, Numbers.IsLessThan);

Group vector = new Group(1, 1); {
Group counter = new Group(1, 1); {
Constant one = new Constant(1);
counter.associateInput(INPUT, one, INPUT);
Cumulate sum_one = new Cumulate(new CumulativeFunction<Number>(Numbers.Addition));
connect(one, sum_one);
counter.associateOutput(OUTPUT, sum_one, OUTPUT);
counter.addProcessors(one, sum_one);
}
SlicerMap slicer = new SlicerMap(new IdentityFunction(1), counter);
ApplyFunction to_normalized_vector = new ApplyFunction(
new FunctionTree(DoublePointCast.instance,
new FunctionTree(Normalize.instance,
new FunctionTree(ToValueArray.instance, new StreamVariable(0)))));
connect(slicer, to_normalized_vector);
vector.associateInput(INPUT, slicer, INPUT);
vector.associateOutput(OUTPUT, to_normalized_vector, OUTPUT);
vector.addProcessors(slicer, to_normalized_vector);
}
Multiset pattern = new Multiset();
pattern.add(new DoublePoint(new double[]{0.7, 0.3}))
.add(new DoublePoint(new double[]{0.3, 0.7}));
TrendDistance<> alarm = new TrendDistance<Multiset,Multiset,Number>(pattern, 9, vector,
new FunctionTree(AbsoluteValue.instance, new FunctionTree(
new DistanceToClosest(new EuclideanDistance()), new StreamVariable(0),
new StreamVariable(1))), 0.15, Numbers.IsLessThan);
17 lines of code

object
(e.g. stream)
feature extrac�on feature vector
(n dimensions)
? ⟨ 3.8, 0.5, 1.1 ⟩F
A recurring process in data mining:
Examples of feature vectors:
Distribu�on of symbols
Sta�s�cal moments
Any other numerical computa�on

<?d{
β d
δP
n
But where does this reference trend
come from?
?

<?d{
β d
δ
n
Op�on #1: it is computed
from the stream itself.
{m
β

<?d{
β d
δ
n
{m
β
fork
The stream
is split
in two

<?d{
β d
δ
n
{m
β
reference trend
The reference is computed over
a window of width m

<?d{
β d
δ
n
{m
β
stream oﬀset
Second stream copy is
trimmed of its ﬁrst m events

<?d{
β d
δ
n
{m
β
The rest of the pipe
works as before

m n
Trend from
"the present"
Trend from
"the past"
vs.
β β
An alarm is triggered when the stream's current
trend becomes "too diﬀerent" from what it was
in the past
⇒ self-correla�on

{
<?d
{
β d
δ
nm

{
<?d
{
β d
δ
nm
Self-Correlated
Trend Distance
(SCTD)

β
f
Σ
0
+
Σ
0
+
÷
1
n
3 6
δ
1
d
≤
| |
21
−
m

{
<?d
{
6
≤| |
21
−
6
3
1
f
Σ
0
+
Σ
0
+
÷
1
last 3 events deviates by more than 1 from
the running average of the 6 before

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream
Average between t−8 and t−3

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream
Average between t−2 and t

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream
Manhattan distance between averages

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream
Manhattan distance between averages
threshold
exceeded

0 2 4 6 8 10 12 14 16 18
0
1
2
3
4
5
6
t
Input stream
The average in
this window
("the past")
the average in
this window
("the present")
is too far
from

+
object feature extrac�on feature vector
(n dimensions)
? ⟨ 3.8, 0.5, 1.1 ⟩F
⟨ 3.8, 0.5, 1.1 ⟩
⟨ 6.5, 0.2, 1.1 ⟩
⟨ 5.0, 0.1, 1.6 ⟩
⟨ 4.4, 0.5, 0.9 ⟩
. . .
set of
feature vectors
C
clustering
algorithm
+
+ +
+
+
+
+
+
++
+
+
+
+
clusters
cluster
centroid
cluster
radius

β α
}{
α
Op�on #2: it is computed ahead of �me
from a set of reference streams.

β α
}{
α
Unpacks a set of streams
and feeds them one by
one
unpacking

β α
}{
α
Feed each event of a stream to
a processor and collect its last
output
dropping

β α
}{
α
Compute a trend over
a stream (same as before)
trend

β α
}{
α
Collate trend
objects
from all
streams into
a set
pack

β α
}{
α
Compute a
global trend
object from
all individual
trends
aggregate

β α
}{
α
This process can be encapsulated into a generic
group processor with 2 parameters.

β
f
Σ
0
+
Σ
0
+
÷
1
α
f
Σ
0
+
Σ
0
+
÷
1

β
f
Σ
0
+
Σ
0
+
÷
1
α
f
Σ
0
+
Σ
0
+
÷
1
running average
of a stream
average of values
in a set

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1
{⟨1,2,1,1⟩,
⟨2,2,1,2⟩,
⟨3,1,1,1⟩}
a set of 3 streams

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1
{⟨1,2,1,1⟩,
⟨2,2,1,2⟩,
⟨3,1,1,1⟩}
a set of 3 streams
1,2,1,1 1¼
{1¼}

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1
{⟨1,2,1,1⟩,
⟨2,2,1,2⟩,
⟨3,1,1,1⟩}
a set of 3 streams
2,2,1,2 1¾
{1¼,1¾}

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1
{⟨1,2,1,1⟩,
⟨2,2,1,2⟩,
⟨3,1,1,1⟩}
a set of 3 streams
3,1,1,1 1½
{1¼,1¾,1½}

}{
f
Σ
0
+
Σ
0
+
÷
1
f
Σ
0
+
Σ
0
+
÷
1
{⟨1,2,1,1⟩,
⟨2,2,1,2⟩,
⟨3,1,1,1⟩}
a set of 3 streams
1½
{1¼,1¾,1½}
the average
of all running
averages

C
The ML pale�e uses Apache Commons Mαth][which supports the following
clustering algorithms:
,
K-Means++
C Fuzzy-K-Means
C DBSCAN
C Mul�-K-Means++
h�ps://commons.apache.org/proper/commons-math/userguide/ml.html

β
f
Σ
0
+
Σ
0
+
÷
1
α
f
2
running average
of a stream
K-means func�on
(with K=2)

}{
f
Σ
0
+
Σ
0
+
÷
1
f
2
{⟨5,6,5,6,7,6⟩,
⟨8,9,8,10,9⟩,
⟨6,6,7,6,6,5⟩,
⟨7,6,6,7,6,6,5⟩,
⟨9,8,10,9⟩}
a set of streams

}{
f
Σ
0
+
Σ
0
+
÷
1
f
2
{⟨5,6,5,6,7,6⟩,
⟨8,9,8,10,9⟩,
⟨6,6,7,6,6,5⟩,
⟨7,6,6,7,6,6,5⟩,
⟨9,8,10,9⟩}
a set of streams
{5.83,
8.8,
6.0,
6.14,
9}
set of running
averages

}{
f
Σ
0
+
Σ
0
+
÷
1
f
2
{⟨5,6,5,6,7,6⟩,
⟨8,9,8,10,9⟩,
⟨6,6,7,6,6,5⟩,
⟨7,6,6,7,6,6,5⟩,
⟨9,8,10,9⟩}
a set of streams
{5.83,
8.8,
6.0,
6.14,
9}
set of running
averages
{5.99,
8.9}
cluster
centroids
5 6 7 8 9 10 11 ......

β α
f
2
distribu�on of
symbols
K-means func�on
(with K=2)
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1

}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1

}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}

}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}
(0.7,0.3),
(0.3,0.7),
(0.7,0.3),
(0.7,0.3),
(0.3,0.7),
(0.3,0.7)
{
}

}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}
(0.7,0.3),
(0.3,0.7),
(0.7,0.3),
(0.7,0.3),
(0.3,0.7),
(0.3,0.7)
{
}
cluster
centroids
{ (0.7,0.3),
(0.3,0.7) }
Fraction of a's
Fractionofb's
(0.7,0.3)
(0.3,0.7)

Kyoto Dataset
56.621025 http 951 6552 4 1.00 0.00 0.00 21 21 0.00 0.00 0.00 RSTO 0 0 0 -1 fda2:69aa:1f1a:3aef:7af3:3027:3045:7ff2 1670 fda2:69aa:1f1a:8fd3:358b:171c:73a3:0322 80 17:33:20 tcp
55.135794 http 1095 56999 0 0.00 0.00 0.00 15 15 0.00 0.00 0.00 RSTO 0 0 0 -1 fda2:69aa:1f1a:3aef:7af3:3027:3045:7ff2 1675 fda2:69aa:1f1a:e591:334a:006d:0292:71a0 80 17:33:22 tcp
0.368001 other 4 0 0 0.00 0.00 0.00 100 100 1.00 0.00 0.00 OTH 0 0 0 -1 fda2:69aa:1f1a:ac6d:7dc6:27b2:07ae:05ec 445 fda2:69aa:1f1a:f8aa:7da2:088f:3b04:1912 18 17:33:28 tcp
h�p://www.takakura.com/Kyoto_data/

Kyoto Dataset
56.621025 http 951 6552 4 1.00 0.00 0.00 21 21 0.00 0.00 0.00 RSTO 0 0 0 -1 fda2:69aa:1f1a:3aef:7af3:3027:3045:7ff2 1670 fda2:69aa:1f1a:8fd3:358b:171c:73a3:0322 80 17:33:20 tcp
0.368001 other 4 0 0 0.00 0.00 0.00 100 100 1.00 0.00 0.00 OTH 0 0 0 -1 fda2:69aa:1f1a:ac6d:7dc6:27b2:07ae:05ec 445 fda2:69aa:1f1a:f8aa:7da2:088f:3b04:1912 18 17:33:28 tcp
h�p://www.takakura.com/Kyoto_data/
Features can be
computed by BeepBeep
on a raw packet capture

Compute distribu�on of number of connec�ons
per hour of the day
h�ps://youtu.be/_Pc3Q_RiFaw

Compute rela�ve frequency of TCP ports
h�ps://youtu.be/1hdtWCpc0Xk

Plo�ng and clustering of session dura�on vs. �me
of the day
h�ps://youtu.be/DLJJDjXCeeQ

Perform K-means clustering on the dura�on of
each session
h�ps://youtu.be/aOnzSMK-E38

P Flimit
f
f
(x,y)
getSourceBytes
getDestinationBytes
[ ]
[ ]
f
ScatterPlotGenerator
f
KMeansSmart
ﬁlePath
kk
refreshInterval
Perform k-means clustering over bytes sent/received
by each session
h�ps://youtu.be/gA_OSVv2Q0g

All the examples use the same generic pa�ern,
yet amount to very diﬀerent calcula�ons
We only need to select (from a list of choices)
a few values, func�ons and processors
Key observation
Predeﬁned
queries hard-coded
into system
Users write
custom queries
in code
Users write
custom queries in
a custom language
FIXED FLEXIBLE
EASY HARD
This can be turned into a wizard!

Next >< Back
BeepBeep Data Mining Wizard
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the data mining pattern you wish to instantiate.
Self-correlated trend distance
Evaluates the similarity of a trend computed on the present
with the same trend computed over a past window.
Pattern-based trend distance
with a reference trend provided externally.

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the data mining pattern you wish to instantiate.
Self-correlated trend distance
with the same trend computed over a past window.
Pattern-based trend distance
with a reference trend provided externally.
{ <?d
{

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the input stream to use as the source
Pre-recorded log
Reads a stream from a ﬁle or a named pipe
Standard input
Reads input events from stdin
TCP connection
Captures packets from a local TCP port
Browse... No ﬁle selected
Port
{ <?d
{

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the input stream to use as the source
Pre-recorded log
Reads a stream from a ﬁle or a named pipe
Standard input
Reads input events from stdin
TCP connection
Captures packets from a local TCP port
Browse... No ﬁle selected
Port
stdin
{ <?d
{

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the time windows for the evaluation of the pattern
Past window (m)
Width of the window where the past trend is computed
Present window (n)
Width of the window where the present trend is computed
Change... Currently 1hr 0min 0sec
stdin
{ <?d
{

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the time windows for the evaluation of the pattern
Past window (m)
Width of the window where the past trend is computed
Present window (n)
Width of the window where the present trend is computed
stdin
{ <?d
{
1h1h
1h
1h

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the element of each event to use for computing the trend
Select the trend to compute over the stream
Running average
The average of the stream over the entire window
Vector of moments
The n ﬁrst statistical moments over the entire window
Distinct occurrences
The number of distinct values observed in the window
Value distribution
The distribution of values observed in the window
Cumulative sum
The sum of all values over the window
Direct value
Size
Other
Source
Destination
2
stdin
{ <?d
{
1h1h
1h
1h

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the element of each event to use for computing the trend
Select the trend to compute over the stream
Running average
The average of the stream over the entire window
Vector of moments
The n ﬁrst statistical moments over the entire window
Distinct occurrences
The number of distinct values observed in the window
Value distribution
The distribution of values observed in the window
Cumulative sum
The sum of all values over the window
Direct value
Size
Other
Source
Destination
2
stdin
{ <?d
{
1h1h
1h
1h
i
Σ
0
+
1
f
src

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Select the distance metric for comparing the present and the
past trends
Manhattan distance
Sum of pairwise absolute diﬀerences in each dimension
Euclidean distance
Geometrical distance in n dimensions
Scalar diﬀerence
Plain subtraction of two numbers
Ratio
Plain division of two numbers
stdin
{ <?d
{
1h1h
1h
1h
i
Σ
0
+
1
f
src

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Trigger an alarm when the distance becomes
the following threshold:
Smaller than
Larger than
0.5
stdin
{ <?d
{
1h1h
1h
1h
i
Σ
0
+
1
f
src

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Trigger an alarm when the distance becomes
the following threshold:
Smaller than
Larger than
0.5
stdin
{ <?d
{
1h1h
1h
1h
½
≤
i
Σ
0
+
1
f
src

Next >< Back
1. Pattern
2. Input stream
3. Windows
4. Trend
5. Distance
6. Threshold
Start< Back
To summarize, you requested the following data mining operation:
Next >Save...
Over a stream coming from the standard input,
extract a ﬁeld called Source,
and compare the distribution of unique values
between the last 1hr 0min 0sec
and the 1hr 0min 0sec that precedes it.
Raise an alert whenever
the Manhattan distance between them
exceeds the value of 0.5.
stdin
{ <?d
{
1h1h
1h
1h
½
≤
i
Σ
0
+
1
f
src

BeepBeep provides mul�ple func�onali�es for
performing data mining on event streams...
distance
metrics
| |
21
− ΔE
processor pipes
i
Σ
0
+
1
f
size
Σ
0
+
1
Σ
0
+
f
t
src
f
∪
+
f
2
clustering
algorithms
...and easy
means of
crea�ng custom
objects.
k
x y
event types and
manipula�on func�ons

Mining event streams with BeepBeep 3

More Related Content

What's hot (20)

Similar to Mining event streams with BeepBeep 3 (20)

More from Sylvain Hallé (20)

Recently uploaded (20)

Mining event streams with BeepBeep 3