Real-Time Data Mining for Event Streams
0 likes210 views
The document discusses data mining techniques for processing event streams in real-time, highlighting methodologies for trend analysis and deviation detection. It emphasizes the importance of defining parameters for trend computation and the use of distance metrics to identify anomalies in the data. The text also mentions an upcoming book that delves deeper into event stream processing and provides numerous code examples.
![Consider streams of symbols a and b.
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-53-320.jpg)
![Consider streams of symbols a and b.
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
computes a
map of the number
of occurrences of
a and b
extracts the
map's values
normalizes the vector
casts into DoublePoint](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-54-320.jpg)
![Consider streams of symbols a and b.
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
The output is a two-dimensional point
(or vector)
( 0.33, 0.66 )
frac�on
of a's frac�on
of b's](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-55-320.jpg)
![β
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
n P δ
.15
d
≤9 ΔE
(0.3,0.7)
(0.7,0.3)
{
}](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-58-320.jpg)
![<?d
{
9
.15
≤
ΔE
(0.3,0.7)
(0.7,0.3)
{
}
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
Alerts whenever the distribu�on of the last 9
events is further than .15 from either category](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-59-320.jpg)
![C
The ML pale�e uses Apache Commons Mαth][which supports the following
clustering algorithms:
,
K-Means++
C Fuzzy-K-Means
C DBSCAN
C Mul�-K-Means++
h�ps://commons.apache.org/proper/commons-math/userguide/ml.html](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-107-320.jpg)
![β α
f
2
distribu�on of
symbols
K-means func�on
(with K=2)
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-115-320.jpg)
![}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-116-320.jpg)
![}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-117-320.jpg)
![}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}
(0.7,0.3),
(0.3,0.7),
(0.7,0.3),
(0.7,0.3),
(0.3,0.7),
(0.3,0.7)
{
}](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-118-320.jpg)
![}{ f
2
i
Σ
0
+
1
f
DP
Σ=1
[ ]
1
a,a,a,a,b,a,b,a,b,a
a,a,a,a,a,b,b,a,a,b
b,b,a,a,a,b,a,a,a,a
a,b,a,b,b,b,b,b,b,a
b,b,b,b,a,a,b,b,a,b
b,a,b,b,b,b,b,b,a,a
{
}
(0.7,0.3),
(0.3,0.7),
(0.7,0.3),
(0.7,0.3),
(0.3,0.7),
(0.3,0.7)
{
}
cluster
centroids
{ (0.7,0.3),
(0.3,0.7) }
Fraction of a's
Fractionofb's
(0.7,0.3)
(0.3,0.7)](https://image.slidesharecdn.com/edoc18-181018121149/85/Real-Time-Data-Mining-for-Event-Streams-119-320.jpg)
Real-Time Data Mining for Event Streams
- 1. DATA MINING event streams for Massiva Roudjane, Djamal Rebaïne Raphaël Khoury, Sylvain Hallé Real-Time Oct. 18th, 2018
- 7. Z Z Z expected normal rou�ne nominal "ok" surprising out of the ordinary strange different "not ok"
- 9. $$$$$ $$$$$ $$$$$ x=104 y=12 x=232 y=21 119.5 s 4 1-2 1955-11-12 APPL MSFT =123.34 =208.56 GOGL AMZN =314.16 =271.82 10432.3 Src Dst =1.2.3.4:403 =5.6.7.8:221 Many elements of so�ware systems can be modelled as pieces of data called events.
- 10. A stream (or trace) is a sequence of events. The rate at which events are produced is called the throughput. A stored copy of a stream is called a log. . . .
- 11. ” ∑ Event streams can be processed in various ways. Some examples: Aggrega�on Pa�ern detec�on Visualiza�on What is the average price of MSFT over 5 days? Display bandwidth usage for the last 24 hours. Does ever collide with ? ” ” “ “ “
- 12. We are interested in two special kinds of computa�on over streams. Compu�ng trends over a stream Finding out if a stream deviates from a given trend
- 13. We are interested in two special kinds of computa�on over streams. Compu�ng trends over a stream Finding out if a stream deviates from a given trend data mining monitoring
- 14. Event stream processing library f Σ f n { n n Apply Cumulate Trim ForkDecimate Group WindowSliceFilter <? + =? − ÷ × < Booleans 2 3 4 π Numbers abc Strings <a><a><a> XML 3 8 a 3 8 a 2 6 c Tuples + ⊇? Functions Sets Plots EVENTS FUNCTIONS PROCESSORS
- 15. Event stream processing library COMPOSITION
- 16. Event stream processing library Event Stream Processing with Sylvain Hallé Log crunching and analysis made easy 300+ pages Over 120 code examples Color illustra�ons Open access (i.e. free) To appear on Presses de l'Université du Québec (Winter 2019) bit.ly/beepbeep-book
- 17. Observa�on: many scenarios correspond to the same generic workflow configured differently
- 18. <?d { β d δP n
- 20. <?d { β d δP n trend processor Computes a "trend" over a finite sequence of events
- 21. <?d { β d δP n sliding window width The trend is computed over the last n events.
- 22. <?d { β d δP n reference trend Used as a basis for comparison with the current stream
- 23. <?d { β d δP n distance metric Computes the "distance" between the computed trend and the reference
- 24. <?d { β d δP n distance threshold Maximum acceptable distance, according to the selected metric
- 25. <?d { β d δP n comparison func�on Checks if distance is above threshold
- 26. <?d { β d δP n output A stream of Booleans. ⊤ is emi�ed whenever the input stream deviates "too much" from P.
- 27. <?d { β d δP n This pa�ern can be encapsulated into a generic group processor taking 6 parameters. Various computa�ons can be achieved by giving different values to these parameters.
- 28. β i f #
- 29. β i f # Calculates the number of dis�nct symbols in the input stream iden�ty func�on map size map symbol → # occurrences
- 30. β i f # n 4 3 P δ - 1 d ≤
- 31. <?d { 3 4 i f # 1 - ≤ Alerts whenever more than 3 dis�nct symbols were seen in the last 4 events
- 32. Group counter = new Group(1, 1); { SlicerMap slicer = new SlicerMap( new IdentityFunction(1), new Passthrough(1)); counter.associateInput(INPUT, slicer, INPUT); ApplyFunction size = new ApplyFunction(Maps.Size); connect(slicer, size); counter.associateOutput(OUTPUT, size, OUTPUT); counter.addProcessors(slicer, size); } TrendDistance<> alarm = new TrendDistance<HashMap,Number,Number>( 3, 4, slicer, Numbers.Subtraction, 1, Numbers.IsLessThan); 8 lines of code
- 34. β f Σ 0 + Σ 0 + ÷ 1 Calculates the running average of the input stream... ...i.e. the average of all values seen so far
- 35. β f Σ 0 + Σ 0 + ÷ 1 n 3 6 P δ ½ d ≤ | | 21 −
- 36. β f Σ 0 + Σ 0 + ÷ 1 n 3 6 P δ ½ d ≤ | | 21 − 21ABS( − ) Manha�an Distance of dimension 1
- 37. <?d { 6 3 ½ ≤ Alerts whenever the running average of the last 3 events deviates by more than ½ from 6 f Σ 0 + Σ 0 + ÷ 1 | | 21 −
- 38. <?d { 6 3 ½ ≤ Alerts whenever the running average of the last 3 events deviates by more than ½ from 6 f Σ 0 + Σ 0 + ÷ 1 | | 21 − 15 lines of code
- 39. β i Σ 0 + 1
- 40. β i Σ 0 + 1 Calculates the number of occurrences of each symbol in the input stream non-normalized probability density func�on a b c 3 1 5
- 42. n 9 P δ 2 d ≤ a b c 6 1 2 Reference pa�ern is a distribu�on (Manha�an) map distance func�on a b c 6 1 2 a b c 3 4 1 = |6−3|+|1−4|+|2−1| = 7
- 44. <?d { β d δP n So far, the reference trend has been a single object.
- 45. What if there could be mul�ple possible trends? PP mul�modal trend <?d { β d δP n So far, the reference trend has been a single object.
- 46. Given a finite set of points P = {p1,p2,...}, define func�on: Δ min p'∈P Δ(p,p')(p) = It is the distance between p and the closest point in P. Δ can be any distance metric: Euclidean distance, Manha�an distance, etc. p1 p2 p3
- 47. β f Σ 0 + Σ 0 + ÷ 1 n 3 6 P δ ½ d ≤ | | 21 − "Alerts whenever the running average of the last 3 events deviates by more than ½ from 6"
- 48. β f Σ 0 + Σ 0 + ÷ 1 n 3 6 P δ ½ d ≤ | | 21 − {6,9} P δ | | 21 − "Alerts whenever the running average of the last 3 events deviates by more than ½ from 6 and 9"
- 49. 3 4 5 6 7 8 9 10 11 ...... In this case, the func�on , along with and , defines two 1-dimensional "balls" of radius ½. The alarm is triggered when the running average does not lie within one of these balls. P d
- 50. 3 4 5 6 7 8 9 10 11 ...... In this case, the func�on , along with and , defines two 1-dimensional "balls" of radius ½. The alarm is triggered when the running average does not lie within one of these balls. P d 6.1, 6.0, 6.2 6.1 x ✓
- 51. 3 4 5 6 7 8 9 10 11 ...... In this case, the func�on , along with and , defines two 1-dimensional "balls" of radius ½. The alarm is triggered when the running average does not lie within one of these balls. P d 6.1, 6.0, 6.2 6.1 x ✓ 7.3, 8.9, 7.5 7.9 x ✗
- 52. 3 4 5 6 7 8 9 10 11 ...... In this case, the func�on , along with and , defines two 1-dimensional "balls" of radius ½. The alarm is triggered when the running average does not lie within one of these balls. P d 6.1, 6.0, 6.2 6.1 x ✓ 7.3, 8.9, 7.5 7.9 x ✗ Mul�-modal trends can also be mul�-dimensional!
- 53. Consider streams of symbols a and b. i Σ 0 + 1 f DP Σ=1 [ ] 1
- 54. Consider streams of symbols a and b. i Σ 0 + 1 f DP Σ=1 [ ] 1 computes a map of the number of occurrences of a and b extracts the map's values normalizes the vector casts into DoublePoint
- 55. Consider streams of symbols a and b. i Σ 0 + 1 f DP Σ=1 [ ] 1 The output is a two-dimensional point (or vector) ( 0.33, 0.66 ) frac�on of a's frac�on of b's
- 56. Consider streams of symbols a and b. Suppose that the observed streams fall in two categories: a,a,a,a,b,a,b,a,b,a a,a,a,a,a,b,b,a,a,b b,b,a,a,a,b,a,a,a,a Roughly 70% of a, 30% of b a,b,a,b,b,b,b,b,b,a b,b,b,b,a,a,b,b,a,b b,a,b,b,b,b,b,b,a,a Roughly 30% of a, 70% of b ( 0.7, 0.3 ) ( 0.3, 0.7 ) 1 2
- 57. Streams can be seen as two-dimensional points: Fraction of a's Fractionofb's 0% 100% 100% 1 2 (0.7,0.3) (0.3,0.7)
- 58. β i Σ 0 + 1 f DP Σ=1 [ ] 1 n P δ .15 d ≤9 ΔE (0.3,0.7) (0.7,0.3) { }
- 59. <?d { 9 .15 ≤ ΔE (0.3,0.7) (0.7,0.3) { } i Σ 0 + 1 f DP Σ=1 [ ] 1 Alerts whenever the distribu�on of the last 9 events is further than .15 from either category
- 61. Fraction of a's Fractionofb's a,a,b,a,a,b,a,b,a ( 0.67, 0.33 )p= p
- 62. Fraction of a's Fractionofb's a,a,b,a,a,b,a,b,a ( 0.67, 0.33 )p= p d=0.047✓
- 63. Fraction of a's Fractionofb's b,a,b,a,b,a,b,a,b ( 0.44, 0.56 )p= p
- 64. Fraction of a's Fractionofb's b,a,b,a,b,a,b,a,b ( 0.44, 0.56 )p= p d=0.2 This stream is "too different" from either category or !1 2 ✗
- 65. object (e.g. stream) feature extrac�on feature vector (n dimensions) ? ⟨ 3.8, 0.5, 1.1 ⟩F A recurring process in data mining: Examples of feature vectors: Distribu�on of symbols Sta�s�cal moments Any other numerical computa�on
- 66. <?d{ β d δP n But where does this reference trend come from? ?
- 67. <?d{ β d δ n Op�on #1: it is computed from the stream itself. {m β
- 68. <?d{ β d δ n Op�on #1: it is computed from the stream itself. {m β fork The stream is split in two
- 69. <?d{ β d δ n Op�on #1: it is computed from the stream itself. {m β reference trend The reference is computed over a window of width m
- 70. <?d{ β d δ n Op�on #1: it is computed from the stream itself. {m β stream offset Second stream copy is trimmed of its first m events
- 71. <?d{ β d δ n Op�on #1: it is computed from the stream itself. {m β The rest of the pipe works as before
- 72. m n Trend from "the present" Trend from "the past" vs. β β An alarm is triggered when the stream's current trend becomes "too different" from what it was in the past ⇒ self-correla�on
- 73. { <?d { β d δ nm This pa�ern can be encapsulated into a generic group processor taking 6 parameters.
- 74. { <?d { β d δ nm This pa�ern can be encapsulated into a generic group processor taking 6 parameters. Self-Correlated Trend Distance (SCTD)
- 76. { <?d { 6 ≤| | 21 − 6 3 1 f Σ 0 + Σ 0 + ÷ 1 Alerts whenever the running average of the last 3 events deviates by more than 1 from the running average of the 6 before
- 77. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream
- 78. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3
- 79. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3
- 80. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3
- 81. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3
- 82. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t
- 83. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t
- 84. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t
- 85. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t
- 86. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t Manhattan distance between averages
- 87. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream Average between t−8 and t−3 Average between t−2 and t Manhattan distance between averages threshold exceeded
- 88. 0 2 4 6 8 10 12 14 16 18 0 1 2 3 4 5 6 t Input stream The average in this window ("the past") the average in this window ("the present") is too far from
- 89. + object feature extrac�on feature vector (n dimensions) ? ⟨ 3.8, 0.5, 1.1 ⟩F ⟨ 3.8, 0.5, 1.1 ⟩ ⟨ 6.5, 0.2, 1.1 ⟩ ⟨ 5.0, 0.1, 1.6 ⟩ ⟨ 4.4, 0.5, 0.9 ⟩ . . . set of feature vectors C clustering algorithm + + + + + + + + ++ + + + + clusters cluster centroid cluster radius
- 90. <?d{ β d δP n But where does this reference trend come from? ?
- 91. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams.
- 92. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams. Unpacks a set of streams and feeds them one by one unpacking
- 93. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams. Feed each event of a stream to a processor and collect its last output dropping
- 94. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams. Compute a trend over a stream (same as before) trend
- 95. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams. Collate trend objects from all streams into a set pack
- 96. β α }{ α Op�on #2: it is computed ahead of �me from a set of reference streams. Compute a global trend object from all individual trends aggregate
- 97. β α }{ α This process can be encapsulated into a generic group processor with 2 parameters.
- 99. β f Σ 0 + Σ 0 + ÷ 1 α f Σ 0 + Σ 0 + ÷ 1 running average of a stream average of values in a set
- 102. }{ f Σ 0 + Σ 0 + ÷ 1 f Σ 0 + Σ 0 + ÷ 1 {⟨1,2,1,1⟩, ⟨2,2,1,2⟩, ⟨3,1,1,1⟩} a set of 3 streams 1,2,1,1 1¼ {1¼}
- 103. }{ f Σ 0 + Σ 0 + ÷ 1 f Σ 0 + Σ 0 + ÷ 1 {⟨1,2,1,1⟩, ⟨2,2,1,2⟩, ⟨3,1,1,1⟩} a set of 3 streams 2,2,1,2 1¾ {1¼,1¾}
- 104. }{ f Σ 0 + Σ 0 + ÷ 1 f Σ 0 + Σ 0 + ÷ 1 {⟨1,2,1,1⟩, ⟨2,2,1,2⟩, ⟨3,1,1,1⟩} a set of 3 streams 3,1,1,1 1½ {1¼,1¾,1½}
- 105. }{ f Σ 0 + Σ 0 + ÷ 1 f Σ 0 + Σ 0 + ÷ 1 {⟨1,2,1,1⟩, ⟨2,2,1,2⟩, ⟨3,1,1,1⟩} a set of 3 streams 1½ {1¼,1¾,1½} the average of all running averages
- 106. + object feature extrac�on feature vector (n dimensions) ? ⟨ 3.8, 0.5, 1.1 ⟩F ⟨ 3.8, 0.5, 1.1 ⟩ ⟨ 6.5, 0.2, 1.1 ⟩ ⟨ 5.0, 0.1, 1.6 ⟩ ⟨ 4.4, 0.5, 0.9 ⟩ . . . set of feature vectors C clustering algorithm + + + + + + + + ++ + + + + clusters cluster centroid cluster radius
- 107. C The ML pale�e uses Apache Commons Mαth][which supports the following clustering algorithms: , K-Means++ C Fuzzy-K-Means C DBSCAN C Mul�-K-Means++ h�ps://commons.apache.org/proper/commons-math/userguide/ml.html
- 108. β α }{ α
- 109. F C }{
- 110. β f Σ 0 + Σ 0 + ÷ 1 α f 2 running average of a stream K-means func�on (with K=2)
- 112. }{ f Σ 0 + Σ 0 + ÷ 1 f 2 {⟨5,6,5,6,7,6⟩, ⟨8,9,8,10,9⟩, ⟨6,6,7,6,6,5⟩, ⟨7,6,6,7,6,6,5⟩, ⟨9,8,10,9⟩} a set of streams
- 113. }{ f Σ 0 + Σ 0 + ÷ 1 f 2 {⟨5,6,5,6,7,6⟩, ⟨8,9,8,10,9⟩, ⟨6,6,7,6,6,5⟩, ⟨7,6,6,7,6,6,5⟩, ⟨9,8,10,9⟩} a set of streams {5.83, 8.8, 6.0, 6.14, 9} set of running averages
- 114. }{ f Σ 0 + Σ 0 + ÷ 1 f 2 {⟨5,6,5,6,7,6⟩, ⟨8,9,8,10,9⟩, ⟨6,6,7,6,6,5⟩, ⟨7,6,6,7,6,6,5⟩, ⟨9,8,10,9⟩} a set of streams {5.83, 8.8, 6.0, 6.14, 9} set of running averages {5.99, 8.9} cluster centroids 5 6 7 8 9 10 11 ......
- 115. β α f 2 distribu�on of symbols K-means func�on (with K=2) i Σ 0 + 1 f DP Σ=1 [ ] 1
- 119. }{ f 2 i Σ 0 + 1 f DP Σ=1 [ ] 1 a,a,a,a,b,a,b,a,b,a a,a,a,a,a,b,b,a,a,b b,b,a,a,a,b,a,a,a,a a,b,a,b,b,b,b,b,b,a b,b,b,b,a,a,b,b,a,b b,a,b,b,b,b,b,b,a,a { } (0.7,0.3), (0.3,0.7), (0.7,0.3), (0.7,0.3), (0.3,0.7), (0.3,0.7) { } cluster centroids { (0.7,0.3), (0.3,0.7) } Fraction of a's Fractionofb's (0.7,0.3) (0.3,0.7)
- 120. BeepBeep provides mul�ple func�onali�es for performing data mining on event streams... distance metrics | | 21 − ΔE processor pipes i Σ 0 + 1 f size Σ 0 + 1 Σ 0 + f t src f ∪ + f 2 clustering algorithms ...and easy means of crea�ng custom objects. k x y event types and manipula�on func�ons