fluent-plugin-norikra #fluentdcasual
14 likes•63,850 views
Talks about fluent-plugin-norikra in Fluentd Casual Talks #3
![Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Shibuya”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age
13年12月13日金曜日](https://image.slidesharecdn.com/fluentdcasual03norikra-131213043342-phpapp01/85/fluent-plugin-norikra-fluentdcasual-15-320.jpg)
![Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Shibuya”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY user.age
13年12月13日金曜日](https://image.slidesharecdn.com/fluentdcasual03norikra-131213043342-phpapp01/85/fluent-plugin-norikra-fluentdcasual-16-320.jpg)
![Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Shibuya”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Kyoto” AND attend.$0 AND attend.$1
GROUP BY user.age
13年12月13日金曜日](https://image.slidesharecdn.com/fluentdcasual03norikra-131213043342-phpapp01/85/fluent-plugin-norikra-fluentdcasual-17-320.jpg)
![After: Norikra
EACH SERVICES!
SELECT
percentiles(api_response_time, [50,90,95,98,99]) AS p
FROM target_service.win:time_batch(1 min)
WITHOUT ANY RESTARTS!
13年12月13日金曜日](https://image.slidesharecdn.com/fluentdcasual03norikra-131213043342-phpapp01/85/fluent-plugin-norikra-fluentdcasual-19-320.jpg)
![in/out_norikra and out_norikra_filter
out_norikra_filter
For Casual Use Cases / Fluentd Centric
Automated features
Server autostart
Query registration/fetches
in/out_norikra
For Norikra Users
[NOT IMPLEMENTED: Distributed Norikra Support ]
13年12月13日金曜日](https://image.slidesharecdn.com/fluentdcasual03norikra-131213043342-phpapp01/85/fluent-plugin-norikra-fluentdcasual-27-320.jpg)
fluent-plugin-norikra #fluentdcasual
- 1. fluent-plugin-norikra Fluentd Casual Talks #3 2013/12/13(Fri) @tagomoris 13年12月13日金曜日
- 2. TAGOMORI Satoshi (@tagomoris) LINE Corp. Hadoop, Fluentd, Norikra, ... 13年12月13日金曜日
- 3. 13年12月13日金曜日
- 4. Norikra: Schema-less Stream Processing with SQL 13年12月13日金曜日
- 5. Norikra(1): Schema-less event stream: Add/Remove data fields whenever you want SQL: No more restarts to add/remove queries w/ JOINs, w/ SubQueries w/ UDF Truly Complex events: Nested Hash/Array, accessible directly from SQL 13年12月13日金曜日
- 6. Norikra(2): Open source software: Licensed under GPLv2 Based on Esper UDF plugins from rubygems.org Ultra-fast bootstrap & small start: 3mins to install/start 1 server 13年12月13日金曜日
- 7. Norikra Queries: (1) SELECT name, age FROM events 13年12月13日金曜日
- 8. Norikra Queries: (1) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Shibuya”} SELECT name, age FROM events {“name”:”tagomoris”,”age”:34} 13年12月13日金曜日
- 9. Norikra Queries: (1) {“name”:”tagomoris”, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Shibuya”} SELECT name, age FROM events nothing 13年12月13日金曜日
- 10. Norikra Queries: (2) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Shibuya”} SELECT name, age FROM events WHERE current=”Shibuya” {“name”:”tagomoris”,”age”:34} 13年12月13日金曜日
- 11. Norikra Queries: (2) {“name”:”kiyoto”, “age”:99, “address”:”CA”, “corp”:”TD”, “current”:”MV”} SELECT name, age FROM events WHERE current=”Shibuya” nothing 13年12月13日金曜日
- 12. Norikra Queries: (3) SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age 13年12月13日金曜日
- 13. Norikra Queries: (3) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Shibuya”} SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age every 5 mins {”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ... 13年12月13日金曜日
- 14. Norikra Queries: (4) {“name”:”tagomoris”, “age”:34, “address”:”Tokyo”, “corp”:”LINE”, “current”:”Shibuya”} SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age SELECT max(age) as max FROM events.win:time_batch(5 mins) every 5 mins {”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ... {“max”:51} 13年12月13日金曜日
- 15. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Shibuya”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY age 13年12月13日金曜日
- 16. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Shibuya”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) GROUP BY user.age 13年12月13日金曜日
- 17. Norikra Queries: (5) {“name”:”tagomoris”, “user:{“age”:34, “corp”:”LINE”, “address”:”Tokyo”}, “current”:”Shibuya”, “speaker”:true, “attend”:[true,true,false, ...] } SELECT user.age, COUNT(*) as cnt FROM events.win:time_batch(5 mins) WHERE current=”Kyoto” AND attend.$0 AND attend.$1 GROUP BY user.age 13年12月13日金曜日
- 18. Before: Fluentd EACH SERVICES <match for.target.service.application.logs> type numeric_monitor unit minute tag service.response output_key_prefix request_api aggregate all monitor_key api_response_time percentiles 50,90,95,98,99 </match> ... AND RESTART OF FLUENTD!!!!!!!!!!!!!! 13年12月13日金曜日
- 19. After: Norikra EACH SERVICES! SELECT percentiles(api_response_time, [50,90,95,98,99]) AS p FROM target_service.win:time_batch(1 min) WITHOUT ANY RESTARTS! 13年12月13日金曜日
- 20. Before: Fluentd <match for.target.service.access.logs.**> type forest subtype datacounter remove_prefix httpstatus.count unit minute output_per_tag yes aggregate tag output_messages yes count_key status pattern1 success ^2dd tag_prefix datacount.httpstatus </match> ... AND RESTART OF FLUENTD!!!!!!!!!!!!!! 13年12月13日金曜日
- 21. After: Norikra SELECT service, count(IF(status / 200 = 2, 1, NULL)) AS success FROM all_logs.win:time_batch(1 min) GROUP BY service WITHOUT ANY RESTARTS! 13年12月13日金曜日
- 27. in/out_norikra and out_norikra_filter out_norikra_filter For Casual Use Cases / Fluentd Centric Automated features Server autostart Query registration/fetches in/out_norikra For Norikra Users [NOT IMPLEMENTED: Distributed Norikra Support ] 13年12月13日金曜日
- 28. Please Try and Enjoy Norikra! 13年12月13日金曜日