SlideShare a Scribd company logo

Fluentd and Embulk Game Server 4

N Masahiro
N Masahiro

This document summarizes Masahiro Nakagawa's presentation on Fluentd and Embulk. Fluentd is a data collector for unified logging that allows for streaming data transfer based on JSON. It is written in Ruby and uses plugins to collect, process, and output data. Embulk is a bulk loading tool that allows high performance parallel processing of data to load it into various databases and storage systems. Both tools use a pluggable architecture to provide flexibility in handling different data sources and targets.

Technology
1 of 69
Downloaded 36 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
Masahiro Nakagawa Apr 18, 2015 Game Server meetup #4 Fluentd / Embulk For reliable transfer
Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > Living at OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of several meetups (Presto, DTM, etc…) > etc…
Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/
What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
Background
Data Analytics Flow Collect Store Process Visualize Data source Reporting Monitoring
Data Analytics Flow Store Process Cloudera Horton Works Treasure Data Collect Visualize Tableau Excel R easier & shorter time ???
TD Service Architecture Time to Value Send query result Result Push Acquire Analyze Store Plazma DB Flexible, Scalable, Columnar Storage Web Log App Log Censor CRM ERP RDBMS Treasure Agent(Server) SDK(JS, Android, iOS, Unity) Streaming Collector Batch / Reliability Ad-hoc /
 Low latency KPI$ KPI Dashboard BI Tools Other Products RDBMS, Google Docs, AWS S3, FTP Server, etc. Metric Insights Tableau, Motion Board etc. POS REST API ODBC / JDBC SQL, Pig Bulk Uploader Embulk,
 TD Toolbelt SQL-based query @AWS or @IDCF Connectivity Economy & Flexibility Simple & Supported
Dive into Concept
Divide & Conquer & Retry error retry error retry retry retry Batch Stream Other stream
Application ･･･ Server2 Application ･･･ Server3 Application ･･･ Server1 FluentLog Server High Latency! must wait for a day... Before…
Application ･･･ Server2 Application ･･･ Server3 Application ･･･ Server1 Fluentd Fluentd Fluentd Fluentd Fluentd In streaming! After…
Why JSON / MessagePack? (1 > Schema on Write (Traditional MPP DB) > Writing data using schema for improving
 query performance > Pros > minimum query overhead > Cons > Need to design schema and workload before > Data load is expensive operation
Why JSON / MessagePack? (2 > Schema on Read (Hadoop) > Writing data without schema and map schema at query time > Pros > Robust over schema and workload change > Data load is cheap operation > Cons > High overhead at query time
Features
Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data

Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data
 Common Concerns Use Case Specific
> default second unit > from data source Event structure(log message) ✓ Time > for message routing > where is from? ✓ Tag > JSON format > MessagePack
 internally > schema-free ✓ Record
Architecture (v0.12 or later) EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
Configuration and operation > No central / master node > @include helps configuration sharing > Operation depends on your environment > Use your deamon / deploy tools > Use Chef in Treasure Data > Apache like syntax
How to use
Setup fluentd (e.g. Ubuntu) $ apt-get install ruby! ! $ gem install fluentd! ! $ edit fluent.conf! ! $ fluentd -c fluent.conf http://docs.fluentd.org/articles/faq#w-what-version-of-ruby-does-fluentd-support
Treasure Agent (td-agent) > Treasure Data distribution of Fluentd > include ruby, popular plugins and etc > Treasure Agent 2 is current stable > Recommend to use v2, not v1 > rpm, deb and dmg > Latest version is 2.2.0 with fluentd v0.12
Setup td-agent $ curl -L http://toolbelt.treasuredata.com/ sh/install-redhat-td-agent2.sh | sh! ! $ edit /etc/td-agent/td-agent.conf! ! $ sudo service td-agent start See: http://docs.fluentd.org/categories/installation
Apache to Mongo tail insert event buffering routing 127.0.0.1 - - [11/Dec/2014:07:26:27] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:30] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:32] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:40] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:27:01] "GET / ... ... Fluentd Web Server 2014-02-04 01:33:51 apache.log { "host": "127.0.0.1", "method": "GET", ... }
Plugins - use rubygems $ fluent-gem search -rd fluent-plugin! ! $ fluent-gem search -rd fluent-mixin! ! $ fluent-gem install fluent-plugin-mongo In td-agent:
 /usr/sbin/td-agent-gem install fluent-plugin-mongo
# receive events via HTTP <source> @type http port 8888 </source> ! # read logs from a file <source> @type tail path /var/log/httpd.log format apache tag apache.access </source> ! # save access logs to MongoDB <match apache.access> @type mongo database apache collection log </match> # save alerts to a file <match alert.**> @type file path /var/log/fluent/alerts </match> ! # forward other logs to servers <match **> @type forward <server> host 192.168.0.11 weight 20 </server> <server> host 192.168.0.12 weight 60 </server> </match> ! @include http://example.com/conf
> Apply filtering routine to event stream > No more tag tricks!
 
 
 
 
 
 Filter <match access.**> @type record_reformer tag reformed.${tag} </match> ! <match reformed.**> @type growthforecast </match> <filter access.**> @type record_transformer … </filter> v0.10: v0.12: <match access.**> @type growthforecast </match>
Before
After or Embulk
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / processing / routing M x N → M + N
Roadmap > v0.10 (old stable) > v0.12 (current stable) > Filter / Label / At-least-once > v0.14 (spring - early summer, 2015) > New plugin APIs, ServerEngine, Time… > v1 (summer - fall, 2015) > Fix new features / APIs https://github.com/fluent/fluentd/wiki/V1-Roadmap
Use-cases
Simple forwarding
# logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag backend.apache </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>
# Ruby! Fluent.open(“myapp”)! Fluent.event(“login”, {“user” => 38})! #=> 2014-12-11 07:56:01 myapp.login {“user”:38} > Ruby > Java > Perl > PHP > Python > D > Scala > ... Client libraries
Less Simple Forwarding - At-most-once / At-least-once
 - HA (failover) - Load-balancing
All data Near realtime and batch combo! Hot data
# logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag web.access </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to ES and HDFS <match web.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
CEP for Stream Processing Norikra is a SQL based CEP engine: http://norikra.github.io/
Container Logging
> Kubernetes ! ! ! ! ! > Google Compute Engine > https://cloud.google.com/logging/docs/install/compute_install Fluentd on Kubernetes / GCE
Treasure Data Frontend Job Queue Worker Hadoop Presto Fluentd Applications push metrics to Fluentd
 (via local Fluentd) Datadog for realtime monitoring Treasure Data for historical analysis Fluentd sums up data minutes
 (partial aggregation)
hundreds of app servers sends event logs sends event logs sends event logs Rails app td-agent td-agent td-agent Google Spreadsheet Treasure Data MySQL Logs are available after several mins. Daily/Hourly Batch KPI visualizationFeedback rankings Rails app Rails app Unlimited scalability Flexible schema Realtime Less performance impact Cookpad ✓ Over 100 RoR servers (2012/2/4)
Slideshare http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/
Log Analysis System And its designs in LINE Corp. 2014 early
Line BusinessConnect http://developers.linecorp.com/blog/?p=3386
Eco-system
fluent-bit > Made for Embedded Linux > OpenEmbedded & Yocto Project > Intel Edison, RasPi & Beagle Black boards > https://github.com/fluent/fluent-bit > Standalone application or Library mode > Built-in plugins > input: cpu, kmsg, output: fluentd > First release at the end of Mar 2015
fluentd-forwarder > Forwarding agent written in Go > Focusing log forwarding to Fluentd > Work on Windows > Bundle TCP input/output and TD output > No flexible plugin mechanizm > We have a plan to add some input/output > Similar product > fluent-agent-lite, fluent-agent-hydra, ik
fluentd-ui > Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui
 
 
 
 
 

Bulk loading ! Parallel processing ! Pluggable architecture http://embulk.org/
The problems at Treasure Data > Treasure Data Service on the Cloud > Customers want to try Treasure Data, but > SEs write scripts to bulk load their data. Hard work :( > Customers want to migrate their big data, but > Hard work :( > Fluentd solved streaming data collection, but > bulk data loading is another problem.
Embulk > Bulk Loader version of Fluentd > Pluggable architecture > JRuby, JVM languages > High performance parallel processing > Share your script as a plugin > https://github.com/embulk
The problems of bulk load > Data cleaning (normalization) > How to normalize broken records? > Error handling > How to remove broken records? > Idempotent retrying > How to retry without duplicated loading? > Performance optimization
HDFS MySQL Amazon S3 Embulk CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓ Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behaviour ✓ Idempotent retrying Plugins Plugins bulk load http://www.embulk.org/plugins/
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4
How to use
Setup embulk (e.g. Linux/Mac) $ curl --create-dirs -o ~/.embulk/bin/embulk -L “http://dl.embulk.org/embulk-latest.jar"! ! $ chmod +x ~/.embulk/bin/embulk! ! $ echo 'export PATH="$HOME/.embulk/bin: $PATH"' >> ~/.bashrc! ! $ source ~/.bashrc
Try example $ embulk example ./try1! ! $ embulk guess ./example.yml -o config.yml! ! $ embulk preview config.yml! ! $ embulk run config.yml
# install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml Guess format & schema in: type: file path_prefix: /path/to/sample_ out:
 type: stdout in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} out:
 type: stdout guess by guess plugins
# install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary +--------------------------------------+---------------+--------------------+ | time:timestamp | uid:long | word:string | +--------------------------------------+---------------+--------------------+ | 2015-01-27 19:23:49 UTC | 32,864 | embulk | | 2015-01-27 19:01:23 UTC | 14,824 | jruby | | 2015-01-28 02:20:02 UTC | 27,559 | plugin | | 2015-01-29 11:54:36 UTC | 11,270 | fluentd | +--------------------------------------+---------------+--------------------+ Preview & fix config
# install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary ! # run $ ./embulk run config.yml -o config.yml exec: {} in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp,
 format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} last_path: /path/to/sample_001.csv.gz out:
 type: stdout Deterministic run
exec: {} in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp,
 format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} last_path: /path/to/sample_01.csv.gz out:
 type: stdout Repeat # install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary ! # run $ ./embulk run config.yml -o config.yml ! # repeat $ ./embulk run config.yml -o config.yml $ ./embulk run config.yml -o config.yml
Use-cases
Quipper from GDS slide
Other cases > Treasure Data > Embulk worker for automatic import > Web services > Send existing logs to Elasticsearch > Business / Batch systems > Database to Database > etc…
Check: treasuredata.com Cloud service for the entire data pipeline

More Related Content

What's hot (20)

PDF
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
Ontico
 
PDF
Recent Updates at Embulk Meetup #3
Muga Nishizawa
 
PDF
[214]유연하고 확장성 있는 빅데이터 처리
NAVER D2
 
PDF
Connection Pooling in PostgreSQL using pgbouncer
Sameer Kumar
 
PDF
Java/Spring과 Node.js의공존
동수 장
 
PDF
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
PDF
Amazed by aws 1st session
Amazon Web Services Korea
 
PDF
MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference prese...
ronwarshawsky
 
PDF
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Ontico
 
PDF
How to build a High Performance PSGI/Plack Server
Masahiro Nagano
 
PDF
Big Master Data PHP BLT #1
Masahiro Nagano
 
PDF
Spring Data MongoDB 介紹
Kuo-Chun Su
 
PDF
Troubleshooting redis
DaeMyung Kang
 
ODP
Meetup cassandra for_java_cql
zznate
 
PDF
Fluentd - Set Up Once, Collect More
Sadayuki Furuhashi
 
PPT
MongoDB Basic Concepts
MongoDB
 
PDF
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
Insight Technology, Inc.
 
PDF
Node.js
Jan Dillmann
 
PPT
Node js presentation
martincabrera
 
PPTX
JahiaOne - Performance Tuning
Jahia Solutions Group
 
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
Ontico
 
Recent Updates at Embulk Meetup #3
Muga Nishizawa
 
[214]유연하고 확장성 있는 빅데이터 처리
NAVER D2
 
Connection Pooling in PostgreSQL using pgbouncer
Sameer Kumar
 
Java/Spring과 Node.js의공존
동수 장
 
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
Amazed by aws 1st session
Amazon Web Services Korea
 
MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference prese...
ronwarshawsky
 
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Ontico
 
How to build a High Performance PSGI/Plack Server
Masahiro Nagano
 
Big Master Data PHP BLT #1
Masahiro Nagano
 
Spring Data MongoDB 介紹
Kuo-Chun Su
 
Troubleshooting redis
DaeMyung Kang
 
Meetup cassandra for_java_cql
zznate
 
Fluentd - Set Up Once, Collect More
Sadayuki Furuhashi
 
MongoDB Basic Concepts
MongoDB
 
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
Insight Technology, Inc.
 
Node.js
Jan Dillmann
 
Node js presentation
martincabrera
 
JahiaOne - Performance Tuning
Jahia Solutions Group
 

Viewers also liked (13)

PPTX
EmbulkとDigdagとデータ分析基盤と
Toru Takahashi
 
PDF
分割と整合性と戦う
Yugo Shimizu
 
PDF
サーバーのおしごと
Yugo Shimizu
 
PDF
負荷がたかいいんだから～♪（仮）
Yohei Hamada
 
PPTX
Imprementation of realtime_networkgame
Satoshi Yamafuji
 
PPTX
MMOのサーバについて 剣と魔法のログレス ～いにしえの女神～ での実装例
Satoshi Yamafuji
 
PPTX
リアルタイムサーバー 〜Erlang/OTPで作るPubSubサーバー〜
Yugo Shimizu
 
PDF
Halo2 におけるHFSM(階層型有限状態マシン) 【ビヘイビアツリー解説】
Youichiro Miyake
 
PDF
負荷対策しておもったことまとめ~JMeterでSocket.IOもいけるでよ~
johgus johgus
 
PDF
Embulk, an open-source plugin-based parallel bulk data loader
Sadayuki Furuhashi
 
PDF
自宅で出来る！ゲームサーバの作り方
光晶 上原
 
PDF
サーバー未経験者がソーシャルゲームを通して知ったサーバーの事
Manabu Koga
 
PDF
ゲームサーバ開発現場の考え方
Daisaku Mochizuki
 
EmbulkとDigdagとデータ分析基盤と
Toru Takahashi
 
分割と整合性と戦う
Yugo Shimizu
 
サーバーのおしごと
Yugo Shimizu
 
負荷がたかいいんだから～♪（仮）
Yohei Hamada
 
Imprementation of realtime_networkgame
Satoshi Yamafuji
 
MMOのサーバについて 剣と魔法のログレス ～いにしえの女神～ での実装例
Satoshi Yamafuji
 
リアルタイムサーバー 〜Erlang/OTPで作るPubSubサーバー〜
Yugo Shimizu
 
Halo2 におけるHFSM(階層型有限状態マシン) 【ビヘイビアツリー解説】
Youichiro Miyake
 
負荷対策しておもったことまとめ~JMeterでSocket.IOもいけるでよ~
johgus johgus
 
Embulk, an open-source plugin-based parallel bulk data loader
Sadayuki Furuhashi
 
自宅で出来る！ゲームサーバの作り方
光晶 上原
 
サーバー未経験者がソーシャルゲームを通して知ったサーバーの事
Manabu Koga
 
ゲームサーバ開発現場の考え方
Daisaku Mochizuki
 
Ad

Similar to Fluentd and Embulk Game Server 4 (20)

PDF
Fluentd - RubyKansai 65
N Masahiro
 
PDF
The basics of fluentd
Treasure Data, Inc.
 
PDF
Fluentd Unified Logging Layer At Fossasia
N Masahiro
 
PDF
Treasure Data and OSS
N Masahiro
 
PDF
Fluentd Project Intro at Kubecon 2019 EU
N Masahiro
 
PDF
Fluentd - road to v1 -
N Masahiro
 
PDF
Fluentd at HKOScon
N Masahiro
 
PPTX
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Data Con LA
 
PDF
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
PDF
Fluentd: Unified Logging Layer at CWT2014
N Masahiro
 
PDF
Fluentd v0.12 master guide
N Masahiro
 
PDF
Fluentd and Docker - running fluentd within a docker container
Treasure Data, Inc.
 
PDF
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
PDF
SQL for Everything at CWT2014
N Masahiro
 
PDF
fluentd -- the missing log collector
Muga Nishizawa
 
PPT
Rapid java backend and api development for mobile devices
ciklum_ods
 
PDF
Fluentd meetup #2
Treasure Data, Inc.
 
PDF
Fluentd unified logging layer
Kiyoto Tamura
 
PDF
Prestogres, ODBC & JDBC connectivity for Presto
Sadayuki Furuhashi
 
PDF
Collect distributed application logging using fluentd (EFK stack)
Marco Pas
 
Fluentd - RubyKansai 65
N Masahiro
 
The basics of fluentd
Treasure Data, Inc.
 
Fluentd Unified Logging Layer At Fossasia
N Masahiro
 
Treasure Data and OSS
N Masahiro
 
Fluentd Project Intro at Kubecon 2019 EU
N Masahiro
 
Fluentd - road to v1 -
N Masahiro
 
Fluentd at HKOScon
N Masahiro
 
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Data Con LA
 
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
Fluentd: Unified Logging Layer at CWT2014
N Masahiro
 
Fluentd v0.12 master guide
N Masahiro
 
Fluentd and Docker - running fluentd within a docker container
Treasure Data, Inc.
 
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
SQL for Everything at CWT2014
N Masahiro
 
fluentd -- the missing log collector
Muga Nishizawa
 
Rapid java backend and api development for mobile devices
ciklum_ods
 
Fluentd meetup #2
Treasure Data, Inc.
 
Fluentd unified logging layer
Kiyoto Tamura
 
Prestogres, ODBC & JDBC connectivity for Presto
Sadayuki Furuhashi
 
Collect distributed application logging using fluentd (EFK stack)
Marco Pas
 
Ad

More from N Masahiro (20)

PDF
Fluentd v1 and future at techtalk
N Masahiro
 
PDF
Fluentd and Distributed Logging at Kubecon
N Masahiro
 
PDF
Fluentd v1.0 in a nutshell
N Masahiro
 
PDF
Fluentd v1.0 in a nutshell
N Masahiro
 
PDF
Presto changes
N Masahiro
 
PDF
Fluentd v0.14 Overview
N Masahiro
 
PDF
Fluentd and Kafka
N Masahiro
 
PDF
fluent-plugin-beats at Elasticsearch meetup #14
N Masahiro
 
PDF
Dive into Fluentd plugin v0.12
N Masahiro
 
PDF
Technologies for Data Analytics Platform
N Masahiro
 
PDF
Docker and Fluentd
N Masahiro
 
PDF
How to create Treasure Data #dotsbigdata
N Masahiro
 
PDF
Treasure Data and AWS - Developers.io 2015
N Masahiro
 
PDF
Can you say the same words even in oss
N Masahiro
 
PDF
I am learing the programming
N Masahiro
 
PDF
Fluentd meetup dive into fluent plugin (outdated)
N Masahiro
 
PDF
D vs OWKN Language at LLnagoya
N Masahiro
 
PDF
Goodbye Doost
N Masahiro
 
KEY
Final presentation at pfintern
N Masahiro
 
ZIP
Kernel VM 5 LT
N Masahiro
 
Fluentd v1 and future at techtalk
N Masahiro
 
Fluentd and Distributed Logging at Kubecon
N Masahiro
 
Fluentd v1.0 in a nutshell
N Masahiro
 
Fluentd v1.0 in a nutshell
N Masahiro
 
Presto changes
N Masahiro
 
Fluentd v0.14 Overview
N Masahiro
 
Fluentd and Kafka
N Masahiro
 
fluent-plugin-beats at Elasticsearch meetup #14
N Masahiro
 
Dive into Fluentd plugin v0.12
N Masahiro
 
Technologies for Data Analytics Platform
N Masahiro
 
Docker and Fluentd
N Masahiro
 
How to create Treasure Data #dotsbigdata
N Masahiro
 
Treasure Data and AWS - Developers.io 2015
N Masahiro
 
Can you say the same words even in oss
N Masahiro
 
I am learing the programming
N Masahiro
 
Fluentd meetup dive into fluent plugin (outdated)
N Masahiro
 
D vs OWKN Language at LLnagoya
N Masahiro
 
Goodbye Doost
N Masahiro
 
Final presentation at pfintern
N Masahiro
 
Kernel VM 5 LT
N Masahiro
 

Recently uploaded (20)

PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 

Fluentd and Embulk Game Server 4

  • 1. Masahiro Nakagawa Apr 18, 2015 Game Server meetup #4 Fluentd / Embulk For reliable transfer
  • 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > Living at OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of several meetups (Presto, DTM, etc…) > etc…
  • 3. Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/
  • 4. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
  • 6. Data Analytics Flow Collect Store Process Visualize Data source Reporting Monitoring
  • 7. Data Analytics Flow Store Process Cloudera Horton Works Treasure Data Collect Visualize Tableau Excel R easier & shorter time ???
  • 8. TD Service Architecture Time to Value Send query result Result Push Acquire Analyze Store Plazma DB Flexible, Scalable, Columnar Storage Web Log App Log Censor CRM ERP RDBMS Treasure Agent(Server) SDK(JS, Android, iOS, Unity) Streaming Collector Batch / Reliability Ad-hoc /
 Low latency KPI$ KPI Dashboard BI Tools Other Products RDBMS, Google Docs, AWS S3, FTP Server, etc. Metric Insights Tableau, Motion Board etc. POS REST API ODBC / JDBC SQL, Pig Bulk Uploader Embulk,
 TD Toolbelt SQL-based query @AWS or @IDCF Connectivity Economy & Flexibility Simple & Supported
  • 10. Divide & Conquer & Retry error retry error retry retry retry Batch Stream Other stream
  • 13. Why JSON / MessagePack? (1 > Schema on Write (Traditional MPP DB) > Writing data using schema for improving
 query performance > Pros > minimum query overhead > Cons > Need to design schema and workload before > Data load is expensive operation
  • 14. Why JSON / MessagePack? (2 > Schema on Read (Hadoop) > Writing data without schema and map schema at query time > Pros > Robust over schema and workload change > Data load is cheap operation > Cons > High overhead at query time
  • 16. Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data

  • 17. Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data
 Common Concerns Use Case Specific
  • 18. > default second unit > from data source Event structure(log message) ✓ Time > for message routing > where is from? ✓ Tag > JSON format > MessagePack
 internally > schema-free ✓ Record
  • 19. Architecture (v0.12 or later) EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
  • 20. Configuration and operation > No central / master node > @include helps configuration sharing > Operation depends on your environment > Use your deamon / deploy tools > Use Chef in Treasure Data > Apache like syntax
  • 22. Setup fluentd (e.g. Ubuntu) $ apt-get install ruby! ! $ gem install fluentd! ! $ edit fluent.conf! ! $ fluentd -c fluent.conf http://docs.fluentd.org/articles/faq#w-what-version-of-ruby-does-fluentd-support
  • 23. Treasure Agent (td-agent) > Treasure Data distribution of Fluentd > include ruby, popular plugins and etc > Treasure Agent 2 is current stable > Recommend to use v2, not v1 > rpm, deb and dmg > Latest version is 2.2.0 with fluentd v0.12
  • 24. Setup td-agent $ curl -L http://toolbelt.treasuredata.com/ sh/install-redhat-td-agent2.sh | sh! ! $ edit /etc/td-agent/td-agent.conf! ! $ sudo service td-agent start See: http://docs.fluentd.org/categories/installation
  • 25. Apache to Mongo tail insert event buffering routing 127.0.0.1 - - [11/Dec/2014:07:26:27] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:30] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:32] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:26:40] "GET / ... 127.0.0.1 - - [11/Dec/2014:07:27:01] "GET / ... ... Fluentd Web Server 2014-02-04 01:33:51 apache.log { "host": "127.0.0.1", "method": "GET", ... }
  • 26. Plugins - use rubygems $ fluent-gem search -rd fluent-plugin! ! $ fluent-gem search -rd fluent-mixin! ! $ fluent-gem install fluent-plugin-mongo In td-agent:
 /usr/sbin/td-agent-gem install fluent-plugin-mongo
  • 27. # receive events via HTTP <source> @type http port 8888 </source> ! # read logs from a file <source> @type tail path /var/log/httpd.log format apache tag apache.access </source> ! # save access logs to MongoDB <match apache.access> @type mongo database apache collection log </match> # save alerts to a file <match alert.**> @type file path /var/log/fluent/alerts </match> ! # forward other logs to servers <match **> @type forward <server> host 192.168.0.11 weight 20 </server> <server> host 192.168.0.12 weight 60 </server> </match> ! @include http://example.com/conf
  • 28. > Apply filtering routine to event stream > No more tag tricks!
 
 
 
 
 
 Filter <match access.**> @type record_reformer tag reformed.${tag} </match> ! <match reformed.**> @type growthforecast </match> <filter access.**> @type record_transformer … </filter> v0.10: v0.12: <match access.**> @type growthforecast </match>
  • 31. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / processing / routing M x N → M + N
  • 32. Roadmap > v0.10 (old stable) > v0.12 (current stable) > Filter / Label / At-least-once > v0.14 (spring - early summer, 2015) > New plugin APIs, ServerEngine, Time… > v1 (summer - fall, 2015) > Fix new features / APIs https://github.com/fluent/fluentd/wiki/V1-Roadmap
  • 35. # logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag backend.apache </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>
  • 36. # Ruby! Fluent.open(“myapp”)! Fluent.event(“login”, {“user” => 38})! #=> 2014-12-11 07:56:01 myapp.login {“user”:38} > Ruby > Java > Perl > PHP > Python > D > Scala > ... Client libraries
  • 37. Less Simple Forwarding - At-most-once / At-least-once
 - HA (failover) - Load-balancing
  • 38. All data Near realtime and batch combo! Hot data
  • 39. # logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag web.access </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to ES and HDFS <match web.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
  • 40. CEP for Stream Processing Norikra is a SQL based CEP engine: http://norikra.github.io/
  • 42. > Kubernetes ! ! ! ! ! > Google Compute Engine > https://cloud.google.com/logging/docs/install/compute_install Fluentd on Kubernetes / GCE
  • 43. Treasure Data Frontend Job Queue Worker Hadoop Presto Fluentd Applications push metrics to Fluentd
 (via local Fluentd) Datadog for realtime monitoring Treasure Data for historical analysis Fluentd sums up data minutes
 (partial aggregation)
  • 44. hundreds of app servers sends event logs sends event logs sends event logs Rails app td-agent td-agent td-agent Google Spreadsheet Treasure Data MySQL Logs are available after several mins. Daily/Hourly Batch KPI visualizationFeedback rankings Rails app Rails app Unlimited scalability Flexible schema Realtime Less performance impact Cookpad ✓ Over 100 RoR servers (2012/2/4)
  • 46. Log Analysis System And its designs in LINE Corp. 2014 early
  • 49. fluent-bit > Made for Embedded Linux > OpenEmbedded & Yocto Project > Intel Edison, RasPi & Beagle Black boards > https://github.com/fluent/fluent-bit > Standalone application or Library mode > Built-in plugins > input: cpu, kmsg, output: fluentd > First release at the end of Mar 2015
  • 50. fluentd-forwarder > Forwarding agent written in Go > Focusing log forwarding to Fluentd > Work on Windows > Bundle TCP input/output and TD output > No flexible plugin mechanizm > We have a plan to add some input/output > Similar product > fluent-agent-lite, fluent-agent-hydra, ik
  • 51. fluentd-ui > Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui
 
 
 
 
 

  • 52. Bulk loading ! Parallel processing ! Pluggable architecture http://embulk.org/
  • 53. The problems at Treasure Data > Treasure Data Service on the Cloud > Customers want to try Treasure Data, but > SEs write scripts to bulk load their data. Hard work :( > Customers want to migrate their big data, but > Hard work :( > Fluentd solved streaming data collection, but > bulk data loading is another problem.
  • 54. Embulk > Bulk Loader version of Fluentd > Pluggable architecture > JRuby, JVM languages > High performance parallel processing > Share your script as a plugin > https://github.com/embulk
  • 55. The problems of bulk load > Data cleaning (normalization) > How to normalize broken records? > Error handling > How to remove broken records? > Idempotent retrying > How to retry without duplicated loading? > Performance optimization
  • 56. HDFS MySQL Amazon S3 Embulk CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓ Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behaviour ✓ Idempotent retrying Plugins Plugins bulk load http://www.embulk.org/plugins/
  • 60. Setup embulk (e.g. Linux/Mac) $ curl --create-dirs -o ~/.embulk/bin/embulk -L “http://dl.embulk.org/embulk-latest.jar"! ! $ chmod +x ~/.embulk/bin/embulk! ! $ echo 'export PATH="$HOME/.embulk/bin: $PATH"' >> ~/.bashrc! ! $ source ~/.bashrc
  • 61. Try example $ embulk example ./try1! ! $ embulk guess ./example.yml -o config.yml! ! $ embulk preview config.yml! ! $ embulk run config.yml
  • 62. # install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml Guess format & schema in: type: file path_prefix: /path/to/sample_ out:
 type: stdout in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} out:
 type: stdout guess by guess plugins
  • 63. # install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary +--------------------------------------+---------------+--------------------+ | time:timestamp | uid:long | word:string | +--------------------------------------+---------------+--------------------+ | 2015-01-27 19:23:49 UTC | 32,864 | embulk | | 2015-01-27 19:01:23 UTC | 14,824 | jruby | | 2015-01-28 02:20:02 UTC | 27,559 | plugin | | 2015-01-29 11:54:36 UTC | 11,270 | fluentd | +--------------------------------------+---------------+--------------------+ Preview & fix config
  • 64. # install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary ! # run $ ./embulk run config.yml -o config.yml exec: {} in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp,
 format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} last_path: /path/to/sample_001.csv.gz out:
 type: stdout Deterministic run
  • 65. exec: {} in: type: file path_prefix: /path/to/sample_ decoders: - {type: gzip} parser: charset: UTF-8 newline: CRLF type: csv delimiter: ',' quote: '"' skip_header_lines: 1 columns: - {name: id, type: long} - {name: account, type: long} - {name: time, type: timestamp,
 format: '%Y-%m-%d %H:%M:%S’} - {name: purchase, type: timestamp, format: ‘%Y%m%d'} - {name: comment, type: string} last_path: /path/to/sample_01.csv.gz out:
 type: stdout Repeat # install $ wget http://dl.embulk.org/embulk-latest.jar -O embulk.jar $ chmod 755 embulk.jar
 ! # guess $ vi example.yml $ ./embulk guess example.yml
 -o config.yml
 ! # preview $ ./embulk preview config.yml $ vi config.yml # if necessary ! # run $ ./embulk run config.yml -o config.yml ! # repeat $ ./embulk run config.yml -o config.yml $ ./embulk run config.yml -o config.yml
  • 68. Other cases > Treasure Data > Embulk worker for automatic import > Web services > Send existing logs to Elasticsearch > Business / Batch systems > Database to Database > etc…
  • 69. Check: treasuredata.com Cloud service for the entire data pipeline