[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫

The Trial and Error in
Releasing GREE Chat
Shun Ozaki, Takayuki Hasegawa
Copyright © GREE, Inc. All Rights Reserved.
Scala Matsuri2014 B-6
GREE's First Scala Product

Self Introduction
• Shun Ozaki
• @wozaki
• Joined in April, 2013
• Android Application
• Takayuki Hasegawa
• @hase1031
• Joined in April, 2013
• NLP, Machine Learning
1/56

GREE Chat
• Released at June 2
2/56

How to build our system for hundreds of
thousands daily users
Agenda
3/56

Goal
Share our knowledge we got through
the development of GREE Chat
• Why Scala?
• Team development
• Decision of frameworks
• Obstacles & workarounds
4/56

Outline
• Reason of Selecting Scala
• Learning Scala in a Team
• Architecture of GREE Chat
• Obstacles
• Summary

• Reason of Selecting Scala
• Learning Scala in a Team
• Architecture of GREE Chat
• Obstacles
• Summary
Outline

Decide the Language
based on the Requirement

Requirements
• Hundreds of thousands daily users
• Real-time response
 Connect with streaming
• Run on a small number of servers
 Utilize server resource effectively
• Maintain for over 5 years
8/56

◎ GREE uses PHP so heavily
• Many libraries and know-how in GREE
△ Streaming
• #connections = #processes
△ Single thread, multi process
• Overhead of spawning an OS process
△ Maintainability
• Dynamic typing
9/56

◎Compatible with concurrent programming
◎One process, many connections
◎High maintainability
• Static typing
• Functional programming (no side effects)
◎Stimulate new technology learning in GREE
• Open to new languages & technologies!
Scala!
10/56

After adopting Scala …

Only two Scala programmers in a team of seven
Shortage of Scala experts
2／
7
13/56

Only two Scala programmers in a team of seven
We must build up our skill!
Shortage of Scala experts
2／
7
14/56

Learning of Scala
1. Self study
2. Study club
3. Pair programming
15/56

1. Self Study
Understand the basic syntax

1. Self Study
Books
17/56

1. Self Study
Documents by Twitter
https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 18/56

1. Self Study
Source code from OSS
19/56

2. Study Club
Share what we learned by ourselves
Introduction to functional programming

2. Study Club
• Each engineer solves the problems
provided by Scala experts
• e.g., Binary Tree, Fibonacci Sequence
21/56

• Example of factorial generation algorithm
2. Study Club
Discussion with Members
Use var Use recursion
22/56

• Example of factorial generation algorithm
Use var Use recursion
2. Study Club
Discussion with Members
No side effect
23/56

3. Pair Programming
Practice by using the knowledge
obtained from study club

3. Pair Programming
• Effective learning
• Learn about symbols (e.g., +, @), which cannot
be searched easily on the Internet
• Supplement the knowledge not taught at
study club
• Complex syntaxes (e.g., implicit, currying)
• Development tools (e.g., sbt, IntelliJ)
25/56

Summary: Adopting Scala
• Reasons of selecting Scala
• Compatible with concurrent programming
• Maintainability because of static typing
• Learning of Scala
• Self Study, Study Club, Pair Programming
• Big burdens on Scala experts
• Needed for teaching team members
• Maintain the quality of codes
26/56

Architecture of Backend (simplified)
• Servers separated by Queues
• API Server: Process users’ requests
• EventBus Server: Process events in Queue
• Stream Server: Supply event to connected users
28/56

Role of Each Server
and Framework

API Server
Process users’ requests
• Enqueue events
• e.g., Send message
Join/leave conversation
• Techniques to process a lot of requests
• Delegate heavy tasks (e.g., Disk I/O) to others
• Logic is written to run asynchronously
• scala.concurrent.Future
30/56

• RPC system for JVM based on Netty
• Support us to write asynchronous logic
• Used by large scale web services
• e.g., Twitter, Tumblr, Foursquare, Pinterest
• Equipped with various clients
• Redis, Memcached
• With retry policy, connection pools
31/56

EventBus Server
Process events in Queue
• Examples of tasks
• Logging
• Inquire GREE internal system
• Store events in DB
• etc.
• Problems of concurrent processing
• Deadlock, race condition, …
32/56

Framework to write concurrent, distributed
logic more easily
• No need to handle shared resources
• Race condition, dead-lock
• Logic separation
• Parent-Child
• Fault tolerance
33/56

Example of Akka
34/56

Example of Akka
Match by Event type
35/56

Example of Akka
Send copy to child
36/56

Example of Akka
log
37/56

Stream Server
Supply events to connected users
• Connect by streaming
• Increase connections as much as possible
• Keep users data in the memory to reduce I/O
• Asynchronous I/O by Finagle, Akka
• Task division by Akka
• Supply events to users
• Update users data
• Send KeepAlive to users
38/56

Summary: Architecture
• Each server is separated by Queue
• API, EventBus, Stream Server
• Framework is selected by considering
asynchronous processing
• Finagle, Akka
• Task is divided by Akka
• Easy to write concurrent logic
39/56

Obstacles
1.JVM
• Full GC
2. Self-created Scala library
• Sharding
• ID Architecture
41/56

Full GC
Problems

• Scala runs on JVM
• We want to prevent “stop the world”
• GC in young generation is faster than full GC
• Try not to let it store object in old generation
Full GC
Young generation Old generation
43/56

Causes of Uncollected
Reference Problem
• Use method that has side effects
• Reuse object everywhere
• Forget to release used resources
44/56

Causes of Uncollected
Reference Problem
• Use method that has side effects
• Reuse object everywhere
• Forget to release used resources
Difficult to recognize by developers
45/56

Short-Lived Object
• Use val variable (final variable)
• Don’t use mutable variable
• Don’t leave references to objects for too long
Create new objects instead of updating old ones
46/56

Sharding: Scalable Storage
47/56

Scale up?
48/56

Scale out!
49/56

Horizontal Partitioning
• Sharding is necessary!
• Considering capacity and #accesses
• We have Cascade, a sharding library
• For PHP, not Scala
• So, we created Aurora and make it OSS
• Aurora and application were developed
simultaneously
• So difficult to integrate them during
development
• Code modification must be done in many parts
50/56

ID Architecture:
UUID and InnoDB
Ref：MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance
http://kccoder.com/mysql/uuid-vs-int-insert-performance/

Ref：MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance
http://kccoder.com/mysql/uuid-vs-int-insert-performance/
UUID
AUTO_INCREMENT #record
#time

Drop UUID, Take Time-Based
• At first, we used UUID as primary key
• But this caused MySQL to insert randomly
• Created a library to generate ID instead
• Referred to Snowflake by Twitter
• Sequential insertion based on sorted created
time
53/56

Summary: Obstacles &
Workaround
• JVM
• Use short-lived objects to avoid Full GC
• Self-created Scala library
• Sharding
• Made Scala library OSS
• Integration of library to application was difficult
• ID Architecture
• Adopt time-based ID for sequential insertion
54/56

Summary
• Reasons for selecting Scala
• Compatible with concurrent programming
• Plan Scala learning within the team
• Learning cost is expensive
• Architecture & Frameworks
• Each server is separated by Queue
• Finagle, Akka
• Shortage of libraries
• We must have made libraries by ourselves
• Needed to pull request to OSS repository
56/56

Thanks!
@tomoyoshi_ogura, @kyo_ago, @takc923
@yoshie_777, @le_chang, @beketa, @j5ik2o
and
ScalaMatsuri staffs

[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to [ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫 (20)

More from gree_tech (20)

Recently uploaded (20)

[ScalaMatsuri] グリー初のscalaプロダクト！チャットサービス公開までの苦労と工夫

Editor's Notes