Real-world Experiences in Scala

Real-world Experiences
in
Scala
How I found happiness
by Amir Karimi | www.dev-frame.com
Aug 2015

Philosophy of Life
It's about the journey not the destination
If you want to be happy

Geeks Philosophy
It's about the tools not the work
If you want to be a happy developer

My Philosophy
I'm happy with Linux and Scala
for now

The Past
●
Fanatic C# Developer for 10 years
●
Lived in MS echo chamber
●
.NET CF, WinForms, WebForms, MVC, WF, WCF, WPF,
Silverlight, LINQ, EF, MS SQL, VS, etc.
●
Love Linux: in my dreams I was developing on my own Linux
machine!

Converting; Why?
●
Tired of MS echo chamber
●
Feel lonely
●
Love Linux and Open Source
●
To run a huge private server with less than 2GB of RAM
– Clean installation of Ubuntu server 14.04 takes less than 50MB of RAM
– Windows Server 2012 Essentials needs at least 2048MB?
●
To observe copyright and licensing (VS, Windows Server/Client, SQL Server,
etc.)

My Ideal Programming
Language
●
Open Source
●
Cross platform (Practically)
●
Ready to use with mature ecosystem (Practical)
●
Strongly-type / Type-safe
●
Lambdas / Closures => Functional (I was thinking about LINQ)
●
Object Oriented (Interfaces, Generics, ...)
●
Modern (Not Verbose: Type inference, Auto Getter/Setter, …)

Available ones
●
C# - Cross Platform (Practically)
●
Python / Ruby / JS - Strongly-type / Type-safe
●
Java - Modern (Not Verbose: Type inference, Auto Getter/Setter,
…) and didn't support lambdas at the time
●
Go - Ready to use with mature ecosystem | Object Oriented
(Interfaces, Generics, …)
●
Scala - What?

Scala Features
●
Type inference
●
Powerful Syntax
●
Functional
●
DSL
●
Immutable Structures
●
Case Classes
●
Pattern Matching
●
Implicits
●
Macros
●
Powerful Type System and
Standard Library
●
etc.

Extra Bonus
● Moore's law dying and the need of concurrent programming
– Facts
●
First 3GHz single core CPU at 2002
●
It's 13 years and there are still just 3GHz CPUs
●
But with more cores (your cellphone would have 4)
●
Thread blocking is very expensive
– Scala and other functional programming languages shine here
– Akka is another bonus
●
Somebody created a Redis clone on top of Akka (CurioDB)
● Big Data
– Spark is on top of Scala and Akka (100x faster than Hadoop)

My New Ecosystem
● Scala
– The only available language which meet my needs
● Play Framework
– Mature, Type-safe (even templates), High Velocity, Async, Iteratees,
Simple to Use
● MongoDB
– Open Source, Fast, Distributed, Schema Free, Not good for everything
● ReactiveMongo
– Blazingly Fast, Async, Iteratees

Deployment Requirement
●
Development Machine
– JDK
– Activator (Play)
– Git (to push the binaries to the
server)
– Bash
●
Production Server
– JRE (I don't use this server as build machine)
– Nginx (as front-end if you have more than one
web apps)
– Virtualmin
●
DNS
●
Main Server
●
Webmail
– MongoDB
– Git (as binary repository for publishing process)

Deployment Process
Development Machine Production Server
Source
Bin Bare Git Repo
“/var/webapps/[app]
service play-[app] restart

Real-world Projects
Tehran International Book Fair
● Website
● Publisher Registration System
● Student Coupon Selling System

Real-world Projects
● Tehran International Book Fair (TIBF) Website
– Up to 40,000 unique visitors daily
– 2,287 publishers
– 361,788 books
● TIBF Publisher Registration System
– Registration process of TIBF
– About 24,000,000,000 IRR ≈ 700,000$ total transactions in three weeks
● TIBF Student Coupon Selling System
– With a quota system based on provinces
– More than 500 request/sec
– About 87,000,000,000 IRR ≈ 2,600,000$ total transactions in three weeks

Challenges
●
TIBF Website
– Large number of RPS on book search page
●
Users can find their favorite book location
●
TIBF Registration System
– Complicated process on top of NoSQL database
●
TIBF Student Coupon Selling System
– Large number of RPS
– Large number of financial transactions
– Integration with Bank
Summary:
●
High availability
●
Large number of RPS
●
Complicated processes on top of NoSQL
● One small server for all of them
– 6 cores
– 6 GB of RAM
– Non-SSD hard disk

Solutions – High Velocity
●
Play is a high velocity Framework by default
●
Using async/non-blocking data-access library: ReactiveMongo
●
Futures and Promises
●
Iteratees (for rendering huge data with minimum resource
usage: RAM or CPU)
●
Everything is perfect; even didn't need caching!

Solutions – NoSQL
●
Manage relationships asynchronously
– Get fair information → Get registration information → Get basic information → …
●
All non-block
– Repeat data
●
Fairs in which the exhibitor is registered
●
Registered exhibitors for each fair
– Search
●
Generate keywords on CUD, don't use RegEx
●
NoSQL is not suitable for a system with complex relationships
●
Reports are painful but amazingly fast
– Need a job to collect data into another collection
– For real-time reports, data should be collected on CUD events

Solutions - Integration
●
Scala is on top of JVM with a huge high quality libraries
– I needed Rijndael-256 which is not common and is not implemented in Java
standard libraries
●
Bouncycastle
●
Also used
– Play's WS as web service client
– Scalaxb to generate WSDL classes (they didn't support REST)
– Again everything is async

Libraries and Tools - 100%
Free●
OS
– Linux (server and development)
●
IDE and Tools
– Activator
– Scala IDE
– RoboMongo
– Git
●
DB
– MongoDB
●
Front-end Server
– Nginx
●
Framework Libraries
– Play Framework
– ReactiveMongo
– Macwire (compile-time DI)
– Silhouette (authentication)
– Scrimage (image resizing)
– Persianutils by Bahman Movaghar
– Bouncycastle
– Xcala; my own application library
– etc.

Well Done!
●
Common
– Full-stack development
– Full async architecture (top to
bottom)
– Very good RAM usage except for
exporting Excel files
●
TIBF Website
– Less than 100ms responses on
peak times
●
TIBF Registration System
– Most of the publishers were happy
– Most of the operators were happy
– I was agile to implement real-time
requests in minutes or hours
●
TIBF Coupon System
– Boom! Things didn't go well
– Although about 5,000 successful
purchases have been done in a couple of
hours...

Failure
●
After lunching the TIBF coupon website, all websites seemed down
●
but the system load was OK!
●
Incomplete payments started to show up
– Verified payments without having invoices marked as paid
– Double payments
●
Stress
– Loneliness
– Failure

Failure
I Failed :(((((
●
Bad feeling about Scala
●
Feels like someone is pouring ice water on my head
●
A weird kind of relief
●
Everything got quiet

Didn't Give Up
●
Took a break, got relaxed and focused
●
Checked the bandwidth
●
Checked the server disk speed
●
Checked the logs
– Nginx: Too many open files
●
Checked the connections
– netstat -na | grep :80 | wc -l
●
About 4,000 open connections!
●
Google
– Default OS limitation for open
files is 4,000
– There is a simple solution for
“Too many open files”
– It was a victory! I could have
reach to the OS limits on a
small server

Fix
●
Add “fs.file-max” to “/etc/sysctl.conf”
●
Set soft and hard limits in “/etc/security/limits.conf” for Nginx and the user
by which Play apps are running
●
Set “worker_rlimit_nofile” in “/usr/local/nginx/conf/nginx.conf”
●
The server begin responding but the RAM was still too low
– Exporting Excel files for operators eats the RAM
– The laptop I used for this presentation has
more RAM than the server did (6GB)
6GB
3M$

Final Relief
●
They gave me a giant server
– 120 Core
– 128 GB of RAM
●
I used at most lest than 1% of this server
resources!
●
Although again I increased the default
OS limitation on number of open files
●
And their network got a serious problem

Bonus Demo
Iteratees in Action

Thank you
Build amazing things with your lovely tools
Iran-SUG => Iranian Scala User Group
Amir Karimi <a.karimi.k@gmail.com>
@4m1rk
www.dev-frame.com

Real-world Experiences in Scala

More Related Content

What's hot (20)

Similar to Real-world Experiences in Scala (20)

Recently uploaded (20)

Real-world Experiences in Scala