Full-stack Web Development with MongoDB, Node.js and AWS

Andrey
Mikhalchuk
Chief Archi tect
Aki ra
Technologies
FULL-STACK WEB
DEVELOPMENT WITH
MONGODB,
NODE.JS AND AWS

ABOUT AKIRA
Founded in 2003
Technology and management consulting
company
80+ employees
HUBZone, SBA 8(a), SDB
HITSP and NIEM Contributing Member
 ISO 9001, CMMI Level 3
Top Secret Facility Clearance
More on http://www.akira-tech.com

ABOUT THIS PRESENTATION
Relational databases -> MongoDB
Integrating with MongoDB
Stress-testing
Complications and solutions

ABOUT THE CLIENT
“Leading source of quality data
about the nation’s people and
economy”
US Census Bureau

WHERE DOES THE QUALITY DATA
COME FROM?
Surveys
Decennial Survey
American Community Survey
 Sur vey of Income and Program Par t icipat ion
 Cur rent Populat ion Sur vey
 Consumer Expendi ture Sur vey
 Nat ional Heal th Inter v iew Sur vey
 Amer ican Hour sing Sur vey
 Amer ican Time Use Sur vey
 Beginning Teacher Longi tudinal Study
 Consumer Expendi ture Sur vey
 Cur rent Populat ion Sur vey
 Housing Vacancy Sur vey
 Ident i t y Thef t Supplement
 Nat ional Ambulator y Medical Care Sur vey

PROBLEM
Most of these surveys use unique
software to collect and process
the data

PROBLEM
Different data storages:
Text files or different format
Binary data files
Relational databases
Incompatible data structures
Dozens of programming languages
Technologies incompatibles closed
architectures

AKIRA GOAL
Single solution for all surveys
Scalable
Inexpensive
Reusable cross-survey data
Works on all devices
Flat learning curve for developers

SURVEY DSL
How to
define a
new survey
©
Akira
Techn
ologie
s,
15

HOW TO DEFINE SURVEY
Akira has developed a prototype of an
engine that takes JSON specification and
turns it into feature-rich survey
JSON is a simple language that analysts
can use to define questions
Developers later can enrich user
experience with custom bells and
whistles
Both analysts and developers can reuse
existing functionality in future surveys
© Akira Technologies, 2013 16

DEFINE SURVEY
1. survey = {
2. name: "decennial",
3. type: 'survey',
4. title: "Decennial Survey",
5. stylesheet: "decennial.css",
6. header: "MongoDB PoC Demo",
7. footer: "Measuring America …",
8. intro: "The Census must count every …",
9. questions: []
10.}

QUESTIONS
1. {
2. name: "num_people_1",
3. quest ion: "How many people were l iving or staying … " ,
4. detai ls: "<B>INCLUDE</B> in this number :<UL><LI >…" ,
5. type: " int",
6. required: t rue,
7. } , {
8. name: "addi t ional_people_2",
9. quest ion: "Were there any addi t ional people staying … " ,
10. detai ls: "Mark al l that apply",
11. type: "checkbox",
12. opt ions: [
13. "Chi ldren, such as newborn babies or foster chi ldren",
14. "Relat ives, such as adul t chi ldren, c o u s i n s …" ,
15. "Nonrelat ives" ,
16. "People staying here temporar i ly",
17. "No addi t ional p e o p l e ”
18. }

BELLS AND WHISTLES
1. val idator : funct ion(s) {
2. n = parseInt ( s, 10 ) ;
3. return n > 0 && n < 5 ? nul l : ” E n te r value in [1,4] r a n g e ”
4. },
1. decorator : funct ion( q ) {
2. for ( i = 1; i <= 4; i++ ) {
3. if( i > parseInt ( $("input#num_people_1") .val ( ) , 10 ) ) {
4. $("div#quest ion_person_"+i ) .hide( ) ;
5. } else {
6. $("div#quest ion_person_"+i ) .show( ) ;
7. }
8. }

ACTION!
 Let’s see how this code works:
http://census.akira-tech.com/survey/8
Contains all Decennial questions
Written in <1 hour
Validates input in realtime
Updates interface dynamically

ACTION!
Take a look at another example:
http://census.akira-tech.com/survey/9
Contains most ACS questions
Written in <1 hour
Supports geolocation

MORE CUSTOMIZATIONS
 We d i d n’ t h ave to c re a te a new d a t a bas e s c h ema
 We d i d n’ t h ave to c re ate new we b i nte r f ac e
 We can customize al l aspects of survey process
 Here is another great example:
http://census.akira-tech.com/test/10000001
http://census.akira-tech.com/test_dashboard/sat_math

MOBILE VERSION
©
Akira
Techn
ologie
s,
26

MOBILE VERSION
 Speaking of geolocation …
 Let’s take a look at iPad version. Android/Windows
versions will look similar
 It looks basically the same, not surprisingly
 Surprisingly the iPad is not even connected to the
internet!
 Let’s complete the sur vey anyway and submit the
results
 Enjoy the result!

OFFLINE MODE
 Let’s shutdown the browser, it’s not in RAM
anymore
 Let’s connect the iPad to the internet
 Get back to the page
 Isn’t what you see is awesome?!
 This is complete of fline mode out of the box
 The inter face is the same ->
 No re-learning
 Single interface for CAPI, WAPI, CATI and more
 Less code to maintain

ANALYTICS
Af ter some
t ime this
system could
col lect bi l l ions
of records.
How do we
process them?
©
Akira
Techn
ologie
s,
31

DATA PROCESSING
Hadoop allows distributed data processing on
thousands servers
Cloudera manager and AWS CLI allow
deploying hundres of servers in minutes
We have deployed cluster of 3 nodes in AWS.
Let’s see how it can be reconfigured

LET’S PROCESS SOME DATA
 There are a lot of options for processing data in
Hadoop:
 Hive – DataWarehouse infrastructure for data query and
analysis
 MapReduce – programming model for large-scale data
processing
 Pig – high-level platform for creating MapReduce programs
 In this demo we chose Pig as the simplest way to
demonstrate the power of Hadoop
 This script loads data from MongoDB into , calculates
statistics and pushes it back to MongoDB

SOME PIG LATIN HERE
1. REGISTER /home/ec2-user/Dist r/mongo-2.10.1. jar ;
2. REGISTER /usr/l ib/hadoop-0.20/l ib/mongo-hadoop-core_0.20.205.0-
1.1.0. jar ;
3. REGISTER /usr/l ib/hadoop-0.20/l ib/mongo-hadoop-pig_0.20.205.0-
1.1.0. jar ;
4. raw = LOAD 'mongodb://master :27017/mongodb_poc. invi tes ' USING
com.mongodb.hadoop.pig.MongoLoader ( 'u__id:charar ray,
token:charar ray, emai l :charar ray, sur vey:charar ray, f i r st_name:charar ray,
last_name:charar ray, address:charar ray, inter view_mode:charar ray,
f r_id: int , processed:charar ray, state:charar ray ' )
5. AS ( id, token, emai l , sur vey, f i r st , last , address, mode, f r,
processed, state) ;
6. raw_l imi ted = LIMIT raw 3000;
7. DUMP raw_l imi ted;
8. raw_f i l tered = FILTER raw BY processed == ' t rue' ;
9. total_processed = FOREACH raw_f i l tered GENERATE COUNT(processed) ;
10. total_by_state = GROUP raw_f i l tered BY state;
11. DUMP total_by_state;
12. DUMP total_processed;
13. STORE total_by_state INTO
'mongodb://master :27017/mongodb_poc.stat ist ics '
14. USING com.mongodb.hadoop.pig.MongoStorage ( ) ;

STATISTICS
 Both MongoDB and Hadoop provide aggregation
framework.
 Hadoop works best for slow crunching humongous quantities of
data
 MongoDB is good for quick calculations on reasonably large
(tens of millions records) scopes
 Processed data is pushed back to Mongo for storing
and future visualization
We used Highcharts (an open source library) for data
visualization

ARCHITECTURE
©
Akira
Techn
ologie
s,
37

HOW MULTIPLE SURVEYS ARE POSSIBLE?
MONGODB!
 We use MongoDB
 Instead of creating complex database we store both
surveys and responses as documents
 Al l surveys are stored in the same col lection.
 If survey questions change, you still can query all versions of
the survey in single query.
 You even can query totally different survey results on core
parameters, like DOB
 The mongo cluster is deployed in the cloud
 We use Amazon Web Services (AWS) as the cloud
platform, but can use any other solution as wel l or bui ld
our own
 L et ’ s t a ke a l o o k a t t h e d e p loyment

THE POC HARDWARE
 In the PoC we use very basic computers:
 Realtime data collection:
 0.613 Gb RAM
 30 Gb HDD
 1 Core
 iPhone 5: 1Gb RAM, 32Gb SSD, Dual -core
 Data processing:
 1.7gb RAM
 6 Gb storage
 1 Core
 Samsung Galaxy S4: 2Gb RAM, 16Gb Storage, Quad-core

WEBSERVER(S)
We use Node.js as the web server
 All code is written in Javascript
 Node.js + MongoDB + Akira Survey DSL = everything
is written in Javascript. Learning curve is almost flat
 Validators in survey definition can be used both on
client side and server side, no need to duplicate code
 Node.js is extremely fast, but Nginx is faster for
static content. We use it as surrogate Content
Delivery Network (CDN)

ACTION!
 L e t ’ s g o to t h e AWS c o n s o l e :
https://console.aws.amazon.com/ec2/v2/home?region=us-west-
2#Instances
 Master, Node1, Node2 al l run the same configuration: Mongo
Config, Mongo Router, Mongo DB + Node. js
 No d e 3 i s i d e n t i cal to ot h e r n o d e s exc e pt i t t y p i c all y d o e s n ’ t
run Mongo config server
 Al l nodes are load-balanced with Elastic Load Balanser (ELB)
 I f we turn on Node 3 it wi l l be automatically included into load
balancing
 Al l nodes share MongoDB content, this is cal led sharding

PRODUCTION CONFIGURATION
 PoC is dif ferent from the production configuration:
 Slow/cheap servers
 Only few nodes in clusters
 Runtime and data warehouse in the same cloud
 Simplified security
 No real CDN
 No Mongo replicas
 Let’s take a look at how this could work in
production

AWS CONSOLE
 Provides status of all your servers
 Allows shutting them down when you don’t need
them and bringing back in minutes
 You can take sof tware running on a server and move
it to a more power ful computer in minutes
 Also provides vital statistics about all your servers

MONGODB
Many free opensource and commercial tools
Here is just one example
Provides comprehensive statistics on all
aspects of a MongoDB cluster performance

HADOOP
Multiple commercial and opensource solutions
for monitoring and managing Hadoop clusters
Cloudera Manager – deploys nodes in EC2 in
bulk
Here is just the standard out-of the box
Hadoop web interface for monitoring cluster
health

STRESS TESTING
 We user Tsung to
stress-test the system,
creating load of up to
40000 simultaneous
users.
 Even on a single laptop
the system was serving
250+ responses per
second with avg
response time < 1/3
sec

SCALABILITY
 What if you need even better per formance?
 Scale ver tically. Every node can be shut down and
restar ted on more power ful hardware up to 32
cores, 117Gb RAM, 2Tb SSD
 Scale horizontally. Hundreds of copies can be
deployed in <1hr. Akira already bui lt the
infrastructure that suppor ts plugging in hundreds of
nodes
 Al l scal ing operations are either automated or could
be automated with AWS CLI scripts
 Add rel iability by adding repl icas to MongoDB
Nodes
CPU Cores

INTEGRATION
Even the
best system
has l imi ted
use i f we
c a n ’ t
integrate i t
wi th other
systems.
©
Akira
Techn
ologie
s,
61

INTEGRATION
 Our PoC code provides complete REST API for manipulating
surveys and responses
 As we have demonstrated before we can easi ly integrate it
with Oracle Service Bus and SOAP-based services
 Data can be extracted into Oracle database both from
MongoDB and Hadoop
 SAS and R can be used to process data, both integrate with
Hadoop and MongoDB
 OPA
 Hadoop can write data into .csv files for OPA batch processing
 OPA batch processor can output to .csv for Hadoop MongoDB
consumption
 OPA Determinations engine can be queried from Hadoop MR tasks

SOFTWARE
Only open source software is used in this PoC
Most software can be used absolutely for free
Some has nominal fee, typically within $1000
range
All software has commercial licenses
providing support.

CLOUD SERVICES
 If you don’t use it, you don’t pay for it
 Hundreds of nodes can be deployed from the images
we have prepared
 Hot-plug nodes can be preconfigured and shutdown
for future use
 This is how much we paid for our servers for 3
weeks:
https://por tal.aws.amazon.com/gp/aws/developer/account?ie=
UTF8&action=activity -summary

SUMMARY
Effortlessly supports multiple surveys
Extremely scalable, leverages cloud
Low-cost open source
Easily integrates with Oracle DB,
Services, OPA
Tested to handle stress loads
Supports online and offline interview
modes

CONTACT
Akira Technologies, Inc
1747 Penn ave NW #600
Washington, DC 20006
P: 202.517.7187
F: 800.589.3129
E: info@akira-tech.com
W: www.akira-tech.com

Full-stack Web Development with MongoDB, Node.js and AWS

More Related Content

Viewers also liked (8)

Similar to Full-stack Web Development with MongoDB, Node.js and AWS (20)

More from MongoDB (20)

Recently uploaded (20)

Full-stack Web Development with MongoDB, Node.js and AWS

Editor's Notes