SlideShare a Scribd company logo
Cloud-Based Architecture and Big Data
Philip Balinov,
DevOps Engineer
SOME DEFINITIONS
►
def. CLOUD COMPUTING
l
- distributed computing over a network
l
- the ability to run a program or application on many connected computers at
the same time.
►
def. BIG DATA
l
data sets so large and complex that it becomes difficult to process using
traditional data processing applications
Retail
Finance
E-Commerce
Telecommunication
B2B
Publishing/Media
Government & NGO
Automotive
Travel
KOMFO'S CLIENTS
KOMFO PLATFORM WORKFLOW OVERVIEW
EXTERNAL PROVIDERS
MODERN COMPANY STRUCTURE
►
Agile software development: Scrum, Kanban
►
Short product feature lifecycle
►
Continuous delivery
►
Self-organizing teams
►
Cross-functional teams
AGILE DEVELOPMENT, CONTD.
►
Scrum
►
Excellent for product development
►
Short, iterative cycles (sprints), commit to deadlines
►
Self-organizing teams
►
Kanban
►
Developed at Toyota to optimize supply-chain
►
Good for ad-hoc tasks, e.g. support, bugfixing
AGILE DEVELOPMENT, CONTD.
►
Agile development requires agile operations
►
Underlying architecture must scale together with the product
►
Continuous integration
►
Quality assurance
►
Deployment
►
The solution: de-couple development and operations
►
DevOps
USE CASE
►
Fast and dependable communication with external providers
►
On-demand resource scaling
►
Flexibility
►
Storage, indexing & analysis of huge amounts of data
►
Dependability
Cloud & Big Data: Lessons Learnt
Cloud & Big Data: Lessons Learnt
CLOUD CONCEPTS
►
Three main service models
►
IaaS – EC2, Rackspace, Azure, HP, Oracle
►
PaaS – Google App Engine, Heroku
►
SaaS – Gmail, Wordpress.com, Salesforce, Office 365
CLOUD CONCEPTS, CONTD
►
Four deployment models
►
Public
►
Private
►
Community
►
Hybrid
CLOUD INTERNALS
SERVERS
STORAGE
EXTERNAL SERVICES
(CDN, NETWORKING, SaaS)
VMs VMs
PROS AND CONS
Dedicated Cloud
+ simplicity
+ performance
+ predictability
+ tried and tested
+ agility
+ ease of use
+ scalability
+ stability
- inflexible
- ineffective
- nightmare in case of disaster
- vendor lock-in
- blackboxed
- fluctuations
- helpless in case of disaster
CLOUD ARCHITECTURE
►
Application servers scale up and down based on load
►
Application software written for parallelism
►
Communication between nodes via messaging service
►
Write for eventual consistency
OK, so we have an (endlessly) scalable cloud app
now.
But didn't we just create a bottleneck elsewhere?
MIX & MATCH
►
Crunch numbers in the cloud
►
Application servers
►
Slow running tasks
►
Temporary services
►
Test servers
MIX & MATCH, CONTD.
►
Traditional servers for:
►
Incompatible apps (single-threaded, memory, disk intensive, specialized
hardware) do not work well in cloud environments
►
Database servers are best kept on dedicated machines in our experience
DATABASES, NOSQL
►
def. NoSQL
l
a mechanism for storage and retrieval of data that is modeled in means other
than the tabular relations used in relational databases.
DATABASES
• Use the best tool for the job depending on the task
• NoSQL Advantages
►
Social networks generate a lot of data
►
Complex interconnections, cyclical dependencies
►
Aggregations must be performed on both new and old data
►
Structure of foreign sources may change on short notice
DATABASES, NOSQL CONTD
►
Riak, Hadoop, MongoDB, Cassandra, Redis
►
In-memory dataset for faster operation
►
No predefined structure
►
Integrated sharding, load-balancing and failover
►
Versatility - can be used for anything from data storage to real-time messaging to search
indexes
DATABASES, LONG TERM STRATEGY
►
Data quickly becomes irrelevant
►
Archive it, but still be accessible
►
Online Data Warehouse solutions
►
Amazon Redshift
►
Keep Everything
►
Terabytes for pennies
WE ARE LOOKING FOR…
Senior Web Developer
Junior Web Developer
Junior QA
DevOps Engineer
Questions?

More Related Content

What's hot (20)

PDF
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
ScyllaDB
 
PDF
From Local to Global
AWS Germany
 
PPTX
Cosmos db made easy with azure functions
Luis Beltran
 
PPTX
IronSource Atom - Redshift - Lessons Learned
Idan Tohami
 
PPT
Clustered Architecture Patterns Delivering Scalability And Availability
ConSanFrancisco123
 
PDF
Shift: Real World Migration from MongoDB to Cassandra
DataStax
 
PDF
Build A Scalable Mobile App
Mohamed Aboul-Fotouh
 
PPTX
Big Data on Cloud Native Platform
Sunil Govindan
 
PPTX
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
iguazio
 
PDF
Migrating to Cassandra
Instaclustr
 
PPTX
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
iguazio
 
PDF
The Journey To Serverless At Home24 - reflections and insights
AWS Germany
 
PDF
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Ruslan Synytsky
 
PPTX
Building big data applications on AWS by Ran Tessler
Idan Tohami
 
PDF
An Introduction To Space Based Architecture
Amin Abbaspour
 
PDF
Caching for Microservices Architectures: Session I
VMware Tanzu
 
PPTX
Kafka: Legacy microservices
bleporini
 
PPTX
RedisConf18 - Video Experience Operational Insights in Real Time.
Redis Labs
 
PDF
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
ScyllaDB
 
PDF
20160309-VDI for Under $100 / User
Atlantis Computing
 
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
ScyllaDB
 
From Local to Global
AWS Germany
 
Cosmos db made easy with azure functions
Luis Beltran
 
IronSource Atom - Redshift - Lessons Learned
Idan Tohami
 
Clustered Architecture Patterns Delivering Scalability And Availability
ConSanFrancisco123
 
Shift: Real World Migration from MongoDB to Cassandra
DataStax
 
Build A Scalable Mobile App
Mohamed Aboul-Fotouh
 
Big Data on Cloud Native Platform
Sunil Govindan
 
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
iguazio
 
Migrating to Cassandra
Instaclustr
 
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
iguazio
 
The Journey To Serverless At Home24 - reflections and insights
AWS Germany
 
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Ruslan Synytsky
 
Building big data applications on AWS by Ran Tessler
Idan Tohami
 
An Introduction To Space Based Architecture
Amin Abbaspour
 
Caching for Microservices Architectures: Session I
VMware Tanzu
 
Kafka: Legacy microservices
bleporini
 
RedisConf18 - Video Experience Operational Insights in Real Time.
Redis Labs
 
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
ScyllaDB
 
20160309-VDI for Under $100 / User
Atlantis Computing
 

Similar to Cloud & Big Data: Lessons Learnt (20)

PDF
NoSQL and Cloud Services - Philip Balinow, Comfo
beITconference
 
PPT
Big Data on The Cloud
Putchong Uthayopas
 
PPTX
The Evolution of Data Engineering Emerging Trends and Scalable Architecture D...
Ashis86
 
PDF
SpringPeople - Introduction to Cloud Computing
SpringPeople
 
PPTX
The rise of “Big Data” on cloud computing
Minhazul Arefin
 
PPTX
INN530 - Assignment 2, Big data and cloud computing for management
Simen Smaaberg
 
PDF
Cloud Computing: The Hard Problems Never Go Away
ZendCon
 
PDF
Addressing dm-cloud
Genoveva Vargas-Solar
 
PPTX
Big Data PPT by Rohit Dubey
Rohit Dubey
 
PPTX
Lecture1 BIG DATA and Types of data in details
AbhishekKumarAgrahar2
 
PDF
cloud computing and big data .pdf
pushpamarasinghe62
 
PPTX
Lecture1
Manish Singh
 
PPT
Big data.ppt
IdontKnow66967
 
PDF
Lecture 1-big data engineering (Introduction).pdf
ahmedibrahimghnnam01
 
PDF
Big data and cloud computing 9 sep-2017
Dr. Anita Goel
 
PDF
Cloud and Bid data Dr.VK.pdf
kalai75
 
PPTX
Above the cloud joarder kamal
Joarder Kamal
 
PPT
Database Management Myths & Reality for the future
A B M Moniruzzaman
 
PPTX
Unushs susus susujss. Ssuusussjjsjsit 4.pptx
AshishHiwale1
 
PPTX
SMAC - Social, Mobile, Analytics and Cloud - An overview
Rajesh Menon
 
NoSQL and Cloud Services - Philip Balinow, Comfo
beITconference
 
Big Data on The Cloud
Putchong Uthayopas
 
The Evolution of Data Engineering Emerging Trends and Scalable Architecture D...
Ashis86
 
SpringPeople - Introduction to Cloud Computing
SpringPeople
 
The rise of “Big Data” on cloud computing
Minhazul Arefin
 
INN530 - Assignment 2, Big data and cloud computing for management
Simen Smaaberg
 
Cloud Computing: The Hard Problems Never Go Away
ZendCon
 
Addressing dm-cloud
Genoveva Vargas-Solar
 
Big Data PPT by Rohit Dubey
Rohit Dubey
 
Lecture1 BIG DATA and Types of data in details
AbhishekKumarAgrahar2
 
cloud computing and big data .pdf
pushpamarasinghe62
 
Lecture1
Manish Singh
 
Big data.ppt
IdontKnow66967
 
Lecture 1-big data engineering (Introduction).pdf
ahmedibrahimghnnam01
 
Big data and cloud computing 9 sep-2017
Dr. Anita Goel
 
Cloud and Bid data Dr.VK.pdf
kalai75
 
Above the cloud joarder kamal
Joarder Kamal
 
Database Management Myths & Reality for the future
A B M Moniruzzaman
 
Unushs susus susujss. Ssuusussjjsjsit 4.pptx
AshishHiwale1
 
SMAC - Social, Mobile, Analytics and Cloud - An overview
Rajesh Menon
 
Ad

Recently uploaded (20)

PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
Blockchain Transactions Explained For Everyone
CIFDAQ
 
PDF
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
PDF
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PDF
July Patch Tuesday
Ivanti
 
PDF
Python basic programing language for automation
DanialHabibi2
 
PDF
CIFDAQ Weekly Market Wrap for 11th July 2025
CIFDAQ
 
PPTX
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
PDF
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
Blockchain Transactions Explained For Everyone
CIFDAQ
 
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
July Patch Tuesday
Ivanti
 
Python basic programing language for automation
DanialHabibi2
 
CIFDAQ Weekly Market Wrap for 11th July 2025
CIFDAQ
 
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
Ad

Cloud & Big Data: Lessons Learnt

  • 1. Cloud-Based Architecture and Big Data Philip Balinov, DevOps Engineer
  • 2. SOME DEFINITIONS ► def. CLOUD COMPUTING l - distributed computing over a network l - the ability to run a program or application on many connected computers at the same time. ► def. BIG DATA l data sets so large and complex that it becomes difficult to process using traditional data processing applications
  • 6. MODERN COMPANY STRUCTURE ► Agile software development: Scrum, Kanban ► Short product feature lifecycle ► Continuous delivery ► Self-organizing teams ► Cross-functional teams
  • 7. AGILE DEVELOPMENT, CONTD. ► Scrum ► Excellent for product development ► Short, iterative cycles (sprints), commit to deadlines ► Self-organizing teams ► Kanban ► Developed at Toyota to optimize supply-chain ► Good for ad-hoc tasks, e.g. support, bugfixing
  • 8. AGILE DEVELOPMENT, CONTD. ► Agile development requires agile operations ► Underlying architecture must scale together with the product ► Continuous integration ► Quality assurance ► Deployment ► The solution: de-couple development and operations ► DevOps
  • 9. USE CASE ► Fast and dependable communication with external providers ► On-demand resource scaling ► Flexibility ► Storage, indexing & analysis of huge amounts of data ► Dependability
  • 12. CLOUD CONCEPTS ► Three main service models ► IaaS – EC2, Rackspace, Azure, HP, Oracle ► PaaS – Google App Engine, Heroku ► SaaS – Gmail, Wordpress.com, Salesforce, Office 365
  • 13. CLOUD CONCEPTS, CONTD ► Four deployment models ► Public ► Private ► Community ► Hybrid
  • 15. PROS AND CONS Dedicated Cloud + simplicity + performance + predictability + tried and tested + agility + ease of use + scalability + stability - inflexible - ineffective - nightmare in case of disaster - vendor lock-in - blackboxed - fluctuations - helpless in case of disaster
  • 16. CLOUD ARCHITECTURE ► Application servers scale up and down based on load ► Application software written for parallelism ► Communication between nodes via messaging service ► Write for eventual consistency
  • 17. OK, so we have an (endlessly) scalable cloud app now. But didn't we just create a bottleneck elsewhere?
  • 18. MIX & MATCH ► Crunch numbers in the cloud ► Application servers ► Slow running tasks ► Temporary services ► Test servers
  • 19. MIX & MATCH, CONTD. ► Traditional servers for: ► Incompatible apps (single-threaded, memory, disk intensive, specialized hardware) do not work well in cloud environments ► Database servers are best kept on dedicated machines in our experience
  • 20. DATABASES, NOSQL ► def. NoSQL l a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.
  • 21. DATABASES • Use the best tool for the job depending on the task • NoSQL Advantages ► Social networks generate a lot of data ► Complex interconnections, cyclical dependencies ► Aggregations must be performed on both new and old data ► Structure of foreign sources may change on short notice
  • 22. DATABASES, NOSQL CONTD ► Riak, Hadoop, MongoDB, Cassandra, Redis ► In-memory dataset for faster operation ► No predefined structure ► Integrated sharding, load-balancing and failover ► Versatility - can be used for anything from data storage to real-time messaging to search indexes
  • 23. DATABASES, LONG TERM STRATEGY ► Data quickly becomes irrelevant ► Archive it, but still be accessible ► Online Data Warehouse solutions ► Amazon Redshift ► Keep Everything ► Terabytes for pennies
  • 24. WE ARE LOOKING FOR… Senior Web Developer Junior Web Developer Junior QA DevOps Engineer