Download to read offline
![Documents Enable Dynamic Schema & Optimal
Performance
Relational MongoDB
{ customer_id : 1,
first_name : "Mark",
last_name : "Smith",
city : "San Francisco",
phones: [
{
number : “1-212-777-1212”,
dnc : true,
type : “home”
},
number : “1-212-777-1213”,
type : “cell”
}]
}
Customer
ID
First Name Last Name City
0 John Doe New York
1 Mark Smith San Francisco
2 Jay Black Newark
3 Meagan White London
4 Edward Daniels Boston
Phone Number Type DNC
Customer
ID
1-212-555-1212 home T 0
1-212-555-1213 home T 0
1-212-555-1214 cell F 0
1-212-777-1212 home T 1
1-212-777-1213 cell (null) 1
1-212-888-1212 home F 2](https://image.slidesharecdn.com/mongodb3-161108101152/75/L-architettura-di-Classe-Enterprise-di-Nuova-Generazione-12-2048.jpg)
![Document Model Benefits
Agility and flexibility
Data model supports business change
Rapidly iterate to meet new requirements
Intuitive, natural data representation
Eliminates ORM layer
Developers are more productive
Reduces the need for joins, disk seeks
Programming is more simple
Performance delivered at scale
{
customer_id : 1,
first_name : "Mark",
last_name : "Smith",
city : "San Francisco",
phones: [
{
number : “1-212-777-1212”,
dnc : true,
type : “home”
},
number : “1-212-777-1213”,
type : “cell”
}]
}](https://image.slidesharecdn.com/mongodb3-161108101152/75/L-architettura-di-Classe-Enterprise-di-Nuova-Generazione-13-2048.jpg)
![Data Store for Raw Dataset
…
Siloed source databases
External feeds
(batch)
Streams
Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png
Transform
Store raw
data
AnalyzeAggregate
Pub-sub,ETL,fileimports
Stream Processing
Users
Other
Systems
Store
raw data
Transform
- Typically just writing
record-by-record from
source data
- Usually just need high
write volumes
- All 3 options handle that
Transform read requirements
- Benefits to reading multiple datasets
sorted [by index], e.g. to do a merge
- Might want to look up across tables
with indexes (and join functionality in
MDB v3.2)
- Want high read performance while
writes are happening
Interactive querying on
the raw data could use
indexes with MongoDB](https://image.slidesharecdn.com/mongodb3-161108101152/75/L-architettura-di-Classe-Enterprise-di-Nuova-Generazione-27-2048.jpg)
Uploaded byMongoDB
L’architettura di Classe Enterprise di Nuova Generazione
This document discusses using MongoDB as part of an enterprise data management architecture. It begins by describing the rise of data lakes to manage growing and diverse data volumes. Traditional EDWs struggle with this new data variety and volume. The document then provides an overview of MongoDB's features like flexible schemas, secondary indexes, and aggregation capabilities that make it suitable for building different layers of an EDM pipeline for tasks like raw data storage, transformation, analysis, and serving data to downstream systems. Example use cases are presented for building a single customer view and for replacing Oracle with MongoDB.
More Related Content
Similar to L’architettura di Classe Enterprise di Nuova Generazione
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=640&height=640&fit=bounds)
L’architettura di Classe Enterprise di Nuova Generazione
- 1. Massimo Brignoli Principal SolutionArchitect [email protected] @massimobrignoli L’architettura di Classe Enterprise di Nuova Generazione
- 2. Agenda • Nascita deiData Lake • Overview di MongoDB • Proposta di un’architettura EDM • Case Study & Scenarios • Data Lake Lessons Learned
- 3. Quanti dati? • Unacosa non manca alla aziende: dati – Flussi dei sensori – Sentiment sui social – Log dei server – App mobile • Analisti stimano una crescita del volume di dati del 40% annuo, 90% dei quali non strutturati. • Le tecnologie tradizionali (alcune disegnate 40 anni fa) non sono sufficienti
- 4. La Promessa del“Big Data” • Scoprire informazioni collezionando ed analizzando i dati porta la promessa di – Un vantaggio competitivo – Risparmio economico • Un esempio diffuso dell’utilizzo della tecnologia Big Data è la “Single View”: aggregare tutto quello che si conosce di un cliente per migliorarne l’ingaggio e i ricavi • Il tradizionale EDW scricchiola sotto il carico, sopraffatto dal volume e varietà dei dati (e dall’alto costo).
- 5. La Nascita deiData Lake • Molte aziende hanno iniziato a guardare verso un’architettura detta Data Lake: – Piattaforma per gestire i dati in modo flessibile – Per aggregare I dati cross-silo in un unico posto – Permette l’esplorazione di tutti i dati • La piattaforma più in voga in questo momento è Hadoop: – Permette la scalabilità orizzontale su hardware commodity – Permette una schema di dati variegati ottimizzato in lettura – Include strati di lavorazione dei dati in SQL e linguaggi comuni – Grandi referenze (Yahoo e Google in primis)
- 6. Perché Hadoop? • HadoopDistributed FileSystem è disegnato per scalare su grandi operazioni batch • Fornisce un modello write-one read-many append-only • Ottimizzato per lunghe scansione di TB o PB di dati • Questa capacità di gestire dati multi-strutturati è usata: – Segmentazione dei clienti per campagne di marketing e recommendation – Analisi predittiva – Modelli di Rischio
- 7. Ma va beneper tutto? • I Data Lake sono disegnati per fornire l’output di Hadoop alle applicazioni online. Queste applicazioni hanno dei requisiti tra cui: – Latenza di risposta in ms – Accesso random su un sottoinsieme di dati indicizzato – Supporto di query espressive ed aggregazioni di dati – Update di dati che cambiano valori frequentemente in real-time
- 8. Hadoop è larisposta a tutto? • Nel nostro mondo guidato ormai dai dati, i millisecondi sono importanti. – Ricercatori IBM affermano che il 60% dei dati perde valore alcuni millisecondi dopo la generazione – Ad esempio identificare una transazione di borsa fraudolenta è inutile dopo alcuni minuti • Gartner predice che il 70% delle installazioni di Hadoop fallirà per non aver raggiunto gli obiettivi di costo e di incremento del fatturato.
- 9. Enterprise Data ManagementPipeline … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems
- 10. In Dettaglio • Joinnon necessarie causano pessime performance • Costoso scalare verticalmente • Lo schema rigido rende difficile il consolidamento di datai variabili o non strutturati • Ci sono differenze nei record da eliminare durante la fase di aggregazione • I processi soventi durano ore durante la notte • I dati sono vecchi per prendere decisioni intraday
- 11. Veloce Overview diMongoDB
- 12. Documents Enable DynamicSchema & Optimal Performance Relational MongoDB { customer_id : 1, first_name : "Mark", last_name : "Smith", city : "San Francisco", phones: [ { number : “1-212-777-1212”, dnc : true, type : “home” }, number : “1-212-777-1213”, type : “cell” }] } Customer ID First Name Last Name City 0 John Doe New York 1 Mark Smith San Francisco 2 Jay Black Newark 3 Meagan White London 4 Edward Daniels Boston Phone Number Type DNC Customer ID 1-212-555-1212 home T 0 1-212-555-1213 home T 0 1-212-555-1214 cell F 0 1-212-777-1212 home T 1 1-212-777-1213 cell (null) 1 1-212-888-1212 home F 2
- 13. Document Model Benefits Agilityand flexibility Data model supports business change Rapidly iterate to meet new requirements Intuitive, natural data representation Eliminates ORM layer Developers are more productive Reduces the need for joins, disk seeks Programming is more simple Performance delivered at scale { customer_id : 1, first_name : "Mark", last_name : "Smith", city : "San Francisco", phones: [ { number : “1-212-777-1212”, dnc : true, type : “home” }, number : “1-212-777-1213”, type : “cell” }] }
- 14. MongoDB Technical Capabilities Application Driver Mongos Primary Secondary Secondary Shard1 Primary Secondary Secondary Shard 2 … Primary Secondary Secondary Shard N db.customer.insert({…}) db.customer.find({ name: ”John Smith”}) 1. Dynamic Document Schema { name: “John Smith”, date: “2013-08-01”, address: “10 3rd St.”, phone: { home: 1234567890, mobile: 1234568138 } } 2. Native language drivers 5. High performance - Data locality - Indexes - RAM 3. High availability 6. Horizontal scalability - Sharding 4. Workload Isolation
- 15.
- 16. 3.2 Features Relevantfor EDM • WiredTiger as default storage engine • In-memory storage engine • Encryption at rest • Document Validation Rules • Compass (data viewer & query builder) • Connector for BI (Visualization) • Connector for Hadoop • Connector for Spark • $lookUp (left outer join)
- 17. Data Governance withDocument Validation Implement data governance without sacrificing agility that comes from dynamic schema • Enforce data quality across multiple teams and applications • Use familiar MongoDB expressions to control document structure • Validation is optional and can be as simple as a single field, all the way to every field, including existence, data types, and regular expressions
- 18. MongoDB Compass For fastschema discovery and visual construction of ad-hoc queries • Visualize schema – Frequency of fields – Frequency of types – Determine validator rules • View Documents • Graphically build queries • Authenticated access
- 19. MongoDB Connector forBI Visualize and explore multi-dimensional documents using SQL-based BI tools. The connector does the following: • Provides the BI tool with the schema of the MongoDB collection to be visualized • Translates SQL statements issued by the BI tool into equivalent MongoDB queries that are sent to MongoDB for processing • Converts the results into the tabular format expected by the BI tool, which can then visualize the data based on user requirements
- 20. Dynamic Lookup Combine datafrom multiple collections with left outer joins for richer analytics & more flexibility in data modeling • Blend data from multiple sources for analysis • Higher performance analytics with less application-side code and less effort from your developers • Executed via the new $lookup operator, a stage in the MongoDB Aggregation Framework pipeline
- 21. Aggregation Framework –Pipelined Analysis Start with the original collection; each record (document) contains a number of shapes (keys), each with a particular color (value) • $match filters out documents that don’t contain a red diamond • $project adds a new “square” attribute with a value computed from the value (color) of the snowflake and triangle attributes • $lookup performs a left outer join with another collection, with the star being the comparison key • Finally, the $group stage groups the data by the color of the square and produces statistics for each group
- 22.
- 23. MongoDB Architecture Patterns 1.Operational Data Store (ODS) 2. Enterprise Data Service 3. Datamart/Cache 4. Master Data Distribution 5. Single Operational View 6. Operationalizing Hadoop System of Record System of Engagement
- 24. Enterprise Data ManagementPipeline … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems
- 25. Come scegliere illayer di Data Management per ognuno degli stage? Processing Layer ? When you want: 1. Secondary indexes 2. Sub-second latency 3. Aggregations in DB 4. Updates of data For: 1. Scanning files 2. When indexes not needed Wide column store (e.g. HBase) For: 1. Primary key queries 2. If multiple indexes & slices not needed 3. Optimized for writing, not reading
- 26. MongoDB Hadoop/Spark Connector Distributed processing/ analytics •Sub-second latency • Expressive querying • Flexible indexing • Aggregations in database • Great for any subset of data • Longer jobs • Batch analytics • Append only files • Great for scanning all data or large subsets in files - MongoDB Hadoop Connector - Spark-mongodb Both provide: • Schema-on-read • Low TCO • Horizontal scale
- 27. Data Store forRaw Dataset … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems Store raw data Transform - Typically just writing record-by-record from source data - Usually just need high write volumes - All 3 options handle that Transform read requirements - Benefits to reading multiple datasets sorted [by index], e.g. to do a merge - Might want to look up across tables with indexes (and join functionality in MDB v3.2) - Want high read performance while writes are happening Interactive querying on the raw data could use indexes with MongoDB
- 28. Data Store forTransformed Dataset … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems AggregateTransform Often benefits to updating data as merging multiple datasets Dashboards & reports can have sub-second latency with indexes Aggregate read requirements - Benefits to using indexes for grouping - Aggregations natively in the DB would help - With indexes, can do aggregations on slices of data - Might want to look up across tables with indexes to aggregate
- 29. Data Store forAggregated Dataset … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems AnalyzeAggregate Dashboards & reports can have sub-second latency with indexes Analytics read requirements - For scanning all of data, could be in any data store - Often want to analyze a slice of data (using indexes) - Querying on slices is best in MongoDB
- 30. Data Store forLast Dataset … Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Transform Store raw data AnalyzeAggregate Pub-sub,ETL,fileimports Stream Processing Users Other Systems Analyze Users Dashboards & reports can have sub-second latency with indexes - At the last step, there are many consuming systems and users - Need expressive querying with secondary indexes - MongoDB is best option for the publication or distribution of analytical results and operationalization of data Other Systems Often digital applications - High scale - Expressive querying - JSON preferred Often RESTful services, APIs
- 31. Architettura EDM Completa … Siloedsource databases External feeds (batch) Streams Data processing pipeline Pub-sub,ETL,fileimports Stream Processing Downstrea m Systems … … Single CSR Application Unified Digital Apps Operational Reporting … … … Analytic Reporting Drivers & Stacks Customer Clusterin g Churn Analysis Predictiv e Analytics … Distributed Processing Governance to choose where to load and process data Optimal location for providing operational response times & slices Can run processing on all data or slices Data Lake
- 32. Example scenarios 1.Single CustomerView a. Operational b. Analytics on customer segments c. Analytics on all customers 2.Customer profiles & clustering 3.Presenting churn analytics on high value customers
- 33. Single View ofCustomer Spanish bank replaces Teradata and Microstrategy to increase business and avoid significant cost Problem Why MongoDB Results Problem Solution Results Took days to implement new functionality and business policies, inhibiting revenue growth Branches needed an app providing single view of the customer and real time recommendations for new products and services Multi-minute latency for accessing customer data stored in Teradata and Microstrategy Built single view of customer on MongoDB – flexible and scalable app easy to adapt to new business needs Super fast, ad hoc query capabilities (milliseconds), and real-time analytics thanks to MongoDB’s Aggregation Framework Can now leverage distributed infrastructure and commodity hardware for lower total cost of ownership and greater availability Cost avoidance of 10M$+ Application developed and deployed in less than 6 months. New business policies easily deployed and executed, bringing new revenue to the company Current capacity allows branches to load instantly all customer info in milliseconds, providing a great customer experience Large Spanish Bank
- 34. Case Study Insurance leadergenerates coveted single view of customers in 90 days – “The Wall” Problem Why MongoDB ResultsProblem Solution Results No single view of customer, leading to poor customer experience and churn 145 years of policy data, 70+ systems, 15+ apps that are not integrated Spent 2 years, $25M trying build single view with Oracle – failed Built “The Wall” pulling in disparate data and serving single view to customer service reps in real time Flexible data model to aggregate disparate data into single data store Churn analysis done with Hadoop with relevant results output to MongoDB Prototyped in 2 weeks Deployed to production in 90 days Decreased churn and improved ability to upsell/cross-sell
- 35. Top 15 Global Bank KickingOut Oracle Global bank with 48M customers in 50 countries terminates Oracle ULA & makes MongoDB database of choice Problem Why MongoDB Results Problem Solution Results Slow development cycles due to RDBMS’ rigid data model hindering ability to meet business demands High TCO for hardware, licenses, development, and support (>$50M Oracle ULA) Poor overall performance of customer- facing and internal applications Building dozens of apps on MongoDB, both net new and migrations from Oracle – e.g., significant portion of retail banking, including customer-facing and backoffice apps, fraud detection, card activation, equity research content mgt.) Flexible data model to develop apps quickly and accommodate diverse data Ability to scale infrastructure and costs elastically Able to cancel Oracle ULA. Evaluating what apps can be migrated to MongoDB. For new apps, MongoDB is default choice Apps built in weeks instead of months or years, e.g., ebanking app prototyped in 2 weeks and in production in 4 weeks 70% TCO reduction
Editor's Notes
- #10 Stream processing is often separate processing layer than the batch processing, but it can be stored into the data stores at various stages
- #11 Could make more visual
- #17 Kernel 3.2 Scope tracking: https://docs.google.com/spreadsheets/d/1L1EbbWoshUIHXBzCh5e3sALtAFxm_dJ52SRPR6GzeAY/edit#gid=0 Release notes for 3.1.6: http://docs.mongodb.org/manual/release-notes/3.1-dev-series/
- #19 Determine validator rules: You can use the tool to figure out what you want to set as validation rules
- #22 $lookup – this creates new documents which contain everything from the previous stage but augmented with data from any document from the second collection containing a matching colored star (i.e., the blue and yellow stars had matching lookup values, whereas the red star had none)
- #23 In terms of reporting, A number of Business Intelligence (BI) vendors have developed connectors to integrate MongoDB as a data source with their suites, alongside traditional relational dbs. This integration provides reporting, visualizations, dash-boarding of MongoDB data
- #25 Stream processing is often separate processing layer than the batch processing, but it can be stored into the data stores at various stages
- #32 Just a logical diagram. Processing could be on same physical servers as storage nodes to minimize data movement