Use Cases for NoSQL in Media
Download as PPTX, PDF0 likes598 views
The document outlines the role of NoSQL databases within the media sector, particularly at Sanoma, emphasizing the need for specialized data solutions to enhance data management, analytics, and application performance. It discusses various NoSQL data models, including key-value, document, column, and graph stores, and highlights the significance of the CAP theorem in understanding the trade-offs in distributed systems. Additionally, it presents Sanoma's implementation of big data technologies and their applications in business intelligence, data science, and content delivery.
Use Cases for NoSQL in Media
- 1. NOSQL in Media Sander Kieft
- 2. About me Manager Core Services at Sanoma Responsible for all common services, including the Big Data platform Work: – Centralized services – Data platform – Search Like: – Work – Water(sports) – Whiskey – Tinkering: Arduino, Raspberry PI, soldering stuff 24 April 20152
- 3. Sanoma, B2C Publishing and Learning company 2+100 2 Finnish newspapers Over 100 magazines 24 April 2015 Presentation name3 5 TV channels in Finland and The Netherlands 200+ Websites 100 Mobile applications on various mobile platforms
- 5. 24 April 2015 Presentation name5 Not Only SQL
- 6. Generic vs specialized solutions 24 April 2015 Presentation name6
- 7. Data models Speed Scalability Partition tolerance Availability / Redundancy Cost per GB Specialized focus 24 April 2015 Presentation name7
- 8. CAP (or Brewster) Theorem says: “it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: – Consistency – Availability – Partition tolerance” CAP Theorem 24 April 2015 Presentation name8 A C P
- 9. CAP Theorem 24 April 2015 Presentation name9 A C P Availability Each client can always read and write Partition Tolerance The system works well despite physical network partitions Consistency All clients always have the same view of the data RDBMS MySQL Postgres MS SQL Oracle NOSQL NOSQL
- 10. Eventual consistency -- Werner Vogels, CTO Amazon
- 12. key-value column document stores map/reduce graph search blob storage Various data models 24 April 2015 Presentation name12
- 13. Key/value stores Photo credits: John Chulick - https://www.flickr.com/photos/chulickphotos/8234894686/
- 14. Key/value stores Storing object on key Based on the Dynamo paper (Werner Vogels) Products: – Riak – Memcache/Membase – Tokyo Cabinet – Redis – Voldemort Use cases: – Counting – Top lists – Caches – Pre-calculated optimizations 24 April 2015 Presentation name14
- 15. Bucket A B C Key/Value buckets 24 April 2015 Presentation name15 User XXXX YYYY ZZZZ Article 100 200 300 Article_<5 min. TIME> 50 100 150
- 16. Real time stats 24 April 2015 Presentation name16
- 19. Document Stores
- 20. Document stores Stores ”records” as documents Versioning Easy sharding (document self contained) Products: – MongoDB – CouchDB – SimpleDB Use case: – CMS – Meta data – Product catalog 24 April 2015 Presentation name20
- 21. From relational data model to document 24 April 2015 Presentation name21 Product Properties Application Property Property
- 22. MyJour Item Based Framework …. CMS Architecture Content Platform 24 April 2015 Presentation name22 Content Platform Core Search Solr Blob Storage (S3 & MT) Article storage MongoDB Analyse CMS CMS Editorial reuse-interface ePub Digital Template system WoodWing Content Portal Feeds Noma Viva PDF Based Framework …. HomeDeco Sources Services Solutions Products ?? ?? ?? ?? eLinea Blendle Google Currents LINDA. nieuws NU.nl search
- 23. Column stores
- 24. Column stores Lineage: Google's BigTable paper Records with many, many columns Distinguish between hot and cold data Versioning Records and columns can be sharded Products: – Hbase – Cassandra – Hypertable Use cases: – Analytics – Messages 24 April 2015 Presentation name24
- 25. Big Data
- 26. Big Data Linage: Google GFS & Map/Reduce Distributed data storage and processing Advanced analytics capabilities on raw data Schema on read Products: Hadoop MPP databases Use cases: – Adhoc querying terabytes of data – Data science Predictive analytics Model training – Calculate recommendations 24 April 2015 Presentation name26
- 27. Big Data at Sanoma Main use case for reporting and analytics, moving to data science A/B MVT testing evaluations Using Qlikview as a front-end Supply data to other environments (SAS, Advertising, Behavioral Targeting) Agile process for adding sources, from raw to intermediate to modeled datawarehouse Sanoma standard data platform, used in all Sanoma countries > 250 Users: dashboard users 40 daily users: analysts & developers 43 source systems, with 125 different sources 400 tables in hive Platform: – Cloudera Hadoop – 40-60 nodes – > 400TB storage – ~2000 jobs/day Typical data node / task tracker: – 1-2 CPU 4-12 cores – 2 system disks (RAID 1) – 4 data disks (2TB, 3TB or 4TB) – 24-32GB RAM 24 April 2015 Presentation name27
- 28. Sanoma Data lake Traditional BI vs Big Data approach 28 24.4.2015 © Sanoma Media
- 30. Search Keyword search can be combined with advanced forms of ranking the results Most of the fields go to an index Facets can be used for analytics Ranker can be replaced with custom logic Products: – Solr – ElasticSearch – Marklogic Use cases: – Content Search – Analytics / Faceted – Percolation 24 April 2015 Presentation name30
- 31. Search 24 April 2015 Presentation name31 Content Q Σ Result ranking
- 32. Search too 24 April 2015 Presentation name32 Content t Σ Result ranking User
- 33. Search too 24 April 2015 Presentation name33 Content Page Σ Result ranking User
- 34. Traditional queries: against index with existing data What if the data does not exist at time of query? Percolation allows registration of queries and then returning the query IDs, e.g. for notification when new matches are available Use case: – Search for a tweet, but after the initial results continuously get newly tweeted items when they come in Search - Percolation 24 April 2015 Presentation name34
- 35. Graph databases
- 36. Graph databases Lineage: Euler and graph theory. Data model: Nodes & edges, both which can hold key-value pairs Products: – AllegroGraph – InfoGrid – Neo4j Use cases: – Social relationships – Content Linking (Entity linking) 24 April 2015 Presentation name36 Jan Smit 3js Nick en Simon Volendam Article 1 Article 2 Article 3
- 37. Blob storage
- 38. Blob storage Endless storage of binary data Storing larger objects then a single machine “Lower” price/GB compared to SAN storage Products – Amazon S3 – CAStor – (Hadoop) Use case: – Media storage – Archiving 24 April 2015 Presentation name38
- 39. Summary
- 40. RDBMS systems are a good enough for many problems For specific problems NOSQL solutions provide a specific solution There’s a variety of NOSQL solutions with different characteristics NOSQL solutions will require a higher engineering effort Summary 24 April 2015 Presentation name40
- 41. Dream NO SQL Architecture – Content Delivery 24 April 201541 CMS Document storage (MongoDB/ CouchDB) Blob storage (S3/ CAStor) Search (ElasticSearch/ Solr) Website / Mobile Application
- 42. Dream NO SQL Architecture - Analytics 24 April 201542 Event collection Message Queue (Kafka / Flume ) Event processing (Storm) Key-value store (Redis) Real time recommendations / targeting Column storage (Cassandra/ Hbase) Real time Dashboarding Big Data (Hadoop) Adhoc reporting & Data science
- 43. CAP Theorem 24 April 2015 Presentation name43 A C P Availability Each client can always read and write Partition Tolerance The system works well despite physical network partitions Consistency All clients always have the same view of the data MySQL Asterdata Postgres Greenplum MS SQL Vertica Oracle Dynamo Cassandra Voldemort SimpleDB Tokyo Cabinet CouchDB KAI Riak Big Table MongoDB Berkeley DB Hypertable Terrastore MemcachDB Hbase Scalaris Redis Data models Relational databases Key-value Column-oriented Document-oriented