Data analytics with NOSQL
5 likes•428 views
The document discusses NoSQL databases as an alternative to relational databases, highlighting their role in managing big data and unstructured information. It addresses the advantages and challenges of NoSQL, such as scalability, performance, and the CAP theorem, while outlining various NoSQL models like key-value stores, document databases, and graph databases. Use cases for NoSQL include log management and fraud detection, emphasizing the importance of selecting the right database based on specific functional and non-functional requirements.
Data analytics with NOSQL
- 1. Data Analytics with NOSQL Mukundan Agaram Chris Weiss
- 2. Some initial thoughts about data... Continual issues with large scale web apps – Data growth + query response time ● Data growth => performance degradation ● Explosion of big data “analytics” use cases – Increase in unstructured data ● More interconnectivity, more formats, lack of structure... ● Document oriented data (XML/JSON) are difficult to manage and search – Distributed server configurations ● Large systems, more distribution and HA Cloud services has aggravated these issues
- 3. Agenda for the night ● What is NOSQL? ● Varieties of NOSQL ● Key Industry Use Cases ● Applications for Data Analytics ● Landscape ● Demos/Walkthroughs ● Closing Discussions
- 4. What is NOSQL? ● “...mechanism for storage and retrieval of data that is modeled in means other than tabular relations used in relational databases.” Wikipedia ● Non SQL or Non-relational ● Not Only SQL ● Technically since late 1960... – E.g. IDMS, IMS, MUMPS, Cache, BerkeleyDB
- 5. What is NOSQL? ● Drivers for modern day NOSQL – Web 2.0 – Big Data – Facebook, Google, Amazon, Expedia etc. – Horizontal scaling to clusters of computers ● Achilles heel for RDBMS – Cost – Provide ● HA ● Partition Tolerance (a.k.a sharding) ● Speed
- 6. NOSQL - Drawbacks and Barriers ● Compromise on consistency (CAP Theorem) ● Custom query languages vs. SQL ● Lack of standardized interfaces ● Existing investments in RDBMS ● Most lack true ACID transactions. – Use an “eventually” consistent model – Data is replicated with a conflict resolution algorithm – Methods for conflict resolution and distribution vary significantly
- 7. CAP Theorem ● a.k.a Brewer's theorem ● Impossible for a distributed computer system to simultaneously provide – Consistency ● all nodes see same data at same time – Availability ● Every request receives a response – Partition Tolerance ● Fault tolerance to partitioning because of network failures
- 8. CAP alignment for NOSQL Source: http://blog.nahurst.com/visual-guide-to-nosql-systems
- 9. NOSQL direction The landscape is morphing... ● Current NOSQL industry focus – Address large distributed systems reactionary to the CAP theorem ● The newer breed of NOSQL address important aspects such as ACID ● There is a new buzz word … – NewSQL
- 11. NOSQL Model Classification Key Value Stores & Caches Data is represented as a collection of (K,V) pairs. In-memory, persistent or eventually persistent. Document Databases Data is stored in JSON document structures. RDF, OWL & Triple Stores Meaningful way to connect information. Can inference over triples (S,P,O). Can be represented graphically. SPARQL Wide Column Databases Extensible record set. Stores data tables as sections of columns. Great for EDW. Graph Databases Stores data as a graph G(V,E). Great for correlation analysis, recommendation engines and fraud detection. Multi-model Databases Combination of one or more varieties of the above.
- 12. NOSQL Models ● Key-Value – Cache (EHCache, BigMemory, Coherence, Memcached) – Store (Redis, Riak, AeroSpike, Oracle NoSQL) ● Document (MongoDB, CouchDB, AmazonDynamoDB) ● Wide Column (Cassandra, HBase, Vertica) ● Graph (Neo4j, Titan, Giraph) ● Multi-model (OrientDB, ArangoDB, Sqrrl)
- 14. Consider NOSQL for... ● Enabling “big data” and “web” scale – Massive distribution through horizontal scaling ● Performant queries (alternatives to RDBMS) – Denormalization and large horizontal scalability ● Massive write volumes (Facebook, Twitter) ● Fast and dynamic access to key data ● Flexible schemas and data types ● Data/Schema Migration ● Developer centric environments
- 15. Consider NOSQL for... ● Diverse data organization options – Hierarchical correlation – Graph correlation – Semantic relationships – Set based analytics ● Caching in end usage format ● Data Archival ● Big Data Analytics – Cumulative metrics and insights – Correlation
- 16. Where RDBMS/SQL is better.. ● OLTP ● Data Integrity ● SQL centricity ● Complex relationships – Exception of graph NOSQL ● Maturity, stability and standardization
- 17. Use Cases ● Log management (unstructured data) ● Data synchronization (online vs. offline sources) – Shopping cart, Field sales/services, PoS, Gaming, Transportation/telemetry ● User profile management ● Customer 360 degree view ● Fraud detection ● Medical/Healthcare diagnosis ● Data Archival ● Recommendation Engines
- 18. Applications for Data Analytics ● Complements (part of) Hadoop and Big Data ● Acts as the persistence infrastructure for larger machine learning use cases – Predictive Analytics – Fraud/Anomaly/Outlier Detection – Recommendation engines ● Provides a back drop for interesting data visualization initiatives – Integrate with visualization packages such as Tableau
- 19. Interesting links ● Redis in Practice: Who's online? www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ ● Inventory list of NOSQL systems www.nosql-database.org ● Database Engine ranking and analytics www.db-engines.com ● Visual guide to NOSQL systems www.blog.nahurst.com/visual-guide-to-nosql-systems
- 20. Case Studies / Demos ● Retail fraud detection – Neo4j – Contrasting with OrientDB – Tinkerpop/Gremlin/Blue Print ● 360 degree single view of voter information – MongoDB ● Schema on read – Hadoop
- 23. Gremlin Blueprints Architecture Neo4j OrientDB TitanGraph ArangoDB
- 24. Qualified Voter – Use Case ● Tracks registration information for all voters in Michigan ● Uses a tabular geography model ● Highly normalized schema – Data partitioned into subsets ● Enable local application instances and row level security ● Expensive queries when doing reporting ● Expensive queries for performing “single view” of voter ● Several tables with tens of millions of records
- 25. Voter Schema
- 26. Find the first 100 voters in Ingham county with status and school district SELECT V.VOTER_IDENTIFICATION_NUMBER,V.FIRST_NAME, V.LAST_NAME, G.CODE AS GENDER, IDS.NAME AS ID_STATUS, UST.NAME AS UOCAVA_STATUS, VA.ADDRESS_LINE_ONE, VA.CITY, VA.ZIP_CODE, DIS.NAME AS SCHOOL_DISTRICT FROM VOTER V, VOTER_ADDRESS VA, GENDER G, IDENTIFICATION_STATUS IDS, UOCAVA_STATUS UST, VOTER_STATUS_TYPE VST, STREET_RANGE SI, DISTINCT_POLITICAL_AREA DPA, DISTINCT_POLITICAL_AREA_DIS DPAD, DISTRICT DIS, DISTRICT_TYPE DT, COUNTY CO WHERE V.ID = VA.VOTER_ID AND V.GENDER_ID = G.ID AND V.IDENTIFICATION_STATUS_ID = IDS.ID AND V.UOCAVA_STATUS_ID = UST.ID AND V.VOTER_STATUS_TYPE_ID = VST.ID AND VST.NAME = 'Active' AND VA.STREET_RANGE_ID = SI.ID AND SI.DISTINCT_POLITICAL_AREA_ID = DPA.ID AND VA.IS_ACTIVE = 'Y' AND DPA.COUNTY_ID = CO.ID AND CO.NAME = 'Ingham' AND DPA.ID = DPAD.DISTINCT_POLITICAL_AREA_ID AND DPAD.DISTRICT_ID = DIS.ID AND DIS.DISTRICT_TYPE_ID = DT.ID AND DT.NAME = 'School' AND ROWNUM <= 100;
- 29. Expensive in terms of IO ● Multiple objects read ● Two stage IO: ● Read index ● Read entire table row ● Selected and WHERE clause columns assembled and then filtered ● Resources for larger volume query would be high – memory, CPU, fast disk
- 30. Parting conclusions ● NOSQL is a mixed bag of fruit ● This space is growing ● There are hundreds of products ● Best value is realized from identifying the correct use case – Functional requirements – Non-functional requirements
- 31. Finally you can use NOSQL for...