Exploring MongoDB & Elasticsearch: Better Together

Editor's Notes

• #4: MongoDB is somewhat the defacto general purpose NoSQL DB and it has added enough new features and made enough improvements to stay there at top of NoSQL offerings Elastic is moving up and it can do things fast As our word expands and changes, the potential use cases for combining data stores – MongoDB and Elasticsearch – also grows. But before we can talk about those current and potential use cases for combining them, we should take a quick look at what each of them are and when to use them individually.
• #5: 2 mins
• #6: People wanted Big Data to go away, they wanted to call it other things or NOT call it things or whatever… EOT IOT IIOT But it’s not going to…
• #8: -Internet of Things / Everything / Industrial IIOT - logs, events, - 2019 ~$1.7 TRILLION $$ -Monitoring and managing those has sprung up whole companies now – -Augmented Reality AR VR MR - THE FUTURE – the next iphone level CHANGE Manufacturing, Training,
• #9: Sorry, not sorry - still love this quote after all of the years - But the truth remains – more and more and more Data Points Requires THINGS (applications, Data Store) to manage them
• #10: We NEED Something to hold the data, to find the data fast, to SHARE the data and MOVE it from one APP to another Process and transform along the way, Analyze it MEANINGS
• #11: NEVER truly schemaless though… If you are NOT thinking about app design before you actually start designing it, you FAIL You are just storing data that will likely never be used and your new shiny NoSQL datastore will just become a data wasteland = MongoDB and Elastic then MONGODB solo next
• #12: Keeo them tied together here – MongoDB is somewhat the defacto general purpose NoSQL DB and it has added enough new features and made enough improvements to stay there at top of NoSQL offerings Elastic is moving up and it can do things fast
• #13: IF something comes straight from wikipedia it HAS to be true MongoDB is the defacto general purpose NoSQL DB #5 Datastore technology over and holding steady there #1 NoSQL Database product
• #14: MongoDB has the market share and the community buy-in to make the difference in supportability to usually take the prize unless you have a really really heavy write application Community Support and Development efforts - drivers, etc. Built in Sharding/Scaling via Replica Sets High writes and heavy reads – can be somewhat mutually exclusive MongoDB scales nearly linearly for heavy read workloads
• #15: 3.4.10 as of Halloween - since released on Halloween, would avoid ;-) no tricks please - 3.4.9 considered a minor release overall but … But what does it look like really? Architecture overview next
• #16: 1 Primary, 2 Secondaries - heartbeat communication for up/down state, replication to secondaries via oplog MongoDB has same kind of potential to scale UP instead of OUT – **NOTE - many people run MongoDB on dedicated larger bare metal hosts and grow by scaling up vertically However, if they continue to grow, they will run into many of the same challenges that traditional RDBMS's have So what about scaling OUT with Mongo? Religious War here
• #17: MongoD’s – the data nodes – the shards - the Replica Sets (primary and 2 secondary members) MongoS’s – Query routers – talk to config servers and MongoD data nodes - get location metadata from config servers to route queries to the correct shard to satisfy a query and return the result Good design to have multiple mongoS query routers in sharded clusters – our environments have 4 Config servers – the Data Dictionary of Mongo - contains cluster/shard metadata – mapping of data set –3.0 and below Always keep exactly and ONLY 3 for PROD env’s. 3.2 and up, is now by default a replica set and is NOW Required to be WT – improves consistency of info in chunk map - aka where data extents reside If you lose or corrupt your configs, the mongoS will not know where the data resides - so can’t retrieve it …so effectively lost
• #18: Too much to cover other than mention for you to look up later WT – new default, also for required config serer replica set vs 3 single db’s as before MMAP - still good for larger result sets or smaller, more frequent write activities, specifically updates Unless you have a lot of CPU and cores to throw at it for WT usage = reminder to talk about percona version that allows us to offer security features that usually only come with the more expensive Enterprise version SSL kerberos LDAP integration *** our experience there
• #19: User Data in Games Inventory Management – update, decrease, increase inventory Shopping carts - tales of the long query and 1000 pairs of shoes CMS – Our expertise running Sitecore on Azure
• #21: A search engine but a whole lot more MUCH more powerful than JUST a search engine GeoAnalytics - Geo near me
• #22: Basically Clusters with Nodes holding Indexes then split across hosts with Shards Holding slices of data held in segments at the lucene chunk level Composed of the data via documents written in JSON
• #24: There are lots of reasons to use multiple components of the Elastic Stack Including for Visualization which we will talk about a bit later. But 1st let’s talk about just elasticsearch
• #25: With Elastic, to increase in scale and add more performance, you increase the Replication Factor Basically ADD NODES -this increases HW resources to improve search performance and improve redundancy The number of replica shards can be changed dynamically on a live cluster, allowing us to scale up or down as demand requires. And Elastic will automatically redistribute as needed nine shards: three primaries and six replicas. This means that we can scale out to a total of nine nodes, again with one shard per node. This would allow us to triple search performance compared to our original three-node cluster.
• #26: here Logging and Log Analysis Basically taking over for Splunk which has become too expensive
• #27: Elasticsearch has made massive improvements to its geospatial capabilities in the last 2 releases It way outperforms the geospatial abilities of MongoDB’s $geoNear and within operators Which is why you would look to combine them – which we will talk about later on But other good uses of Elasticsearch combined with elements of its Elastic STACK
• #28: But other good uses of Elasticsearch combined with elements of its Elastic STACK BUT Now to Summarize those 2 – MongoDB and Elasticsearch
• #29: Summarize those 2 Both store data objects that have key-value pair, both allow querying that body of objects. But both come from 2 different camps and are made for different purposes. Elastic - Great with full and fuzzy text searching Slow when adding ‘new’ Data -  aka creating new indexes Uses indexes to help you find the data  - fast Completes complex search queries quickly  Interacts well directly other associated technologies – kibana, beats, logstash, etc. and other NoSQL and SQL DB’s 
• #30: In the end, it is about the ability to store data, aggregate things, pass it along. Then ANALYZE and USE that data analysis for whatever purpose you desire So let’s look at these 2 together now
• #31: - When your data has a lot of arrays - to perform queries that required many different $and clauses on an field with an array as a value.  MANY Smaller shards as they need additional write scopes 2nd case  - Fuzzy - If you want to do a search on the word chicken in a menu application: 
• #33: Examples of How we combine MongoDB and Elasticserch CURRENTLY at ObjectRocket
• #34: POTENTIAL and or Theoretical New Use Cases Possibilities and Potential Combination uses are very broad – New emerging markets and areas – from cryptocurrency peripherals for persistence to
• #35: Use MongoDB to Analyze cryptocurrency price swings and intervals - https://medium.com/@serbanmihai/aggregate-mongodb-data-with-node-js-and-mongoose-cryptocurrency-financial-time-series-ae739b4c9485
• #36: node.js (v7.8 or greater) Persistence is achieved using MongoDB tribeca - very low latency cryptocurrency market making trade bot with a full featured web client, backtester, and supports direct connectivity to several crypto coin exchanges  - reacts to market data by placing and canceling orders in under a millisecond
• #37: Fantasy Football wine sites -If you want to do a search and possibly a match on the words wine & red db.comments.find( { $and: [ { $text: { $search: "win" },  { $text: { $search: "red" }  }  ] } ) WON’T work $text special MongoDB operator - only use once per query,
• #38: Endless opportunities here to combine with other data stores - grab those result sets, store the primary results in MongoDB, perform additional aggregations to further refine them Post online for massive around the world use by colleagues Use Elasticsearch again to keep frequently searched combinations nearby/fast
• #39: Endless opportunities here to combine with other data stores - grab those result sets, store the primary results in MongoDB, perform additional aggregations to further refine them Post online for massive around the world use by colleagues Use elasticsearch again to keep frequently searched combinations nearby/fast
• #41: Hiring DBA’s and CDE’s