Using Compass to Diagnose Performance Problems in Your Cluster

Editor's Notes

• #2: Welcome to my talk on using Compass to diagnose performance problems in MongoDB. Thank you for coming to MongoDB World this year and thank you for coming to my talk. I'm excited to have the opportunity to discuss Compass and performance with you today.
• #3: Georgia Tech and University of Stuttgart grad. Long-time software developer and tinkerer. Compass Technical Mentor. I work for MongoDB on the Technical Services in NA. The worldwide Technical Services team at MongoDB provides guidance, insight and support for all customer use cases and all MongoDB products, including Compass. Attribution: ryan harvey -- https://www.flickr.com/photos/ryanh/7509976606/in/photolist-crCAph-pUWKYx-pqePxH-ocHgxR-9jnR2B-2yYmQA-bnMxe1-e1UHry-bAfqYx-DZBrcY-6kskYE-fM1bFg-9AjS9x-aDh5nS-3wM6aD-bcwj7F-bcwiNK-6roFBU-8YqrKr-72WwLe-aBJdHB-oPGYyS-9uz2cU-azaqFx-8v5BgU-F733W-9mWPDq-aEQLs3-bnkyhm-dkca89-fKYvGK-gNxZxv-eb4cvE-qpZWpC-kKUvmL-8YtH3G-98qbQb-98n1t6-8Yvxm4-jXtqZV-a6DAUD-qtjq5x-a6DzUB-bcwmu2-a1U4pc-kmAa9p-8YKAJJ-cdhsNf-jXvncG-bYwEeA
• #4: Motivation Performance Concepts Three Scenarios, Slow Query, Inefficient Query, Frequent Query Examine Workloads Search for Problem Queries Explain Query Plans Analyze Collection Schema with Compass Create Targeted Index with Compass Attribution: US Air Force -- https://www.flickr.com/photos/usairforce/33048702564/
• #5: To start, I would like to get at the WHY behind this talk. Why optimize for performance and why use the Compass tool?
• #6: Question: Can I see a show of hands, who thinks that performance is important to their business? A faster database makes your application faster. A faster application makes your customers and users happier and more engaged, which leads to good things like more revenue. Many organizations, including the ones shown here, have found a strong positive correlation between speed, engagement and revenue. You can read more at the wpostats website. Google DoubleClick found 2x more revenue for faster mobile sites. https://www.doubleclickbygoogle.com/articles/mobile-speed-matters/ Pinterest improved signups and traffic 15% each with 40% wait improvement. https://medium.com/@Pinterest_Engineering/driving-user-growth-with-performance-improvements-cfc50dafadd7 Walmart found 2% better conversion for 1s performance improvement. https://www.slideshare.net/devonauerswald/walmart-pagespeedslide Attribution: TaxCredits.net -- https://www.flickr.com/photos/76657755@N04/7214596024/in/photostream/
• #7: The GUI for MongoDB. Visually explore your data. Run ad hoc queries in seconds. Interact with your data with full CRUD functionality. View and optimize your query performance. Compass empowers you to make smarter decisions about indexing, document validation, and more. On the surface, Compass is a Desktop Database Client that provides a Graphical Admin Panel Schema Explorer Document Explorer CRUD operations and Real-time Server Stats
• #8: However, at a deeper level, the Compass design is based on years of experience working with customers to improve their database deployments. The lead engineer for the Compass team, Thomas Rueckstiess, is a former technical services engineer. This tool allows you to rapidly visualize and interpret MongoD performance statistics and explain plan results which might take hours with at text only tool.
• #9: Which concepts are important for MongoDB performance? Fundamental performance related concepts that apply to databases…
• #10: Working Set can be thought of as the list of things you need to get work done during the day. Your office is similar to RAM and the self-storage unit is like Disk. How fast and efficient would you be if most things you need are in self-storage? For a database program, the working set includes the documents, indexes and pages needed to satisfy the most common queries. Access Time (latency) Transfer Rate Attribution: Martin Cathrae -- https://www.flickr.com/photos/suckamc/4276413497/in/photolist-7vTJKc-2Q32fB-wa2vQ-4A14HM-6DNKtq-fDhvum-fDcfk1-Pnk6Y-uVie7-ezdr-eEUWr5-4C5faJ-4LiN8U-bo1uK4-4JGZ8X-bo1vmR-4JGZbi-bo1xKz-6cQtxv-7vGrC5-4Dty3o-27N7af-9VHzM1-4bfE5W-4jvZQU-adZXgC-27HGaz-4WDC-3X9P4z-4a7S-b3pwp6-7kjYPw-98Z2H9-5whRwW-6VwcaJ-9hsRzq-eEUQnC-yFwUH-5yZkKy-6VjSrS-34KHNL-4xrRHJ-4pJ4cU-4KhU8o-4Erhbj-5nUpvY-cyN4N-6Mp5t-33k6jr-3jMQeW Joseph North -- https://www.flickr.com/photos/joephotos/246500309/in/photolist-nMnXk-QK8a24-Ucrcyv-5MGN3p-Ugw4SA-5Hnu69-4HNbFN-65dFqW-dmVz2q-vduCE-diPfC5-5nsg3x-bxTbDc-5XQ6M4-2Vs7Sm-nNDvR2-dmTKK9-dmTDtP-nkC4Wp-2ZQUSn-dmVegw-dmTLUh-qwukjH-dmTBr9-4F6mnZ-HkYizf-5Hnu3y-r38RvU-dN8UGD-dmVbMq-bC8emS-ozCZQ9-dmU2qt-9gnYJE-fg82tJ-dmTGUm-oBEJun-898YWc-7eznXU-ajvEgB-dmTB2C-fSzRMo-fwdyPM-c9BtzG-9moxrU-baqnv6-6hwkGt-7z2hSs-oB2sYM-biC91r
• #11: What’s good, what’s bad? Consider two slides. Discuss/mention all three: access, time cost and transfer rate. Compare on graph. Attribution: reynermedia -- https://www.flickr.com/photos/89228431@N06/11080408955/ Eirien -- https://www.flickr.com/photos/ilweranta/8996280560/in/photolist-eGYhMw-TKqLua-24iYC6-ehCvbY-gmbhFP-6aqfiw-ehwsmM-7Jy3RX-8TCGJU-ehwJiT-oRL71m-ehwHVv-c7Zvk1-ehwwuk-5sVWLm-oziZ4t-cnyWB-2iVQHc-8fJQ5r-N5A6F-cnyS6-67dSQX-ehwvLV-ehCgJQ-258XF-agyDyv-ehCdhw-ehCxaq-ehChu3-ehwMea-ehCyPw-6mp62D-mhhPJ-JqN6W-aDKcaM-ehwwSg-cnyLA-cZeRxY-ehCsTu-ehCe5W-82K8xC-2j1bJ7-6aqfhw-2iW2Gv-gJXaz-82hF4R-2iVTSx-88iceA-99kMKq-6xDNqk
• #12: Two additional performance concepts are query shape and index. The Query Shape is a generalization of the structure of a query. Specifically, it is compose of a combination of query predicate, sort, and projection specifications. It represents the outline or shape of different specific queries that can be fulfilled with the same query plan, index and execution path. The MongoDB query planner uses the query shape as a type of query category when evaluating competing query plans as well as or caching the winning query plan and matching that plan against subsequent queries. Index – Conceptually a database index is similar to the index in the back of a book. It allows for a fast search based on a keyword with a reference to the full page or record. In a database, an index is a data structure similar to the red path through the maze. The index allows fast access to the goal without searching every possibility. Indexes provide: Fast Read (Search) via B-Tree Write cost due to B-Tree maintenance Can prevent replacement (pollution) of working set
• #13: Without Index Working Set Eviction from Cache Linear Collection Scan Faster Updates With Index Only Index Path in Cache O(log(N)) Search Updates Maintain Indexes
• #14: Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass. Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM. Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
• #15: Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass. Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM. Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
• #16: Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass. Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM. Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
• #17: In our first scenario, we are going to use Compass to examine a workload with a slow query. Q: Has anyone in the audience ever encountered a slow query in their application?
• #18: The application in scenario one is an e-commerce web site with product data stored in MongoDB. A collection of attributes describes product characteristics and the application searches for attributes by name. Attribution: Scott Lewis -- https://www.flickr.com/photos/99781513@N04/14491021598/in/photolist-9M2QU4-83BuMZ-Qu2Fhh-o5wgRY
• #19: An experiment requires measurement. Measure the workload both before and after any changes in order to draw a valid conclusion. Find a Slow Query Real Time Server Statistics Hottest Collections Slowest Queries Investigate the Slow Query Explain Plan Tab How do we do this in Compass? Documentation https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/ https://docs.mongodb.com/manual/tutorial/optimize-query-performance-with-indexes-and-projections/ Attribution: brittgow -- https://www.flickr.com/photos/brittgow/4782569782/in/photolist-8hBVnG-8hyCFt
• #20: Use real time server statistics to look for hot collections and slow queries. Find the query shape and use the explain plan tab to look for bottlenecks.
• #21: Ask the audience. Who has seen a slow query scan the entire collection? “webshop.attributes”, uses the most resources. Queries for “webshop.attributes.name” are scanning the entire collection. Only approximately 40 thousand of one million documents returned. Relatively few updates occur on “webshop.attributes”. Attribution: Michael Gil -- https://www.flickr.com/photos/msvg/5143096005/in/photolist-8QtHg4-o5efHJ-8p3E5-fcRoPN-os3UFT-V5oFwZ-ecGiaW-bBrEAg-4UkSjm-4UJPWC-BsTqmE-7whZWk-7yAvdf-fh7UJg-71kBVK-pSSwtL-cKsFNE-V1UR8h-mKgVDa-xHhGv-nKDHT-4UkSjf-4fQmCt-V5otwg-UPYAo1-arQCo9-hSgPq-iDiP8b-TQPTer-V5oFFr-ro5X3-nKDKa-BywLa-a7Fixt-fvRxn-bhcLyH-oDYyin-6GSZZ8-4UkSj9-d3vbWs-8WW33y-N39Y6-V5oFgP-8rrhrk-fPhtZQ-55N7WV-4UkUgY-PrPdag-9L4Bb8-4Xy1RX
• #22: Run an experiment! Carefully measure the workload before the experiment. Create a Targeted Index with Compass Measure Compare Results Real Time Server Stats Change in Duration Operations per Second Explain Plan Tab Documents Examined vs. Returned Index Used Decide Whether to Keep the Index. If Not Improved, Drop Index. Attribution: Max Pixel -- http://maxpixel.freegreatpicture.com/Aim-Focus-Dartboard-S-Target-Goal-Aiming-Arrow-1551504
• #23: Create an index that improves query performance of the slow query identified earlier. Creating an index is a very resource intensive process. Always test index creation in a test environment before production. Wait for index creation to complete. Verify that query performance improved.
• #24: With index, query on “name” only scans documents returned. Slowest query execution time improved from ~3500ms to ~500ms. Operations per second improved from ~5 to ~15. If the improvement holds for all workloads, keep the index... Attribution: DVIDSHUB -- https://www.flickr.com/photos/dvids/2897817247/in/photolist-5q557k-Hu4yVv-Jq9Sn2-Hu4yyZ-Hu4xvM-Hu4zyp-MFSyFN-Jgc8FA-nDrrYm-nT2SbE-NEDRdG-UWje3H-UWje9K-UWjehv-UFYn1N-HRh4R3-Lg7rif-JQQXhP-HUwfod-6zwr78-M9Q5xX-9vbTZg-NyChcC-BCrk3e-Mn5KXD-MxZfFL-MxZeQY-D5gg4j-aFV75p-Hmtvuy-HvPV9o-Jkh8MY-JgBmPv-HXJM9L-wx2WoT-wsbZar-wr3axA-wfESkZ-wxCTKX-wf5b6i-vzGP3G-wr5Fd3-wfwLrS-wwv5Tm-vuKsb5-wa4FDE-vuKx4S-wprYTN-vuUCxi-wpvpyJ
• #25: You should add an Index when careful experiments in a test environment under realistic workload indicate that it will improve performance. Attribution: https://freeiconshop.com/icon/search-document-icon-outline/
• #26: Indications Read Activity Outweighs Write Activity Dataset Does Not Fit in RAM Frequent or Time Critical Query Big Data, Business Intelligence and Analytics Contraindications Write Activity Outweighs Read Activity Infrequent or Unimportant Query Logging, Data Collection Attribution: https://freeiconshop.com/icon/search-document-icon-outline/ https://freeiconshop.com/icon/edit-document-icon-outline/
• #27: Use Compass to Recognize and Fix Queries Using Excessive Bandwidth
• #28: Scenario two involves a database of high-tech startup companies. The application is doing a search for companies in a particular category. A faceted search will sometimes generate this type of query.
• #29: The investigative process is similar to the first scenario. However, this time we will need to be more observant and pay closer attention to the details provided by Compass. Attribution: U.S. Naval Forces Central Command/U.S. Fifth Fleet -- https://www.flickr.com/photos/navcent/9303824295/in/photolist-fb9wPH-r8Rrwy-nZG4bk-dXEM3i-qTrTTY-huCynx-qqJajb-r8MDJX-nJnAnf-dMjwii-9yN2BH-qMwUo3-huDkBQ-dXLnaA-qpkW26-aFvbLx-qDbxej-qH7Zkn-aFvbTz-oWapfU-gxqnJn-iidD7g-b8yvbc-e3gh3s-qYptk6-oy9Cpp-pLvA9t-ogDCUw-iidnX9-qDjc1z-pXxQi6-qpkX4g-dk6r4o-iidFbM-PaqAVW-aVvrzK-dmGZ1c-ppn5x3-r4ZrhC-qYizBh-5DBPAT-grhrKt-o7Z1dG-eKkNUq-dknpJz-cpPr5o-atdftB-DKE1zx-Jebhuw-aY7NTn
• #30: In this workload, there are fewer slowest queries. Clicking a slow query gives a query shape of { "category_code": "___" }. However, there is already an index on "category_code". What else can we observe about the workload? Approx. 30 operations/sec using approx. 100,000KB/s or 100MB/s. We can switch over to the tech.companies collection and select the Schema tab to see the fields present in the sampled documents. There are many fields and some are sub-documents or arrays. The average document size is 3.8KB. If a large number of companies matches a category, then each query may return a lot of data.
• #31: The query is using an excessive amount of network bandwidth. It appears this is leading to poor performance. What can we do to improve the performance of this query? Attribution: Brent Moore -- https://www.flickr.com/photos/brent_nashville/240072751/
• #32: The default query in MongoDB returns the entire document for all documents matching the filter (predicate) fields. If the application only requires a subset of these fields, we can update the database client to instruct the server to only return the required fields. If the documents in the result set are large and numerous, then this projection of the required fields can significantly reduce network demands and improve performance. The next video shows the workload after changing the client application to add this project. Run an experiment! Use a Projection to Only Return Required Fields. Original query: db.companies.find( { "category_code" : "cleantech" } ) Updated query: db.companies.find( { "category_code" : "cleantech" }, { "name" : 1, "email_address" : 1, "category_code" : 1 } ) Compare Results Real Time Server Stats Network Utilization Operations per Second Decide Whether to Keep the Projection. Attribution: Stephendann -- https://www.instagram.com/p/WT-FPNNwb9/
• #33: The update for this experiment is implemented in the database application. There is no change to the MondoDB server. This video clip shows the workload after the projection has been added to the query.
• #34: With projection, about 10% of the network utilization (14KB/s vs 140KB/s). With projection, about 4 times the operations per second (160 vs 40). If the application does not need the fields and there is measurable improvement, keep the projection. The Voyager aircraft shown here was the first to fly non-stop around the world in 1986. It was piloted by Dick Rutan and Jeana Yeager and flew for 9 days, 3 minutes and 44 seconds. We can assume this vehicle was finely tuned to be as efficient as possible. Voyager Round-the-World On Dec. 23, 1986, the experimental airplane Voyager, piloted by Dick Rutan and Jeana Yeager, completed the first nonstop, around-the-world flight without refueling. It landed safely at Edwards Air Force Base in California.
• #35: In this screenshot, the benchmarking script is updated to run the workload first without the projection, then pause and run the workload with the projection. This side-by-side comparison shows the performance improvement that is sometimes possible using a projection to only return the document fields needed by the application.
• #36: There are actually two different network optimization strategies available. The first strategy uses projection to Only Return Required Fields as seen in the video. The relevant fields are shown as red columns in the diagram. The second strategy limits the documents requested from the server using either a Sort+Limit or a Range Query to return only Relevant Documents. This might be useful when displaying the top-ten items. Make the selection of the top-ten on the server so that other matching documents are not sent back across the network. The relevant documents are shown as blue rows in the diagram on this slide.
• #37: The best indication to use a network optimization strategy is the collection of improved performance results from a relevant, controlled test. This suitcase represents your network. If the network load is a bottleneck, then adjust the application to request less data or provision a bigger network. Indications High Network Usage Result Sets with Many Documents Result Sets with Large, Complex Documents Contraindications Application Requires All Fields and All Documents Attribution: Catherine -- https://www.flickr.com/photos/spiritinme/4246498337/in/photolist-7tfq1z-NLGy-bofSDY-bBaMjT-bBaLEX-bofSXN-ofAuJ7-bBaLJt-bohsvw-bofSJ9-nY7VUu-bofUkj-bofSHf-bBaM1Z-bofTky-bBaL6B-bohsG3-npu3mt-bBaMxR-bofSSf-bBcmkp-bBaLVZ-bofTj1-bBaLGt-bBaL84-bBaLk6-bofT5C-bBcm6K-nFK9b9-bofSWo-bBcnbT-bBaKZx-bBaM8H-bofT45-bofTHd-nY7Rj7-bBaLYr-nFWUDS-6ojcoc-nWbXnN-GZbdUh-nptzuJ-nDVApS-o11Tax-nh4ohL-nFFouF-7WKAdx-nY8EtE-nyyPDN-nptFW1
• #38: This suitcase represents your network. If the network load is a bottleneck, then adjust the application to request less data or provision a bigger network. If your use case requires all documents and all fields. Indications High Network Usage Result Sets with Many Documents Result Sets with Large, Complex Documents Contraindications Application Requires All Fields and All Documents Attribution: Catherine -- https://www.flickr.com/photos/spiritinme/4246498337/in/photolist-7tfq1z-NLGy-bofSDY-bBaMjT-bBaLEX-bofSXN-ofAuJ7-bBaLJt-bohsvw-bofSJ9-nY7VUu-bofUkj-bofSHf-bBaM1Z-bofTky-bBaL6B-bohsG3-npu3mt-bBaMxR-bofSSf-bBcmkp-bBaLVZ-bofTj1-bBaLGt-bBaL84-bBaLk6-bofT5C-bBcm6K-nFK9b9-bofSWo-bBcnbT-bBaKZx-bBaM8H-bofT45-bofTHd-nY7Rj7-bBaLYr-nFWUDS-6ojcoc-nWbXnN-GZbdUh-nptzuJ-nDVApS-o11Tax-nh4ohL-nFFouF-7WKAdx-nY8EtE-nyyPDN-nptFW1 versageek-- https://www.flickr.com/photos/versageek/6098761546/in/photolist-ahVKk9-8KN9iA-ayDpNj-5rwri-9admbX-ayDq5A-Ge6GhA-7EX141-ezV6kP-a7e3zZ-6mjKiN-6wAGCq-5vh9cc-au9WUa-nL5g91-Gv3jvG-5p9iyT-a6CAHV-74J24d-eaUVZC-dBZ9Ye-gfJCHs-FHTJoR-SVvPRY-mB3zD2-5EJhju-boCARN-mkrcYv-2x4pH-AA9sx-24sRPz-bpXyrQ-8T7QME-5UJnXP-4VhE9-6qagst-4vM6at-TXWRb4-9ZKD9Y-bpUKCa-45DQvn-Ge6H5N-FHTJrg-62bKi6-dizVRY-55AyXL-4Eq6Lo-5dnxaG-4rtEyk-SFSVU3
• #39: In our third and final scenario, we will use Compass to find, analyze and accelerate a high-frequency Query.
• #40: Question: Are there any Chicago Fire Soccer fans in the house? The workload for scenario three involves a fanclub database. The application is performing an affinity search to find fans with the same age and membership status. Attribution: goatling -- https://www.flickr.com/photos/mmmchoco/4654658647/
• #41: Our investigation in scenario three will be similar to the first two, however, we are going to dig deeper into the Compass tool. Specifically we will combine the server statistics with the full query shape, including the projection from the application. We will consider how the query is using an existing index and how it may be accelerated even further. Attribution: Travis Goodspeed -- https://www.flickr.com/photos/travisgoodspeed/2768232925/
• #42: After connecting Compass to the database, we can click on the performance tab to view the real time server statistics. There we see that mongodb.fanclub is the collection using the most resources. Reviewing one of the slow queries, we can see the query shape includes a search or filter against age and membership status. There is an index scan in use, but the index only covers the age field of the documents. We can copy the filter portion of the query shape to use in the next video.
• #43: Going to the explain plan tab in the mongodb.fanclub collection, we can paste in the query filter predicate from real time server stats. Next, we will use the new options available in Compass 1.7 to also add the query projection from the client application. Click apply to ask the server how it will execute the query. This time, the server will include the projection part of the query in the results. The explain plan results show almost 24 thousand documents examined and about 16 thousand returned to the client application. Execution time was about 2.5 seconds. The index scan on age executed fairly quickly. Most of the time was consumed in the fetch stage that loaded and scanned 24 thousand documents. Since the membership status field is not covered by the available index, the query scans more documents that it returns to the client.
• #44: The full query is shown at the top of this slide. An index is being used, but it does not include both fields in the filter predicate. So, adding the membership status field to the index is one option. There is already a projection in use to limit the result size.
• #45: A covered query uses an index with the entire query shape included in the index. To achieve this goal, all filter, sort and projection fields will be included in the covering index. This allows the MongoDB server to fulfill the request without fetching any documents from the underlying collection. By responding directly from the index content, performance can be enhanced significantly. For this high frequency query, we will need to add four fields to the index: “age”, “membership_status”, “name”, and “email”.
• #46: For optimal performance, we need to verify that the new Index will fit comfortably in RAM on the database server. Covering indexes can be very large, so this is an important check.
• #47: There are a number of rules governing when an index may be used for a specific query. For this reason, we will use explain plan to verify that the query planner is taking advantage of the new index.
• #48: As with any careful experiment, we must compare the behavior and results before and after our changes in order to reach a solid conclusion. Ideally, we will analyze database performance under all workloads for the application.
• #49: Create a new index covering the age, membership status, name and email fields from the query shape. Do not create indexes directly on databases with production workloads. Creating indexes is very resource intensive and sometimes leads to temporary, but significant, performance degredation. You should use a tool or follow a manual process to add indexes to production servers in a rolling fashion without production workload present. The new index is displayed with zero usage and a size of 33.9 MB. Having build the database server, I know that this index will fit easily in RAM on the server. Returning to the explain plan tab, we past the filter and projection documents in and ask the server to re-evaluate the query with the new index present. From the results, we can see zero documents examined. This means the database server did not need to load any documents from the collection itself. This is possible because all information needed to answer the query is included in the index structure. We see that about 16 thousand index keys were examined and the same number of documents returned to the client application. Additionally, the execution time was reduced to only 81ms and the results show that the query was covered by an index. The database server is able to answer the query by examing zero documents from the full colleciton because the entire contents of the response is available in the covering index.
• #50: With the new index in place, return to the real time server stats and we see increased operations per second as well as high network usage. The fanclub collection is still using the most resources, however, there are fewer slowest operations reported and they are completing faster. If we click on a slow operation, we can see the same query shape. However, the new index is now reported as in use.
• #51: On the left, we have a screenshot of the server stats with only the one "age" field indexed. On the right, the same stats with all four fields indexed to cover the query completely. The maximum operations per second reached 117 for the covered query, but only 12 for the single field. Active read and writes decreased because they are now completing so much faster. Network bandwidth used increase due to answering more queries per second.
• #52: The covered query strategy accelerates a small, critical or frequent query by including the entire query shape in the structure of a custom index. Electromagnetic railgun belonging to the Office of Naval Research. Attribution: Office of Naval Research -- https://www.flickr.com/photos/usnavyresearch/34063946264/in/photostream/
• #53: As with the other strategies, the best signal to use the strategy is positive results from a controlled test against relevant workload. In general, covered queries work best for a small, critical or frequent query that has a relatively small result set. In most cases, it makes the most sense to first test the use of a projection to improve performance before considering a covered query. Attribution: Joe Goldberg -- https://www.flickr.com/photos/goldberg/179511846/in/photostream/
• #54: If the new index size no longer fits in RAM, the covered query strategy will not deliver optimal performance. Additionally, adding another index may have unacceptable performance costs for write operations. The tradeoff between query speed improvement and update speed impacts must be evaluated on a case-by-case basis. Possibility of too many indexes… It is possible for too many indexes to reduce performance, particularly if they do not all fit in RAM. Maybe the workload on a collection has changed and new indexes were added but the indexes supporting the old workload were not removed. In that case, you should review the index usage information on the indexes tab in Compass to determine whether those early indexes are still in use. If they are not, run an experiment where the early indexes are removed and find out whether that improves performance in your environment. Attribution: Joe Goldberg -- https://www.flickr.com/photos/goldberg/179511846/in/photostream/ zoetnet -- https://www.flickr.com/photos/zoetnet/7929093836/
• #55: What did we learn?
• #56: The "The Fastest Code is the Code that Does Not Run" applies to the inefficient query in scenario two. As useful as these heuristics may be, you will be well served to conduct precise tests that include measuring the before and after scenarios for database updates. Examine your workload to find hints as to which Indexes are important. This examination may also uncover requests for data that goes unused by the application.
• #58: The people listed here provided feedback that was instrumental in getting this presentation to a useful state. I almost certainly missed a name or two and I apologize for that in advance. Thank you so much for your help and candid advice. BTW, MongoDB is a great place to work. As you can see from all the people who helped me prepare this presentation, we have a very collaborative environment. We are currently hiring strong engineers for many positions. See me afterward if you are interested in finding out more about working for MongoDB.