SlideShare a Scribd company logo

MongoDB World 2016: Deciphering .explain() Output

MongoDB
MongoDB

The document discusses different explain modes for MongoDB queries and aggregations. It begins with an overview of explain() and query plans, then covers the default "queryPlanner" mode which shows the winning and rejected plans. It also mentions the "executionStats" and "allPlansExecution" modes which provide more runtime statistics. The document aims to help understand how queries and aggregations are executed and troubleshoot performance issues.

Technology
1 of 129
Downloaded 81 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
Deciphering Explain Output Charlie Swanson
Charlie Swanson Software Engineer - Query Team
Goals Of This Talk Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project
Non-goals of This Talk Picking the best indexes for your needs Index Selection It's mostly extends naturally Sharded ClustersSorry, we don't have time… Fixing Problems created by ChangHoon Baek from Noun Project created by David Marioni from Noun Project created by factor[e] design initiative from Noun Project
Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
MongoDB World 2016: Deciphering .explain() Output
MongoDB World 2016: Deciphering .explain() Output
Monitoring the Spread
Our Task MongoDB created by Christopher Holm-Hansen from Noun Project
Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
01: What Is explain()?
MongoDB World 2016: Deciphering .explain() Output
Example Explain Output > db.foo.find({"nFollowers": {$gte: 1000}}).explain() { "queryPlanner" : { "winningPlan" : { "stage" : "FETCH", "inputStage" : { "stage" : "IXSCAN", "keyPattern" : {"nFollowers" : 1}, } }, "rejectedPlans" : [ ] } }
Why Do You Care? Your Application MongoDB created by Mike Ashley from Noun Project
Why Do You Care? Your Application MongoDB Documents Please!
Why Do You Care? Your Application MongoDB Hmm… Let me think about that…
Why Do You Care? Your Application MongoDB Ah! Here are your results!
Why Do You Care? Your Application MongoDB What took you so long?!
Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
Overview of Query Explain • Query Plans • explain() Output Format • Different Verbosities
Query Plans //  db.tweets.ensureIndex({hashtags:  1});   db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN (hashtags) SORT
Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Returns the IDs of tweets with a hashtag of "MDBW16"
Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Converts from an ID to a full BSON object
Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Sorts the BSON objects by the field "nFavorites"
Query Plans db.tweets.find({$or:  [          {hashtag:  {"#MongoDB"}},          {username:  {$in:  [              "@MongoDB",              "@MongoDBEng",                "@MongoDBcareers"          ]}}   ]}); INDEX SCAN {hashtags: 1} OR FETCH INDEX SCAN {username: 1}
Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },     …   }
Explain Output >  db.collection.find(…).explain()   {queryPlanner:  {        winningPlan:  {            stage:  "SORT",            inputStage:  {                stage:  "FETCH",                    inputStage:  {                        stage:  "IXSCAN"                    }                }            }        }   }} FETCH INDEX SCAN {hashtags: 1} SORT
Explain Output >  db.collection.find(…).explain()   {   …                        stage:  "IXSCAN"                        keyPattern:  {hashtags:  1},                        indexBounds:  {                 a:  [  "[4.0,  inf.0]"  ]
                      },                        …  //  Other  index  scan                              //  specific  stats.   …   }} FETCH INDEX SCAN keyPattern: { hashtags: 1 } indexBounds: […] … SORT
Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  […]     },     …   } FETCH INDEX SORT
>  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} Explain Output FETCH INDEX SORT COLLECTI ON SCAN SORT
Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  [                //  Encodes  rejected  plan(s).            ]     },     …   } FETCH INDEX SORT COLLECTI ON SCAN SORT
Applying This Information
Is your query using the index you expect?
Is your query using the index you expect? FETCH SORT ✓ COLLECTION SORT ✗ INDEX SCAN keyPattern: {nFollowers:
Is your query using an index to provide the sort?
Is your query using an index to provide the sort? FETCH INDEX SCAN ✓ ✗FETCH INDEX SCAN SORT
Is your query using an index to provide the sort? ✓SORT_MERGE INDEX SCAN FETCH INDEX SCAN
Is your query using an index to provide the projection?
PROJECTI INDEX SCAN ✓ ✗ PROJECTI INDEX SCAN FETCH Is your query using an index to provide the projection?
Is your query using an index to provide the projection? //  With  index   //  {nFollowers:  1,  username:  1}.   db.users.find(          {nFollowers:  {$gte:  1000}},          {_id:  0,  nFollowers:  1,  username:  1}   ) ✓ PROJECTI INDEX SCAN keyPattern: { nFollowers: 1, username: 1 }
Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? !
Some Questions We Can't Yet Answer • How selective is your index? • What is the most expensive part of your plan? • Why was your winning plan chosen? !
Query Explain Modes • "queryPlanner" (default) • "executionStats" • "allPlansExecution"
Query Explain Modes • "queryPlanner" (default) >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
• "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓
• "executionStats" Query Explain Modes >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
• "executionStats" Query Explain Modes created by Mike Ashley from Noun Project created by Creative Stall from Noun Project
Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {    //  New!       …,       "executionStages":  {…}     }     …   }
Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {      "queryPlanner"  :  {  /*  Same  as  before.  */  },      "executionStats":  {       //  Top-­‐level  stats.       "executionStages":  {              stage:  "SORT",              //  Sort  stats.              inputStage:  {                  //  etc,  etc.
            }          }      }     …   } FETCH SORT INDEX SCAN keyPattern:
Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   { …, "executionStats" : { // Top-level stats. "nReturned" : 390000, "executionTimeMillis" : 4431, "totalKeysExamined" : 390000, "totalDocsExamined" : 390000, "executionStages" : {…} }, } FETCH SORT INDEX SCAN keyPattern:
Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, … "sortPattern" : { "nFollowers" : 1 }, "memUsage" : 20280000, "memLimit" : 33554432, "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } ? Explain Mode: "executionStats" FETCH SORT INDEX SCAN keyPattern:
Execution Stats: works, advanced, etc. • These are all PlanStages • SortStage • FetchStage • IndexScanStage FETCH SORT INDEX SCAN keyPattern:
Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work() FETCH SORT INDEX SCAN keyPattern:
Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work()   • work() returns one of:   • ADVANCED   • NEED_TIME   • IS_EOF FETCH SORT INDEX SCAN keyPattern:
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work()
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work()
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() work()
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() ADVANCED ID
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCED {…}
Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCEDNEED_TIME
Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {"executionStats":  {     "executionStages":  {              stage:  "SORT",              //  Sort  stats,  includes  "works",  "advanced",  …              inputStage:  {                  stage:  "FETCH",                  //  Fetch  stats,  includes  "works",  "advanced",  …                  inputStage:  {
                    //  etc,  etc.                  }
            }          }      }      …   } FETCH … SORT … INDEX SCAN …
• "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓
Applying This Information
How selective is your index?
How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats"  :  {   "nReturned" : 314, "totalKeysExamined" : 2704, // < 8% matched … } FETCH INDEX SCAN keyPattern: {createdDate:
How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 12:03 12:04 12:06 … INDEX SCAN FETCH INDEX SCAN keyPattern: {createdDate:
How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 {createdDate:  12:02,      favorites:  [        "@MongoDB",        "@taylorswift"   ]} 12:03 12:04 12:06 … FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:03 12:02 12:04 12:06 … {createdDate:  12:03,      favorites:  [        "@eliothorowitz",        "@taylorswift"   ]} FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
What's the most expensive part of your plan?
What's the most expensive part of your plan? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH executionTimeMillisEstimate: 431 INDEX SCAN executionTimeMillisEstimate:
What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH works: 2705 advanced: 314 needTime: 2391 //  314/2705  ≈  8% INDEX SCAN
Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? • How selective is your index? • Which part of your plan is the most expensive? !
Some Questions We Can't Yet Answer • Why was your winning plan chosen? !
Why was your winning plan chosen? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   • We had an index on {favorites: 1}, would that have been faster? !
• "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ 😃
Query Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} ❓
Query Explain Mode: "allPlansExecution"
• MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage work() work() work()
• MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage ADVANCED NEED_TIME ADVANCED
• MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage Advances: 78 22 50
• MultiPlanStage::pickBestPlan() Query Planning … MultiPlanStage Advances: 78 22 50
Explain Mode: "allPlansExecution" >  db.tweets.find(…).explain("allPlansExecution")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {       …,       "executionStages":  {…},            "allPlansExecution":  […]    //  New!     }     …   }
Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {…}                           },     "rejectedPlans"  :  [       {…},            …     ],        "executionStats":  {
          "executionStages":  {…}            "allPlansExecution":  [
              {…},                {…},                …            ]        }   }}
Explain Mode: "allPlansExecution" db.tweets.explain("allPlansExecution").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats":  {            "allPlansExecution":  [                    {nReturned:  34,                      executionStages:  {  /*  Index  Scan  on  "favorites"  */  }                    },                    {nReturned:  101,                      executionStages:  {  /*  Index  Scan  on  "createdDate"  */  }                    }            ]     }     …   }
• "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ ✓
One Final Warning explain() output can get VERY large
Compass Shoutout
Compass Shoutout
Compass Shoutout
Query Summary • Query Plans • Explain Format • Different Verbosities
Different Verbosities Diagnostics Query 03 Overview Make explain() exciting Motivation 01 Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks
Aggregation Pipelines • Like PlanStages, input of one stage feeds into the next • db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) MATCH GROUP{…}{…}{…} {…}
db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
Aggregation Pipelines • Aggregation delegates document selection to the query system This is just a query! MATCH GROUP{…}{…}{…} {…}
Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓
Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
Explain Output {      "$cursor"  :  {            "query"  :  {  /*  Same  as  $match  */  },        "queryPlanner"  :  {…}      }   }   GROCURSO
Explain Output {      "$cursor"  :  {            "query"  :  {…},        "queryPlanner"  :  {              "winningPlan":  {…},              "rejectedPlans":  […]          }      }   }   GROCURSO
Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}     ],   }   GROCURSO
Explain Output db.tweets.explain().aggregate([        {$match:  {…}},        {$group:  {…}},        {$sort:  {…}},        {$project:  {…}},   ])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}       {"$sort"  :  {/*  Same  as  $sort  specification  */}}       {"$project"  :  {/*  Same  as  $project  specification  */}}     ],   }   GROCURSO PROJECSOR
Explain Output
Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓ ✓
Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations
Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations Disclaimer: All discussions assume MongoDB 3.2
Aggregation & Indexes 1. Index Provided Sorts 2. Index Provided Projections
Index Provided Sorts //  Compute  the  most  favorited  #MDBW16  tweets.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])  
Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},     ],   }   CURSO
Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [        {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  "sort"  :  {nFavorites:  -­‐1},                  "queryPlanner"  :  {                      "winningPlan":  {  /*  Index  Scan  on  nFavorites  */  }                  }
          }     ],   }   CURSO
Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  //  No  "sort"  field.                  "queryPlanner"  :  {          "winningPlan":  {  /*  No  index  on  nFavorites  */  },      }   },   {$sort:  {nFavorites:  -­‐1}}     ],   }   CURSO SORT
Index Provided Projections //  Compute  the  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])  
Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {…}},     ],   }   CURSO GROUP SORT
Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                "fields"  :  {_id:  0,  username:  1},                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1,  username:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                //  No  "fields"  entry                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations ✓
Other Pipeline Optimizations //  Compute  the  top-­‐10  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])  
Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},     ],   }   CURS GROU SORT*
Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$project:  {…}},        {$limit:  10}   ])   {     "stages"  :  [            …            {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},            {$project:  {…}}     ],   }   CURS GROU SOR PROJE
Aggregation: Summary • Explain output includes query section • Sometimes the query is doing a lot of your work • We don't always execute the stages in the order you specified
Summary Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project
MongoDB World 2016: Deciphering .explain() Output

More Related Content

Similar to MongoDB World 2016: Deciphering .explain() Output (20)

PPTX
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PPTX
Performance Tuning and Optimization
MongoDB
 
PPTX
Webinar: Index Tuning and Evaluation
MongoDB
 
PPTX
Query Optimization in MongoDB
Hamoon Mohammadian Pour
 
PDF
Deciphering Explain Output
MongoDB
 
PDF
Indexing and Query Performance in MongoDB.pdf
Malak Abu Hammad
 
PPTX
MongoDB and Indexes - MUG Denver - 20160329
Douglas Duncan
 
PPTX
#MongoDB indexes
Daniele Graziani
 
PPTX
Webinar: Performance Tuning + Optimization
MongoDB
 
PDF
Mongoseattle indexing-2010-07-27
MongoDB
 
PDF
Indexing and Query Optimizer (Richard Kreuter)
MongoDB
 
PDF
Introduction to-mongo db-execution-plan-optimizer-final
M Malai
 
PDF
Introduction to Mongodb execution plan and optimizer
Mydbops
 
PDF
Mdb dn 2016_06_query_primer
Daniel M. Farrell
 
PPTX
Automated Slow Query Analysis: Dex the Index Robot
MongoDB
 
PPTX
Running Production MongoDB Lightning Talk
chrisckchang
 
PPTX
MongoDB.local Sydney 2019: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PDF
Query planner
Miguel Angel Nieto
 
PDF
Mongo indexes
Mehmet Çetin
 
PDF
Indexing and Performance Tuning
MongoDB
 
MongoDB.local DC 2018: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
Performance Tuning and Optimization
MongoDB
 
Webinar: Index Tuning and Evaluation
MongoDB
 
Query Optimization in MongoDB
Hamoon Mohammadian Pour
 
Deciphering Explain Output
MongoDB
 
Indexing and Query Performance in MongoDB.pdf
Malak Abu Hammad
 
MongoDB and Indexes - MUG Denver - 20160329
Douglas Duncan
 
#MongoDB indexes
Daniele Graziani
 
Webinar: Performance Tuning + Optimization
MongoDB
 
Mongoseattle indexing-2010-07-27
MongoDB
 
Indexing and Query Optimizer (Richard Kreuter)
MongoDB
 
Introduction to-mongo db-execution-plan-optimizer-final
M Malai
 
Introduction to Mongodb execution plan and optimizer
Mydbops
 
Mdb dn 2016_06_query_primer
Daniel M. Farrell
 
Automated Slow Query Analysis: Dex the Index Robot
MongoDB
 
Running Production MongoDB Lightning Talk
chrisckchang
 
MongoDB.local Sydney 2019: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
Query planner
Miguel Angel Nieto
 
Mongo indexes
Mehmet Çetin
 
Indexing and Performance Tuning
MongoDB
 

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
Ad

Recently uploaded (20)

PDF
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
PDF
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PPTX
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
PDF
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Ad

MongoDB World 2016: Deciphering .explain() Output

  • 3. Goals Of This Talk Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project
  • 4. Non-goals of This Talk Picking the best indexes for your needs Index Selection It's mostly extends naturally Sharded ClustersSorry, we don't have time… Fixing Problems created by ChangHoon Baek from Noun Project created by David Marioni from Noun Project created by factor[e] design initiative from Noun Project
  • 5. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  • 6. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  • 10. Our Task MongoDB created by Christopher Holm-Hansen from Noun Project
  • 11. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  • 12. 01: What Is explain()?
  • 14. Example Explain Output > db.foo.find({"nFollowers": {$gte: 1000}}).explain() { "queryPlanner" : { "winningPlan" : { "stage" : "FETCH", "inputStage" : { "stage" : "IXSCAN", "keyPattern" : {"nFollowers" : 1}, } }, "rejectedPlans" : [ ] } }
  • 15. Why Do You Care? Your Application MongoDB created by Mike Ashley from Noun Project
  • 16. Why Do You Care? Your Application MongoDB Documents Please!
  • 17. Why Do You Care? Your Application MongoDB Hmm… Let me think about that…
  • 18. Why Do You Care? Your Application MongoDB Ah! Here are your results!
  • 19. Why Do You Care? Your Application MongoDB What took you so long?!
  • 20. Overview Make explain() exciting Motivation 01 Different Verbosities Diagnostics Query 03Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  • 21. Overview of Query Explain • Query Plans • explain() Output Format • Different Verbosities
  • 22. Query Plans //  db.tweets.ensureIndex({hashtags:  1});   db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN (hashtags) SORT
  • 23. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Returns the IDs of tweets with a hashtag of "MDBW16"
  • 24. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Converts from an ID to a full BSON object
  • 25. Query Plans db.tweets.find({hashtags:  "#MDBW16"})        .sort({nFavorites:  -­‐1}); FETCH INDEX SCAN {hashtags: 1} SORT Sorts the BSON objects by the field "nFavorites"
  • 26. Query Plans db.tweets.find({$or:  [          {hashtag:  {"#MongoDB"}},          {username:  {$in:  [              "@MongoDB",              "@MongoDBEng",                "@MongoDBcareers"          ]}}   ]}); INDEX SCAN {hashtags: 1} OR FETCH INDEX SCAN {username: 1}
  • 27. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },     …   }
  • 28. Explain Output >  db.collection.find(…).explain()   {queryPlanner:  {        winningPlan:  {            stage:  "SORT",            inputStage:  {                stage:  "FETCH",                    inputStage:  {                        stage:  "IXSCAN"                    }                }            }        }   }} FETCH INDEX SCAN {hashtags: 1} SORT
  • 29. Explain Output >  db.collection.find(…).explain()   {   …                        stage:  "IXSCAN"                        keyPattern:  {hashtags:  1},                        indexBounds:  {                 a:  [  "[4.0,  inf.0]"  ]
                      },                        …  //  Other  index  scan                              //  specific  stats.   …   }} FETCH INDEX SCAN keyPattern: { hashtags: 1 } indexBounds: […] … SORT
  • 30. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  […]     },     …   } FETCH INDEX SORT
  • 31. >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} Explain Output FETCH INDEX SORT COLLECTI ON SCAN SORT
  • 32. Explain Output >  db.collection.find(…).explain()   {     "queryPlanner"  :  {       …       "winningPlan"  :  {                //  Encodes  selected  plan.          },       "rejectedPlans"  :  [                //  Encodes  rejected  plan(s).            ]     },     …   } FETCH INDEX SORT COLLECTI ON SCAN SORT
  • 34. Is your query using the index you expect?
  • 35. Is your query using the index you expect? FETCH SORT ✓ COLLECTION SORT ✗ INDEX SCAN keyPattern: {nFollowers:
  • 36. Is your query using an index to provide the sort?
  • 37. Is your query using an index to provide the sort? FETCH INDEX SCAN ✓ ✗FETCH INDEX SCAN SORT
  • 38. Is your query using an index to provide the sort? ✓SORT_MERGE INDEX SCAN FETCH INDEX SCAN
  • 39. Is your query using an index to provide the projection?
  • 40. PROJECTI INDEX SCAN ✓ ✗ PROJECTI INDEX SCAN FETCH Is your query using an index to provide the projection?
  • 41. Is your query using an index to provide the projection? //  With  index   //  {nFollowers:  1,  username:  1}.   db.users.find(          {nFollowers:  {$gte:  1000}},          {_id:  0,  nFollowers:  1,  username:  1}   ) ✓ PROJECTI INDEX SCAN keyPattern: { nFollowers: 1, username: 1 }
  • 42. Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? !
  • 43. Some Questions We Can't Yet Answer • How selective is your index? • What is the most expensive part of your plan? • Why was your winning plan chosen? !
  • 44. Query Explain Modes • "queryPlanner" (default) • "executionStats" • "allPlansExecution"
  • 45. Query Explain Modes • "queryPlanner" (default) >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
  • 46. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓
  • 47. • "executionStats" Query Explain Modes >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }}
  • 48. • "executionStats" Query Explain Modes created by Mike Ashley from Noun Project created by Creative Stall from Noun Project
  • 49. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {    //  New!       …,       "executionStages":  {…}     }     …   }
  • 50. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {      "queryPlanner"  :  {  /*  Same  as  before.  */  },      "executionStats":  {       //  Top-­‐level  stats.       "executionStages":  {              stage:  "SORT",              //  Sort  stats.              inputStage:  {                  //  etc,  etc.
            }          }      }     …   } FETCH SORT INDEX SCAN keyPattern:
  • 51. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   { …, "executionStats" : { // Top-level stats. "nReturned" : 390000, "executionTimeMillis" : 4431, "totalKeysExamined" : 390000, "totalDocsExamined" : 390000, "executionStages" : {…} }, } FETCH SORT INDEX SCAN keyPattern:
  • 52. Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, … "sortPattern" : { "nFollowers" : 1 }, "memUsage" : 20280000, "memLimit" : 33554432, "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
  • 53. db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } ? Explain Mode: "executionStats" FETCH SORT INDEX SCAN keyPattern:
  • 54. Execution Stats: works, advanced, etc. • These are all PlanStages • SortStage • FetchStage • IndexScanStage FETCH SORT INDEX SCAN keyPattern:
  • 55. Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work() FETCH SORT INDEX SCAN keyPattern:
  • 56. Execution Stats: works, advanced, etc. • These are all PlanStages • Each PlanStage implements work()   • work() returns one of:   • ADVANCED   • NEED_TIME   • IS_EOF FETCH SORT INDEX SCAN keyPattern:
  • 57. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work()
  • 58. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work()
  • 59. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() work()
  • 60. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() work() ADVANCED ID
  • 61. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCED {…}
  • 62. Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: work() ADVANCED ADVANCEDNEED_TIME
  • 63. Explain Mode: "executionStats" db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } FETCH SORT INDEX SCAN keyPattern:
  • 64. Explain Mode: "executionStats" >  db.tweets.find(…).explain("executionStats")   {"executionStats":  {     "executionStages":  {              stage:  "SORT",              //  Sort  stats,  includes  "works",  "advanced",  …              inputStage:  {                  stage:  "FETCH",                  //  Fetch  stats,  includes  "works",  "advanced",  …                  inputStage:  {
                    //  etc,  etc.                  }
            }          }      }      …   } FETCH … SORT … INDEX SCAN …
  • 65. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓
  • 67. How selective is your index?
  • 68. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats"  :  {   "nReturned" : 314, "totalKeysExamined" : 2704, // < 8% matched … } FETCH INDEX SCAN keyPattern: {createdDate:
  • 69. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 12:03 12:04 12:06 … INDEX SCAN FETCH INDEX SCAN keyPattern: {createdDate:
  • 70. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:02 {createdDate:  12:02,      favorites:  [        "@MongoDB",        "@taylorswift"   ]} 12:03 12:04 12:06 … FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  • 71. How selective is your index? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   12:03 12:02 12:04 12:06 … {createdDate:  12:03,      favorites:  [        "@eliothorowitz",        "@taylorswift"   ]} FETCH FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  • 72. What's the most expensive part of your plan?
  • 73. What's the most expensive part of your plan? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   FETCH filter: { favorites: "@eliothorowitz" INDEX SCAN keyPattern: {createdDate:
  • 74. What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH executionTimeMillisEstimate: 431 INDEX SCAN executionTimeMillisEstimate:
  • 75. What's the most expensive part of your plan? db.tweets.explain("executionStats").find(…)   FETCH works: 2705 advanced: 314 needTime: 2391 //  314/2705  ≈  8% INDEX SCAN
  • 76. Some Questions We Can Answer • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? • How selective is your index? • Which part of your plan is the most expensive? !
  • 77. Some Questions We Can't Yet Answer • Why was your winning plan chosen? !
  • 78. Why was your winning plan chosen? db.tweets.explain("executionStats").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   • We had an index on {favorites: 1}, would that have been faster? !
  • 79. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ 😃
  • 80. Query Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                           },     "rejectedPlans"  :  [       {"stage"  :  "SORT",         "inputStage"  :  {…}       }}                …     ]   }} ❓
  • 81. Query Explain Mode: "allPlansExecution"
  • 86. Explain Mode: "allPlansExecution" >  db.tweets.find(…).explain("allPlansExecution")   {     "queryPlanner"  :  {       …       "winningPlan"  :  {…},       "rejectedPlans"  :  […]     },      "executionStats":  {       …,       "executionStages":  {…},            "allPlansExecution":  […]    //  New!     }     …   }
  • 87. Explain Mode: "allPlansExecution" >  db.collection.find(…).explain()   {"queryPlanner"  :  {     "winningPlan"  :  {…}                           },     "rejectedPlans"  :  [       {…},            …     ],        "executionStats":  {
          "executionStages":  {…}            "allPlansExecution":  [
              {…},                {…},                …            ]        }   }}
  • 88. Explain Mode: "allPlansExecution" db.tweets.explain("allPlansExecution").find({      createdDate:  {$gte:  <today>},      favorites:  "@eliothorowitz"   })   {      "executionStats":  {            "allPlansExecution":  [                    {nReturned:  34,                      executionStages:  {  /*  Index  Scan  on  "favorites"  */  }                    },                    {nReturned:  101,                      executionStages:  {  /*  Index  Scan  on  "createdDate"  */  }                    }            ]     }     …   }
  • 89. • "queryPlanner" (default) • "executionStats" • "allPlansExecution" Query Explain Modes ✓ ✓ ✓
  • 90. One Final Warning explain() output can get VERY large
  • 94. Query Summary • Query Plans • Explain Format • Different Verbosities
  • 95. Different Verbosities Diagnostics Query 03 Overview Make explain() exciting Motivation 01 Why Do You Care? What is explain()? 02 Similarities & Differences Aggregation 04
  • 96. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks
  • 97. Aggregation Pipelines • Like PlanStages, input of one stage feeds into the next • db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) MATCH GROUP{…}{…}{…} {…}
  • 98. db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
  • 99. db.tweets.aggregate([
  {$match:  {hashtag:  "#MDBW16}},
  {$group:  {
      _id:  "$author",
      nTweets:  {$sum:  1}
  }}
 ]) Collection Scan?!? Are you kidding me?? Aggregation Pipelines MATCH GROUP{…}{…}{…} {…}
  • 100. Aggregation Pipelines • Aggregation delegates document selection to the query system This is just a query! MATCH GROUP{…}{…}{…} {…}
  • 101. Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
  • 102. Aggregation Pipelines • Aggregation delegates document selection to the query system GROU {…}CURSOR
  • 103. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓
  • 104. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
  • 105. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}}     ],   }   GROCURSO
  • 106. Explain Output {      "$cursor"  :  {            "query"  :  {  /*  Same  as  $match  */  },        "queryPlanner"  :  {…}      }   }   GROCURSO
  • 107. Explain Output {      "$cursor"  :  {            "query"  :  {…},        "queryPlanner"  :  {              "winningPlan":  {…},              "rejectedPlans":  […]          }      }   }   GROCURSO
  • 108. Explain Output • db.tweets.explain().aggregate([{$match:  {…}},  {$group:  {…}}])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}     ],   }   GROCURSO
  • 109. Explain Output db.tweets.explain().aggregate([        {$match:  {…}},        {$group:  {…}},        {$sort:  {…}},        {$project:  {…}},   ])   {     "stages"  :  [       {"$cursor"  :  {query:  {…},  queryPlanner:  {…}}},       {"$group"  :  {/*  Same  as  $group  specification  */}}       {"$sort"  :  {/*  Same  as  $sort  specification  */}}       {"$project"  :  {/*  Same  as  $project  specification  */}}     ],   }   GROCURSO PROJECSOR
  • 111. Aggregation: Overview • Aggregation Pipelines • Explain Output • Tips and Tricks ✓ ✓
  • 112. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations
  • 113. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations Disclaimer: All discussions assume MongoDB 3.2
  • 114. Aggregation & Indexes 1. Index Provided Sorts 2. Index Provided Projections
  • 115. Index Provided Sorts //  Compute  the  most  favorited  #MDBW16  tweets.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])  
  • 116. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},     ],   }   CURSO
  • 117. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [        {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  "sort"  :  {nFavorites:  -­‐1},                  "queryPlanner"  :  {                      "winningPlan":  {  /*  Index  Scan  on  nFavorites  */  }                  }
          }     ],   }   CURSO
  • 118. Index Provided Sorts db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},
      {$sort:  {nFavorites:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {                  "query"  :  {hashtag:  "#MDBW16"},                  //  No  "sort"  field.                  "queryPlanner"  :  {          "winningPlan":  {  /*  No  index  on  nFavorites  */  },      }   },   {$sort:  {nFavorites:  -­‐1}}     ],   }   CURSO SORT
  • 119. Index Provided Projections //  Compute  the  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])  
  • 120. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {…}},     ],   }   CURSO GROUP SORT
  • 121. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                "fields"  :  {_id:  0,  username:  1},                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1,  username:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
  • 122. Index Provided Projections db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},   ])   {     "stages"  :  [       "$cursor"  :  {                "query"  :  {hashtag:  "#MDBW16"},                //  No  "fields"  entry                "queryPlanner"  :  {          "winningPlan":  {  /*  Index  Scan  on  {hashtag:  1}  */  },   },
 {$group:  {…}},   {$sort:  {…}}     ],   } CURS GROU SORT
  • 123. Tips & Tricks: Overview 1. Making the most of your indexes 2. Pipeline Optimizations ✓
  • 124. Other Pipeline Optimizations //  Compute  the  top-­‐10  users  who  tweeted  about  #MDBW16  most  often.   db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])  
  • 125. Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$limit:  10}   ])   {     "stages"  :  [       {"$cursor"  :  {…}},       {"$group"  :  {…}},       {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},     ],   }   CURS GROU SORT*
  • 126. Other Pipeline Optimizations db.tweets.explain().aggregate([        {$match:  {hashtag:  "#MDBW16"},        {$group:  {_id:  "$username",  count:  {$sum:  1}}},        {$sort:  {count:  -­‐1}},        {$project:  {…}},        {$limit:  10}   ])   {     "stages"  :  [            …            {"$sort"  :  {sortKey:  {count:  -­‐1},  limit:  NumberLong(10)}},            {$project:  {…}}     ],   }   CURS GROU SOR PROJE
  • 127. Aggregation: Summary • Explain output includes query section • Sometimes the query is doing a lot of your work • We don't always execute the stages in the order you specified
  • 128. Summary Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices created by Francielly Constantin Senra from Noun Project created by Ramon Sandino from Noun Project