SlideShare a Scribd company logo

Tips and Tricks for Avoiding Common Query Pitfalls

MongoDB
MongoDB

The document discusses some common query pitfalls in MongoDB and provides recommendations to avoid performance issues. It addresses the issues of using blocking stages like $sort, and how the $or operator cannot use a single index effectively. For $sort, creating a composite index that includes the sort key can allow sorting to occur without blocking. For $or queries, creating separate indexes for each clause of the $or is recommended to allow the query optimizer to use them independently.

Technology
1 of 119
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
Tips and Tricks for Avoiding Common Query Pitfalls
Tips and Tricks for Avoiding Common Query Pitfalls Adam Schwartz Technical Services, MongoDB
Adam Schwartz Manager, Technical Services Engineering, EMEA MongoDB adam.schwartz@mongodb.com
The Power of Query Optimization Query tuning results in: ● Improved performance ● Reduced resource utilization Tuning commonly leads to: ● Improved stability and predictability ● Smaller hardware footprint It is not uncommon to observe efficiency improvements $gte 99%
Roadmap ● Three ideas to be mindful of: – Use of Blocking Stages – Use of the $or operator – Using Case-Insensitivity
Game of Drones A Brand New Battle Royale Game - Launching May 15th Tales of pre-launch testing Acme Games Introduces ...
Getting Ready for Launch! MongoDB Certified Developer and DBA Stakeholders Others at Acme Games
Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros
Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros Indexes support the efficient execution of queries in MongoDB
Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros SueAce … …Bob Indexes support the efficient execution of queries in MongoDB
Stakeholder Concerns ● App Being stress tested ● Concern over the current performance
Stakeholder Concern #1 Developers created index: db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find({gamerTag: "Ace"}).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 })
Stakeholder Concern #1 Developers created index: db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 }) "Clearly MongoDB is not webscale!"
Avoiding Query Pitfalls: Blocking Operations
Blocking Operation ● Formally: "An operation which must process all input before it can begin to produce any output" ● Opposite of the often desirable “fully pipelined” plan which can stream results back as soon as they are found ● Blocking is commonly observed when a sort is added to query
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Sorting with Blocking
Indexes that support sorting db.games.createIndex({gamerTag: 1, score: -1})
Sorting without Blocking
Sorting without Blocking
Sorting without Blocking
Sorting without Blocking
Blocking Stages $sort In aggregation and find $group $bucket $count $facet Are there any other blocking operations?
Aggregation Pipeline Stages
Working with Blocking Stages For sorting: Add a supporting index Worth the overhead in almost all circumstances For other stages: Do you need the blocking stage? Offload to secondary member
Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) "Clearly MongoDB is not webscale!"
Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 })
Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 }) "That’ll work great!"
Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] })
Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) We just created an index with both those fields… Can it be used for the $or?
Avoiding Query Pitfalls: $or
Query on games: db.games.find({ $and : [ {gamerTag:"Ace"}, {score:{$gt:9000}} ] }) Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "Ace", score: 500}, {gamerTag: "Bob", score: 9500}, {gamerTag: "Bob", score: 500} $and Example
Groups of Documents score: {$gt: 9000}gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 }
$and Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
$and Venn Diagram (logical) { gamerTag: "Bob", score: 500 } db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
$and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
$and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
$and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
$and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
Query on games: db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt:9000}} ] }) Matching games: {gamerTag: "Ace", score: 9500}, {gamerTag: "Ace", score: 500}, {gamerTag: "Bob", score: 9500} Non-matching games: {gamerTag: "Bob", score: 500} $or Example
$or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
$or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
$or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
$or Venn Diagram (logical) score: {$gt: 9000}gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) { gamerTag: "Bob", score: 9500 } score: {$gt: 9000}
$or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
$or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]} Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ]}
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]} Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ]} So is there anything we can do to improve the performance of this query?
Recommendations Use multiple indexes! db.games.createIndex({gamerTag: 1}) db.games.createIndex({score: 1})
$or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
$or : multiple Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob {gamerTag:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
$or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
Recommendations Use multiple indexes! db.games.createIndex({gamerTag: 1}) db.games.createIndex({score: 1})We already have the {gamerTag:1, score:-1} index. Do we need both of these new indexes?
Recommendations Use multiple indexes! db.games.createIndex({gamerTag: 1}) db.games.createIndex({score: 1}) db.games.createIndex({ gamerTag: 1, score:-1 }) Works with sorting Generate a SORT_MERGE plan
Stakeholder Concern #2 db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt: 9000}} ] }) Having the right index is critical
Stakeholder Concern #2 db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt: 9000}} ] }) Having the right index is critical "Super!!"
Stakeholder Concern #3 “Wait wait wait, we can’t even FIND the gamers!” A basic search on gamerTag takes several seconds already: db.games.find({gamerTag: /^Ace$/i}) “This query is SLOWER with the index than it is without it!”
Avoiding Query Pitfalls: Case Insensitive Queries
db.games.find({ gamerTag: /^Ace$/ }) //equivalent to db.games.find({ gamerTag: "Ace" }) Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "ACE", score: 500}, {gamerTag: "aCe", score: 9500}, {gamerTag: "ace", score: 0}, {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Sensitive
Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "ACE", score: 500}, {gamerTag: "aCe", score: 9500}, {gamerTag: "ace", score: 0}, {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Sensitive "indexBounds" : { "gamerTag" : [ "["Ace", "Acf")", "[/^Ace$/, /^Ace$/]" ] } ace aCe acxyz Ace Ace mdb ACE Bob
db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: "^Ace$", $options: "i" } }) //equivalent to db.games.find({ gamerTag: "Ace"}) .collation({locale:'en', strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Insensitive
db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: "^Ace$", $options: "i" } }) //equivalent to db.games.find({ gamerTag: "Ace"}) .collation({locale:'en', strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Would a $text search behave the same? Case Insensitive
Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Insensitive "indexBounds" : { "gamerTag" : [ "["", {})", "[/^Ace$/i, /^Ace$/i]" ] } ace aCe acxyz Ace Ace mdb ACE Bob
Recommendations Case insensitive index! Collations available since 3.4
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex({gamerTag: 1},{collation:{locale:'en',strength:2}}) > db.games.find( {gamerTag: "Ace"}).collation({locale: 'en',strength:2}) { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5d"), "gamerTag" : "Ace", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5e"), "gamerTag" : "ACE", "score" : 500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5f"), "gamerTag" : "aCe", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b60"), "gamerTag" : "ace", "score" : 0 }
Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex({gamerTag: 1},{collation:{locale:'en',strength:2}}) or Store a transformed (eg toLower()) copy of the field
Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2})
Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2})
Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2}) "Wow, MongoDB can do anything!!"
Avoiding Query Pitfalls: Summary
Work Smartly • Understand the business logic • Index appropriately • Is it the right index for the query? • Be cognizant of 3 issues: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
Work Smartly • Understand the business logic • Index appropriately • Is it the right index for the query? • Be cognizant of 3 issues: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
Before/After Metric Comparison
Countdown to Release of GameOfDrones Powered by an optimized MongoDB environment, GameOfDrones is sure to be a hit!
Become the HERO!
Questions
Thank You!
Tips and Tricks for Avoiding Common Query Pitfalls

More Related Content

What's hot (16)

PDF
The Ring programming language version 1.5.3 book - Part 62 of 184
Mahmoud Samir Fayed
 
PDF
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
PDF
SCBCN17 - El camino hacia la programación declarativa
Gerard Madorell
 
PDF
WIA 2019 - Unearth the Journey of Implementing Vision Based Deep Learning Sol...
Women in Analytics Conference
 
PPT
Heroku Waza 2013 Lessons Learned
Simon Bagreev
 
PDF
WordCamp Bristol 2019 - WordPress custom theme building
Jonny Allbut
 
PDF
The Ring programming language version 1.5.4 book - Part 51 of 185
Mahmoud Samir Fayed
 
PPTX
Symfony 1, mi viejo amigo
Jose Antonio Pio
 
DOCX
Web lab programs
SRINIVASUNIVERSITYEN
 
PDF
What Have The Properties Ever Done For Us
Miklós Martin
 
PDF
Javascript quiz
DM's College, Assagao Goa
 
PPT
MySQLConf2009: Taking ActiveRecord to the Next Level
Blythe Dunham
 
PDF
Aggregation Pipeline Power++: MongoDB 4.2 파이프 라인 쿼리, 업데이트 및 구체화된 뷰 소개 [MongoDB]
MongoDB
 
PDF
CSSO – compress CSS (english version)
Roman Dvornov
 
PDF
The Ring programming language version 1.10 book - Part 62 of 212
Mahmoud Samir Fayed
 
PDF
PyCon Siberia 2016. Не доверяйте тестам!
Ivan Tsyganov
 
The Ring programming language version 1.5.3 book - Part 62 of 184
Mahmoud Samir Fayed
 
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
SCBCN17 - El camino hacia la programación declarativa
Gerard Madorell
 
WIA 2019 - Unearth the Journey of Implementing Vision Based Deep Learning Sol...
Women in Analytics Conference
 
Heroku Waza 2013 Lessons Learned
Simon Bagreev
 
WordCamp Bristol 2019 - WordPress custom theme building
Jonny Allbut
 
The Ring programming language version 1.5.4 book - Part 51 of 185
Mahmoud Samir Fayed
 
Symfony 1, mi viejo amigo
Jose Antonio Pio
 
Web lab programs
SRINIVASUNIVERSITYEN
 
What Have The Properties Ever Done For Us
Miklós Martin
 
Javascript quiz
DM's College, Assagao Goa
 
MySQLConf2009: Taking ActiveRecord to the Next Level
Blythe Dunham
 
Aggregation Pipeline Power++: MongoDB 4.2 파이프 라인 쿼리, 업데이트 및 구체화된 뷰 소개 [MongoDB]
MongoDB
 
CSSO – compress CSS (english version)
Roman Dvornov
 
The Ring programming language version 1.10 book - Part 62 of 212
Mahmoud Samir Fayed
 
PyCon Siberia 2016. Не доверяйте тестам!
Ivan Tsyganov
 

Similar to Tips and Tricks for Avoiding Common Query Pitfalls (20)

PPTX
MongoDB.local Dallas 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PPTX
MongoDB.local Sydney 2019: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PPTX
MongoDB.local Seattle 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
PDF
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PDF
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
Lisa Roth, PMP
 
PDF
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
PDF
MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PDF
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
PPTX
Mongo db mug_2012-02-07
Will Button
 
PDF
SWP - A Generic Language Parser
kamaelian
 
PDF
An Introduction to Tinkerpop
Takahiro Inoue
 
PDF
MongoDB World 2019: How to Keep an Average API Response Time Less than 5ms wi...
MongoDB
 
PPTX
Mongo Baseball .NET
David Hoerster
 
PDF
Mongo indexes
Mehmet Çetin
 
PPTX
Alexander Mostovenko "Modern approach to localization in javascript with the ...
OdessaJS Conf
 
PPT
Graph Databases
Josh Adell
 
PDF
A3 sec -_regular_expressions
a3sec
 
PPTX
Microsoft NERD Talk - R and Tableau - 2-4-2013
Tanya Cashorali
 
MongoDB.local Dallas 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
MongoDB.local Sydney 2019: Tips and Tricks for Avoiding Common Query Pitfalls
MongoDB
 
MongoDB.local Seattle 2019: Tips & Tricks for Avoiding Common Query Pitfalls
MongoDB
 
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
MongoDB .local London 2019: Tips and Tricks++ for Querying and Indexing MongoDB
Lisa Roth, PMP
 
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
MongoDB .local Munich 2019: Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
MongoDB
 
Mongo db mug_2012-02-07
Will Button
 
SWP - A Generic Language Parser
kamaelian
 
An Introduction to Tinkerpop
Takahiro Inoue
 
MongoDB World 2019: How to Keep an Average API Response Time Less than 5ms wi...
MongoDB
 
Mongo Baseball .NET
David Hoerster
 
Mongo indexes
Mehmet Çetin
 
Alexander Mostovenko "Modern approach to localization in javascript with the ...
OdessaJS Conf
 
Graph Databases
Josh Adell
 
A3 sec -_regular_expressions
a3sec
 
Microsoft NERD Talk - R and Tableau - 2-4-2013
Tanya Cashorali
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
Ad

Recently uploaded (20)

PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
July Patch Tuesday
Ivanti
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PDF
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
PDF
Advancing WebDriver BiDi support in WebKit
Igalia
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
July Patch Tuesday
Ivanti
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
Advancing WebDriver BiDi support in WebKit
Igalia
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 

Tips and Tricks for Avoiding Common Query Pitfalls

  • 2. Tips and Tricks for Avoiding Common Query Pitfalls Adam Schwartz Technical Services, MongoDB
  • 3. Adam Schwartz Manager, Technical Services Engineering, EMEA MongoDB [email protected]
  • 4. The Power of Query Optimization Query tuning results in: ● Improved performance ● Reduced resource utilization Tuning commonly leads to: ● Improved stability and predictability ● Smaller hardware footprint It is not uncommon to observe efficiency improvements $gte 99%
  • 5. Roadmap ● Three ideas to be mindful of: – Use of Blocking Stages – Use of the $or operator – Using Case-Insensitivity
  • 6. Game of Drones A Brand New Battle Royale Game - Launching May 15th Tales of pre-launch testing Acme Games Introduces ...
  • 7. Getting Ready for Launch! MongoDB Certified Developer and DBA Stakeholders Others at Acme Games
  • 8. Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros
  • 9. Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros Indexes support the efficient execution of queries in MongoDB
  • 10. Stakeholder Concerns ● Game nearly complete ● Acme has learned a lot from the MongoDB Certified Pros SueAce … …Bob Indexes support the efficient execution of queries in MongoDB
  • 11. Stakeholder Concerns ● App Being stress tested ● Concern over the current performance
  • 12. Stakeholder Concern #1 Developers created index: db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find({gamerTag: "Ace"}).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 })
  • 13. Stakeholder Concern #1 Developers created index: db.games.createIndex({ gamerTag: 1 }) This query takes several seconds to execute: db.games.find( { gamerTag: "Ace" } ).sort({score:-1}) Adding the index on score does not help! db.games.createIndex({ score: -1 }) "Clearly MongoDB is not webscale!"
  • 15. Blocking Operation ● Formally: "An operation which must process all input before it can begin to produce any output" ● Opposite of the often desirable “fully pipelined” plan which can stream results back as soon as they are found ● Blocking is commonly observed when a sort is added to query
  • 34. Indexes that support sorting db.games.createIndex({gamerTag: 1, score: -1})
  • 39. Blocking Stages $sort In aggregation and find $group $bucket $count $facet Are there any other blocking operations?
  • 41. Working with Blocking Stages For sorting: Add a supporting index Worth the overhead in almost all circumstances For other stages: Do you need the blocking stage? Offload to secondary member
  • 42. Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) "Clearly MongoDB is not webscale!"
  • 43. Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 })
  • 44. Stakeholder Concern #1 Performance of db.games.find({gamerTag: "Ace"}).sort({score:-1}) db.games.createIndex({ gamerTag: 1, score:-1 }) "That’ll work great!"
  • 45. Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] })
  • 46. Stakeholder Concern #2 The $and version of a query returns quickly: db.games.find({ $and : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) But the $or version is slow: db.games.find({ $or : [ { gamerTag:"Ace" }, { score:{$gt:9000}} ] }) We just created an index with both those fields… Can it be used for the $or?
  • 48. Query on games: db.games.find({ $and : [ {gamerTag:"Ace"}, {score:{$gt:9000}} ] }) Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "Ace", score: 500}, {gamerTag: "Bob", score: 9500}, {gamerTag: "Bob", score: 500} $and Example
  • 49. Groups of Documents score: {$gt: 9000}gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 }
  • 50. $and Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
  • 51. $and Venn Diagram (logical) { gamerTag: "Bob", score: 500 } db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 52. $and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 53. $and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 54. $and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 55. $and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 56. $and Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 57. $and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
  • 58. $and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
  • 59. $and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
  • 60. $and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
  • 61. $and Venn Diagram (logical) db.games.find( { $and : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000.0)" ] }
  • 62. Query on games: db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt:9000}} ] }) Matching games: {gamerTag: "Ace", score: 9500}, {gamerTag: "Ace", score: 500}, {gamerTag: "Bob", score: 9500} Non-matching games: {gamerTag: "Bob", score: 500} $or Example
  • 63. $or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
  • 64. $or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
  • 65. $or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
  • 66. $or Venn Diagram (logical) score: {$gt: 9000}gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) { gamerTag: "Bob", score: 9500 } score: {$gt: 9000}
  • 67. $or Venn Diagram (logical) gamerTag: "Ace" { gamerTag: "Ace", score: 9500 } { gamerTag: "Ace", score: 500 } { gamerTag: "Bob", score: 9500 } { gamerTag: "Bob", score: 500 } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) score: {$gt: 9000}
  • 68. $or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 69. $or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 70. $or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 71. $or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 72. $or Venn Diagram (logical) gamerTag: "Ace" score: {$gt: 9000} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
  • 73. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
  • 74. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
  • 75. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
  • 76. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
  • 77. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]}
  • 78. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]} Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ]}
  • 79. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500 Expected Index Bounds: "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]" ], "score" : [ "[inf.0, 9000)" ]} Actual (Hinted) Index Bounds: "indexBounds" : { "gamerTag" : [ "[MinKey, MaxKey]" ], "score" : [ "[MaxKey, MinKey]" ]} So is there anything we can do to improve the performance of this query?
  • 81. $or : single Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) Ace Bob {gamerTag:1 , score:-1} 500 9500 500 9500
  • 82. $or : multiple Index visualization db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 83. $or : multiple Index visualization Ace db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 84. $or : multiple Index visualization Ace Bob db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 85. $or : multiple Index visualization Ace Bob {gamerTag:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 86. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 87. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 88. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] } db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]})
  • 89. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
  • 90. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
  • 91. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
  • 92. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
  • 93. $or : multiple Index visualization Ace Bob {gamerTag:1} 500 9500 {score:1} db.games.find( { $or : [ { gamerTag: "Ace" }, { score: {$gt: 9000}} ]}) "indexBounds" : { "gamerTag" : [ "["Ace", "Ace"]” ] } "indexBounds" : { "score" : [ "(9000.0, inf.0]” ] }
  • 94. Recommendations Use multiple indexes! db.games.createIndex({gamerTag: 1}) db.games.createIndex({score: 1})We already have the {gamerTag:1, score:-1} index. Do we need both of these new indexes?
  • 95. Recommendations Use multiple indexes! db.games.createIndex({gamerTag: 1}) db.games.createIndex({score: 1}) db.games.createIndex({ gamerTag: 1, score:-1 }) Works with sorting Generate a SORT_MERGE plan
  • 96. Stakeholder Concern #2 db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt: 9000}} ] }) Having the right index is critical
  • 97. Stakeholder Concern #2 db.games.find({ $or : [ {gamerTag: "Ace"}, {score: {$gt: 9000}} ] }) Having the right index is critical "Super!!"
  • 98. Stakeholder Concern #3 “Wait wait wait, we can’t even FIND the gamers!” A basic search on gamerTag takes several seconds already: db.games.find({gamerTag: /^Ace$/i}) “This query is SLOWER with the index than it is without it!”
  • 99. Avoiding Query Pitfalls: Case Insensitive Queries
  • 100. db.games.find({ gamerTag: /^Ace$/ }) //equivalent to db.games.find({ gamerTag: "Ace" }) Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "ACE", score: 500}, {gamerTag: "aCe", score: 9500}, {gamerTag: "ace", score: 0}, {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Sensitive
  • 101. Matching games: {gamerTag: "Ace", score: 9500} Non-matching games: {gamerTag: "ACE", score: 500}, {gamerTag: "aCe", score: 9500}, {gamerTag: "ace", score: 0}, {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Sensitive "indexBounds" : { "gamerTag" : [ "["Ace", "Acf")", "[/^Ace$/, /^Ace$/]" ] } ace aCe acxyz Ace Ace mdb ACE Bob
  • 102. db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: "^Ace$", $options: "i" } }) //equivalent to db.games.find({ gamerTag: "Ace"}) .collation({locale:'en', strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Insensitive
  • 103. db.games.find({ gamerTag: /^Ace$/i }) //equivalent to db.games.find({ gamerTag: { $regex: "^Ace$", $options: "i" } }) //equivalent to db.games.find({ gamerTag: "Ace"}) .collation({locale:'en', strength:2}) Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Would a $text search behave the same? Case Insensitive
  • 104. Matching games: { gamerTag: "Ace", score: 9500 }, { gamerTag: "ACE", score: 500 }, { gamerTag: "aCe", score: 9500 }, { gamerTag: "ace", score: 0 } Non-matching games: {gamerTag: "Bob", score: 500}, {gamerTag: "acxyz", score: 9500 }, {gamerTag: "Ace mdb", score: 9500} Case Insensitive "indexBounds" : { "gamerTag" : [ "["", {})", "[/^Ace$/i, /^Ace$/i]" ] } ace aCe acxyz Ace Ace mdb ACE Bob
  • 106. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex({gamerTag: 1},{collation:{locale:'en',strength:2}}) > db.games.find( {gamerTag: "Ace"}).collation({locale: 'en',strength:2}) { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5d"), "gamerTag" : "Ace", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5e"), "gamerTag" : "ACE", "score" : 500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b5f"), "gamerTag" : "aCe", "score" : 9500 } { "_id" : ObjectId("5b29dbee6c7d4f531bf73b60"), "gamerTag" : "ace", "score" : 0 }
  • 107. Recommendations Case insensitive index! Collations available since 3.4 db.games.createIndex({gamerTag: 1},{collation:{locale:'en',strength:2}}) or Store a transformed (eg toLower()) copy of the field
  • 108. Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2})
  • 109. Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2})
  • 110. Stakeholder Concern #3 db.games.find({gamerTag: "Ace"}) .collation({locale:'en', strength:2}) "Wow, MongoDB can do anything!!"
  • 112. Work Smartly • Understand the business logic • Index appropriately • Is it the right index for the query? • Be cognizant of 3 issues: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
  • 113. Work Smartly • Understand the business logic • Index appropriately • Is it the right index for the query? • Be cognizant of 3 issues: • Blocking Stages • Usage of $or • Case sensitivity • Leverage the Performance Advisor
  • 115. Countdown to Release of GameOfDrones Powered by an optimized MongoDB environment, GameOfDrones is sure to be a hit!