Optimizing Slow Queries with Indexes and Creativity
4 likes4,245 views
This document discusses optimizing slow queries in MongoDB through the use of indexes and schema design. It begins with examples of common performance issues like inefficient reads from table scans and writes causing locks. Each example shows the slow query, explains the problem, and proposes a solution like adding an index. It emphasizes that new indexes only scale up to a point and sometimes the best approach is to redesign the schema or add hardware. The talk provides guidance on measuring performance, finding slow queries, determining if they can be indexed, and rephrasing problems in terms of desired queries.
![log
[conn73454] query my_db.my_collection query: { $query: {
publisher: "US Weekly" }, orderby: { publishAt: -1 } }
ntoreturn:5 ntoskip:0 nscanned:33236 scanAndOrder:1
keyUpdates:0 numYields: 21 locks(micros) r:317266
nreturned:5 reslen:3127 178ms
Example 1](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-7-320.jpg)
![tail
[conn72593] remove my_db.my_collection
query: { status: "some chuck of text" }
keyUpdates:0 numYields: 15 locks(micros)
w:213415 210ms
Example 2](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-11-320.jpg)
![tail
[initandlisten] connection accepted from ....
[conn4229724] authenticate: { authenticate: ....
[initandlisten] connection accepted from ....
[conn4229725] authenticate: { authenticate: .....
[conn4229717] query ..... 102ms
[conn4229725] query ..... 140ms
amazingly quiet
Example 3](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-15-320.jpg)
![currentOp
> db.currentOP()
{
"inprog" : [
{
"opid" : 66178716,
"lockType" : "read",
"secs_running" : 760,
"op" : "query",
"ns" : "my_db.my_collection",
"query" : {
keywords: $in: [“keyword1”, “keyword2”],
tags: $in: [“tags1”, “tags2”]
},
orderby: {
“created_at”: -1
},
"numYields" : 21
}
]
}
Example 3](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-16-320.jpg)
![Finding Offenders
Find the time of the slowest query of the day:
grep '[0-9]{3,100}ms$' $MONGODB_LOG | awk '{print $NF}' | sort -n
Example 4](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-20-320.jpg)
![Slowest Map Reduce
my_db.$cmd command: {
mapreduce: "my_collection",
map: function() {},
query: { $or: [
{ object.type: "this" },
{ object.type: "that" }
],
time: { $lt: new Date(1359025311290), $gt: new Date(1358420511290) },
object.ver: 1,
origin: "tnh"
},
out: "my_new_collection",
reduce: function(keys, vals) { ....}
} ntoreturn:1 keyUpdates:0 numYields: 32696 locks(micros)
W:143870 r:511858643 w:6279425 reslen:140 421185ms
Example 4](https://image.slidesharecdn.com/mongodbaustin-winslett-130304155345-phpapp02/85/Optimizing-Slow-Queries-with-Indexes-and-Creativity-21-320.jpg)
Optimizing Slow Queries with Indexes and Creativity
- 1. Optimizing Slow Queries with Indexes and Creativity Chris Winslett [email protected]
- 2. My Background • For the past year, I’ve looked at MongoDB logs at least once every day. • We routinely answer the question “how can I improve performance?”
- 3. Who’s this talk for? • New to MongoDB • Seeing some slow operations, and need help debugging • Running database operations on a sizeable deploy • I have a MongoDB deployment, and I’ve hit a performance wall
- 4. What should you learn? Know where to look on a running MongoDB to uncover slowness, and discuss solutions. MongoDB has performance “patterns”. How to think about improving performance. And . . .
- 5. Schema Design Design with the end in mind.
- 6. First, a Simple One Alerted due to high CPU query getmore command res faults locked db ar|aw netIn netOut conn time 129 4 7 126m 2 my_db:0.0% 3|0 27k 445k 42 15:36:54 64 4 3 126m 0 my_db:0.0% 5|0 12k 379k 42 15:36:55 65 7 8 126m 0 my_db:0.1% 3|0 15k 230k 42 15:36:56 65 3 3 126m 1 my_db:0.0% 3|0 13k 170k 42 15:36:57 66 1 6 126m 1 my_db:0.0% 0|0 14k 262k 42 15:36:58 32 8 5 126m 0 my_db:0.0% 5|0 5k 445k 42 15:36:59 a truncated mongostat
- 7. log [conn73454] query my_db.my_collection query: { $query: { publisher: "US Weekly" }, orderby: { publishAt: -1 } } ntoreturn:5 ntoskip:0 nscanned:33236 scanAndOrder:1 keyUpdates:0 numYields: 21 locks(micros) r:317266 nreturned:5 reslen:3127 178ms Example 1
- 8. Solution We are fixing this query { $query: { publisher: "US Weekly" }, orderby: { publishedAt: -1 } } With this index db.my_collection.ensureIndex({“publisher”: 1, publishedAt: -1}, {background: true}) I would show you the logs, but now they are silent. Example 1
- 9. The Pattern Inefficient Read Queries from in-memory table scans cause high CPU load Caused by not matching indexes to queries. Example 1
- 10. Example 2 query delete res faults locked db idx miss % qr|qw ar|aw netIn netOut conn 25 6 346m 0 my_db:188.1% 0 0|0 0|2 25k 45k 117 24 6 346m 0 my_db:188.6% 0 0|0 0|1 27k 44k 117 24 6 346m 0 my_db:184.3% 0 0|0 0|1 21k 36k 117 24 6 346m 0 my_db:190.9% 0 0|0 0|1 20k 33k 117 19 4 346m 0 my_db:191.5% 0 0|0 0|0 21k 41k 117 a truncated mongostat
- 11. tail [conn72593] remove my_db.my_collection query: { status: "some chuck of text" } keyUpdates:0 numYields: 15 locks(micros) w:213415 210ms Example 2
- 12. Solution This is the slow query db.my_collection.remove({status: “some chunk of text”}) With this index db.my_collection.ensureIndex({status: 1}) Example 2
- 13. The Pattern Inefficient write queries cause high lock. Caused by losing track of your indexes / queries. Example 2
- 14. Example 3 Alerted on high CPU insert query update delete getmore command faults locked % idx miss % qr|qw ar|aw *0 *0 *0 *0 0 1|0 1422 0 0 0|0 50|0 *0 6 *0 *0 0 6|0 575 0 0 0|0 51|0 *0 3 *0 *0 0 1|0 1047 0 0 0|0 50|0 *0 2 *0 *0 0 3|0 1660 0 0 0|0 50|0 a truncated mongostat
- 15. tail [initandlisten] connection accepted from .... [conn4229724] authenticate: { authenticate: .... [initandlisten] connection accepted from .... [conn4229725] authenticate: { authenticate: ..... [conn4229717] query ..... 102ms [conn4229725] query ..... 140ms amazingly quiet Example 3
- 16. currentOp > db.currentOP() { "inprog" : [ { "opid" : 66178716, "lockType" : "read", "secs_running" : 760, "op" : "query", "ns" : "my_db.my_collection", "query" : { keywords: $in: [“keyword1”, “keyword2”], tags: $in: [“tags1”, “tags2”] }, orderby: { “created_at”: -1 }, "numYields" : 21 } ] } Example 3
- 17. Solution Return Stability to Database > db.currentOP().inprog.filter(function(row) { return row.secs_running > 100 && row.op == "query" }).forEach(function(row) { db.killOp(row.opid) }) Disable query, and refactor schema. Example 3
- 18. Refactoring I have one word for you, “Schema”
- 19. Example 4 A map reduce has gradually run slower and slower.
- 20. Finding Offenders Find the time of the slowest query of the day: grep '[0-9]{3,100}ms$' $MONGODB_LOG | awk '{print $NF}' | sort -n Example 4
- 21. Slowest Map Reduce my_db.$cmd command: { mapreduce: "my_collection", map: function() {}, query: { $or: [ { object.type: "this" }, { object.type: "that" } ], time: { $lt: new Date(1359025311290), $gt: new Date(1358420511290) }, object.ver: 1, origin: "tnh" }, out: "my_new_collection", reduce: function(keys, vals) { ....} } ntoreturn:1 keyUpdates:0 numYields: 32696 locks(micros) W:143870 r:511858643 w:6279425 reslen:140 421185ms Example 4
- 22. Solution Problem Query is slow because it has multiple multi-value operators: $or, $gte, and $lte Solution Change schema to use an “hour_created” attribute: hour_created: “%Y-%m-%d %H” Create an index on “hour_created” with followed by “$or” values. Query using the new “hour_created.” Example 4
- 23. Words of caution 2 / 4 solutions were to add an index. New indexes as a solution scales poorly.
- 24. Sometimes . . . It is best to do nothing, except add shards / add hardware. Go back to the drawing board on the design.
- 25. Bad things happen to good databases? • ORMs • Manage your indexes and queries. • Constraints will set you free.
- 26. Road Map for Refactoring • Measure, measure, measure. • Find your slowest queries and determine if they can be indexed • Rephrase the problem you are solving by asking “How do I want to query by data?”
- 27. Thank you! • Questions? • E-mail me: [email protected]