Why Is My Solr Slow?: Presented by Mike Drob, Cloudera
0 likes•1,384 views
This document summarizes a presentation about using Apache HTrace to trace slow queries in Solr. It introduces HTrace as a distributed tracing library that generates spans with metadata about requests. The presentation shows how to configure Solr and HTrace to trace Solr queries, providing more detailed timing information than the default debug parameter. Tracing can identify where time is spent across servers and calls to other systems. The overhead of tracing is small, around 25 nanoseconds per trace. A demo then shows an example trace of a simple search query in Solr.
Why Is My Solr Slow?: Presented by Mike Drob, Cloudera
- 1. OCTOBER 11-14, 2016 • BOSTON, MA
- 2. Why is my Solr slow? Mike Drob Software Engineer, Cloudera
- 3. Who Am I? ● Apache HTrace (Incubating) Committer ● Software Engineer @ Cloudera ● Middle School LEGO Robotics Mentor
- 4. Outline ● The need for tracing ● HTrace introduction ● Examples with Solr ● Finding the slow server!
- 5. Scenario ● User complains that queries are “slow” ● Caches? ● Or faceting? ● Or sorting? ● Or...?
- 6. The State of the Art Run a query with debug=timing: "timing":{ "time":178.0, "prepare":{"time":7.0, "query":{"time":4.0}} "process":{"time":160.0, "query":{"time":118.0}, "facet":{"time":33.0}}
- 7. What is Apache HTrace ● Distributed Tracing Library ● Primitive Type is the “Span” ● Unique ID ● Source + Description ● Start/Stop Time ● Additional Metadata ● See Also: Dapper, Zipkin, Wingtips
- 10. How Do I Trace? ● Can Trace Any “Unit Of Work” ● Method Calls ● Threads ● RPCs doQuery(); try (TraceScope scope = tracer.newScope(“work”)) { doQuery(); }
- 11. What Does It Look Like? ● Trace of q=*:* and resulting sub-queries ● Data modeled on debug=timing
- 12. Aside: Benefits over debug=timing ● Separate time-lines for each server ● Ability to mark annotations ● Data stored for later analysis ● Trace non-query requests ● Trace calls to other systems
- 16. Configuration ● New section in solr.xml <trace> <str name="span.receiver.classes">org.apache.htrace.impl.HTracedSpanReceiver</str> <str name="htraced.receiver.address">trace-server:9075</str> <str name="sampler.classes">org.apache.htrace.core.ProbabilitySampler</str> </trace> ● Sampler Options ● Advanced: Force Trace … traceid=<64-bit> … ● Advanced: Buffer Options
- 17. Performance ● Each trace add ~25ns of overhead ● Search handler can generate ~20 traces ● Traces are buffered in memory ● May consider multiple recievers
- 18. Future Work ● Add tracing to clients (SolrJ) ● Leverage MDC logging ● Async Requests
- 19. Demo
- 21. Demo (Backup)
- 22. Demo (Backup)
- 23. Demo (Backup)
- 24. Demo (Backup)