State-of-the-Art Drupal Search with Apache Solr

Editor's Notes

• #2: This presentation was given on Sunday, February 7, 2010, in Brussels, at the FOSDEM conference.
• #3: Two years ago, at a conference called FOSDEM, the Apache Solr module was introduced. Coincidentally, it was the day I started working for Acquia.
• #4: It wasn’t really ready for prime time. In all I’d say it was 20% software and 80% vaporware.
• #5: I also had more hair back then, which was not only longer and thicker, but less grey.
• #6: A lot has changed.
• #7: Since then Acquia has launched a hosted search service based on Apache Solr: http://acquia.com/products-services/acquia-search
• #8: It’s been a big successs. Hosted by Acquia, using Amazon cloud architecture for great performance and high availability. If you want to try it all out and get up and running quickly, the Acquia Stack Installer and a free or basic subscription will get you there in around 5 minutes.
• #9: http://acquia.com/acquia-search http://acquia.com/downloads
• #10: Another important change is the funding and founding of Lucid Imagination. http://www.lucidimagination.com/About
• #11: Like Acquia, Lucid Imagination is a venture funded software company based in the Boston area. Like Acquia they have core committers on their team. Like Acquia they’re in the business of providing support and services for their open source project.
• #12: They have succeeded in raising the profile and awareness of Solr, and have also accelerated the pace of Solr development. Lucid Imagination has also captured a lot of interest from government clients.
• #13: Acquia and Lucid Imagination represent two good reasons to choose Solr. It is a good long-term technology platform decision.
• #14: One of the big changes of the past two years has affected all of us quite strongly. For some time, Drupal.org has been running Solr as its main search component.
• #15: Around 50% of page requests to Drupal.org involve the Solr server in some way. These include site search, the project listings page, and the issue queue listings.
• #16: Here is a search page. Note the opportunities to sort, and filter on facets including content type and author.
• #17: Here’s the modules listing page. It’s especially helpful that you can filter on Drupal version type, project type, or do a keyword search that is limited just to modules.
• #18: Here’s the issue queue advanced search. This too is powered by Solr.
• #19: The advantages of Drupal.org switching to Solr search include a much better search experience. There’s faceting. There’s better relevancy. Better performance. Better scalability.
• #20: Of course, one of the exciting announcements from last year was that President Obama’s website, Whitehouse.gov, had switched to Drupal.
• #21: I bet you can guess that was quite happy about this.
• #22: And quite proud.
• #23: Proud of Drupal, and of Acquia,
• #24: and of the Obama administration for working to foster openness in government.
• #25: Two years ago the idea and concept of faceted search - the ability to easily drill down into search results - was new. Now everybody wants it, it’s become a de facto standard in new projects.
• #26: Two years ago I promised we’d get spelling suggestions. We have them, though we’re still learning how to tune and improve them.
• #27: I will give you a tip, though: find solr/conf/spellings.txt and add problem words to it. The default file that comes with Solr only has the words “pizza” and “history”.
• #28: Someone here could make a contributed module to generate a spellings.txt. You could use taxonomy terms, vocabularies, synonyms, content types, user names, custom input forms, and even online dictionaries to generate the file.
• #29: Two years ago I promised more control over tuning search results. Now there are all sorts of boosting and customization options. You can exclude content types from the index. You can boost or reduce the importance of individual fields or HTML elements when searching.
• #30: This screen shows how you can use node attributes, like whether a node is promoted to the front page, or is sticky, to influence search rankings.
• #31: This screen shows how you can boost or diminish the ranking of individual content types, or exclude content types from being indexed altogether.
• #32: And this screen shows how you can use the HTML markup itself to give extra weight to some elements, or diminish the value of others.
• #33: Two years ago I promised the ability to do content recommendation with Solr. That is now a reality and it works really well, leading to far lower bounce rates and more time spent on your site
• #34: When Dries enabled content recommendation on http://Buytaert.net I spent about two hours re-discovering things that he’d written over the years. Every article had more context and background. You can always find something related and interesting to read.
• #35: And with this pending patch, you have even greater control over content recommendation. You can limit the recommendations to certain content types, or certain taxonomy terms, or boost certain words. You can make as many different recommendation blocks as you need.
• #36: We now have Views 3, File search, multisite search, comment search, cck date facets, statistics, autocomplete, and the glorious display suite.
• #37: Views 3 builds custom solr queries (using the apachesolr_views module). Views 3 then displays query results, with all the Views goodies you’re familiar with (tables, grids, carousels, slideshows etc.) You can build custom search forms with exposed filters, and faceting works as well.
• #38: Here you can see an example view that has an exposed keyword filter, a taxonomy facet block, and a table display including a sortable title column. All made using the views user interface.
• #39: Read more about it at acquia dot com, node nine one one six six seven. http://acquia.com/node/911667
• #40: It’s possible to search for text inside of uploaded files as well. There’s a brand new blog post on Acquia.com about this topic that I published today. http://acquia.com/node/1129446
• #41: I searched for “merlinofchaos” and the text was found in a the zipped up tarball of Views3 which I attached to a node using a filefield.
• #42: Here are examples of the word “Drupal” being found in a Microsoft Word file and this very Keynote presentation that you’re watching.
• #43: Multisite search: sites share an index and can be filtered on the “Site” facet. You can either search across all your sites, or on the site that you’re currently on.
• #44: Comment search: Comments get indexed as 1st class citizens and get their own search results. Even solved the problem of linking to a comment on page X. This is only available in the DRUPAL-6--2 branch of ApacheSolr.
• #45: CCK Date Facets: Any CCK date field becomes a facet filter. You can drill down - year - month - day - hour.
• #46: This is a patch, currently, and needs testing. Please help. http://drupal.org/node/558160
• #47: Too few people analyze what’s happening on their site regarding search. When you look at normal analytics you see a lot of incoming links and keyword searches from sites like Google. Do you, however, analyze the keywords that people use on your own site search? This is golden information as they’re telling you exactly what they’re looking for. Do you retrace their steps and look at the search results they see? Is it what you’d expect? Are they finding what they’re looking for?
• #48: The statistics module gives you insight to how many searches, search performance (.0001 second average - .543 sec maximum)
• #49: And also which search filters are most often being applied.
• #50: Here you can see how many searches from a music site are being filtered by genre or instrumentation.
• #51: Ready for use. Go try it out.
• #52: Autocomplete uses the contents of the index to suggest terms to you, as you type. If you finish one term it will suggest a common second term to go with it.
• #53: Just a few days ago the display suite 1.0 was released. This is now the easiest way to customize your search results.
• #54: Here, for example, are search results with full teasers and images.
• #55: Solr 1.5 is just around the corner. The team at Lucid has set a brisk pace. It will include native geospatial seaerch. It will have an autocomplete request handler. It will have the very exciting eDismax request handler - full lucene syntax and perhaps opportunity for Lucene API module integration. (http://drupal.org/project/luceneapi)
• #56: People often want to search non-Drupal sites along with their Drupal sites. This can be done by crawling those sites with Nutch and using the Nutch/Solr integration. Acquia support for multisite, file and geospatial search are on the way. (In the presentation I asked a show of hands for which of these three features would be most popular: People responded 1 - File, 2 - Multisite, and 3 - Geospatial in that order.
• #57: There are urgent needs and immediate opportunities to help. #1 on the list is to assist with the Drupal.org redesign and relaunch. http://drupal.org/node/704062 Much of the new site is driven by Solr #2 is the glaring absence of test suites. This slows development due to regressions. #3 is the upgrade to Drupal 7.
• #58: Like any two year long party, there’s a bit of of housecleaning that now has to be done. Better APIs, more abstraction, better documentation.