Boosting Documents in Solr by Recency, Popularity, and User Preferences

Editor's Notes

• #2: Attendees with come away from this presentation with a good understanding and access to source code for boosting and/or filtering documents by recency, popularity, and personal preferences. My solution improves upon the common "recip" based solution for boosting by document age. The framework also supports boosting documents by a popularity score, which is calculated and managed outside the index. I will present a few different ways to calculate popularity in a scalable manner. Lastly, my solution supports the concept of a personal document collection, where each user is only interested in a subset of the total number of documents in the index. My presentation will provide a good example of how to filter and/or boost results based on user preferences, which is a very common requirement of many Web applications.
• #3: The one thing I’d like you to come away with today is confidence that Solr has powerful boosting capabilities built-in, but they require some fine-tuning and experimentation. Some simple recipes for complementing core Solr functionality to do: I. Boost documents by age (recency / freshness boost) II. Boost documents by popularity III. Filter results based on User Preferences (Personalized collection)
• #4: Currently working at the National Renewable Energy Laboratory on building an infrastructure for storing and analyzing large volumes of smart grid related energy data using Hadoop technologies. Been doing search work for the past 5 years including a Lucene based search solution of eLearning content, Solr based solution for online magazine content and a FAST to Solr migration for a real estate portal. My other area of interest is in Mahout; I've contributed a few bug fixes and several pages on the wiki including working with Grant Ingersoll on benchmarking Mahout's distributed clustering algorithms in the Amazon cloud. Technical Blog: http://thelabdude.blogspot.com/ Currently working on JSF2 components for Solr.
• #5: All other things being equal, more recent documents are better What’s not covered is how to determine if you should apply the boost. That’s a more in-depth topic that is the focus of academic research, especially in relation to Web search. News and most magazine articles Business documents – perhaps a less aggressive boost function identification of recency sensitive queries before ranking. see: http://technicallypossible.wordpress.com/2011/03/13/identifying-queries-which-demand-recency-sensitive-results-in-web-search/
• #6: Careful! TrieFields make it more efficient to do range searches on numeric fields indexed at full precision, but it doesn't actually do anything to round the fields for people who genuinely want their stored and index values to only have second/minute/hour/day precision regardless of what the initial raw data looks like. Currently, Solr doesn't have anything built-in to round a date down to a different precision, such as minute / hour. Thus, you may need to do this yourself prior to indexing a document. see SOLR-741 // from commons DateUtils Date published = DateUtils.round(item.getPublishedOnDate(), Calendar.HOUR);
• #8: Solr 1.4+ the recommended approach is to use the recip function with the ms function: There are approximately 3.16e10 milliseconds in a year, so one can scale dates to fractions of a year with the inverse, or 3.16e-11 recip(ms(NOW/HOUR,pubdate),3.16e-11,1,1) For standard query parser, you could do: q={!boost b=recip(ms(NOW/HOUR,pubdate),3.16e-11,1,1)}wine This uses the built-in boost function query. This uses a Lucene FieldCache under the covers on the pubdate field (stored in the index as long). The ms(NOW/HOUR) uses less precise measure of document age (rounding clause), which helps reduce memory consumption. Lessons: 1 - {!boost b=} syntax breaks spell-checking so you need to use spellcheck.q to be explicit 2 - Use edismax because it multiplies the boost whereas dismax adds "bf" 3 - Use a tdate field when indexing 4 - Use ms(NOW/HOUR) and less precision when indexing 5 - Use max(boost,0.20) - to bottom out the age penalty
• #9: A reciprocal function with recip(x,m,a,b) implementing a/(m*x+b). m,a,b are constants, x is any numeric field or arbitrarily complex function. When a and b are equal, and x>=0, this function has a maximum value of 1 that drops as x increases. Increasing the value of a and b together results in a movement of the entire function to a flatter part of the curve. These properties can make this an ideal function for boosting more recent documents – see http://wiki.apache.org/solr/FunctionQuery
• #10: identification of recency sensitive queries before ranking. see: http://technicallypossible.wordpress.com/2011/03/13/identifying-queries-which-demand-recency-sensitive-results-in-web-search/
• #11: Score made of number of unique views in a time slot + avg rating / # of comments, etc. Must be computed outside of the index; refreshed periodically Probably don’t want to mix this with age boost as an older document might be really popular for some weird reason; think of old videos that become popular on YouTube Age – probably not as an old doc might get popular identification of recency sensitive queries before ranking. see: http://technicallypossible.wordpress.com/2011/03/13/identifying-queries-which-demand-recency-sensitive-results-in-web-search/
• #12: Bar chart illustrates time slots Popularity score favors more recent content Document A is most popular; B was popular but is now on the decline and C has enjoyed consistent interest for a longer period but scores a little lower than A because of the recent interest in A
• #15: Most likely use case would be to use log-file analysis > Ideal problem for MapReduce Question the audience – who has heard of MapReduce?