Building Smarter Search Applications Using Built-In Knowledge Graphs and Query Introspection: Presented by Ted Sullivan, Lucidworks
4 likes•1,189 views
This document discusses improving search precision through better phrase detection, such as recognizing noun phrases using autophrasing. It also describes implementing query autofiltering to map noun and verb phrases in queries to metadata fields, and providing a suggester component that leverages faceted metadata to provide contextual suggestions.
Building Smarter Search Applications Using Built-In Knowledge Graphs and Query Introspection: Presented by Ted Sullivan, Lucidworks
- 2. Ted Sullivan Building Smarter Search Applications Using Built-In Knowledge Graphs and Query Introspection lucidworks.com Senior Solutions Architect
- 3. Ted Sullivan Building Smarter Search Applications Using Built-In Knowledge Graphs (aka Solr!) and Query Introspection lucidworks.com Senior Solutions Architect
- 4. Relevance - Precision - Recall Do we put the cart before the horse? Precision/Recall determine what matches and what doesn’t. Relevance then computes the “best” matches from what is left. Without focusing more on precision/recall first, we tend to have Garbage In/Garbage Out This is especially true in faceted search - relevance tuning can fix the first few pages but the facets cannot be fixed!!!
- 5. Improving Precision starts … with better phrase detection Embarrassing “noise” hits are often due to phrase cross matches. Synonyms can improve recall tremendously but they need some help in Solr when they are multi-term Stop words can be important for disambiguation within phrases: “The Lady Is A Tramp” vs. “The Lady And The Tramp” “To Be Or Not To Be”
- 6. Better Search: Detecting Noun Phrases The basic technique is called “autophrasing” – recognizing when more than one word represents just one thing. Autophrasing – uses an extra knowledge-base file “autophrases.txt” Query Autofiltering – uses the phrases that are stored as metadata values in the index. A Novel Approach to Natural Language Processing: Mapping Noun and Verb phrases to metadata fields “Who’s in The Who?”
- 7. Multi-term Synonym Demo autophrases.txt new york new york state empire state new york city new york new york big apple ny ny city of new york state of new york ny state synonyms.txt new_york => new_york_state, new_york_city, big_apple, new_york_new_york, ny_ny, nyc,empire_state,ny_state, state_of_new_york new_york_state,empire_state,ny_state, state_of_new_york new_york_city,big_apple,new_york_new_york, ny_ny,nyc, city_of_new_york
- 8. Multi-term Synonym Demo This document is about new york state. This document is about new york city. There is a lot going on in NYC. I heart the big apple. The empire state is a great state. New York, New York is a hellova town. I am a native of the great state of New York. New York New York City New York State /select /autophrase
- 9. Query Autofiltering Implementation Use Lucene FieldCache to build a map of field values to field names (string fields) Add synonym mappings from synonyms.txt and stemming to this value(s) -> field(s) map Use this map to discover noun phrases in the query that correspond to field values in the index – longest contiguous phrase wins Build filter or boost queries based on these discovered mappings SOLR-7539 created 5/30/2015 - One comment so far “+1” - Bill Bell - Thanks Bill
- 10. Query Autofiltering – Basic Behavior q = red socks -> fq=color:red&fq=product_type:socks or bq=(color:red AND product_type:socks)^20 q = red lion socks -> fq=brand:”Red Lion”&fq=product_type:socks q = scarlet chaise lounge -> color:red AND product_type:”Lounge Chair” q = white dress shirts -> color:white AND product_type:”dress shirt” q = white linen shirts -> ((brand:"White Linen" OR (color:white AND material:linen)) AND product_category:shirts) q = white and grey dress shirts ((product_type:"dress shirt" OR ((product_type:dress OR product_category:dress) AND (product_type:shirt OR product_category:shirt))) AND (color:(white OR grey) OR colors:(white AND grey)))
- 11. Query Autofiltering – Language And Logic Logical or “Boolean” operators (named after mathematician George Boole) have precise meaning in Set Theory and Computer Science Search is about returning a set of records that match a set of terms AND - Intersection ( && ) OR - Union ( | | ) NOT - Exclusion ( ! ) In language - the meaning of “and” and “or” is contextual - sometimes they are synonyms and sometimes they are antonyms! - depends on the cardinality (single or multi-value) of an noun property or attribute!
- 12. A Music Ontology Song Songwriter Genre Performer Recording Guitarist Pianist VocalistProducer Record Label Band Album
- 13. Natural Language Processing - Lite (Front-end NLP) Precise Free-text searching of Structured Metadata Query Autofilter can take Natural Language Queries and turn them into structured Boolean Queries. Now processes both noun and verb/adjective phrases Verb phrase mapping enables better selection of field names Beatles Songs written by George Harrison Willie Dixon Songs covered by Led Zeppelin ==> Look Mom - No SQL!!
- 14. Natural Language Processing - Lite Noun phrases that map to fieldName/fieldValue pairs Bob Dylan Songs composer:”Bob Dylan” OR performer:”Bob Dylan” Verb phrase patterns that map to field names: Songs written by Bob Dylan ==> composer:”Bob Dylan” “Who’s in The Who?” ==> memberOfGroup:”The Who” Songs Bob Dylan covered vs covers of Bob Dylan Songs
- 15. Short Demo of Query Autofiltering NLP-Lite
- 16. A Suggester for Query Autofiltering Create multi-field suggestions using “Pivot Facet Patterns” that can be processed by Query Autofiltering Use facets - at index time - to extract suggestion meta phrases and context.
- 17. Steps in building the Suggester Processing - Denormalizing the Graph Create searchable metadata from object links Process graph relationships, apply business rules Creating the Pivot Patterns ${name_s} ${recording_type}s ==> Bob Dylan Songs ${genre} ${recording_type}s ==> Progressive Rock Albums ${genre} ${musician_type}s ==> Rock Drummers, Heavy Metal Bands Calculating which recordings are covers Finding Related entities (i.e. John Lennon <=> Paul McCartney) ${members_ss} ${musician_type}s ==> Paul McCartney Bands
- 18. Repurposing Facets for relationship mining Using pivot facets to generate multi-field phrases Only get linguistically sensible permutations! Facets provide Specification and Context Traditionally used for visualization and navigation. We can repurpose this to make a smarter suggester! Security Trimming of suggestions Only show suggestions that can return results given the current user’s entitlements
- 19. A Suggester that learns what the user is looking for Suggester now brings back metadata with the suggestion Front end can cache this metadata and use it to boost subsequent typeahead queries based on what the user selected. Searching for Beatles Songs - “Baby’s In Black” and “Baby You’re A Rich Man” are now boosted over all of the other song titles that start with “Baby”
- 20. Thank you! lucidworks.com Ted Sullivan Senior Solutions Architect