Searching for NLP: Using Elasticsearch to Create MVPs of NLP-enabled User Experiences

Editor's Notes

• #7: This isn’t a master class on what to do exactly. This is intended to stir your creativity, prod at some things that you hadn’t thought about before, and get you minded in the right direction.
• #8: This isn’t a master class on what to do exactly. This is intended to stir your creativity, prod at some things that you hadn’t thought about before, and get you minded in the right direction.
• #10: Bad news: you are the reason they can’t get to it Good news: you are the only way they will be able to get to it Their life is in your hands
• #11: Bad news: you are the reason they can’t get to it Good news: you are the only way they will be able to get to it Their life is in your hands
• #12: Bad news: you are the reason they can’t get to it Good news: you are the only way they will be able to get to it Their life is in your hands
• #14: Word embeddings: GloVe Word2Vec Bag of words FastText
• #15: Polysemy: I got the invite to do this talk I got anxious Hope you can say afterwards, “I got it”
• #16: The group grew: Bert, Ernie, Big Bird, etc.
• #17: It’s getting a bit out of hand
• #18: As much as your users likely love Sesame Street… they don’t care about how bleeding edge your solution is They’ll be grouchier than Oscar when your solution doesn’t work. They’ll be happy as Elmo when it does—regardless of how.
• #19: As much as your users likely love Sesame Street… they don’t care about how bleeding edge your solution is They’ll be grouchier than Oscar when your solution doesn’t work. They’ll be happy as Elmo when it does—regardless of how.
• #20: Cute, lovable puppet characters notwithstanding
• #23: There is likely a very fat initial part of your tail wherein you can get outsized gains and improvements. You won’t solve all the issues, but you’ll solve 80% of them with only 20% of the work or investment. Or perhaps you’ll solve the problems that equate to 80% of the user value, company bottom-line, etc. The point is this: focus on the wins, not the hows; the value, not the tech.
• #24: You can probably spend 1-2 days tops and comb through some logs to find areas of opportunity for your application. If you can’t see it clearly then how could you spec it clearly?
• #28: Dismax
• #29: E.g. Dismax Optimize what you have before you build new, costly tech that needs to be optimized “Pediatric cardiologist 46220” will have a greater chance of being properly tuned to relevance once we appropriate our data and shift to the terms-centric approach found in a dismax query Suggested reading: Relevant Search by Doug Turnbull and John Berryman
• #30: It’s not cheating to use your app layer, other technologies, etc. Redis + Zip
• #31: It’s not cheating to use your app layer, other technologies, etc. Redis + Zip Maybe instead of just ranking higher on a zip match you want to filter on it. Regex and modify your query to be a facet. Or maybe you don’t want to filter out other zips but have concentric rings of sorting done based on your user’s submitted zip code. Adding this layer is a trivial amount of work for your engineers, it’s a trivial amount of impact to your infrastructure, e.g. Redis storage, and it’s a trivial amount of added latency to the overall request time—but it’s a non-trivial upgrade to your user’s overall experience.
• #32: Facets are features. They’re facts about your data, simple truths that help you navigate it. This is why users use them when your precision isn’t good and your recall is really high: facets are the features they wish they’d given you or that you’d discerned. This being the case, you could be very aggressive and stuff keywords, highly-sought-after terms and phrases, into a special junk drawer for your documents and have those fields boosted in your dismax query. If nothing else, it can help you see the shape of your data a bit better in terms of density & distribution which then will help you to best facilitate the “cheapLP” solution needed to get users to the data Note, of course, that facets can be one of the first stops for your new ML/NLP engineer to get relevant feature data for their models Ted Sullivan from Lucidworks
• #35: Wikipedia: “heart attackers” isn’t a phrase, “heart attack” is, leverage for phrase recognition Don’t build a huge algorithm to know what “near me” means -- there are plenty of fat head areas of opportunity Point-wise mutual information is a means of measuring associations in information theory, e.g. “heart” and “attack” vs. “heart attack”
• #36: The reality is that there are a LOT of problems to solve If you don’t focus on one at a time then you probably won’t get to any of them. Sesame Street is smart in that they take one letter and one number at a time and teach it to kids. Boiling the ocean is frustrating. Solving real user problems is a lot of fun, especially if you do it in a way that is cost-effective and creates a rapidity or momentum to your stream of value-delivery