Berlin buzzwords 2013 - Faceting analyzed fields with some sprinkles of probability theory
0 likes•656 views
Faceting analyzed fields using probability theory allows for trending topic analysis and other insights. The document discusses using fielddata filters to reduce the number of terms and documents in an index based on frequency thresholds. This improves performance for trending topic analysis by dropping infrequent terms and documents with too many terms.
Berlin buzzwords 2013 - Faceting analyzed fields with some sprinkles of probability theory
- 1. Faceting analyzed fields with some sprinkles of probability theory conjures trending topic analysis and other interesting insights Boaz Leskes Elasticsearch @bleskes work done for Buzzcapture
- 2. Trending?
- 7. topic reference P(w|T) = kDt|w 2 Dtk kDtk
- 9. topic reference P(w|T) = kDt|w 2 Dtk kDtk
- 10. P(w|T) = kDt|w 2 Dtk kDtk
- 11. brown dog fox quick 2 5 10 12 5 6 12 13 2 5 6 10 12 13 brown dog fox quick
- 12. In our index. • Terms = 12GB • “Arrows” = 41GB
- 13. { tweet: { type: "string", analyzer: "whitespace" fielddata: { filter: { regex: "^#.*", frequency: { min: 10 } } } } } Drop terms which occur too little
- 14. Drop docs with too many terms
- 16. iculture 10,122 floor 8,998 cover 6,874 toy 4,402 ground 3,841 4.0 7,878 4.1 4,292 rtacties 4,078 jelly 2,905 bean 2,857