Use Cases for Elastic Search Percolator
5 likes3,883 views
The document discusses the use of Elastic Search's percolator feature. The percolator allows storing queries in an index and then indexing documents to retrieve matching queries. This is opposite of traditional search where documents are indexed and queries retrieve them. The percolator works in real-time so queries can immediately be used. Examples are provided of adding percolator queries and using them to match documents. Additional use cases discussed include alerting, notifications, and monitoring systems.
![10
Redis
Search Meetup 2015
{"matches":["g2_123456700",
"g1_123456700"],
"event":{
"service":"processPurchaseEvent",
"logstash_checksum":"cef6adb614de4623bbd809197fb79e3f",
…
{"alias":"123456700","aliasType":1}}
…
}
}
Data for redis](https://image.slidesharecdn.com/espercolatorusecases-160119170141/85/Use-Cases-for-Elastic-Search-Percolator-10-320.jpg)
![13
Logstash plugin
Search Meetup 2015
Configuration
output {
elasticsearch {
host => " HOSTNAME :9200"
protocol => "http"
document_id => "%{logstash_checksum}"
}
percolator {
host => "HOSTNAME"
port => "9200"
redis_host => ["localhost"]
document_id => "%{logstash_checksum}"
pattern_index => "percolators"
index => "data"
data_type => "channel"
}
}](https://image.slidesharecdn.com/espercolatorusecases-160119170141/85/Use-Cases-for-Elastic-Search-Percolator-13-320.jpg)
Use Cases for Elastic Search Percolator
- 1. Use Cases for Elastic Search Percolator Search Meetup 2016
- 2. 2 Percolator Search Meetup 2015 Traditionally you design documents based on your data, store them into an index, and then define queries to retrieve these documents. TheTheTheThe percolator works in the opposite direction.percolator works in the opposite direction.percolator works in the opposite direction.percolator works in the opposite direction. First you store queries into an index and then you define documents in order to retrieve these queries. The percolator and most of its features work in realtime - once a percolate query is indexed it can immediately be used in the percolate API.
- 3. 3 Search Meetup 2015 Near – time decisions for rewarding customer in PAYBACK programs
- 4. 4 Search Meetup 2015 Business target Near – time decisions for rewarding customer in PAYBACK programs source: https://www.flickr.com/photos/mdalmuld/9559878695
- 5. 5 Search Meetup 2015 Gamification More achievements and more emotions with games. source: https://www.flickr.com/photos/129941959@N06/15664857154/
- 6. 6 Implementation details ElasticElasticElasticElastic SearchSearchSearchSearch as real time Data Container with all events PercolatorPercolatorPercolatorPercolator as real time Rule Engine RedisRedisRedisRedis as queuing system LogstashLogstashLogstashLogstash for event pushing DropwizardDropwizardDropwizardDropwizard for RESTfull standalone application HTML5 and AngularJSHTML5 and AngularJSHTML5 and AngularJSHTML5 and AngularJS for responsive GUI Search Meetup 2015 Tools / frameworks
- 8. 8 Percolator Search Meetup 2015 curl -XPUT 'localhost:9200/data/.percolator/g1_123456700' -d '{ "query" : { "match" : { "service" : "processPurchaseEvent", "alias" : "123456700" } } }' Creation of percolator query
- 9. 9 Percolator Search Meetup 2015 curl -XGET 'localhost:9200/data/logs/_percolate' -d '{ "doc" : { "service" : "processPurchaseEvent", "alias" : "123456700" . . . } }' Usage of percolator
- 11. 11 Java application Search Meetup 2015 Creation of percolators
- 12. 12 Logstash plugin Search Meetup 2015 Ruby code for using percolator queries
- 13. 13 Logstash plugin Search Meetup 2015 Configuration output { elasticsearch { host => " HOSTNAME :9200" protocol => "http" document_id => "%{logstash_checksum}" } percolator { host => "HOSTNAME" port => "9200" redis_host => ["localhost"] document_id => "%{logstash_checksum}" pattern_index => "percolators" index => "data" data_type => "channel" } }
- 14. 14 Percolator details stored per index in the .percolator type active percolator queries are kept in memory at percolate time, the document from the request gets parsed into a Lucene document and is stored in a in-memory Lucene index percolator queries are executed on single document in-memory index multi percolate and percolation of existing documents is possible Search Meetup 2015 Under the hood
- 15. 15 Alerting Use Cases Scaling the PAYBACK platform for the US: plenti.com
- 17. 17 Alerting Use Case easy with JSON structured logs simplest use case: percolate on FATAL and ERROR log levels alerts can be triggered for displaying on dashboards also easy for enhanced use cases such as Monitoring of invariants: create alerts whenever system goes outside of expected normal ranges shift monitoring from reactive to proactive – be able to look for exceptional behavior before it exhibits itself as a production problem Search Meetup 2015 Using with ELK
- 18. 18 Other use cases Search Meetup 2015 Alerting based notifications Having corresponding data in elastic search you can notify customer about • new product release • price change of a product (is below threshold) • weather changes (snow, rain etc) • stock changes • what do you use?
- 19. 19 Search Meetup 2015 Questions & Answers source: https://www.flickr.com/photos/21496790@N06/5065834411/