Introduction to Apache Solr
6 likes4,383 views
This document provides an introduction to Apache Solr, including: - Solr is an open-source search engine and REST API built on Lucene for indexing and searching documents. - Solr architecture includes nodes, cores, schemas, and concepts like SolrCloud which uses Zookeeper for coordination across collections and shards. - Documents are indexed, queried, updated, and deleted via the REST API or client libraries. Queries support various types including range, date, boolean, and proximity queries. - Installation and configuration of standalone Solr involves downloading, extracting, and running bin/solr scripts to start the server and create cores. - Resources for learning more include tutorials, documentation, and integration options
![INDEXING
MULTIPLE
DOCS (JSON)
REST API example:
curl -X POST -H 'Content-Type: application/json' 'http://localhost:
8983/solr/my_collection/update' --data-binary '
[
{
"id": "1",
"title": "Doc 1"
},
{
"id": "2",
"title": "Doc 2"
}
]'
24](https://image.slidesharecdn.com/solr-presentation-151018095953-lva1-app6891/85/Introduction-to-Apache-Solr-24-320.jpg)
![SEARCHING:
RANGE
QUERIES
▸ Allow the selection of documents whose fields
fall within a range
▸ Ranges with [] are inclusive at both sides
▹ price:[0 TO 100]
▹ price:[0 TO *]
▹ price:[* TO 100]
▸ Range queries with {} are exclusive
▹ price:{0 TO 100}
▸ Can mix { and ]
▹ price:[0 TO 100}
27](https://image.slidesharecdn.com/solr-presentation-151018095953-lva1-app6891/85/Introduction-to-Apache-Solr-27-320.jpg)
![SEARCHING:
OTHER
QUERIES (2)
▸ Phrase/proximity queries:
▹ "Sheldon Couper" matches only Sheldon
Couper
▹ "Sheldon Couper"~1 matches Sheldon Lee
Couper
▸ Multi-term queries:
▹ title:Ιωάννης Μακρυγιάννης
▹ title:(Ιωάννης Μακρυγιάννης)
▸ Combine them:
▹ +this -title:that +price:[* TO 100] –
name:"Sheldon Couper"
30](https://image.slidesharecdn.com/solr-presentation-151018095953-lva1-app6891/85/Introduction-to-Apache-Solr-30-320.jpg)
![SEARCHING:
FILTER
QUERIES
▸ Limit the possible responses to the main query
▸ Do not change ordering or scoring
▸ Can be based on any query type
▸ Example:
&fq=category:music
&fq=price:[0 TO 100]
&fq=rating:[3 TO *]
34](https://image.slidesharecdn.com/solr-presentation-151018095953-lva1-app6891/85/Introduction-to-Apache-Solr-34-320.jpg)
![RUN SOLR
44 bob@bobos-PC$ /opt/solr-5.3.1 $ bin/solr start -p 8983
Waiting up to 30 seconds to see Solr running on port 8983
[/]
Started Solr server on port 8983 (pid=6240). Happy
searching!
(in Windows use: bin/solr.cmd)](https://image.slidesharecdn.com/solr-presentation-151018095953-lva1-app6891/85/Introduction-to-Apache-Solr-44-320.jpg)
Introduction to Apache Solr
- 1. Introduction to Apache Solr Thessaloniki Java Meetup 2015-10-16 Christos Manios
- 2. 2Contents 1. What is Solr 2. Solr Architecture / Concepts 3. Install / Configure 4. Index, Query, Update, Delete data 5. Solr integration 6. Solr resources 7. SolrCloud
- 3. WHAT IS SOLR (and why we care so much about it!) 3
- 4. WHAT IS SOLR ▸ A search engine ▸ A REST API ▸ Built on Lucene ▸ Open Source ▸ Blazing-fast ▸ Scalable ▸ Fault tolerant 4
- 5. WHY SOLR ▸ Text search faster than RDBMS ▸ Solr knows about languages ▸ Specific features: ▹ Highlighting ▹ Faceting ▹ Scoring/Boost and many more !! 5
- 6. SOLR TIMELINE 6 1999 2004 2010 2015 Doug Cutting creates Lucene Version 5.0 Yonik Seeley creates Solr Lucene and Solr merge 2012 Solr 4 and introduction of SolrCloud
- 7. WHO USES SOLR LinkedIn DuckDuckGo IBM Websphere Commerce AT&T Apple eBay MTV Networks Magento O.T.S. Instagram Nasa Netflix Disney Buy.com Adobe SAP Hybris Bloomberg 7 and many more!
- 8. Does Solr fit in our application? “Well…. it depends!” 8
- 10. BASIC CONCEPTS ▸ Standalone application server (Jetty powered) ▸ Document oriented ▸ Schema (less) ▸ Not ACID (document atomicity) 10
- 11. ARCHITECTURE 11
- 15. SOLR CONCEPTS AND TERMINOLOGY ▸ Node ▸ Core ▸ Schema ▸ ConfigSet ▸ SolrCloud ▹ Collection ▹ Shard ▹ Zookeeper 15
- 16. SCHEMA ▸ Every Solr core has a schema ▸ Defined in schema.xml ▸ Contains: ▹ Fields ▹ Data types ▹ Analysers 16
- 17. MANAGED SCHEMA ▸ Solr supports schemaless mode ▸ Not recommended for production ▸ Performance implications 17
- 18. FIELD TYPES ▸ int, float, long, double ▸ date ▸ string ▸ text (multilingual ** ) ▸ location 18
- 19. COMMON FIELD ATTRIBUTES ▸ indexed ▸ stored ▸ type ▸ multivalued 19 Example: <field name="id" type="string" indexed="true" stored="true" required="true" multiValued=" false" />
- 20. DYNAMIC FIELDS ▸ Fields not explicitly defined in schema ▸ Field names must match a pattern ▸ Field names prefixed or suffixed with a wildcard ▸ Make schema dynamic <dynamicField name="*_i" type="tint" indexed="true" stored="true"/> 20
- 22. INDEXING You can index one or more documents using: ▸ bin/post command ▸ REST api ▸ SolrJ or other libraries ▸ DataImportHandler 22
- 23. INDEXING (2) REST API example: curl http://localhost:8983/solr/my_collection/update -H "Content-Type: text/xml" --data-binary ' <add> <doc> <field name="id">012ab1</field> <field name="authors">Patrick Eagar</field> <field name="subject">Sports</field> <field name="dd">796.35</field> <field name="isbn">0002166313</field> <field name="yearpub">1982</field> <field name="publisher">Collins</field> </doc> </add>' 23
- 24. INDEXING MULTIPLE DOCS (JSON) REST API example: curl -X POST -H 'Content-Type: application/json' 'http://localhost: 8983/solr/my_collection/update' --data-binary ' [ { "id": "1", "title": "Doc 1" }, { "id": "2", "title": "Doc 2" } ]' 24
- 25. SEARCHING REST API example: curl http://192.168.1.2:8983/solr/javameetup/select?q=*%3A* &sort=creatorName_txtel_diav+desc &rows=10 &fl=id &wt=json &indent=true 25
- 26. SEARCHING: QUERY PARSERS Solr has the following query parsers: ▸ Standard (lucene) ▸ Dismax ▸ Edismax 26
- 27. SEARCHING: RANGE QUERIES ▸ Allow the selection of documents whose fields fall within a range ▸ Ranges with [] are inclusive at both sides ▹ price:[0 TO 100] ▹ price:[0 TO *] ▹ price:[* TO 100] ▸ Range queries with {} are exclusive ▹ price:{0 TO 100} ▸ Can mix { and ] ▹ price:[0 TO 100} 27
- 28. SEARCHING: DATE QUERIES ▸ Date format: 2015-10-16T19:19:59Z ▸ Dates are stored in UTC. ▸ Date math ▹ NOW ▹ NOW/YEAR ▹ NOW/HOUR ▹ NOW/MONTH ▹ NOW/SECOND 28
- 29. ▸ Boolean queries: ▹ +this -that ▹ this AND that ▸ Field queries: ▹ title: Bob SquarePants ▹ company: Nickelodeon SEARCHING: OTHER QUERIES 29
- 30. SEARCHING: OTHER QUERIES (2) ▸ Phrase/proximity queries: ▹ "Sheldon Couper" matches only Sheldon Couper ▹ "Sheldon Couper"~1 matches Sheldon Lee Couper ▸ Multi-term queries: ▹ title:Ιωάννης Μακρυγιάννης ▹ title:(Ιωάννης Μακρυγιάννης) ▸ Combine them: ▹ +this -title:that +price:[* TO 100] – name:"Sheldon Couper" 30
- 31. SEARCHING: FUZZY & WILDCARD QUERIES ▸ Sometimes we don't know exactly what you are looking: ▹ It starts with pro: pro* ▹ It ends with tion: *tion ▹ Not sure about a letter: j?t ▸ Something like chris: ▹ chris~ ▹ chris~0.9 ▸ Regular expression: /H.*t/ matches Hornet 31
- 32. SEARCHING: RELEVANCY Relevancy is the quality of results returned from a query, encompassing both what documents are found, and their relative ranking (the order that they are returned to the user.) 32
- 33. SEARCHING: RELEVANCY EXAMPLE ▸ Find all people with name “Κώστας” and return politicians first: ▹ q=name:”Κώστας” +occupation: Politician~100 33
- 34. SEARCHING: FILTER QUERIES ▸ Limit the possible responses to the main query ▸ Do not change ordering or scoring ▸ Can be based on any query type ▸ Example: &fq=category:music &fq=price:[0 TO 100] &fq=rating:[3 TO *] 34
- 35. SEARCHING: SORTING ▸ Solr can sort by ▹ Score ▹ A value in a field ▹ A function ▸ In ascending or descending order ▸ Multiple fields: &sort=name asc,age desc 35
- 38. UPDATE DOCUMENT EXAMPLE 38 ▸ Solr performs atomic (partial) updates. ▹ It marks the old version of the document as deleted ▹ It adds the new version of the document. ▹ Updates are based on the unique ID. ▹ Not possible to update by query.
- 39. DELETE DOCUMENTS 39 ▸ Delete documents by query (WARNING! The following deletes all docs!! http://192.168.1.1:8983/solr/update? commit=true&stream.body=<delete><query>*: *</query></delete>
- 40. SEARCH SPEED? 40
- 41. SEARCH SPEED PARAMETERS 41 It depends on: 1. Document size 2. Field cardinality 3. RAM assigned to JVM 4. Indexing rate (updates / sec) 5. Query rate (queries / sec) 6. Query quality Be careful or it will become:
- 43. INSTALL SOLR ▸ Download from a mirror ▸ Unzip ▸ Run 43 bob@bobos-PC$ ls solr* solr-5.3.1.zip bob@bobos-PC$ unzip -q solr-5.3.1.zip bob@bobos-PC$ cd solr-5.3.1/
- 44. RUN SOLR 44 bob@bobos-PC$ /opt/solr-5.3.1 $ bin/solr start -p 8983 Waiting up to 30 seconds to see Solr running on port 8983 [/] Started Solr server on port 8983 (pid=6240). Happy searching! (in Windows use: bin/solr.cmd)
- 45. CREATE A NEW CORE 45 $ bin/solr create_core -c javameetup -d basic_configs Setup new core instance directory: /opt/solr-5.3.1/server/solr/javameetup
- 47. RESOURCES ▸ Official Lucene page: ▹ http://lucene.apache.org ▸ Official Solr page: ▹ http://lucene.apache.org/solr 47
- 48. RESOURCES (2) Solr official resources page provides links to: ▸ Tutorials ▸ Release documentation ▸ Reference guide ▸ Mailing lists 48
- 49. SOLR INTEGRATIO N 49 Solr is integrated with multiple languages via libraries: ▸ Java (solrj, spring-data-solr) ▸ Python ▸ PHP ▸ .NET ▸ Go for a full list see here.
- 50. SOLR INTEGRATIO N (2) 50 Solr can be combined with big data software such as: ▸ Apache Hadoop ▸ Apache Cassandra ▸ Apache Spark ▸ Apache Mahout
- 51. SolrCloud 51
- 52. INDEX SIZE ▸ Be constantly aware of your index size:52
- 53. 2,100,000,000 maximum number of documents per core or shard 53 For more, consider SolrCloud solution!
- 54. SOLRCLOUD CHARACTERISTI CS ▸ Distributed search ▸ Sharding ▸ Fault tolerance ▸ High availability ▸ Apache Zookeeper coordinates: ▹ shard leader election ▹ updates distribution to shard leaders 54
- 55. SOLRCLOUD ADMIN PAGE 55 Collection with one shard
- 56. SOLRCLOUD ADMIN PAGE (2) 56 Collection with one shard
- 57. 57 Questions? About me: ▸ https://manios.org ▸ https://github.com/manios