SlideShare a Scribd company logo

Apache Solr crash course

Tommaso Teofili
Tommaso Teofili

The document provides an overview and agenda for an Apache Solr crash course. It discusses topics such as information retrieval, inverted indexes, metrics for evaluating IR systems, Apache Lucene, the Lucene and Solr APIs, indexing, searching, querying, filtering, faceting, highlighting, spellchecking, geospatial search, and Solr architectures including single core, multi-core, replication, and sharding. It also provides tips on performance tuning, using plugins, and developing a Solr-based search engine.

Technology
1 of 72
Downloaded 1,396 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Apache Solr crash course Tommaso Teofili
Agenda • IR • Solr • Tips&tricks • Case study • Extras
Information Retrieval • “Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers)” - P. Nayak, Stanford University
Inverted index • Each document has an id and a list of terms • For each term t we must store a list of all documents that contain t • Identify each document by its id
IR Metrics • How good is an IR system? • Precision: Fraction of retrieved docs that are relevant to user’s information need • Recall: Fraction of relevant docs in collection that are retrieved
Apache Lucene • Information Retrieval library • Inverted index of documents • Vector space model • Advanced search options (synonims, stopwords, similarity, proximity)
Lucene API - indexing • Lucene indexes are built on a Directory • Directory can be accessed by IndexReaders and IndexWriters • IndexSearchers are built on top of Directories and IndexReaders • IndexWriters can write Documents inside the index • Documents are made of Fields • Fields have values • Directory > IndexReader/Writer > Document > Field
Lucene API - searching • Open an IndexSearcher on top of an IndexReader over a Directory • Many query types: TermQuery, MultiTermQuery, BooleanQuery, WildcardQuery, PhraseQuery, PrefixQuery, MultiPhraseQuery, FuzzyQuery, NumericRangeQuery, ... • Get results from a TopDocs object
Apache Solr • Ready to use enterprise search server • REST (and programmatic) API • Results in XML, JSON, PHP, Ruby, etc... • Exploit Lucene power • Scaling capabilities (replication, distributed search, ...) • Administration interface • Customizable via plugins
Apache Solr 3.1.0
Apache Solr - admin UI
Solr - project status • Solr 3.1.0 version released in March 2011 • Lucene/Solr is now a single project • Huge community • Backed by Lucid Imagination
Solr basic configuration • schema.xml • contains types definitions for field analysis (field type+tokenizers+filters) • contains field definitions • solrconfig.xml • contains the Solr instance configuration
Solr - schema.xml • Types (with index/query Analyzers) • Fields with name, type and options • Unique key • Dynamic fields • Copy fields
Solr - content analysis • define documents’ model • each document consists of fields • each field • has attributes telling Solr how to handle its contents • contains free text, keywords, dates, numbers, etc.
Solr - content analysis • Analyzer: create tokens using a Tokenizer and, eventually, some filters (TokenFilters) • Each field can define an Analyzer at ‘query’ time and another at ‘index’ time, or the same in both cases • Each field can be indexed (searchable), stored (possibly fetched with results), multivalued, required, etc.
Solr - content analysis • Commonly used tokenizers: • WhitespaceTokenizerFactory • StandardTokenizerFactory • KeywordTokenizerFactory • PatternTokenizerFactory • HTMLStripWhitespaceTokenizerFactory • HTMLStripStandardTokenizerFactory
Solr - content analysis • Commonly used TokenFilters: • SnowballPorterFilterFactory • StopFilterFactory • LengthFilterFactory • LowerCaseFilterFactory • WordDelimiterFilterFactory • SynonymFilterFactory • PatternReplaceFilterFactory • ReverseWildcardFilterFactory • CharFilterFactories (Mapping,HtmlString)
Debugging analysis
Solr - solrconfig.xml • Data directory (where Solr will write the Lucene index) • Caches configuration: documents, query results, filters • Request handlers definition (search/update handlers) • Update request processor chains definition • Event listeners (newSearcher, firstSearcher) • Fine tuning parameters • ...
Solr - indexing • Update requests on index are given with XML commands via HTTP POST • <add> to insert and update • <add> <doc boost="2.5"> • <field name="employeeId">05991</field> • </doc></add> • <delete> to remove by unique key or query • <delete><id>05991</id></delete> • <delete><query>office:Bridgewater</query></delete> • <commit/> reopen readers on the new index version • <optimize/> optimize index internal structure for faster access
Solr - basic indexing • REST call - XML/JSON • curl ‘http://localhost:8983/solr/update? commit=true’ -H "Content-Type: text/xml" -- data-binary '<add><doc><field name="id">testdoc</field></doc></add>' • curl 'http://localhost:8983/solr/update/json? commit=true' -H 'Content-type:application/json' -d ' { "add": {"doc": {"id" : "TestDoc1", "title" : "test1"} } }’
Solr - binary files indexing • Many documents are produced in (properietary) binary formats : PDF, RTF, XLS, etc. • Apache Tika integrated in Solr REST service for indexing such documents • curl "http://localhost:8983/solr/update/ extract?literal.id=doc1&commit=true" -F "myfile=@tutorial.html"
Solr - index analysis • Luke is a tool for navigating Lucene indexes • For each field : top terms, distinct terms, terms histogram, etc. • LukeRequestHandler : • http://localhost:8983/solr/admin/luke? wt=xslt&tr=luke.xsl
Solr - data import handler • DBMS • FileSystem • HTTP
Solr - searching
Solr - searching
Solr - query syntax • query fields with fieldname:value • + - AND OR NOT operators • Range queries on date or numeric fields, ex: timestamp:[* TO NOW] • Boost terms, ex: people^4 profits • Fuzzy search, ex: roam~0.6 • Proximity search, ex: “apache solr”~2 • ...
Solr - basic search • parameters: • q: the query • start: offset of the first result • rows: max no. of results returned • fl: comma separated list of fields to return • defType: specify the query parser • debugQuery: enable query debugging • wt: result format (xml, json, php, ruby, javabin, etc)
Solr - query parsers • Most used: • Default Lucene query parser • DisMax query parser • eDisMax query parser
Solr - highlighting • can be done on fields with stored=”true” • returns a snippet containing the higlighted terms for each doc • enabled with hl=true&hl.fl=fieldname1,fieldname2
Solr - sorting results • Sorting can be done on the "score" of the document, or on any multiValued="false" indexed="true" field provided that field is either non-tokenized (ie: has no Analyzer) or uses an Analyzer that only produces a single term • add parameter &sort=score desc, inStock desc, price asc • can sort on function queries (see later)
Solr - filter queries • get a subset of the index • place it in a cache • run queries for such a “filter” in memory • add parameter &fq=category:hardware • if multiple fq parameters the query will be run against the intersection of the specified filters
Solr - facets • facet by: • field value • arbitrary queries • range • can facet on fields with indexed=”true”
Solr - function queries • allow deep customization of ranking : • http://localhost:8983/solr/select/? fl=score,id&q=DDR&sort=termfreq (text,memory)%20desc • functions : sum, sub, product , div ,pow, abs, log, sqrt, map, scale, termfreq, ...
Solr - query elevation • useful for “marketing” • configure the top results for a given query regardless of the normal Lucene scoring • http://localhost:8983/solr/elevate?q=best %20product&enableElevation=true
Solr - spellchecking • collects suggestions about input query • eventually correct user query with “suggested” terms
Solr - spellchecking • build a spellcheck index dynamically • return suggested results • http://localhost:8983/solr/spell?q=hell ultrashar&spellcheck=true&spellcheck.collate=true &spellcheck.build=true • useful to create custom query converters • <queryConverter name="queryConverter" class="org.apache.solr.spelling.SpellingQueryConv erter"/>
Solr - similarity • get documents “similar” to a given document or a set of documents • Vector Space Model • http://localhost:8983/solr/select? q=apache&mlt=true&mlt.fl=manu,cat&mlt. mindf=1&mlt.mintf=1&fl=id,score
Solr - geospatial search • index location data • query by spatial concepts and sort by distance • find all documents with store position at no more than 5km than a specified point • http://localhost:8983/solr/select? &indent=true&fl=name,store&q=*:*&fq={!geofilt %20sfield=store}&pt=45.15,-93.85&d=5
Solr - field collapsing • group resulting documents on per field basis • http://localhost:8983/solr/select? &indent=true&fl=id,name&q=solr +memory&group=true&group.field=man u_exact • useful for displaying results in a smart way • see SOLR-236
Solr - join • new feature (SOLR-2272) • many users ask for it • quite of a paradigm change • http://localhost:8983/solr/select?q={!join +from=manu_id_s%20to=id} ipod&fl=id,manu_id&debugQuery=true
Solr statistics
Solr statistics
Solr statistics
Solr replication
Solr Architectures • Simple • Multicore • Replication • Sharded
Solr - MultiCore • Define multiple Solr cores inside one only Solr instance • Each cores maintain its own index • Unified administration interface • Runtime commands to create, reload, load, unload, delete, swap cores • Cores can be thought as ‘collections’ • Allow no downtime while deploying new features/ bugfixes
Solr - Replication • It’s useful in case of high traffic to replicate a Solr instance and split (with eventually a VIP in front) the search load • Master has the “original” index • Slave polls master asking the latest version of index • If slave has a different version of the index asks the master for the delta (rsync like) • In the meanwhile indexes remain available • No impact of indexing on search (almost)
Solr - Shards • When an index is too large, in terms of space or memory required, it can be useful to define two or more shards • A shard is a Solr instance and can be searched or indexed independently • At the same time it’s possible to query all the shards having the result be merged from the sub-results of each shard • http://localhost:8983/solr/select?shards=localhost:8983/ solr,localhost:7574/solr&q=category:information • Note that the document distribution among indexes is up to the user (or who feeds the indexes)
Solr - Architectures • When to use each? • KISS principle • High query load : replication • Huge index : shard
Solr - Architectures • High queries w/ large indexes : shard + replication
Solr - Architectures • Tips & Tricks: • Don’t share indexes between Master and Slaves on distributed file systems (locking) • Anyway get rid of distributed file systems (slow) • Lucene/Solr is I/O intensive thus behaves better with quick disks • Always use MultiCore - hot deploy of changes/ bugfixes • Replication is network intensive • Check replication poll time and indexing rates
Tips&Tricks • Solr based SE development process • Plugins • Performance tuning • Deploy
Process - t0 analysis • Analyze content • Analyze queries • Analyze collections • Pre-existing query/index load (if any) • Expected query/index load • Desired throughput/avg response time • First architecture
Process - n-th iteration • index 10-15% content • search stress test (analyze peaks) - use SolrMeter • quality tests from stakeholders (accuracy, recall) • eventually add/reconfigure features • check http://wiki.apache.org/solr/FieldOptionsByUseCase and make sure fields used for faceting/sorting/highlighting/ etc. have proper options • need to change field types/analysis/options - rebuild the index
Solr - Plugins • QParserPlugin • RequestHandler (Search/UpdateHandler) • UpdateRequestProcessor • ResponseWriter • Cache
Performance tuning • A huge tuning is done in schema.xml • Configure Solr caches • Set auto commit where possible • Play with mergeFactor
Performance tuning • The number of indexed fields greatly increases memory usage during indexing, segment merge time, optimization times, index size • Stored fields impact on index size, search time, ... • set omitNorms=”true” where it makes sense (disabling length normalization and index time boosting) • set omitTermFreqAndPositions=”true” if no queries on this field using positions or should not influence score
Performance tuning • FilterCache - unordered document ids for caching filter queries • QueryResultCache - ordered document ids for caching queries results (caching only the returned docs) • DocumentCache - stores stored fields (at least <max_results> * <max_concurrent_queries> • Setup autowarming - keep caches warm after commits
Performance tuning • Choose correct cache implementation FastLRUCache vs LRUCache • FastLRUCache has faster gets and slower puts in single threaded operation and thus is generally faster than LRUCache when the hit ratio of the cache is high (> 75%)
Performance tuning • Explicit warm sorted fields • Often check cache statistics • JVM options - don’t let the OS without memory! • mergeFactor - impacts on the number of index segments created on the disk • low mF : smaller number of index files, which speeds up searching but more segment merges slow down indexing • high mF : generally improves indexing speed but gets less frequent merges, resulting in a collection with more index files which may slow searching
Performance tuning • set autocommit where possible, this will avoid close and reopen of IndexReaders everytime a document is indexed - can choose max number of documents and/or time to wait before automatically do the commit • finally...need to get your hand dirty!
Deploy • SolrPackager by Simone Tripodi! • It’s a Maven archetype • Create standalone/multicore project • Each project will generate a master and a slave instance • Define environment dependent properties without having to manage N config files • ‘mvn -Pdev package’ // will create a Tomcat package for the development environment
Case study
Case study
Case Study • Architecture analysis • Plugin development • Testing and support
Challenges • Architecture • Schema design
Challenge • Architecture • 4B docs of ~4k each • ~3 req/sec overall • 3 collections: • |archive| = 3B • |2010-2011| = 1M • |intranet| = 0.9B
Challenge • Content analysis • get the example Solr schema.xml • optimize the schema in order to enable both stemmed and unstemmed versions of fields: author, title, text, cat • add omitNorms=”true” where possible • add a field ‘html_content’ which will contain an HTML text but will be searched as clean text • all string fields should be lowercased
Extras • Clustering (Solr-Carrot2) • Named entity extraction (Solr-UIMA) • SolrCloud (Solr-Zookeeper) • ManifoldCF • Stanbol EntityHub • Solandra (Solr-Cassandra)
THANKS!

More Related Content

What's hot (20)

PDF
How Solr Search Works
Atlogys Technical Consulting
 
PDF
Introduction to Apache Solr
Christos Manios
 
PPTX
Transactional operations in Apache Hive: present and future
DataWorks Summit
 
PDF
TFA Collector - what can one do with it
Sandesh Rao
 
PPTX
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
PPTX
Apache Solr
Semih Hakkıoğlu
 
PDF
Introduction to Spark with Python
Gokhan Atil
 
PDF
Physical Plans in Spark SQL
Databricks
 
PPTX
Introduction to Apache Spark
Rahul Jain
 
PDF
Modularized ETL Writing with Apache Spark
Databricks
 
PDF
Deep Dive: Memory Management in Apache Spark
Databricks
 
PDF
Solr for Indexing and Searching Logs
Sematext Group, Inc.
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
PDF
Introduction to PySpark
Russell Jurney
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
PDF
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
Databricks
 
PDF
Fine Tuning and Enhancing Performance of Apache Spark Jobs
Databricks
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PDF
Oracle 21c: New Features and Enhancements of Data Pump & TTS
Christian Gohmann
 
PPTX
ORC File - Optimizing Your Big Data
DataWorks Summit
 
How Solr Search Works
Atlogys Technical Consulting
 
Introduction to Apache Solr
Christos Manios
 
Transactional operations in Apache Hive: present and future
DataWorks Summit
 
TFA Collector - what can one do with it
Sandesh Rao
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
Apache Solr
Semih Hakkıoğlu
 
Introduction to Spark with Python
Gokhan Atil
 
Physical Plans in Spark SQL
Databricks
 
Introduction to Apache Spark
Rahul Jain
 
Modularized ETL Writing with Apache Spark
Databricks
 
Deep Dive: Memory Management in Apache Spark
Databricks
 
Solr for Indexing and Searching Logs
Sematext Group, Inc.
 
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
Introduction to PySpark
Russell Jurney
 
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
Databricks
 
Fine Tuning and Enhancing Performance of Apache Spark Jobs
Databricks
 
Understanding Query Plans and Spark UIs
Databricks
 
Oracle 21c: New Features and Enhancements of Data Pump & TTS
Christian Gohmann
 
ORC File - Optimizing Your Big Data
DataWorks Summit
 

Similar to Apache Solr crash course (20)

KEY
Apache Solr - Enterprise search platform
Tommaso Teofili
 
PPTX
Introduction to Apache Lucene/Solr
Rahul Jain
 
PPTX
Apache Solr Workshop
JSGB
 
PDF
Get the most out of Solr search with PHP
Paul Borgermans
 
PPTX
Introduction to Lucene & Solr and Usecases
Rahul Jain
 
PDF
Introduction to Solr
Erik Hatcher
 
PDF
Apache Solr Workshop
Saumitra Srivastav
 
PDF
Search Engine-Building with Lucene and Solr, Part 1 (SoCal Code Camp LA 2013)
Kai Chan
 
PDF
Basics of Solr and Solr Integration with AEM6
DEEPAK KHETAWAT
 
PPTX
Search Engine Building with Lucene and Solr (So Code Camp San Diego 2014)
Kai Chan
 
PDF
Search Engine-Building with Lucene and Solr
Kai Chan
 
PPTX
Assamese search engine using SOLR by Moinuddin Ahmed ( moin )
'Moinuddin Ahmed
 
PDF
Solr Powered Lucene
Erik Hatcher
 
PDF
Information Retrieval - Data Science Bootcamp
Kais Hassan, PhD
 
PDF
Find it, possibly also near you!
Paul Borgermans
 
PPTX
Introduction to Lucene and Solr - 1
YI-CHING WU
 
PDF
Solr Masterclass Bangkok, June 2014
Alexandre Rafalovitch
 
PDF
Solr search engine with multiple table relation
Jay Bharat
 
PPTX
Apache solr
Péter Király
 
PDF
Rapid Prototyping with Solr
Erik Hatcher
 
Apache Solr - Enterprise search platform
Tommaso Teofili
 
Introduction to Apache Lucene/Solr
Rahul Jain
 
Apache Solr Workshop
JSGB
 
Get the most out of Solr search with PHP
Paul Borgermans
 
Introduction to Lucene & Solr and Usecases
Rahul Jain
 
Introduction to Solr
Erik Hatcher
 
Apache Solr Workshop
Saumitra Srivastav
 
Search Engine-Building with Lucene and Solr, Part 1 (SoCal Code Camp LA 2013)
Kai Chan
 
Basics of Solr and Solr Integration with AEM6
DEEPAK KHETAWAT
 
Search Engine Building with Lucene and Solr (So Code Camp San Diego 2014)
Kai Chan
 
Search Engine-Building with Lucene and Solr
Kai Chan
 
Assamese search engine using SOLR by Moinuddin Ahmed ( moin )
'Moinuddin Ahmed
 
Solr Powered Lucene
Erik Hatcher
 
Information Retrieval - Data Science Bootcamp
Kais Hassan, PhD
 
Find it, possibly also near you!
Paul Borgermans
 
Introduction to Lucene and Solr - 1
YI-CHING WU
 
Solr Masterclass Bangkok, June 2014
Alexandre Rafalovitch
 
Solr search engine with multiple table relation
Jay Bharat
 
Apache solr
Péter Király
 
Rapid Prototyping with Solr
Erik Hatcher
 
Ad

More from Tommaso Teofili (18)

PDF
Affect Enriched Word Embeddings for News IR
Tommaso Teofili
 
PDF
Flexible search in Apache Jackrabbit Oak
Tommaso Teofili
 
PDF
Data replication in Sling
Tommaso Teofili
 
PDF
Search engines in the industry
Tommaso Teofili
 
PDF
Scaling search in Oak with Solr
Tommaso Teofili
 
PDF
Text categorization with Lucene and Solr
Tommaso Teofili
 
PPT
Machine learning with Apache Hama
Tommaso Teofili
 
KEY
Adapting Apache UIMA to OSGi
Tommaso Teofili
 
PDF
Oak / Solr integration
Tommaso Teofili
 
PPTX
Domeo, Text Mining, UIMA and Clerezza
Tommaso Teofili
 
PDF
Natural Language Search in Solr
Tommaso Teofili
 
PDF
Apache UIMA - Hands on code
Tommaso Teofili
 
PDF
Apache UIMA Introduction
Tommaso Teofili
 
ODP
OSS Enterprise Search EU Tour
Tommaso Teofili
 
PDF
Information Extraction with UIMA - Usecases
Tommaso Teofili
 
PDF
Apache UIMA and Metadata Generation
Tommaso Teofili
 
PDF
Data and Information Extraction on the Web
Tommaso Teofili
 
KEY
Apache UIMA and Semantic Search
Tommaso Teofili
 
Affect Enriched Word Embeddings for News IR
Tommaso Teofili
 
Flexible search in Apache Jackrabbit Oak
Tommaso Teofili
 
Data replication in Sling
Tommaso Teofili
 
Search engines in the industry
Tommaso Teofili
 
Scaling search in Oak with Solr
Tommaso Teofili
 
Text categorization with Lucene and Solr
Tommaso Teofili
 
Machine learning with Apache Hama
Tommaso Teofili
 
Adapting Apache UIMA to OSGi
Tommaso Teofili
 
Oak / Solr integration
Tommaso Teofili
 
Domeo, Text Mining, UIMA and Clerezza
Tommaso Teofili
 
Natural Language Search in Solr
Tommaso Teofili
 
Apache UIMA - Hands on code
Tommaso Teofili
 
Apache UIMA Introduction
Tommaso Teofili
 
OSS Enterprise Search EU Tour
Tommaso Teofili
 
Information Extraction with UIMA - Usecases
Tommaso Teofili
 
Apache UIMA and Metadata Generation
Tommaso Teofili
 
Data and Information Extraction on the Web
Tommaso Teofili
 
Apache UIMA and Semantic Search
Tommaso Teofili
 
Ad

Recently uploaded (20)

PPTX
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
PPTX
Digital Circuits, important subject in CS
contactparinay1
 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
Digital Circuits, important subject in CS
contactparinay1
 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 

Apache Solr crash course

  • 1. Apache Solr crash course Tommaso Teofili
  • 2. Agenda • IR • Solr • Tips&tricks • Case study • Extras
  • 3. Information Retrieval • “Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers)” - P. Nayak, Stanford University
  • 4. Inverted index • Each document has an id and a list of terms • For each term t we must store a list of all documents that contain t • Identify each document by its id
  • 5. IR Metrics • How good is an IR system? • Precision: Fraction of retrieved docs that are relevant to user’s information need • Recall: Fraction of relevant docs in collection that are retrieved
  • 6. Apache Lucene • Information Retrieval library • Inverted index of documents • Vector space model • Advanced search options (synonims, stopwords, similarity, proximity)
  • 7. Lucene API - indexing • Lucene indexes are built on a Directory • Directory can be accessed by IndexReaders and IndexWriters • IndexSearchers are built on top of Directories and IndexReaders • IndexWriters can write Documents inside the index • Documents are made of Fields • Fields have values • Directory > IndexReader/Writer > Document > Field
  • 8. Lucene API - searching • Open an IndexSearcher on top of an IndexReader over a Directory • Many query types: TermQuery, MultiTermQuery, BooleanQuery, WildcardQuery, PhraseQuery, PrefixQuery, MultiPhraseQuery, FuzzyQuery, NumericRangeQuery, ... • Get results from a TopDocs object
  • 9. Apache Solr • Ready to use enterprise search server • REST (and programmatic) API • Results in XML, JSON, PHP, Ruby, etc... • Exploit Lucene power • Scaling capabilities (replication, distributed search, ...) • Administration interface • Customizable via plugins
  • 11. Apache Solr - admin UI
  • 12. Solr - project status • Solr 3.1.0 version released in March 2011 • Lucene/Solr is now a single project • Huge community • Backed by Lucid Imagination
  • 13. Solr basic configuration • schema.xml • contains types definitions for field analysis (field type+tokenizers+filters) • contains field definitions • solrconfig.xml • contains the Solr instance configuration
  • 14. Solr - schema.xml • Types (with index/query Analyzers) • Fields with name, type and options • Unique key • Dynamic fields • Copy fields
  • 15. Solr - content analysis • define documents’ model • each document consists of fields • each field • has attributes telling Solr how to handle its contents • contains free text, keywords, dates, numbers, etc.
  • 16. Solr - content analysis • Analyzer: create tokens using a Tokenizer and, eventually, some filters (TokenFilters) • Each field can define an Analyzer at ‘query’ time and another at ‘index’ time, or the same in both cases • Each field can be indexed (searchable), stored (possibly fetched with results), multivalued, required, etc.
  • 17. Solr - content analysis • Commonly used tokenizers: • WhitespaceTokenizerFactory • StandardTokenizerFactory • KeywordTokenizerFactory • PatternTokenizerFactory • HTMLStripWhitespaceTokenizerFactory • HTMLStripStandardTokenizerFactory
  • 18. Solr - content analysis • Commonly used TokenFilters: • SnowballPorterFilterFactory • StopFilterFactory • LengthFilterFactory • LowerCaseFilterFactory • WordDelimiterFilterFactory • SynonymFilterFactory • PatternReplaceFilterFactory • ReverseWildcardFilterFactory • CharFilterFactories (Mapping,HtmlString)
  • 20. Solr - solrconfig.xml • Data directory (where Solr will write the Lucene index) • Caches configuration: documents, query results, filters • Request handlers definition (search/update handlers) • Update request processor chains definition • Event listeners (newSearcher, firstSearcher) • Fine tuning parameters • ...
  • 21. Solr - indexing • Update requests on index are given with XML commands via HTTP POST • <add> to insert and update • <add> <doc boost="2.5"> • <field name="employeeId">05991</field> • </doc></add> • <delete> to remove by unique key or query • <delete><id>05991</id></delete> • <delete><query>office:Bridgewater</query></delete> • <commit/> reopen readers on the new index version • <optimize/> optimize index internal structure for faster access
  • 22. Solr - basic indexing • REST call - XML/JSON • curl ‘http://localhost:8983/solr/update? commit=true’ -H "Content-Type: text/xml" -- data-binary '<add><doc><field name="id">testdoc</field></doc></add>' • curl 'http://localhost:8983/solr/update/json? commit=true' -H 'Content-type:application/json' -d ' { "add": {"doc": {"id" : "TestDoc1", "title" : "test1"} } }’
  • 23. Solr - binary files indexing • Many documents are produced in (properietary) binary formats : PDF, RTF, XLS, etc. • Apache Tika integrated in Solr REST service for indexing such documents • curl "http://localhost:8983/solr/update/ extract?literal.id=doc1&commit=true" -F "myfi[email protected]"
  • 24. Solr - index analysis • Luke is a tool for navigating Lucene indexes • For each field : top terms, distinct terms, terms histogram, etc. • LukeRequestHandler : • http://localhost:8983/solr/admin/luke? wt=xslt&tr=luke.xsl
  • 25. Solr - data import handler • DBMS • FileSystem • HTTP
  • 28. Solr - query syntax • query fields with fieldname:value • + - AND OR NOT operators • Range queries on date or numeric fields, ex: timestamp:[* TO NOW] • Boost terms, ex: people^4 profits • Fuzzy search, ex: roam~0.6 • Proximity search, ex: “apache solr”~2 • ...
  • 29. Solr - basic search • parameters: • q: the query • start: offset of the first result • rows: max no. of results returned • fl: comma separated list of fields to return • defType: specify the query parser • debugQuery: enable query debugging • wt: result format (xml, json, php, ruby, javabin, etc)
  • 30. Solr - query parsers • Most used: • Default Lucene query parser • DisMax query parser • eDisMax query parser
  • 31. Solr - highlighting • can be done on fields with stored=”true” • returns a snippet containing the higlighted terms for each doc • enabled with hl=true&hl.fl=fieldname1,fieldname2
  • 32. Solr - sorting results • Sorting can be done on the "score" of the document, or on any multiValued="false" indexed="true" field provided that field is either non-tokenized (ie: has no Analyzer) or uses an Analyzer that only produces a single term • add parameter &sort=score desc, inStock desc, price asc • can sort on function queries (see later)
  • 33. Solr - filter queries • get a subset of the index • place it in a cache • run queries for such a “filter” in memory • add parameter &fq=category:hardware • if multiple fq parameters the query will be run against the intersection of the specified filters
  • 34. Solr - facets • facet by: • field value • arbitrary queries • range • can facet on fields with indexed=”true”
  • 35. Solr - function queries • allow deep customization of ranking : • http://localhost:8983/solr/select/? fl=score,id&q=DDR&sort=termfreq (text,memory)%20desc • functions : sum, sub, product , div ,pow, abs, log, sqrt, map, scale, termfreq, ...
  • 36. Solr - query elevation • useful for “marketing” • configure the top results for a given query regardless of the normal Lucene scoring • http://localhost:8983/solr/elevate?q=best %20product&enableElevation=true
  • 37. Solr - spellchecking • collects suggestions about input query • eventually correct user query with “suggested” terms
  • 38. Solr - spellchecking • build a spellcheck index dynamically • return suggested results • http://localhost:8983/solr/spell?q=hell ultrashar&spellcheck=true&spellcheck.collate=true &spellcheck.build=true • useful to create custom query converters • <queryConverter name="queryConverter" class="org.apache.solr.spelling.SpellingQueryConv erter"/>
  • 39. Solr - similarity • get documents “similar” to a given document or a set of documents • Vector Space Model • http://localhost:8983/solr/select? q=apache&mlt=true&mlt.fl=manu,cat&mlt. mindf=1&mlt.mintf=1&fl=id,score
  • 40. Solr - geospatial search • index location data • query by spatial concepts and sort by distance • find all documents with store position at no more than 5km than a specified point • http://localhost:8983/solr/select? &indent=true&fl=name,store&q=*:*&fq={!geofilt %20sfield=store}&pt=45.15,-93.85&d=5
  • 41. Solr - field collapsing • group resulting documents on per field basis • http://localhost:8983/solr/select? &indent=true&fl=id,name&q=solr +memory&group=true&group.field=man u_exact • useful for displaying results in a smart way • see SOLR-236
  • 42. Solr - join • new feature (SOLR-2272) • many users ask for it • quite of a paradigm change • http://localhost:8983/solr/select?q={!join +from=manu_id_s%20to=id} ipod&fl=id,manu_id&debugQuery=true
  • 47. Solr Architectures • Simple • Multicore • Replication • Sharded
  • 48. Solr - MultiCore • Define multiple Solr cores inside one only Solr instance • Each cores maintain its own index • Unified administration interface • Runtime commands to create, reload, load, unload, delete, swap cores • Cores can be thought as ‘collections’ • Allow no downtime while deploying new features/ bugfixes
  • 49. Solr - Replication • It’s useful in case of high traffic to replicate a Solr instance and split (with eventually a VIP in front) the search load • Master has the “original” index • Slave polls master asking the latest version of index • If slave has a different version of the index asks the master for the delta (rsync like) • In the meanwhile indexes remain available • No impact of indexing on search (almost)
  • 50. Solr - Shards • When an index is too large, in terms of space or memory required, it can be useful to define two or more shards • A shard is a Solr instance and can be searched or indexed independently • At the same time it’s possible to query all the shards having the result be merged from the sub-results of each shard • http://localhost:8983/solr/select?shards=localhost:8983/ solr,localhost:7574/solr&q=category:information • Note that the document distribution among indexes is up to the user (or who feeds the indexes)
  • 51. Solr - Architectures • When to use each? • KISS principle • High query load : replication • Huge index : shard
  • 52. Solr - Architectures • High queries w/ large indexes : shard + replication
  • 53. Solr - Architectures • Tips & Tricks: • Don’t share indexes between Master and Slaves on distributed file systems (locking) • Anyway get rid of distributed file systems (slow) • Lucene/Solr is I/O intensive thus behaves better with quick disks • Always use MultiCore - hot deploy of changes/ bugfixes • Replication is network intensive • Check replication poll time and indexing rates
  • 54. Tips&Tricks • Solr based SE development process • Plugins • Performance tuning • Deploy
  • 55. Process - t0 analysis • Analyze content • Analyze queries • Analyze collections • Pre-existing query/index load (if any) • Expected query/index load • Desired throughput/avg response time • First architecture
  • 56. Process - n-th iteration • index 10-15% content • search stress test (analyze peaks) - use SolrMeter • quality tests from stakeholders (accuracy, recall) • eventually add/reconfigure features • check http://wiki.apache.org/solr/FieldOptionsByUseCase and make sure fields used for faceting/sorting/highlighting/ etc. have proper options • need to change field types/analysis/options - rebuild the index
  • 57. Solr - Plugins • QParserPlugin • RequestHandler (Search/UpdateHandler) • UpdateRequestProcessor • ResponseWriter • Cache
  • 58. Performance tuning • A huge tuning is done in schema.xml • Configure Solr caches • Set auto commit where possible • Play with mergeFactor
  • 59. Performance tuning • The number of indexed fields greatly increases memory usage during indexing, segment merge time, optimization times, index size • Stored fields impact on index size, search time, ... • set omitNorms=”true” where it makes sense (disabling length normalization and index time boosting) • set omitTermFreqAndPositions=”true” if no queries on this field using positions or should not influence score
  • 60. Performance tuning • FilterCache - unordered document ids for caching filter queries • QueryResultCache - ordered document ids for caching queries results (caching only the returned docs) • DocumentCache - stores stored fields (at least <max_results> * <max_concurrent_queries> • Setup autowarming - keep caches warm after commits
  • 61. Performance tuning • Choose correct cache implementation FastLRUCache vs LRUCache • FastLRUCache has faster gets and slower puts in single threaded operation and thus is generally faster than LRUCache when the hit ratio of the cache is high (> 75%)
  • 62. Performance tuning • Explicit warm sorted fields • Often check cache statistics • JVM options - don’t let the OS without memory! • mergeFactor - impacts on the number of index segments created on the disk • low mF : smaller number of index files, which speeds up searching but more segment merges slow down indexing • high mF : generally improves indexing speed but gets less frequent merges, resulting in a collection with more index files which may slow searching
  • 63. Performance tuning • set autocommit where possible, this will avoid close and reopen of IndexReaders everytime a document is indexed - can choose max number of documents and/or time to wait before automatically do the commit • finally...need to get your hand dirty!
  • 64. Deploy • SolrPackager by Simone Tripodi! • It’s a Maven archetype • Create standalone/multicore project • Each project will generate a master and a slave instance • Define environment dependent properties without having to manage N config files • ‘mvn -Pdev package’ // will create a Tomcat package for the development environment
  • 67. Case Study • Architecture analysis • Plugin development • Testing and support
  • 69. Challenge • Architecture • 4B docs of ~4k each • ~3 req/sec overall • 3 collections: • |archive| = 3B • |2010-2011| = 1M • |intranet| = 0.9B
  • 70. Challenge • Content analysis • get the example Solr schema.xml • optimize the schema in order to enable both stemmed and unstemmed versions of fields: author, title, text, cat • add omitNorms=”true” where possible • add a field ‘html_content’ which will contain an HTML text but will be searched as clean text • all string fields should be lowercased
  • 71. Extras • Clustering (Solr-Carrot2) • Named entity extraction (Solr-UIMA) • SolrCloud (Solr-Zookeeper) • ManifoldCF • Stanbol EntityHub • Solandra (Solr-Cassandra)