SlideShare a Scribd company logo
Lucene And Solr Document
Classification
Alessandro Benedetti, Software Engineer, Sease Ltd.
Alessandro Benedetti
● Search Consultant
● R&D Software Engineer
● Master in Computer Science
● Apache Lucene/Solr Enthusiast
● Semantic, NLP, Machine Learning Technologies passionate
● Beach Volleyball Player & Snowboarder
Who I am
● Classification
● Lucene Approach
● Solr Integration
● Demo
● Extensions
● Future Work
Agenda
“Classification is the problem of identifying
to which of a set of categories
(sub-populations) a new observation
belongs, on the basis of a training set of data
containing observations (or instances)
whose category membership is known. “
Wikipedia
Classification
● E-mail spam filter
● Document categorization
● Sexually explicit content detection
● Medical diagnosis
● E-commerce
● Language identification
Real World Use Cases
● Supervised learning
● Labelled training samples
● Documents modelled as
feature vectors
● Term occurrences as features
● Model predicts unseen documents
label
Basics Of Text Classification
Apache Lucene
Apache LuceneTM
is a high-performance, full-featured text search engine library
written entirely in Java.
It is a technology suitable for nearly any application that requires full-text search,
especially cross-platform.
Apache Lucene is an open source project available for free download.
● Lucene index has complex data structures
● Lot of organizations have already indexes in place
● Pre existent data can be used to classify
● No need to train a model from a separate training set
● From training set to Inverted index
Apache Lucene For Classification
● Advanced configurable text analysis
● Term frequencies
● Term positions
● Document frequencies
● Norms
● Part of speech tags and custom payload
Apache Lucene For Classification
● Given an index with labelled documents
● Each document has a class field
● Given an unknown document in input
● Given a set of relevant fields
● Search the top K most similar documents
● Fetch the classes from the retrieved documents
● Return most occurring class(es)
● Class ranking in retrieved documents is important !
K Nearest Neighbours
● KNN uses Lucene More Like This
● Lucene query component
● Extract interesting terms* from the input document fields
● Build a Lucene query
● Run the query against the search index
● Resulting documents are “the similar documents”
* an interesting term is a term :
- occurring frequently in the seed document (high term frequency)
- but quite rare in the corpus (high inverted document frequency)
More Like This
Assumptions
● Term occurrences are probabilistic independent features
● Terms positions are irrelevant ( bag of words )
Calculate the probability score of each available class C
● Prior ( #DocsInClassC / #Docs )
● Likelihood ( P(d|c) = P(t1, t2,..., tn|c) == P(t1|c) * P(t2|c) * … * P(tn|c))
Where given term t
P(t|c) = TF(t) in documents of class c +1 /
#terms in all documents of class c + #docs of class c
Assign top scoring class
Naive Bayes Classifier
● Documents are the Lucene unit of information
● Documents are a map field -> value
● Each field may be analysed differently
(different tokenization and token filtering)
● Each field may have a different weight for the classification
(affecting differently the similarity score)
Document Classification
Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project.
Its major features include powerful full-text search, hit highlighting, faceted
search and analytics, rich document parsing, geospatial search, extensive
REST APIs as well as parallel SQL.
Apache Solr
Index Time Integration - SOLR-7739
● Ingest the document
● Assign the class
● Set the class as a field value
● Index the document
Request Handler Integration (TO DO) - SOLR-7738
Return an assigned class :
● Given a text and a field
● Given an input document
● Given an indexed document id
Solr Integration
● Pipeline of processors
● Each single document flows
through the chain
● Each processor is executed once
● Last processor triggers the
update command
Update Request Processor Chain
● Update Component
● Configurable Singleton Factory
● Single instance per request thread
● Process a single Document
● SolrCloud compatible*
* Pre processor / Post processor
Update Request Processor
● Access the Index Reader
● A Lucene Document Classifier is instantiated
● A class is assigned by the classifier
● A new field is added to the original Document, with the class
● The document goes through the next processing steps
Classification Update Request Processor
...
<initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell,/browse">
<lst name="defaults">
<str name="df">text</str>
<str name="update.chain">classification</str>
</lst>
</initParams>
...
Solrconfig.xml - Update Handler
...
<updateRequestProcessorChain name="classification">
<processor class="solr.ClassificationUpdateProcessorFactory">
...
</processor>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
...
Solrconfig.xml - Chain configuration
<processor class="solr.ClassificationUpdateProcessorFactory">
<str name="inputFields">title^1.5,content,author</str>
<str name="classField">cat</str>
<str name="algorithm">knn</str>
<str name="knn.k">20</str>
<str name="knn.minTf">1</str>
<str name="knn.minDf">5</str>
</processor>
N.B. classField must be stored
Solrconfig.xml - K nearest neighbour classifier config
<processor class="solr.ClassificationUpdateProcessorFactory">
<str name="inputFields">title^1.5,content,author</str>
<str name="classField">cat</str>
<str name="algorithm">bayes</str>
</processor>
N.B. classField must be Indexed (take care of analysis)
Solrconfig.xml - Naive Bayes classifier config
● Lucene >= 6.0
● Solr >= 6.1
● Classification needs a training set ->
An index with initially human assigned classes is required
Solr Classification - Important Notes
● Sci-Fi StackExchange dataset
● Roughly 18.000 questions and answers
● Roughly 6.000 tagged
● 70 % Training Set + 30% test set
Solr Classification - Demo
● Index the training set documents
(this is our ground truth)
● Index the test set
(classification will happen automatically at indexing time)
● Evaluate the test set
(a simple java app to verify that the automatically assigned classes
are consistent with what expected)
Solr Classification - Demo
● True Positive : Predicted class == actual class
● False Positive : Predicted class != actual class
● True Negative : Not predicted class != actual class
● False Negative : Not predicted class == actual class
Precision = TP / TP+FP
Recall = TP / TP+FN
Solr Classification - System Evaluation Metrics
● Index the training set documents
(this is our ground truth)
● Index the test set
(classification will happen automatically at indexing time)
● Evaluate the test set
(a simple java app to verify that the automatically assigned classes
are consistent with what expected)
Solr Classification - Demo
MaxOutputClasses 1
[System Global Accuracy]0.5095676824946846
[System Globel Recall]0.2686846038863976
TP{star-wars}59
FP{star-wars}75
FN{star-wars}7
[Precision (of predicted)]{star-wars}0.44029850746268656
[Recall for class)]{star-wars}0.8939393939393939
TP{harry-potter}147
FP{harry-potter}137
FN{harry-potter}3
[Precision (of predicted)]{harry-potter}0.5176056338028169
[Recall for class]{harry-potter}0.98
Solr Classification - Demo - Full Dataset
MaxOutputClasses 5
[System Global Accuracy]0.20481927710843373
[System Globel Recall]0.5399850523168909
TP{star-wars}66
FP{star-wars}400
FN{star-wars}0
[Precision (of predicted)]{star-wars}0.14163090128755365
[Recall for class)]{star-wars}1.0
TP{harry-potter}150
FP{harry-potter}584
FN{harry-potter}0
[Precision (of predicted)]{harry-potter}0.20435967302452315
[Recall for class]{harry-potter}1.0
Solr Classification - Demo - Full Dataset
MaxOutputClasses 1
[System Global Accuracy]0.9907407407407407
[System Globel Recall]0.6750788643533123
TP{star-wars}64
FP{star-wars}0
FN{star-wars}2
[Precision (of predicted)]{star-wars}1.0
[Recall for class)]{star-wars}0.9696969696969697
TP{harry-potter}150
FP{harry-potter}2
FN{harry-potter}0
[Precision (of predicted)]{harry-potter}0.9868421052631579
[Recall for class]{harry-potter}1.0
Solr Classification - Demo - Partial Dataset
MaxOutputClasses 5
[System Global Accuracy]0.24259259259259258
[System Globel Recall]0.8264984227129337
TP{star-wars}66
FP{star-wars}52
FN{star-wars}0
[Precision (of predicted)]{star-wars}0.559322033898305
[Recall for class)]{star-wars}1.0
TP{harry-potter}150
FP{harry-potter}48
FN{harry-potter}0
[Precision (of predicted)]{harry-potter}0.7575757575757576
[Recall for class]{harry-potter}1.0
Solr Classification - Demo - Partial Dataset
Multi classes support
● Class field may be multi valued
● Assign multiple classes
● Not only the top scoring but top N (parameter)
Split human/auto assigned classes
● classTrainingField
● classOutputField
Default : use the same field
Solr Classification - Extensions SOLR-8871
Classification Context Filtering
● Reduce the document space to consider ->
reduce the training set
● Useful when only a subset of the index may be interesting for
classification
● Consider only the human labelled documents as training data
Solr Classification - Extensions SOLR-8871
Individual Field Weighting
● When classifying, each field has a different importance
e.g.
title vs content
● Set a different boost per field
● Knn compatible
● Bayes compatible
Solr Classification - Extensions SOLR-8871
● Numeric Field Support (Knn)
(Euclidean distance based)
● Lat lon support (Knn)
(geo distance based)
● SolrCloud support
(use the entire sharded index as training set)
Solr Classification - Future Work
Questions ?
● Special thanks to Tommaso Teofili,
Apache committer who followed the developments and made possible the
contributions.
● And to the
Audience :)

More Related Content

What's hot (20)

PDF
Apache flink
pranay kumar
 
PPTX
Kafka connect 101
Whiteklay
 
PDF
Observability: Beyond the Three Pillars with Spring
VMware Tanzu
 
PDF
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
HostedbyConfluent
 
PDF
Apache Kafka - Martin Podval
Martin Podval
 
PDF
Hadoop et son écosystème - v2
Khanh Maudoux
 
PDF
Magnet Shuffle Service: Push-based Shuffle at LinkedIn
Databricks
 
PDF
Consuming RealTime Signals in Solr
Umesh Prasad
 
PDF
Pacemaker + PostgreSQL レプリケーション構成(PG-REX)のフェイルオーバー高速化
kazuhcurry
 
PDF
Apache Kafka
Diego Pacheco
 
PPTX
Worst Splunk practices...and how to fix them
Splunk
 
PDF
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
PDF
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
PPTX
OpenTelemetry For Architects
Kevin Brockhoff
 
PDF
Modern Data Flow
confluent
 
PDF
XStream: stream processing platform at facebook
Aniket Mokashi
 
PPTX
Apache Flink Deep Dive
DataWorks Summit
 
PPTX
Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases
Shivji Kumar Jha
 
PDF
分散トレーシング技術について(Open tracingやjaeger)
NTT Communications Technology Development
 
PDF
Introduction to Apache Kafka and Confluent... and why they matter
confluent
 
Apache flink
pranay kumar
 
Kafka connect 101
Whiteklay
 
Observability: Beyond the Three Pillars with Spring
VMware Tanzu
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
HostedbyConfluent
 
Apache Kafka - Martin Podval
Martin Podval
 
Hadoop et son écosystème - v2
Khanh Maudoux
 
Magnet Shuffle Service: Push-based Shuffle at LinkedIn
Databricks
 
Consuming RealTime Signals in Solr
Umesh Prasad
 
Pacemaker + PostgreSQL レプリケーション構成(PG-REX)のフェイルオーバー高速化
kazuhcurry
 
Apache Kafka
Diego Pacheco
 
Worst Splunk practices...and how to fix them
Splunk
 
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
OpenTelemetry For Architects
Kevin Brockhoff
 
Modern Data Flow
confluent
 
XStream: stream processing platform at facebook
Aniket Mokashi
 
Apache Flink Deep Dive
DataWorks Summit
 
Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases
Shivji Kumar Jha
 
分散トレーシング技術について(Open tracingやjaeger)
NTT Communications Technology Development
 
Introduction to Apache Kafka and Confluent... and why they matter
confluent
 

Similar to Apache Lucene/Solr Document Classification (20)

PDF
PostgreSQL 9.4, 9.5 and Beyond @ COSCUP 2015 Taipei
Satoshi Nagayasu
 
PDF
Lucene for Solr Developers
Erik Hatcher
 
PDF
Lucene for Solr Developers
Erik Hatcher
 
PDF
Introduction to Solr
Erik Hatcher
 
PPTX
Hacking Lucene for Custom Search Results
OpenSource Connections
 
PDF
advancedR.pdf
RukhsanaTaj2
 
PDF
Advanced r
Dieudonne Nahigombeye
 
PDF
Advanced R cheat sheet
Dr. Volkan OBAN
 
PDF
Basic R Learning
Kumar P
 
PPTX
The ELK Stack - Launch and Learn presentation
saivjadhav2003
 
PDF
Full Text Search In PostgreSQL
Karwin Software Solutions LLC
 
PDF
ELK-Stack-Essential-Concepts-TheELKStack-LunchandLearn.pdf
cadejaumafiq
 
PDF
Information Retrieval - Data Science Bootcamp
Kais Hassan, PhD
 
PPTX
The Apache Solr Smart Data Ecosystem
Trey Grainger
 
PPTX
Apache solr
Dipen Rangwani
 
PDF
Data Science with Solr and Spark
Lucidworks
 
PPT
Advanced full text searching techniques using Lucene
Asad Abbas
 
PPTX
Android Database
Dr Karthikeyan Periasamy
 
PPTX
Examiness hints and tips from the trenches
Ismail Mayat
 
PPTX
Search Engine Capabilities - Apache Solr(Lucene)
Manish kumar
 
PostgreSQL 9.4, 9.5 and Beyond @ COSCUP 2015 Taipei
Satoshi Nagayasu
 
Lucene for Solr Developers
Erik Hatcher
 
Lucene for Solr Developers
Erik Hatcher
 
Introduction to Solr
Erik Hatcher
 
Hacking Lucene for Custom Search Results
OpenSource Connections
 
advancedR.pdf
RukhsanaTaj2
 
Advanced R cheat sheet
Dr. Volkan OBAN
 
Basic R Learning
Kumar P
 
The ELK Stack - Launch and Learn presentation
saivjadhav2003
 
Full Text Search In PostgreSQL
Karwin Software Solutions LLC
 
ELK-Stack-Essential-Concepts-TheELKStack-LunchandLearn.pdf
cadejaumafiq
 
Information Retrieval - Data Science Bootcamp
Kais Hassan, PhD
 
The Apache Solr Smart Data Ecosystem
Trey Grainger
 
Apache solr
Dipen Rangwani
 
Data Science with Solr and Spark
Lucidworks
 
Advanced full text searching techniques using Lucene
Asad Abbas
 
Android Database
Dr Karthikeyan Periasamy
 
Examiness hints and tips from the trenches
Ismail Mayat
 
Search Engine Capabilities - Apache Solr(Lucene)
Manish kumar
 
Ad

More from Sease (20)

PPTX
Hybrid Search with Apache Solr Reciprocal Rank Fusion
Sease
 
PPTX
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
Sease
 
PPTX
From Natural Language to Structured Solr Queries using LLMs
Sease
 
PPTX
Hybrid Search With Apache Solr
Sease
 
PPTX
Multi Valued Vectors Lucene
Sease
 
PPTX
When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...
Sease
 
PDF
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
PDF
Introducing Multi Valued Vectors Fields in Apache Lucene
Sease
 
PPTX
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Sease
 
PPTX
How does ChatGPT work: an Information Retrieval perspective
Sease
 
PDF
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
PPTX
Neural Search Comes to Apache Solr
Sease
 
PPTX
Large Scale Indexing
Sease
 
PDF
Dense Retrieval with Apache Solr Neural Search.pdf
Sease
 
PDF
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Sease
 
PDF
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Sease
 
PPTX
How to cache your searches_ an open source implementation.pptx
Sease
 
PDF
Online Testing Learning to Rank with Solr Interleaving
Sease
 
PDF
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Sease
 
PDF
Advanced Document Similarity with Apache Lucene
Sease
 
Hybrid Search with Apache Solr Reciprocal Rank Fusion
Sease
 
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
Sease
 
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Hybrid Search With Apache Solr
Sease
 
Multi Valued Vectors Lucene
Sease
 
When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...
Sease
 
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
Introducing Multi Valued Vectors Fields in Apache Lucene
Sease
 
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
Sease
 
How does ChatGPT work: an Information Retrieval perspective
Sease
 
How To Implement Your Online Search Quality Evaluation With Kibana
Sease
 
Neural Search Comes to Apache Solr
Sease
 
Large Scale Indexing
Sease
 
Dense Retrieval with Apache Solr Neural Search.pdf
Sease
 
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
Sease
 
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
Sease
 
How to cache your searches_ an open source implementation.pptx
Sease
 
Online Testing Learning to Rank with Solr Interleaving
Sease
 
Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...
Sease
 
Advanced Document Similarity with Apache Lucene
Sease
 
Ad

Recently uploaded (20)

PDF
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PDF
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PDF
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
PDF
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
PDF
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 

Apache Lucene/Solr Document Classification

  • 1. Lucene And Solr Document Classification Alessandro Benedetti, Software Engineer, Sease Ltd.
  • 2. Alessandro Benedetti ● Search Consultant ● R&D Software Engineer ● Master in Computer Science ● Apache Lucene/Solr Enthusiast ● Semantic, NLP, Machine Learning Technologies passionate ● Beach Volleyball Player & Snowboarder Who I am
  • 3. ● Classification ● Lucene Approach ● Solr Integration ● Demo ● Extensions ● Future Work Agenda
  • 4. “Classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. “ Wikipedia Classification
  • 5. ● E-mail spam filter ● Document categorization ● Sexually explicit content detection ● Medical diagnosis ● E-commerce ● Language identification Real World Use Cases
  • 6. ● Supervised learning ● Labelled training samples ● Documents modelled as feature vectors ● Term occurrences as features ● Model predicts unseen documents label Basics Of Text Classification
  • 7. Apache Lucene Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. Apache Lucene is an open source project available for free download.
  • 8. ● Lucene index has complex data structures ● Lot of organizations have already indexes in place ● Pre existent data can be used to classify ● No need to train a model from a separate training set ● From training set to Inverted index Apache Lucene For Classification
  • 9. ● Advanced configurable text analysis ● Term frequencies ● Term positions ● Document frequencies ● Norms ● Part of speech tags and custom payload Apache Lucene For Classification
  • 10. ● Given an index with labelled documents ● Each document has a class field ● Given an unknown document in input ● Given a set of relevant fields ● Search the top K most similar documents ● Fetch the classes from the retrieved documents ● Return most occurring class(es) ● Class ranking in retrieved documents is important ! K Nearest Neighbours
  • 11. ● KNN uses Lucene More Like This ● Lucene query component ● Extract interesting terms* from the input document fields ● Build a Lucene query ● Run the query against the search index ● Resulting documents are “the similar documents” * an interesting term is a term : - occurring frequently in the seed document (high term frequency) - but quite rare in the corpus (high inverted document frequency) More Like This
  • 12. Assumptions ● Term occurrences are probabilistic independent features ● Terms positions are irrelevant ( bag of words ) Calculate the probability score of each available class C ● Prior ( #DocsInClassC / #Docs ) ● Likelihood ( P(d|c) = P(t1, t2,..., tn|c) == P(t1|c) * P(t2|c) * … * P(tn|c)) Where given term t P(t|c) = TF(t) in documents of class c +1 / #terms in all documents of class c + #docs of class c Assign top scoring class Naive Bayes Classifier
  • 13. ● Documents are the Lucene unit of information ● Documents are a map field -> value ● Each field may be analysed differently (different tokenization and token filtering) ● Each field may have a different weight for the classification (affecting differently the similarity score) Document Classification
  • 14. Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search and analytics, rich document parsing, geospatial search, extensive REST APIs as well as parallel SQL. Apache Solr
  • 15. Index Time Integration - SOLR-7739 ● Ingest the document ● Assign the class ● Set the class as a field value ● Index the document Request Handler Integration (TO DO) - SOLR-7738 Return an assigned class : ● Given a text and a field ● Given an input document ● Given an indexed document id Solr Integration
  • 16. ● Pipeline of processors ● Each single document flows through the chain ● Each processor is executed once ● Last processor triggers the update command Update Request Processor Chain
  • 17. ● Update Component ● Configurable Singleton Factory ● Single instance per request thread ● Process a single Document ● SolrCloud compatible* * Pre processor / Post processor Update Request Processor
  • 18. ● Access the Index Reader ● A Lucene Document Classifier is instantiated ● A class is assigned by the classifier ● A new field is added to the original Document, with the class ● The document goes through the next processing steps Classification Update Request Processor
  • 19. ... <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell,/browse"> <lst name="defaults"> <str name="df">text</str> <str name="update.chain">classification</str> </lst> </initParams> ... Solrconfig.xml - Update Handler
  • 20. ... <updateRequestProcessorChain name="classification"> <processor class="solr.ClassificationUpdateProcessorFactory"> ... </processor> <processor class="solr.RunUpdateProcessorFactory"/> </updateRequestProcessorChain> ... Solrconfig.xml - Chain configuration
  • 21. <processor class="solr.ClassificationUpdateProcessorFactory"> <str name="inputFields">title^1.5,content,author</str> <str name="classField">cat</str> <str name="algorithm">knn</str> <str name="knn.k">20</str> <str name="knn.minTf">1</str> <str name="knn.minDf">5</str> </processor> N.B. classField must be stored Solrconfig.xml - K nearest neighbour classifier config
  • 22. <processor class="solr.ClassificationUpdateProcessorFactory"> <str name="inputFields">title^1.5,content,author</str> <str name="classField">cat</str> <str name="algorithm">bayes</str> </processor> N.B. classField must be Indexed (take care of analysis) Solrconfig.xml - Naive Bayes classifier config
  • 23. ● Lucene >= 6.0 ● Solr >= 6.1 ● Classification needs a training set -> An index with initially human assigned classes is required Solr Classification - Important Notes
  • 24. ● Sci-Fi StackExchange dataset ● Roughly 18.000 questions and answers ● Roughly 6.000 tagged ● 70 % Training Set + 30% test set Solr Classification - Demo
  • 25. ● Index the training set documents (this is our ground truth) ● Index the test set (classification will happen automatically at indexing time) ● Evaluate the test set (a simple java app to verify that the automatically assigned classes are consistent with what expected) Solr Classification - Demo
  • 26. ● True Positive : Predicted class == actual class ● False Positive : Predicted class != actual class ● True Negative : Not predicted class != actual class ● False Negative : Not predicted class == actual class Precision = TP / TP+FP Recall = TP / TP+FN Solr Classification - System Evaluation Metrics
  • 27. ● Index the training set documents (this is our ground truth) ● Index the test set (classification will happen automatically at indexing time) ● Evaluate the test set (a simple java app to verify that the automatically assigned classes are consistent with what expected) Solr Classification - Demo
  • 28. MaxOutputClasses 1 [System Global Accuracy]0.5095676824946846 [System Globel Recall]0.2686846038863976 TP{star-wars}59 FP{star-wars}75 FN{star-wars}7 [Precision (of predicted)]{star-wars}0.44029850746268656 [Recall for class)]{star-wars}0.8939393939393939 TP{harry-potter}147 FP{harry-potter}137 FN{harry-potter}3 [Precision (of predicted)]{harry-potter}0.5176056338028169 [Recall for class]{harry-potter}0.98 Solr Classification - Demo - Full Dataset
  • 29. MaxOutputClasses 5 [System Global Accuracy]0.20481927710843373 [System Globel Recall]0.5399850523168909 TP{star-wars}66 FP{star-wars}400 FN{star-wars}0 [Precision (of predicted)]{star-wars}0.14163090128755365 [Recall for class)]{star-wars}1.0 TP{harry-potter}150 FP{harry-potter}584 FN{harry-potter}0 [Precision (of predicted)]{harry-potter}0.20435967302452315 [Recall for class]{harry-potter}1.0 Solr Classification - Demo - Full Dataset
  • 30. MaxOutputClasses 1 [System Global Accuracy]0.9907407407407407 [System Globel Recall]0.6750788643533123 TP{star-wars}64 FP{star-wars}0 FN{star-wars}2 [Precision (of predicted)]{star-wars}1.0 [Recall for class)]{star-wars}0.9696969696969697 TP{harry-potter}150 FP{harry-potter}2 FN{harry-potter}0 [Precision (of predicted)]{harry-potter}0.9868421052631579 [Recall for class]{harry-potter}1.0 Solr Classification - Demo - Partial Dataset
  • 31. MaxOutputClasses 5 [System Global Accuracy]0.24259259259259258 [System Globel Recall]0.8264984227129337 TP{star-wars}66 FP{star-wars}52 FN{star-wars}0 [Precision (of predicted)]{star-wars}0.559322033898305 [Recall for class)]{star-wars}1.0 TP{harry-potter}150 FP{harry-potter}48 FN{harry-potter}0 [Precision (of predicted)]{harry-potter}0.7575757575757576 [Recall for class]{harry-potter}1.0 Solr Classification - Demo - Partial Dataset
  • 32. Multi classes support ● Class field may be multi valued ● Assign multiple classes ● Not only the top scoring but top N (parameter) Split human/auto assigned classes ● classTrainingField ● classOutputField Default : use the same field Solr Classification - Extensions SOLR-8871
  • 33. Classification Context Filtering ● Reduce the document space to consider -> reduce the training set ● Useful when only a subset of the index may be interesting for classification ● Consider only the human labelled documents as training data Solr Classification - Extensions SOLR-8871
  • 34. Individual Field Weighting ● When classifying, each field has a different importance e.g. title vs content ● Set a different boost per field ● Knn compatible ● Bayes compatible Solr Classification - Extensions SOLR-8871
  • 35. ● Numeric Field Support (Knn) (Euclidean distance based) ● Lat lon support (Knn) (geo distance based) ● SolrCloud support (use the entire sharded index as training set) Solr Classification - Future Work
  • 37. ● Special thanks to Tommaso Teofili, Apache committer who followed the developments and made possible the contributions. ● And to the Audience :)