An introduction to Elasticsearch's advanced relevance ranking toolbox

1
Advanced Relevance Ranking
Gilad Gal
Product Manager, Elasticsearch
ElasticON

Agenda Slide
Use color to highlight
Performance when ranking
Relevance ranking evaluation
Specific query types and their usefulness
Vector similarity in relevance ranking
Relevance ranking research and roadmap
1
2
3
4
5

Docs first & stats separately - better performance
Separate the query that returns the results from the stats/agg query
A Z
A X
A Y
A X
B Y
B Z
B X
B Y
B X
A B Z
Top
Ranking
A Z
A X
A Y
A X
B Y
B Z
B X
B Y
B X
A B Z
Result
count
Skipping
uncompetitive
results
Full index scan
Review the results while
facets and total count load
Faster UI response where
it matters!

Docs first & stats separately - better performance
Separate the query that returns the results from the stats/agg query
If no aggregation or hit count (beyond configurable k results)
Performance impact:
Term queries: 3x to 7x faster
Conjunction (and) queries: 3% to 7x faster
Disjunction (or) queries: 8% to 15x faster
Running Lucene's benchmark suite

• Set of representative queries
– Representative ! Frequent
– Larger is better, e.g.:
– 10^3 is probably reasonable
– 10^1 is probably too little
• Set of correct results per query
– Set size per query that covers most users’ interest,
e.g. 510
– Examine more results; potentially expand the query
• Some metrics use score and “not in set”,
others require a set
!
!
!
?
Tagged Data Curation
?
?
?
?
?
? ?
?
?
? ?
?
?
?
?
?
?
?
?
?
?
!
!
!
?
!
!
!

Supported Metrics
• Precision at K
• Mean Reciprocal Rank
• Discounted Cumulative Gain

Precision at K
• Number of correct results within the
first k results divided by k
• For one query:
• For a set of |Q|queries:
Supported Metrics

Mean Reciprocal Rank
• |Q|  number of queries in test set
• Ranki
:= position of the first correct result for query i
Supported Metrics

• Cumulative Gain
– Sum of ranking scores till position P
– Problem: different sorting of records 1..P would yield the
same CGP
– Relating to
– Position within k (i indicates the position)
– Ranking score (reli:= ranking score of record in position i)
• Higher positions are more influential
• Average DCG across multiple queries
– Ideally keep the same number of results in the result set
– Issue: different queries have different impact due to
average higher/lower score
Discounted Cumulative Gain

Supported Metrics
• Precision at K -
Simple to understand
• Mean Reciprocal Rank -
First correct answer
• Discounted Cumulative
Gain - Relevance
ranking order

BM25F
The oversimplified history of term frequency algorithm
TF/IDF
Frequent terms are
less indicative than
rare terms
TF
Rank by frequency
of the query term in
the document
BM25
Adjust for lengthy
text being more
probable to contain
the query term and
limits TF impact for
common words
BM25F
Adjust for multiple
fields with varying
field importance

Combined Fields Query
• Benefit:
– Improved relevance ranking when ranking
on multiple fields
• How
– Treat as if unified into a single field
– Support boosting per field
– Use BM25F
Using BM25F

The importance of the record’s intrinsic value
• Records differ in their value, regardless of query
• Typical examples:
– URL length
– User reviews score
– Popularity
– Number of links to / citations of
– Algorithms:
• Hubs and authorities (HITS
• PageRank
Static but crucial
Pre-calculated numbers

Rank Feature Query
• New datatypes:
– rank_feature
– rank_features (vector of rank_feature variables)
• Add to the relevance ranking score
• Allow normalization prior to adding:
– Saturation
– Logarithm
– Sigmoid

Script Score Query
• Based on all fields
• Including _score
• Normalize
– Saturation
– Logarithm
– Sigmoid
• Write your own painless or use efficient
ready-made functions
Define the function for relevance ranking score

Distance Feature Query
• Benefits:
– Newer data is frequently more relevant
– Geographical proximity as a relevance indicator in
non-geo searches
• Easy to use, sophisticated under the hood:
– Designed to leverage top-k faster queries for
improved performance
– Supports normalization through Saturation
Geo and Time Proximity: closer is frequently more relevant

Result Pinning
• Pinning the highest ranking results per query
• Facilitates tailoring SERPs for frequent queries
What is it good for?

Vector Similarity
• Vector distance as similarity indicator
– Embedding for natural language processing
– Feature vectors for image analysis (using
convolutional neural networks)
• Different types of distances/similarities
used for different needs
– Dot Product
– Cosine
– Manhattan
– Euclidean
What is it good for?
x1
x2
x2048
...
rather
than this
This

Vector Manhattan & Euclidean Distances
• Euclidean:
• Manhattan:
p = [p1
, p2
, p3
, …, pn
]
q = [q1
, q2
, q3
, …, qn
]
What is it?

Vector Similarity Functions
Use deep learning models for relevance ranking
Natural Language
Processing Model
Text documents
Queries
Documents
with vectors
Text documents
Documents
Queries
Queries
a1
a2
…
an
a1
a2
…
an
a1
a2
…
an
Text
Convolutional
Neural Network
Image
Embeddings
Feature vectors
a1
a2
…
an
a1
a2
…
an
For Example:
Search by
vector similarity &
text/keywords/num
Ingest documents

Agenda
Performance when ranking
Relevance ranking evaluation
Specific query types and their usefulness
Vector similarity in relevance ranking
Relevance ranking research and roadmap
1
2
3
4
5

Vector Similarity Usage with Elasticsearch
Use deep learning models for relevance ranking
• Image search
– CNN (convolutional neural networks)
• Natural language processing
– BERT, Word2vec, GloVe
https://image-search.eden.elstc.co/home

Approximate Nearest Neighbours
Vector similarity performance and scalability
• Vector similarity as an addition to term frequency ranking works great
– Very typical situation
– Scales well
– Limit the set using term frequency
– Re-rank based on vector similarity
• ANN for performance and scalability
– Ranking only by vector similarity
– Extending the term frequency based set with vector similarity results
• Based on HNSW support in Lucene
Tentative
Roadmap
Layer 2
Layer 1
Layer 0

Embeddings Creation
Translating text into vectors
• Pre-trained deep learning model support,
e.g. for BERT
• Out of the box translation of text into
embeddings
– Docs at ingest time
– Queries at query time
• Useful for NLP
– Search by meaning
– Venture beyond bag of keywords
– E.g. for helpdesk and documentation search
Tentative
Roadmap
Source: Siobhán Grayson in Wikimedia.org -
https://commons.wikimedia.org/wiki/File:TSNE_visualisation_of_word_e
mbeddings_generated_using_19th_century_literature.pdf

Tuning Ranking With Machine Learning
Configuring ranking parameters using ranking evaluation
• Tuning relevance ranking using machine learning
– Field boosting (see “Improving search relevance with data-driven query optimization” by Josh Devins)
– BM25 parameters
– Normalization parameters
• Possible today using ranking evaluation API, but run outside
Elasticsearch
• Considered for the roadmap including a queries curation mechanism
to obtain a tagged set for training/testing/validation
Tentative
Roadmap

Summary
• Lots of mechanism
• Designed to work together
• To facilitate the relevance ranking that best suits your need
BM25F & combined_fields
query
rank_feature query
Normalization: saturation,
logarithm, or sigmoid
distance_feature query
pinned query
Ranking by vector
similarity
Precision at K Mean Reciprocal Rank
Discounted Cumulative
Gain
Embedding creation using
pre-trained models
Tuning ranking with
learning to rank
Approximate Nearest
Neighbours using HNSW

An introduction to Elasticsearch's advanced relevance ranking toolbox

32
ElasticON Solution Series
Presenter Name
Title, Team Name
Event Name | Region

33
Quote slide ipsum do sed
Lorem ipsum dolar sit amet,
consectur adipiscing elit
sed do eiumod ipsum
epsum dolar sit
 First, Lastname | Title, Company Name

34
Place a quote from someone
really, really important and it
will shrink to ﬁt this space…
Author Name Here
“

36
Transition Slide Title
Short and Sweet

37
Short and Sweet

38
Short and Sweet

39
Short and Sweet

40
Short and Sweet

41
Short and Sweet

42
Goes Here and Can Be
a Few Lines Long
Subtitle goes here in sentence case

43
Goes Here and Can Be
a Few Lines Long
Subtitle goes here in sentence case

44
Safe Harbor
Statement
This presentation includes forward-looking
statements that are subject to risks and
uncertainties. Actual results may diﬀer materially
as a result of various risk factors included in the
reports on the Forms 10-K, 10-Q, and 8-K, and in
other ﬁlings we make with the SEC from time to
time. Elastic undertakes no obligation to update
any of these forward-looking statements.

45
Safe Harbor
Statement

46
Closing slide

47
Closing slide

Bullet title (Inter 24 pt)
• Try to keep your use of bullet slides to a minimum
• Be creative and think visually
• If you need to source something copy and paste the text box at the
bottom left onto your page
Subtitle sentence case (Inter 18pt)

Bullet slide title treatment can be up to two lines in
length (Inter bold 24 pt)
• Bullets are sentence case (Inter 18pt)
– Second-line bullets are Inter 14pt
• Third-line bullets are Inter 12pt
• Limit the number of bullets on a slide
• Text highlights are orange, but not underlined
• Try not to go below the recommended font sizes

‒ Second-line bullets are Inter 14pt
‒ Third-line bullets are Inter 12pt

really, really important and it will
shrink to fit this space…
Author Name Here

Author Name Here
really, really important and it will
shrink to fit this space…

Chart Slide With Multiple Colors
Sub-title or chart title here in sentence case

Pie Chart Slide With Multiple Colors
62%
Supporting text
goes here under
the number
62%
Supporting text
goes here under
the number

Pie Chart Slide With Multiple Colors

Transition Slide Title Goes
Here and Can Be a Few
Lines Long
Subtitle goes here in sentence
case

Short and Sweet

1M 1M 1M
HEADER HERE
Supporting text
goes here under
the number
HEADER HERE
Supporting text
goes here under
the number
HEADER HERE
Supporting text
goes here under
the number
Big Number Treatment

1M 1M 1M
HEADER HERE
Supporting text
goes here under
the number
HEADER HERE
Supporting text
goes here under
the number
HEADER HERE
Supporting text
goes here under
the number
Big Number Treatment (Dark Mode)

Table Layout Treatment
Subtitle text placeholder sentence case
HEADER HEADER HEADER HEADER
Information Information Information Information
Option 1

Option 2

Option 3

Option 4

Please use this area
for content, screen
shot, or quote; the
next few slide show
examples

We mine and analyze
4 billion events every
day to detect security
hacks and threats.

75
With organic logging growing 50%
year over year, and monitoring
infrastructure spend at nearly 10%,
one rogue log can ruin the platform.
The checks and balances necessary
to make sure we don’t hit that
roadblock are built with the Elastic
Stack and Beats.
TEXT GOES HERE IN ALL CAPS
Additional text goes here to support the content and can
be a couple lines in length and sits bottom left aligned

76
With organic logging growing 50%
year over year, and monitoring
infrastructure spend at nearly 10%,
one rogue log can ruin the
platform. The checks and balances
necessary to make sure we don’t
hit that roadblock are built with the
Elastic Stack and Beats.
TEXT GOES HERE IN ALL CAPS
Additional text goes here to support the content and can
be a couple lines in length and sits bottom left aligned

”
The Elastic Stack is critical to us. Every day
millions of users and customers worldwide
trust Box to execute mission-critical
business functions.
“

Some text can go here
Some text can go here

You can use
this area for a
text treatment
that supports
your chosen
imagery

Slide Title Here With
a Few Bullets
Subtitle goes here
• Bullet one goes here in
sentence case and no period
• Bullets should be kept short
and sweet; stay focused
• Use bullets to help break up
content that you need to
have on the screen

Slide Title Here With
a Few Bullets
Subtitle goes here
● Bullet one goes here in
sentence case and no
period
● Bullets should be kept short
and sweet; stay focused
● Use bullets to help break up
content that you need to
have on the screen

Slide Title Here
With Key Points
Subtitle goes here
Header Here
Body copy goes here and just increase the
indent level to get to the proper formatting
Header Here
Header Here
Header Here
LOGGING METRICS APM
ADVANCED
SEARCH
SECURITY
ANALYTICS
DATA
SCIENCE
FOUNDATION
SPECIALIZATIONS

Slide Title Here
With Key Points
Subtitle goes here
Header Here
Header Here
Header Here
Header Here

Image Treatment With Caption Layout
How to add your own photos and crop properly…
Your image will populate the
container but you will likely need
to adjust the crop. Double click
on the image to adjust. Use the
blue dots to adjust the size.
Click on the grayed out portion
of the image and drag to the
left or right until you are happy
with the crop.
1 2 3
Right click on the image and go
to replace image. Select a new
image from your machine.

Agenda Slide
Enter title for section one here and use sentence case
1
Enter title for section three here and use sentence case
3
Enter title for section four here and use sentence case
4
Enter title for section five here and use sentence case
5
Enter title for section two here and use sentence case
2
Option 1A
NOTE THIS SLIDE IS NOT IN THE LAYOUT OPTIONS.
ALWAYS START A NEW PRESENTATION USING THE
CORPORATE TEMPLATE AND ADD YOUR CONTENT
TO THIS SLIDE.

Bullet slide title treatment can be up to two lines in length (Inter bold 24 pt)
○ Second-line bullets are Inter 14pt
■ Third-line bullets are Inter 12pt
Agenda Slide
1
2
Option 1B
TO THIS SLIDE.
3
4
5

Agenda Slide
1
2
3
4
5
Option 2
TO THIS SLIDE.

Agenda Slide
1
2
3
4
5
Option 3
TO THIS SLIDE.

Agenda Slide
1
2
3
4
5
Option 4
TO THIS SLIDE.

Process Diagram Treatment, 5 Ideas
See style page for more color options
1 2 3 4 5
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number

Process Diagram Treatment, 5 Ideas + Highlight
1 2 3 4 5
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number

Supporting text
goes here under
the number
1 2 3 4
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number

Supporting text
goes here under
the number
1 2 3
Supporting text
goes here under
the number
Supporting text
goes here under
the number

1 2 3 4
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
5
Supporting text
goes here under
the number

Process Diagram Treatment, 5 Ideas + Highlight
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
1 2 3 4 5

1 2 3 4
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number

1 2 3
Supporting text
goes here under
the number
Supporting text
goes here under
the number
Supporting text
goes here under
the number

Title Here Title Here Title Here
• One bullet here
• Two bullet here
• Three bullet here
• One bullet here
• Two bullet here
• One bullet here
• Two bullet here
Box With Bullet Treatment

• One bullet here
• Two bullet here
• One bullet here
• Two bullet here
• One bullet here
• Two bullet here
Box With Bullet Treatment with Color Choice

• One bullet here
• Two bullet here
Title Here
• One bullet here
• Two bullet here
Title Here
• One bullet here
• Two bullet here
Title Here
Box Bullet Treatment

• One bullet here
• Two bullet here
• One bullet here
• Two bullet here
• One bullet here
• Two bullet here
Box Bullet Treatment with Color Scheme

Thank You
Search. Observe. Protect.

Screenshot Treatment With Browser Window
How to drop in your screen shot…
The browser window is like a
frame so anything you drop
behind it will show through.
Drop in your screen shot, go
to the format menu and crop
it to show only what you
want.
2
Last, be sure to right click on
your screen shot, go to order
and send to back.
3
1

Screenshot Treatment With Title and Browser Window

NOTE USE THIS LAYOUT
FOR PLACING ONE FULL
BLEED SCREENSHOT

Use This Slide for Code, Light Version
Use template colors to highlight
curl –XPUT localhost:9200/
_template/twitter –d ‘
{
“template” : “twitter_*”,
“setting” : {
“number_of_shards” : 4,
“number_of_replicas” : 1
}
}’

Use This Slide for Code, Dark Version
Use template colors to highlight
curl –XPUT localhost:9200/
_template/twitter –d ‘
{
“template” : “twitter_*”,
“setting” : {
“number_of_shards” : 4,
“number_of_replicas” : 1
}
}’

Color Palette
254
197
20
47
67
145
250
115
79
240
78
152
151
156
171
67
71
83
0
119
204
0
191
179
PRIMARY

Styles and Treatments
SHAPES
LOGOS
Please use logos according
to brand guidelines. These
logos can be sized up and
down without losing quality.
Please press shift before
sizing to keep proper
proportions.
Various template colors can
be used for shapes. Shapes
should have a 3pt line stroke.

Video or Large Image Treatment
Sub header goes here

3 solutions
Elastic Enterprise Search Elastic Security
Elastic Observability

Elastic Enterprise Search
Workplace Search App Search Site Search

Logs Metrics APM Uptime

Endpoint SIEM
Elastic Security

3 solutions powered by 1 stack
Kibana
Elasticsearch
Beats Logstash
Elastic Stack

The Elastic Stack
Reliably and securely take data from
any source, in any format, then search,
analyze, and visualize it in real time.

Deploy anywhere.
SaaS Orchestration
Elastic Cloud
on Kubernetes
Elastic Cloud Elastic Cloud
Enterprise
Kibana
Elasticsearch
Beats Logstash
Powered by
the stack
3 solutions
Deployed
anywhere

Deploy anywhere.
SaaS Orchestration
Elastic Cloud Elastic Cloud on
Kubernetes
Elastic Cloud
Enterprise

Subscription Options
ELASTIC CLOUD
FREE PAID
Open Source
Features
Free Proprietary
Features
Paid Proprietary Features
+
Elastic Support
PAID
OPEN SOURCE BASIC GOLD PLATINUM ENTERPRISE
SELFMANAGED
SaaS

Resource-based Pricing
Endpoint Security
No endpoint-based pricing
SIEM
No seat/ingest-based pricing
APM
No agent-based pricing
Metrics
No host-based pricing
Logs
No ingest-based pricing
App Search
No docs-based pricing
Site Search
No query-based pricing
Workplace Search
No user-based pricing

31 Solution Logos
ENTERPRISE
SEARCH
OBSERVABILITY SECURITY
Elastic Logo + Tagline
FULL COLOR
REVERSE

Product Logos
ELASTIC CLOUD
ON KUBERNETES
ECK
KIBANA LOGSTASH
ELASTICSEARCH
BEATS ELASTIC CLOUD
ELASTIC
CLOUD
ENTERPRISE

Product Logos
APM
APP SEARCH
WORKPLACE
SEARCH
METRICS SIEM
LOGS
SITE SEARCH ENDPOINT

Iconography Usage
Product Feature Icons
Do not use these icons for
anything other than what
they are created for.
Product Feature Icons are created
to correlate with a specific feature
within the product and are not
flexible in use. Please see labels as
a guide.
Generic Icons
These icons are made to fit across
multiple concepts within reason.
See labels as a general guide.
Please use discretion.
Training Icons
Do not use these icons for
anything other than what
they are created for.
Training Icons are created to
correlate with a specific feature
within the training relm and are not
flexible in use. Please see labels as
a guide.

Feature Icons
winlogbeat heartbeat packetbeat metricbeat functionbeat filebeat auditbeat index patterns Index
management
Life cycle
management
create single job create advanced
job
create multi
metric job
create population
job
machine
learning
advanced
settings
apm sql visualize dashboards
canvas upgrade assistant management security analytics add data search
profiler
users and
roles
saved objects reporting security settings
grok debugger language clients infra console discover dev tools watcher rollups cross cluster
replication
data visualizer
metrics monitoring notebook logging spaces logstash pipeline gis application timelion graph --

Training Icons
apm metrics Security analytics logging
specialization
Engineering 1 Engineering 2 certification Advanced search Data science
subscription on-demand Instructor led
stack

Generic Icons
training support subscription
customers
structured schema schemaless rapid query
execution
sql No sql Horizontal scale
flexible data
model
downloads custom consulting community community
members
Sophisticated query
language
node idea chart
news user reliable extensible upgrade IoT plugin scale real-time high-five
location distributed visibility plan E commerce family vacation presentation education guide book
benefits certificate video contribution target Health monitor overlap conversation speaker government

Generic Icons
To do Source code Color outside
of the lines
blog Send
message
docs mobile browser Love
letter
connection

Generic Icons
training support subscription
customers
structured schema schemaless rapid query
execution
sql No sql Horizontal scale
flexible data
model
downloads custom consulting community community
members
Sophisticated query
language
node idea Light bulb
news user reliable extensible upgrade IoT plugin scale real-time high-five
location distributed visibility plan E commerce family vacation presentation education guide book
benefits certificate video contribution target Health monitor overlap conversation speaker government

An introduction to Elasticsearch's advanced relevance ranking toolbox

More Related Content

What's hot (20)

Similar to An introduction to Elasticsearch's advanced relevance ranking toolbox (20)

More from Elasticsearch (20)

Recently uploaded (20)

An introduction to Elasticsearch's advanced relevance ranking toolbox