Search at Twitter: Presented by Michael Busch, Twitter
6 likes•10,129 views
Twitter processes over 500 million tweets per day and more than 2 billion search queries per day. The company uses a search architecture based on Lucene with custom extensions. This includes an in-memory real-time index optimized for concurrency without locks, and a schema-based document factory. Future work includes support for parallel index segments and additional Lucene features.
![Concurrency - Definitions
• Pessimistic locking
• A thread holds an exclusive lock on a resource, while an action is
performed [mutual exclusion]
• Usually used when conflicts are expected to be likely
• Optimistic locking
• Operations are tried to be performed atomically without holding a lock;
conflicts can be detected; retry logic is often used in case of conflicts
• Usually used when conflicts are expected to be the exception](https://image.slidesharecdn.com/searchattwitter-michaelbusch-141215114222-conversion-gate02/85/Search-at-Twitter-Presented-by-Michael-Busch-Twitter-27-320.jpg)
Search at Twitter: Presented by Michael Busch, Twitter
- 1. Search @twitter Michael Busch @michibusch [email protected] [email protected]
- 2. Search @twitter Agenda ‣ Introduction - Search Architecture - Lucene Extensions - Outlook
- 4. Introduction
- 5. Introduction Twitter has more than 284 million monthly active users.
- 6. Introduction 500 million tweets are sent per day.
- 7. Introduction More than 300 billion tweets have been sent since company founding in 2006.
- 8. Introduction Tweets-per-second record: one-second peak of 143,199 TPS.
- 9. Introduction More than 2 billion search queries per day.
- 10. Search @twitter Agenda - Introduction ‣ Search Architecture - Lucene Extensions - Outlook
- 13. RT index Search Architecture RT stream Analyzer/ Partitioner RT index (Earlybird) Blender Archive index RT index Mapreduce Analyzer raw tweets Tweet archive HDFS Search requests writes searches analyzed tweets analyzed tweets raw tweets
- 14. RT index Search Architecture Tweets Analyzer/ Partitioner RT index (Earlybird) Blender Archive index RT index queue HDFS Search requests Updates Deletes/ Engagement (e.g. retweets/favs) writes searches Mapreduce Analyzer
- 15. RT index Search Architecture RT index (Earlybird) Social graph Social Blender Archive index RT index User search Search requests writes searches • Blender is our Thrift service aggregator • Queries multiple Earlybirds, merges results Social graph graph
- 16. Search Architecture RT index (Earlybird) Archive index User search
- 17. Search Architecture RT index (Earlybird) Archive index • For historic reasons, these used to be entirely different codebases, but had similar features/ technologies • Over time cross-dependencies were introduced to share code User search Lucene
- 18. Search Architecture RT index (Earlybird) Archive index User search Lucene Extensions Lucene • New Lucene extension package • This package is truly generic and has no dependency on an actual product/index • It contains Twitter’s extensions for real-time search, a thin segment management layer and other features
- 19. Search @twitter Agenda - Introduction - Search Architecture ‣ Lucene Extensions - Outlook
- 22. Lucene Extension Library • Abstraction layer for Lucene index segments • Real-time writer for in-memory index segments • Schema-based Lucene document factory • Real-time faceting
- 23. Lucene Extension Library • API layer for Lucene segments • *IndexSegmentWriter • *IndexSegmentAtomicReader • Two implementations • In-memory: RealtimeIndexSegmentWriter (and reader) • On-disk: LuceneIndexSegmentWriter (and reader)
- 24. Lucene Extension Library • IndexSegments can be built ... • in realtime • on Mesos or Hadoop (Mapreduce) • locally on serving machines • Cluster-management code that deals with IndexSegments • Share segments across serving machines using HDFS • Can rebuild segments (e.g. to upgrade Lucene version, change data schema, etc.)
- 25. Lucene Extension Library HDFS EEEaararlyrlylbybbirirdirdd Mesos Hadoop (MR) RT pipeline
- 26. RealtimeIndexSegmentWriter • Modified Lucene index implementation optimized for realtime search • IndexWriter buffer is searchable (no need to flush to allow searching) • In-memory • Lock-free concurrency model for best performance
- 27. Concurrency - Definitions • Pessimistic locking • A thread holds an exclusive lock on a resource, while an action is performed [mutual exclusion] • Usually used when conflicts are expected to be likely • Optimistic locking • Operations are tried to be performed atomically without holding a lock; conflicts can be detected; retry logic is often used in case of conflicts • Usually used when conflicts are expected to be the exception
- 28. Concurrency - Definitions • Non-blocking algorithm Ensures, that threads competing for shared resources do not have their execution indefinitely postponed by mutual exclusion. • Lock-free algorithm A non-blocking algorithm is lock-free if there is guaranteed system-wide progress. • Wait-free algorithm A non-blocking algorithm is wait-free, if there is guaranteed per-thread progress. * Source: Wikipedia
- 29. Concurrency • Having a single writer thread simplifies our problem: no locks have to be used to protect data structures from corruption (only one thread modifies data) • But: we have to make sure that all readers always see a consistent state of all data structures -> this is much harder than it sounds! • In Java, it is not guaranteed that one thread will see changes that another thread makes in program execution order, unless the same memory barrier is crossed by both threads -> safe publication • Safe publication can be achieved in different, subtle ways. Read the great book “Java concurrency in practice” by Brian Goetz for more information!
- 30. Java Memory Model • Program order rule Each action in a thread happens-before every action in that thread that comes later in the program order. • Volatile variable rule A write to a volatile field happens-before every subsequent read of that same field. • Transitivity If A happens-before B, and B happens-before C, then A happens-before C. * Source: Brian Goetz: Java Concurrency in Practice
- 31. Concurrency RAM 0 int x; Cache Thread 1 Thread 2 time
- 32. Concurrency Cache 5 RAM 0 int x; Thread 1 Thread 2 x = 5; Thread A writes x=5 to cache time
- 33. Concurrency Cache 5 RAM 0 int x; Thread 1 Thread 2 x = 5; time while(x != 5); This condition will likely never become false!
- 34. Concurrency RAM 0 int x; Cache Thread 1 Thread 2 time
- 35. Concurrency RAM 0 int x; Thread A writes b=1 to RAM, because b is volatile 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1;
- 36. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; Read volatile b int dummy = b; while(x != 5);
- 37. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); happens-before • Program order rule: Each action in a thread happens-before every action in that thread that comes later in the program order.
- 38. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); happens-before • Volatile variable rule: A write to a volatile field happens-before every subsequent read of that same field.
- 39. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); happens-before • Transitivity: If A happens-before B, and B happens-before C, then A happens-before C.
- 40. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); This condition will be false, i.e. x==5 • Note: x itself doesn’t have to be volatile. There can be many variables like x, but we need only a single volatile field.
- 41. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); Memory barrier • Note: x itself doesn’t have to be volatile. There can be many variables like x, but we need only a single volatile field.
- 43. Demo
- 44. Concurrency RAM 0 int x; 5 x = 5; 1 Cache Thread 1 Thread 2 time volatile int b; b = 1; int dummy = b; while(x != 5); Memory barrier • Note: x itself doesn’t have to be volatile. There can be many variables like x, but we need only a single volatile field.
- 45. Concurrency IndexWriter IndexReader time write 100 docs maxDoc = 100 in IR.open(): read maxDoc search upto maxDoc write more docs maxDoc is volatile
- 46. Concurrency IndexWriter IndexReader time write 100 docs maxDoc = 100 in IR.open(): read maxDoc search upto maxDoc write more docs maxDoc is volatile happens-before • Only maxDoc is volatile. All other fields that IW writes to and IR reads from don’t need to be!
- 47. Wait-free • Not a single exclusive lock • Writer thread can always make progress • Optimistic locking (retry-logic) in a few places for searcher thread • Retry logic very simple and guaranteed to always make progress
- 48. In-memory Real-time Index • Highly optimized for GC - all data is stored in blocked native arrays • v1: Optimized for tweets with a term position limit of 255 • v2: Support for 32 bit positions without performance degradation • v2: Basic support for out-of-order posting list inserts
- 49. In-memory Real-time Index • Highly optimized for GC - all data is stored in blocked native arrays • v1: Optimized for tweets with a term position limit of 255 • v2: Support for 32 bit positions without performance degradation • v2: Basic support for out-of-order posting list inserts
- 50. In-memory Real-time Index • RT term dictionary • Term lookups using a lock-free hashtable in O(1) • v2: Additional probabilistic, lock-free skip list maintains ordering on terms • Perfect skip list not an option: out-of-order inserts would require rebalancing, which is impractical with our lock-free index • In a probabilistic skip list the tower height of a new (out-of-order) item can be determined without knowing its insert position by simply rolling a dice
- 51. In-memory Real-time Index • Perfect skip list
- 52. In-memory Real-time Index • Perfect skip list Inserting a new element in the middle of this skip list requires re-balancing the towers.
- 53. In-memory Real-time Index • Probabilistic skip list
- 54. In-memory Real-time Index • Probabilistic skip list Tower height determined by rolling a dice BEFORE knowing the insert location; tower height never has to change for an element, simplifying memory allocation and concurrency.
- 55. Schema-based Document factory • Apps provide one ThriftSchema per index and create a ThriftDocument for each document • SchemaDocumentFactory translates ThriftDocument -> Lucene Document using the Schema • Default field values • Extended field settings • Type-system on top of DocValues • Validation
- 56. Schema-based Document factory Schema Lucene Document SchemaDocument Factory Thrift Document • Validation • Fill in default values • Apply correct Lucene field settings
- 57. Schema-based Document factory Schema Lucene Document SchemaDocument Factory Thrift Document • Validation • Fill in default values • Apply correct Lucene field settings Decouples core package from specific product/index. Similar to Solr/ElasticSearch.
- 58. Search @twitter Agenda - Introduction - Search Architecture - Lucene Extensions ‣ Outlook
- 60. Outlook
- 61. Outlook • Support for parallel (sliced) segments to support partial segment rebuilds and other cool posting list update patterns • Add remaining missing Lucene features to RT index • Index term statistics for ranking • Term vectors • Stored fields
- 62. Questions? Michael Busch @michibusch [email protected] [email protected]
- 64. Backup Slides
- 65. Searching for top entities within Tweets • Task: Find the best photos in a subset of tweets • We could use a Lucene index, where each photo is a document • Problem: How to update existing documents when the same photos are tweeted again? • In-place posting list updates are hard • Lucene’s updateDocument() is a delete/add operation - expensive and not order-preserving
- 66. Searching for top entities within Tweets • Task: Find the best photos in a subset of tweets • Could we use our existing time-ordered tweet index? • Facets!
- 67. Searching for top entities within Tweets Query Doc ids Inverted index Term id Term label Forward Doc id index Document Metadata Facet index Doc id Term ids
- 68. Storing tweet metadata Facet Doc id index Term ids
- 69. 5 15 9000 9002 100000 100090 Matching doc id Facet index Term ids Top-k heap Id Count 48239 8 31241 2 Query Searching for top entities within Tweets
- 70. 5 15 9000 9002 100000 100090 Matching doc id Facet index Term ids Top-k heap Id Count 48239 15 31241 12 85932 8 6748 3 Query Searching for top entities within Tweets
- 71. Searching for top entities within Tweets 5 15 9000 9002 100000 100090 Matching doc id Facet index Term ids Top-k heap Id Count 48239 15 31241 12 85932 8 6748 3 Query Weighted counts (from engagement features) used for relevance scoring
- 72. Searching for top entities within Tweets 5 15 9000 9002 100000 100090 Matching doc id Facet index Term ids Top-k heap Id Count 48239 15 31241 12 85932 8 6748 3 Query All query operators can be used. E.g. find best photos in San Francisco tweeted by people I follow
- 73. Searching for top entities within Tweets Inverted Term id index Term label
- 74. Searching for top entities within Tweets Id Count Label Count pic.twitter.com/jknui4w 45 pic.twitter.com/dslkfj83 23 pic.twitter.com/acm3ps 15 pic.twitter.com/948jdsd 11 pic.twitter.com/dsjkf15h 8 pic.twitter.com/irnsoa32 5 48239 45 31241 23 85932 15 6748 11 74294 8 3728 5 Inverted index
- 75. Summary • Indexing tweet entities (e.g. photos) as facets allows to search and rank top-entities using a tweets index • All query operators supported • Documents don’t need to be reindexed • Approach reusable for different use cases, e.g.: best vines, hashtags, @mentions, etc.