Sphinx - High performance full-text search for MySQL
Download as PPTX, PDF2 likes3,366 views
The document discusses Sphinx, an open source full-text search engine. It begins with an overview of full-text search and what Sphinx is - a high performance search engine that integrates well with SQL databases. The document then covers Sphinx's workflow, including indexing data, searching via its API or SphinxQL, and its query syntax. It also discusses how Sphinx scales horizontally across nodes and clusters.
![Sphinx workflow - Query syntax
❖ Boolean search AND OR NOT:
hello | world hello & world hello -world
❖ Per-field search
@title hello, @body world
❖ Field combination
@(title, body) hello world
❖ Search within first N words
@body[50] hello
❖ Phrase search
“hello world”
26](https://image.slidesharecdn.com/sphinx-highperformancefulltextsearchformysql-160627030553/85/Sphinx-High-performance-full-text-search-for-MySQL-26-320.jpg)
Sphinx - High performance full-text search for MySQL
- 1. Sphinx - High performance full-text search for MySQL Nguyen Van Vuong - Framgia
- 2. Agenda ❖ Full-text search ❖ What’s Sphinx ? ❖ Why Sphinx ? ❖ Sphinx workflow ➢ Indexing ➢ Searching ➢ Query syntax ❖ How does it scale ? ❖ More about Sphinx ❖ References 2
- 4. Full-text search ❖ Full-text search is one of the techniques for searching a document or database stored ➢ Examines all of the words ➢ Tries to match the search query Articles id (integer) title (varchar) content (text) tag (varchar) 4 ❖ Example
- 5. Full-text search ❖ Full-text search is one of the techniques for searching a document or database stored ➢ Examines all of the words ➢ Tries to match the search query 5 ❖ Example SELECT * FROM articles WHERE MATCH (title, content) AGAINST ('database' IN NATURAL LANGUAGE MODE)
- 6. Full-text search - Term search vs Full-text search ❖ Search keywords: “I ate pizza yesterday” ❖ Term search ➢ No analysis phase ➢ Operate on a single term 6
- 7. Full-text search - Term search vs Full-text search ❖ Full-text search ➢ Tokenizer/analyzer ■ Breaking keywords down by whitespace and punctuation ■ Charset table ➢ Morphology preprocessors ■ Normalize both "dogs" and "dog" to "dog" ● Eat, eating, eaten, ate 7
- 9. What’s Sphinx ? ❖ Sphinx is a mythical creature with the head of a human and the body of a lion 9
- 10. What’s Sphinx ? ❖ Sphinx is a mythical creature with the head of a human and the body of a lion 10
- 11. What’s Sphinx ? ❖ Full-text search engine ❖ Free open source (GPL v2) ❖ Begin 10 years ago ❖ High performance ❖ Integrate well with SQL databases ❖ API exist for Perl, C#, Ruby, Java, PHP ❖ Available for Linux, Windows, Mac OS 11
- 12. Why Sphinx ? 12
- 13. Why sphinx ? ❖ Quick to learn ❖ Easy to use ❖ Simple to maintain 13
- 14. Why sphinx ? ❖ Speed ➢ 50x-100x faster than MySQL Fulltext ➢ Up to 1000x faster than MySQL in extreme cases (eg. large result set with GROUP BY) ❖ Feature-rich ➢ Relevancy (BM25) ➢ Synonyms ➢ Stopwords ➢ Real-time index ➢ ... 14
- 15. Why sphinx ? ❖ Scalable ➢ Aggregates search results from many sources ➢ Fully transparent to calling application ➢ Built-in load balancing ❖ Easy to Integrate ➢ SphinxApi ➢ SphinxSQL 15
- 17. Spinx workflow 17 Application Database Sphinx Daemon Sphinx Indexer Sphinx Index 1. Search query 2. Search results (IDs) 3.FetchdocbyID
- 18. Sphinx workflow - Indexing ❖ Configuration ➢ sphinx.conf ❖ Data sources 18
- 19. ❖ Character level ➢ Charset_table ■ Use ranges: a...z, U+410...U+42F ➢ Ngram_chars ■ Hieroglyphs as separate tokens ● Chinese, Japanese, … ● Unicode charset CJKV Sphinx workflow - Indexing 19
- 20. Sphinx workflow - Indexing ❖ Word level ➢ Stopwords ■ Avoid wasting index space ■ Example ● Don’t want to search for (like “I”, “Am”, “An”, etc) ➢ Stemming ■ Single word can appear in many forms when used in different contexts 20
- 21. Sphinx workflow - Indexing ❖ Building index 21 $ sudo service sphinxsearch start $ sudo indexer --config <file> --all $ sudo indexer --config <file> --rotate
- 22. Sphinx workflow - Searching ❖ Configuring search daemon 22 searchd { listen = localhost:9312 listen = 9306:mysql log = /var/log/sphinxsearch/searchd.log query_log = /var/log/sphinxsearch/query.log read_timeout = 5 client_timeout = 300 max_children = 30 persistent_connections_limit = 30 pid_file = /var/run/sphinxsearch/searchd.pid ... }
- 23. Sphinx workflow - Searching ❖ Sphinx Api ➢ Perl, C#, Ruby, Java, PHP ➢ Example in PHP 23
- 24. Sphinx workflow - Searching ❖ SphinxQL ➢ Connect via MySQL Client ➢ Query like MySQL 24 $ mysql -h<ip> -P<port_of_sphinx> SELECT * FROM myindex WHERE MATCH ('@(title,content) find me fast');
- 25. Sphinx workflow - Searching ❖ SphinxQL ➢ Connect via MySQL Client 25
- 26. Sphinx workflow - Query syntax ❖ Boolean search AND OR NOT: hello | world hello & world hello -world ❖ Per-field search @title hello, @body world ❖ Field combination @(title, body) hello world ❖ Search within first N words @body[50] hello ❖ Phrase search “hello world” 26
- 27. Sphinx workflow - Query syntax 27 ❖ Per field relevancy ranking weights SPH_MATCH_ALL SPH_MATCH_ANY SPH_MATCH_FULLSCAN ❖ Proximity search "people passion"~3 ❖ GEO distance search (with syntax for mi/km/m) GEODIST(0.659298124, -2.136602399, latitude, longitude)
- 28. How does it scale ? 28
- 29. How does it scale ? ❖ Distribution is done horizontally ➢ Search is performed across different nodes ❖ Set up an index on multiple servers 29
- 30. How does it scale ? ❖ Adding distributed index configuration ➢ First server (192.168.1.1) 30 index master { type = distributed # Local index to be searched local = items # Remote agent (index) to be searched agent = 192.168.1.2:9312:items-2 }
- 32. More about Sphinx ❖ Biggest known Sphinx cluster ➢ Indexes 25+ billion documents ➢ Over 9TB of data ➢ 1+ million searches/day 32 ❖ Busiest known Sphinx cluster ➢ 300+ million search queries/day. ❖ Books
- 33. References ❖ Sphinx document (v2.2.1) ❖ Sphinx Search Beginner's Guide - Abbas Ali ❖ Meet the Sphinx - Andrew Aksyonoff ❖ Advanced fulltext search with Sphinx - Adrian Nuta ❖ Search Big Data with MySQL and Sphinx - Mindaugas Zukas 33
- 34. 34 Thank you