Integrate ManifoldCF with Solr

Editor's Notes

• #3: Start : 0:00. End : 1:05 Hi,Thank you, Moi : Aurélien MAZOYER Co founder of France Labs, open source company based in France. We offer consulting on search technologies, icosystem Datafari, intranet search solution. Say a few word about datafari at the end of talk. Topic, Survey : how many of you have ever use manifoldcf ?
• #4: Start : 1:05. End : 1:40 3 parts Overview of MCF. Explain a Case study on the integration of MCF with Solr in order to search file What happens with mcf
• #5: Start 1:40 to 2:45 CF stands for Connector framework. That means that it is a tool that help you to connect heterodjinious Push the data to your favorite search engine Keep it syncronize Take access right into account to perform authenticated Provides a Complete UI and REST API
• #6: Start 2:45 to 3:15 Karl wright when he worked for Apache Top Level Project since 2012. Active project : last release this summer.
• #7: Start 3:15 to 4:25 Plenty of connectors included in ManifoldCF What is called You can write your own (ManifoldCF In action give you the best practice to write your own connector) Domain controller, such as an active directory Search engine
• #8: Start 4:25 to 5:03 Contains Different components You can see the different connectors for the interaction with the external world Administration interface Talk about it in a few slide. Cannot see here a underlying database, backbone of the solution
• #9: Start 5:03 to 5:30 Actually do the crawling job
• #10: Start 5:30 to 6:26 You add the username in parameter provide the security tokens for a specific user. It gives For example the sid of the user in Active Directory, and the sid of all groups that he belongs to
• #11: Start 6:26 to 7:14 Also web application. Administrate MCF. To begin, you will have to create Crawl Job. Start the job. Once you are done, you are now able to start your crawl
• #12: Start 7:14 to 8:00 What can be done in the admin Put new config Send command Respect REST standards
• #13: Start 8:00 to 8:49 Very simple to test it. Extract the binary distribution, open example directory Not unfamiliar TO solr users Not recommanded way to run it in production (mainly because of the HSQL database)
• #14: Start 8:49 to 10:00 Component we described in different processes. The database is very important. One of the recommanded databaseSynchronize via local folder on the machine or with zookeeper.
• #15: Start 10:00 to 11:07 Here is our scenario Let’s imagine An intranet Users who authenticate against AD They put their files on a shared folders. You have access rights on folders based on the user. But specific permission for some users. Of course it is a mess so they need a good search engine to find theirs documents Quite simple, not very unusuable, but it can be a nightmare if you don’t have to right tool We are here in a full proprietary environment. But we will see that MCF and solr can deal with it.
• #16: Start 11:07 to 12:00 A few words Autorisation, when user runs a solr query Nither solr nor mcf will do this job Up to the front end application
• #17: Start 12:00 to 12:34 Go back to the big picture Step 1 JCIFS connector fetches documents with theirs access control and push to Solr Extracting handler
• #18: Start 12:34 to 13:00 Step 2 : Frontend sends an authenticated query Retrieves the security tokens linked to the current user Then, runs a normal search and filter the result set with the help the document acces control list and user security tokens
• #19: Start 13:00 to 13:44 How can we actually implement that. ON the mcfside Windows share connection. Some few step to do (download last version of JCIFS library, uncomment the windows share line in the connectors config file)
• #20: Start 13:44 to 14:03 Authority connector should be belong to an authority group
• #21: Start 14:03 to 14:18 That’s it for manifold
• #22: Start 14:18 to 15:00 Told you Front end is in charge of the authentication LDAP protocol to authenticate TomcatSPNEGO (Active directory). Spénégo : use single sign on
• #23: Start 15:00 to 15:57 Add fields that will contains the access control list of the document Declare the MCF plugin Configure the endpoint of the authority service Add a filter query that uses this plugin in your search handler This is for the search handler
• #24: Start 15:57 to 16:30 For the update handler. It is a default extracting handler that integrate apache tika. As a reminder, since Solr 5, extracting handler can run tesseract to extract content from images. Solr can do this job.
• #25: Start 16:30 to 17:22 In new version of Manifold. It can also be done In fact, processing pipeline. You can do field mapping but also tika extraction. Perfect if you don’t want to send big files over the network
• #26: Start 17:22 to 18:11 Now we will try to understand what is going on under the hood during our crawl These two crawling models are available with manifoldCF. To avoid indexing Discover new documents, remove old ones.
• #27: Start 18:11 to 18:33 Some repository works well with incremental crawling Others don’t Unfortunatly our windows share won’t be able to answer
• #28: Start 18:33 to 20:00 Therefore JCIFS connector How do windows share connetor handle incremental If it is a file Next slide is version attribute Fetch from the windows share
• #29: Start 20:00 to 20:50 For each document Last version Depends on the repository
• #30: Start 20:50 to 21:04 This was for step 1. more We can repeat these 2 steps in order to keep our data syncrhonize. We have covered how to configure this and we ve describe of it works under the hood. Now it is in production mode and you want to be sure of what is going on
• #31: Start 21:04 to 22:33 Many informations UI or API Send alert if something went wrong or just if the crawl is finished
• #32: Start 22:33 to 23:00 You also have a tab that shows you an history of all the different activities. Document status, for example if you want to see if a document has already been ingested in the current crawl Maximum bandwitch will give you information of crawling performance
• #33: Start 23:00 to 23:16 Unfortunatly somtimes facing obvious Crawled repository that is overloaded It can be because of the network. You should packet with wireshark Solr server : for example if the autocommit frequency is too high. Mcf database is an important component, be sure that you followed the best pratices in the documentation
• #34: Start 23:16 to 24:25 Maybe it is because of the configuration of your connector Two main parameters that can have an impact on performance Throttling : Fixing hard limit on fetching document (usefull if you are doing web crawling don’t don’t to be ban by the webmaster) Max connections that will be done to the system. It can be a good idea if we want to do web crawling to increase this value But windows share won’t work very well with a of connection, so in our scenario we should use a small value
• #35: Start 24:25 to 25:15 In the job settings, you can filter document that you want to index For an intranet file share, you probably don’t want to index the last Star wars movie that an employee wanted to share with their colleagues
• #36: Start 25:15 to 26:00 That was some example of performance issues. But unfortunatly, It can be even worst, you can face errors If you are facing errors A thread dump can give you information on
• #37: Start 26:00 to 28:08 One common problem is when the account you use for crawl doesn’t It must be able to read everything and to read ACLs for each file It can need special right, such as Print operator. As we just saw, we can use exclusion regex or size limit Be also sure to add ignore tika exception in solr JCIFS errors linked or not to network issues timeout. Sometimes be solve while increase jcifs timeout Sometimes you can have to increase solr time out issues Big processing
• #38: Start 28:08 to 28:45 What can happen in real life To conclude. Massive web crawling : Nutch is the best tool for you Then, go for it.
• #39: Start 28:45 to 29:18 Here are some references That is now freely available You can have a look at our blog posts, that you how to run through the different steps that I covered in the file search scenario I described
• #40: Start 29:18 to 29:40 If you are too lazy to integrate Solr and ManifoldCF by yourself
• #41: Start 29:40 to 30:00 Thank you very much for your attention, Be pleased to answer any question you may have