Dynamic sitemaps
Download as PPTX, PDF0 likes192 views
This document discusses the implementation of dynamic sitemaps for the Penn State University Libraries Blacklight catalog (BlackCat) in order to allow search engines like Google to index the catalog's over 7.5 million records. It describes generating a unique signature hash for each record, querying Solr for "signature starts with" values to create dynamic sitemap pages, and updating the robots.txt file. While Google is aware of all the records, indexing has been slow. Lessons learned include Google's mysterious indexing processes, problems with high traffic from Bing that required throttling, and the overall slow growth of search engine indexing for such a large dynamic repository.
Dynamic sitemaps
- 1. Dynamic Sitemaps Blacklight Virtual Summit May 8, 2020 Charlie Morris Lead Web Developer Penn State University Libraries Libraries Strategic Technologies Discovery, Access and Web Services
- 2. Context on PSU Libraries • Blacklight catalog (project name “BlackCat”) in Beta until Fall • Vendor provided search interface remains the primary catalog product for the Libraries • 7.5+ million records • Solr 7.4, running in cloud mode, Blacklight 7+, Traject for ETL • 100,000+ students across commonwealth and around the world
- 3. Letting the bots in • Initially disallowed all bots in robots.txt • As part of phased releasing closer to stable release we invited the bots in
- 4. November 5, 2019 Prior to sitemap, removed deny all for robots
- 5. How do people find you? • Probably through a search engine. • Probably Google. • This is not a revelation. • Search engines like sitemaps, especially critical for a site made up entirely of dynamic links
- 6. A critical feature that is low hanging fruit • Let users find content in channels they trust and use on a daily basis (not defending these search engines, more that they are the critical path for users) • Why not compete with Amazon? Could save patrons some money and increase use of library resources • This isn’t a new revelation, of course, it’s more like ”low hanging fruit” • Note: no sitemap option in core Blacklight
- 7. The challenge of sitemaps on a large repository • < 50,000 • Solely dynamic links
- 8. Prior work • Static sitemap generators • https://github.com/jronallo/blacklight-sitemap • https://github.com/kjvarga/sitemap_generator • Operate by a scheduled task generating static files
- 9. A different approach: dynamic sitemaps • Jack Reed of Stanford University Libraries and others create a POC • Live query Solr for sitemap data • Use a Rails’ controller to dictate what is displayed • Use a Rails’ view to control the sitemap template • Penn State University Libraries’ PR for the work: • https://github.com/psu-libraries/psulib_blacklight/pull/511
- 10. The Query Recipe • Necessary piece: a unique base 16 (hexadecimal) encoded hash for each record indexed in Solr (call it the “signature”) • lucene as the query parser • Query parameter for “the signature starts with…” (q) • Return the id and timestamp fields (fl) • Make sure Solr isn’t attempting to calculate facets (facet) • Specify a large number (rows) to prevent paging More on query parameters from the Solr RefGuide
- 11. Making the signature with Solr <updateProcessor class="solr.processor.SignatureUpdateProcessorFactory" name="add_hash_id"> <bool name="enabled">true</bool> <str name="signatureField">hashed_id_si</str> <bool name="overwriteDupes">false</bool> <str name="fields">id</str> <str name="signatureClass">solr.processor.Lookup3Signature</str> </updateProcessor> More on this from the Solr RefGuide Add this to your UpdateProcessorChain
- 12. “Signature starts with” for “Dynamic leaves” • Depending on size of the index, tell Solr to create links to queries that start with every combination of hexadecimal values for X placeholders • Example: 0 to F for one placeholder = 16 “leaves” • GET /sitemap: shows a list of 16 links to leaves like /sitemap/0 • GET /sitemap/0: a sitemap with every document that has a signature that starts with 0
- 13. PSU Libraries Example: 4096 leaves
- 14. Update robots.txt Crawl-delay: 10 Sitemap: http://catalog.libraries.psu.edu/sitemap.xml
- 16. But hey…
- 18. More on slow growth Google has known about 7+ million documents since November, but growth is about 10,000 items per month, at this rate it will take 62 years for Google to finish up
- 19. Light analysis • About 20-50 visits a day • 4,967 visits since launching it late November • 4.4% of all traffic • Screenshot below is daily visits over time from search engines via Matomo Analytics (hey it used to be zero!)
- 20. Lessons learned
- 21. Google is mysterious • Slow growth despite the fact that they know about all records • Search of site:catalog.libraries.psu.edu still only shows a few thousand records despite Google’s dashboard reporting over 30 thousand
- 22. Bing is problematic • Bing needed to be throttled, it hit us very hard to the point of a DOS like behavior (thankful to have Sematext Performance Monitoring to tattle on Bing) • Used Bing webmaster tools to gain finer control over when the bot is allowed to visit and how often • Also set crawl delay to 10 in robots.txt (Google ignores this because it’s smart enough to not DOS you) • Not sure which of the above two factors solved the issue
- 24. Future • Keep watching growth in Google Search Console • Keep monitoring Matomo Analytics • Discuss with others about their experiences in attempting to have their repositories indexed by Google and others