Advanced Technical SEO - Index Bloat & Discovery: from Facets to Javascript Frameworks - SMX Munich 2016
17 likes•12,065 views
The document discusses advanced technical SEO issues, focusing on index bloat and its impact on crawl budget, particularly for e-commerce and large site architectures. It outlines methods for preventing index bloat, including managing URL parameters, AJAX refinements, and utilizing pre-rendering techniques to improve discoverability. Recommendations emphasize the importance of consistent URL signals and thorough testing to optimize site performance and avoid wasted crawl budget.
Advanced Technical SEO - Index Bloat & Discovery: from Facets to Javascript Frameworks - SMX Munich 2016
- 1. The Latest in Advanced Technical SEO Index Bloat & Discovery: from Facets to Frameworks
- 3. TeamClients
- 4. index bloat
- 5. index bloat
- 6. crawl budget
- 11. Today’s Session • Technical SEO issues around e-commerce / large site architecture • Preventing index bloat & preserving crawl budget as a core methodology • Current solutions & upcoming threats (JS, AJAX, new frameworks, pre-rendering)
- 13. Index Bloat Prevention A bloated index = if indexed URLS > “unique pages”
- 14. Index Bloat Prevention On an ecommerce site: A bloated index = if indexed URLS > sum(CAT+PDP+Static)
- 15. Index Bloat Prevention On a ‘content’ site: A bloated index = if indexed URLS > sum(Articles+Static)
- 17. cannibalization
- 18. Index Bloat Prevention: Cannibalization
- 21. Index Bloat Prevention: Sorts & Facets
- 22. Index Bloat Prevention: Sorts & Filters http://www.site.com/guys/tees/?pref n1=bvAverageRating&prefn2=col orGroup&prefv3=LG&srule=sortin gNewArrival&prefv1=4&prefv2=RE D&prefn3=size
- 23. Index Bloat Prevention: Sorts & Filters <link rel="canonical" href=”http://www.site.com/guys/tees/" /> • Basic Solution: Strip out the unnecessary parameters
- 24. Solution: Filtering Out All Facet Params • PROS: – Avoids diluted / dupe URLs (request, not directive) • CONS: – If you want/need specific parameters indexed and exposed (size, color), need properly coded canonical tag logic, recipe for major leak and confusion. – Considerations w/ pagination & view-all page
- 25. Crawl Budget: Facet Parameter URLs
- 26. Crawl Budget: Facet Parameter URLs
- 28. JS / AJAX Indexation
- 29. Index Bloat VS Discovery: JS + AJAX
- 30. Index Bloat Prevention: JS + AJAX AJAX Refinement V1 = NO URL CHANGE
- 31. Index Bloat Prevention: JS + AJAX AJAX Refinement V1 - NO URL CHANGE, but inactive, different href= URL exists
- 32. AJAX Facet Refinements V1 (NO URL CHANGE) • PROS: – Theoretically no parameters exposed to bloat the index • CONS: – Users can’t share refined / filtered content to friends, no accurate bookmarking. (Terrible UX) – Googlebot will still crawl hidden href=' or other JS framework links like Angular: ng-href= (check canonical logic!!)
- 33. Index Bloat Prevention: JS + AJAX AJAX Refinement V2 = html 5 history.pushState()
- 34. Index Bloat Prevention: JS + AJAX html 5 history.pushState() http://www.site.com/guys/tees/?color=green&size=large
- 35. Consistent URL Signals - Navigation Ideal consistency: Navigation URLs = Pushstate() URLs = Canonical URLs = XML Sitemap URLs =
- 36. Consistent URL Signals - Navigation Ideal consistency: Navigation URLs = Pushstate() URLs ≠ Canonical URLs = XML Sitemap URLs =
- 37. Index Bloat Prevention: JS + AJAX Google preferred pushstate URL version, we had to reinforce (via normal inline href=‘’, canonical, xml sitemap)
- 38. AJAX Facet Refinements V2 (PushState URL Change) • PROS: – Users can now share /bookmark the correct content – Added to browser history • CONS: – Still need to have consistent canonical structure due to Googlebot crawling pushstate() – Different hidden URL structure via AJAX facets may require further unpredictable canonicalization logic / further dev work
- 39. Indexing AJAX & JS Frameworks
- 40. Indexing AJAX & JS Frameworks
- 44. Indexing AJAX & JS Frameworks What method exists that we know still works?
- 45. Indexing AJAX & JS Frameworks HTML SNAPSHOT
- 46. <head> <meta name="fragment" content="!"> Google / Bing crawls with: _escaped_fragment_= Indexing AJAX & JS: HTML Snapshot
- 47. Indexing AJAX & JS: HTML Snapshot
- 48. Indexing AJAX & JS: HTML Snapshot
- 49. Pre or Realtime Rendered (to users & bots) Indexing AJAX & JS: How To Decide? HTML SNAPSHOT _escaped_fragment_= Trust Googlebot VALIDATE! Progressive Enhancement ‘Dumbed down’ HTML Template 3rd Party Service (prerender.io) Server side (phantomJS / headless browser) Pre-Rendered (to bots)
- 50. Pre or Realtime Rendered (to users & bots) Indexing AJAX & JS: How To Decide? HTML SNAPSHOT _escaped_fragment_= Trust Googlebot VALIDATE! Progressive Enhancement ‘Dumbed down’ HTML Template 3rd Party Service (prerender.io) Pre-Rendered (to bots) Server side (phantomJS / headless browser)
- 51. Indexing AJAX & JS: HTML Snapshot • Upon crawl of URL with _escaped_fragment_=, serve ’dumbed down’ HTML version of page. • Not pre-rendered, rather simplified. • For example, on ecommerce à a view-all category listing with no dynamic facets. Amazing results from our clients.
- 52. Indexing AJAX & JS: How To Decide? HTML SNAPSHOT _escaped_fragment_= Trust Googlebot VALIDATE! Progressive Enhancement ‘Dumbed down’ HTML Template 3rd Party Service (prerender.io) Pre or Realtime Rendered (to users & bots) Pre-Rendered (to bots) Server side (phantomJS / headless browser)
- 53. Indexing AJAX & JS: Pre-rendering Upon crawl of URL with _escaped_fragment_= 1. prerender.io – middleware via reverse proxy that serves a pre-rendered, cached HTML page to bots OR 2. Server side – the server pre-rendered the JS in cached html pages to serve to bots or does it in real-time (headless browser).
- 54. Indexing AJAX & JS: Prerender.io
- 55. Indexing AJAX & JS: Prerender.io
- 56. Indexing AJAX & JS: BromBone
- 57. Indexing AJAX & JS: Server Prerender
- 58. Server side (phantomJS / headless browser) Pre or Realtime Rendered (to users & bots) Indexing AJAX & JS: How To Decide? HTML SNAPSHOT _escaped_fragment_= Trust Googlebot VALIDATE! Progressive Enhancement ‘Dumbed down’ HTML Template 3rd Party Service (prerender.io) Pre-Rendered (to bots)
- 59. Indexing AJAX & JS: Server Side bit.ly/javascriptseo
- 60. Indexing AJAX & JS: Server Side bit.ly/javascriptseobit.ly/javascriptseo
- 61. Indexing AJAX & JS: Server Side bit.ly/javascriptseobit.ly/javascriptseo
- 62. Server side (phantomJS / headless browser) Pre or Realtime Rendered (to users & bots) Indexing AJAX & JS: How To Decide? HTML SNAPSHOT _escaped_fragment_= Trust Googlebot VALIDATE! Progressive Enhancement ‘Dumbed down’ HTML Template 3rd Party Service (prerender.io) Pre-Rendered (to bots)
- 63. Indexing AJAX & JS: Trust Googlebot read these first…
- 64. Testing JS Indexation: Jscrawlability.com
- 65. Validation & Testing: Discovery vs Bloat
- 67. Testing: Fetch & Render JS / AJAX
- 68. Testing: Slice and Dice the Index Advanced Site Operators site:yoursite.com –inurl:cat.jsp -inurl:prod.jsp –inurl:store.jsp
- 69. Testing: Slice and Dice the Index Advanced Site Operators site:yoursite.com inurl:size inurl:cat.jsp -inurl:cid
- 70. Testing: Slice and Dice the Index Advanced Site Operators site:yoursite.com inurl:pdp intext:”write a review”
- 71. Testing: Automate Bloat + Discovery Check
- 72. Testing: Automate Bloat + Discovery Check
- 73. Testing: Search Analytics for Bloat / Discovery
- 74. Testing: Go To The Source: Server Logs!
- 75. Summing It Up • Index Bloat, Crawl Budget, & Testing: Large sites are prone to serious index bloat and wasted crawl budget. Needs diligent testing and an OCD-like attention to detail with the basics. Test often & automate! • JS/AJAX: Pushstate(), JS Frameworks and AJAX present both discovery and bloat challenges. Know the options: short term fixes like HTML snapshot (G+B), and long term re-designs with modern frameworks w/ built in server side rendering.
- 77. References: • Can You Now Trust Google To Crawl Ajax Sites? • Search Engine Optimization Best Practices for AJAX URLs | Webmaster Blog • We Tested How Googlebot Crawls Javascript And Here's What We Learned • Prerender - AngularJS SEO, BackboneJS SEO, or EmberJS SEO • SMX Munich Advanced Technical SEO Brainstorm - Google Docs • www.simoahava.com/seo/dynamically-added-meta-data-indexed-google-crawlers/ • Speakers | Search Marketing Expo – SMX Munich • JavaScript + SEO: Better Together — Medium • SEO AJAX Crawlability in a Responsive Publisher World • SEO Strategies for JavaScript-Heavy Single Page Applications or AJAX Sites | Search Engine Watch • The Basics of JavaScript Framework SEO in AngularJS - Builtvisible • Can Search Engines Crawl Javascript? • https://www.w3.org/wiki/Graceful_degradation_versus_progressive_enhancement#Graceful_degradatio n_and_progressive_enhancement_in_a_nutshell • SEO and JS: New Challenges • BromBone | SEO for your AngularJS, EmberJS, or BackboneJS website. • DIY AngularJS SEO with PhantomJS (the easy way!) | Lawsonry • https://scotch.io/tutorials/angularjs-seo-with-prerender-io
- 78. Image Credits: fat-american-1.jpg (1280×955) bigbrands1.jpg (570×383) consistencydemotivator_large.jpeg (480×338) 04-godfather-keep-friend.jpg (518×300) 4da1a1a23dba011a7ba6918986a6b818302b949ae694b27d559cf8e733 08bf7b.jpg (604×392) the-17-craziest-cannibal-attacks-in-history-u2.jpg (520×272) taxonomy-types-800x450.png (800×450) wireframes-homecat.png (1000×460) Check-yoself.jpg (800×1025) Dangerous-Curve-Ahead-Sign-K-6513.gif (400×400) crawlerserver2.png (884×445) beach.png (1196×838)