SearchLove London 2017 | Jon Myers | Mobile-First Preparedness: What We've Learned From Crawling the Top 1 Million Websites
6 likes6,284 views
This document outlines insights from a study on mobile-first preparedness based on crawling the top 1 million websites. It highlights the impact of mobile configurations on search engine optimization, indicating that responsive design is optimal and separate mobile sites face significant configuration issues. Additionally, it discusses the low adoption of HTTPS across these sites and emphasizes the future shift towards progressive web apps.
SearchLove London 2017 | Jon Myers | Mobile-First Preparedness: What We've Learned From Crawling the Top 1 Million Websites
- 1. @jondmyers @DeepCrawl Mobile-First Preparedness- what we've learned from crawling the top 1 million websites. Jon Myers – Chief Growth Officer @jondmyers @DeepCrawl Source: http://glooadvertising.com.au/is-your-website-mobile-ready/
- 2. @jondmyers @DeepCrawl Google is our world…
- 3. @jondmyers @DeepCrawl Google is our world… 2 Trillion searches per year!
- 4. @jondmyers @DeepCrawl Google is our world… 15% of searches new each day!
- 5. @jondmyers @DeepCrawl Google is our world… 60% of search are on Mobiles!
- 7. @jondmyers @DeepCrawl Sooooooo 2015! And before! @jondmyers @DeepCrawl
- 8. @jondmyers @DeepCrawl Google is thinking about it... @jondmyers @DeepCrawl
- 9. @jondmyers @DeepCrawl Helpful to see… @jondmyers @DeepCrawl
- 10. @jondmyers @DeepCrawl 1 page at a time could take a while!… @jondmyers @DeepCrawlSource: http://shop.adamjk.com/product/1-page-at-a-time
- 11. @jondmyers @DeepCrawl History of my devices…
- 12. @jondmyers @DeepCrawl History of Mobile… WAP HTML 1999 2007 2009 2014 2015 2017 Separate Mobile Pages (Mobile Site or Dynamic Delivery) Responsive Design Deep App Linking AMP & Progressive Web Apps Mobile- first Indexing
- 13. @jondmyers @DeepCrawl Time to get SH*T done… @jondmyers @DeepCrawlSource: https://www.successimpulse.com/products/less-talk-more-action?variant=44575800839
- 14. @jondmyers @DeepCrawl Heavy Data ahead… @jondmyers @DeepCrawlSource: https://betanews.com/2015/09/14/8-early-warning-signs-of-problems-in-your-data-governance-plan/
- 15. @jondmyers @DeepCrawl Being Majestic… @jondmyers @DeepCrawl Source: https://lafleur.marketing/blog/six-mad-men-marketing-tips-don-draper/
- 16. @jondmyers @DeepCrawl What we did, why? And only we could… @jondmyers @DeepCrawlSource: https://markfisherfitness.com/what-could-have-been/
- 17. @jondmyers @DeepCrawl History of Mobile SEO Configuration… Desktop Mobile Responsive AMP pages Mobile first One version for desktop devices One to rule them all - one version designed to work equally good on desktop and mobile Dedicated light weight version designed for a fast loading Dedicated mobile pages served on a separate URL e.g. m.domain or dynamically served on the same URL Mobile becomes the PRIMARY version.
- 18. @jondmyers @DeepCrawl Types of Mobile Configuration… 1. No Mobile Configuration (desktop only) 2. Responsive 3. Dynamic 4. Desktop + Dedicated Mobile 5. Desktop + AMP 6. Responsive/Dynamic + AMP 7. Desktop + Dedicated Mobile + AMP 8. Responsive + Dedicated Mobile + AMP 9. Responsive + Dynamic + Dedicated Mobile + AMP 10. Mobile/AMP only 11. Mobile + AMP
- 19. @jondmyers @DeepCrawl Impact of Mobile-first… -CRITICAL- - Desktop only (Heavily Affected) -HIGH PRIORITY- - Desktop + Dedicated Mobile (Affected) -HIGH PRIORITY- - Dynamic (Affected e.g. Content Issues) -LOW PRIORITY- - Responsive (Not affected) Impact of Mobile-first on different Mobile Configurations:
- 20. @jondmyers @DeepCrawl Responsive Dynamic Separate Mobile Would expect Google to use both user-agents on the same URL to validate that the same content is returned. Google needs to crawl with both user agents to validate the mobile version. Hint: Use the Vary HTTP header! Google needs to crawl the dedicated mobile URLs with a mobile user agent to validate the pages and confirm the content matches the desktop pages. How does Google crawl different configurations?
- 21. @jondmyers @DeepCrawl Responsive design is the optimal situation… “Responsive design is pretty much the optimal situation because regardless of which device we use to crawl the site we will see the same thing. We’ll get the same content and the structured data, the same videos, the same images and the same internal links. It’s all there.” Reference: https://youtu.be/7Aq9bFdfMuE?t=54m53s
- 22. @jondmyers @DeepCrawl How does the world look?... Source: http://abcnews.go.com/topics/news/doomsday-clock.htm
- 23. @jondmyers @DeepCrawl How are sites set up for mobile?...
- 24. @jondmyers @DeepCrawl Mobile Configuration Breakdown… Breakdown of all sites where mobile configuration could be identified Significant no. dynamic sites – work to ensure meta data is kept consistent across different versions of page. Responsive design dominates - Path of least resistance moving to Mobile-first Index Separate Mobile unpopular – high maintenance to ensure desktop/mobile equivalence 15,000 sites 190,000 sites 795,000 sites
- 25. @jondmyers @DeepCrawl Separate Mobile Sites – Are they up to scratch?... Desktop Page Mobile Page Rel=canonical 61% of Separate Mobile sites have correct canonical to desktop page.
- 26. @jondmyers @DeepCrawl Separate Mobile Sites – Are they up to scratch?... Desktop Page Mobile Page No canonical 21% of Separate Mobile sites had no canonical to desktop equivalent.
- 27. @jondmyers @DeepCrawl Separate Mobile Sites – Are they up to scratch?... Desktop Page Mobile Page Other page? 18% of Separate Mobile sites had canonicals pointing to wrong page
- 28. @jondmyers @DeepCrawl Separate Mobile Sites – Desktop/Mobile Equivalence… Of 15,000 Separate Mobile sites returning 200 response codes: Desktop Page Mobile Page Matching title & meta descriptions Desktop Page Mobile Page Matching meta descriptions Desktop Page Mobile Page Matching title 7.7% 7.9% 5.8%
- 29. @jondmyers @DeepCrawl Separate Mobile Issues… 1% of separate mobile sites had noindex meta tag 25% of all Separate Mobile sites returned a non-200 status code 21% returned 3xx redirection 2% Separate Mobile sites returned 4xx errors 1% returned 5xx errors
- 30. @jondmyers @DeepCrawl Google pushing for responsive web… https://webmasters.googleblog.com/2017/09/how-to-move-from- m-dot-urls-to.html Separate Mobile Google needs to crawl the dedicated mobile URLs with a mobile user agent to validate the pages and confirm the content matches the desktop pages.
- 31. @jondmyers @DeepCrawl Order of Mobile-first Indexing… “If you have a vary header which returns a different page for a mobile user agent, Google will use that as the mobile page instead of the responsive page.” Reference:https://www.youtube.com/watch?v=8NyZypIfO zI&t=36m31s
- 32. @jondmyers @DeepCrawl Dynamic configurations… Initial analysis found ?% of websites that serve different content based on user agent do not have a Vary: User-agent header. Dynamic – same URL, different HTML dependent on user agent
- 33. @jondmyers @DeepCrawl Let’s talk about speed… Source: https://play.google.com/store/apps/details?id=com.ea.game.nfs14_row
- 34. @jondmyers @DeepCrawl Fetch times – The need for speed… Percentage of sites split by fetch time Fetch time includes time taken to fetch URL and display the HTTP response. This doesn’t include the time taken to request or run any associated resources (such as images or scripts) on the page.
- 35. @jondmyers @DeepCrawl But are the top million sites really that fast?...
- 36. @jondmyers @DeepCrawl Fetch times – Configuration breakdown… Dynamically servedResponsive Separate Mobile Average fetch time by configuration (secs): Separate mobile sites have the slowest fetch times. Responsive sites have fastest fetch times of three mobile configurations.
- 37. @jondmyers @DeepCrawl Fetch time with mobile desktop user agents… 19% of sites had a fetch time between 1- 2 seconds. 68% of sites had a fast fetch time – below one second. 13% of sites had a slow fetch time exceeding 2 seconds
- 38. @jondmyers @DeepCrawl What does Google have to say about site performance?... Source: https://www.thinkwithgoogle.com/about/
- 39. @jondmyers @DeepCrawl Google research – Performance per vertical… Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/ Consumer sites have faster more responsive web servers. Business and finance lagging behind. Technology my arse!
- 40. @jondmyers @DeepCrawl Google research – Performance per vertical… Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/ Google says to aim for 2.4 seconds load time for a page!!!???!!
- 41. @jondmyers @DeepCrawl Lets Talk Page Size…
- 42. @jondmyers @DeepCrawl Google research – Page Size… Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/ According to Google, sites across all verticals are on average larger than the 500kb recommendation. Again, technology sites aren’t leading the way.
- 43. @jondmyers @DeepCrawl What did we find out about HTML size?... 94%<=200k 5%200k - 500k
- 44. @jondmyers @DeepCrawl Accelerated Mobile Pages… Percentage of sites with AMP version of homepage: Responsive 0.63% Dynamic 0.61% Separate Mobile 0.86%
- 45. @jondmyers @DeepCrawl What about HTTPS?...
- 46. @jondmyers @DeepCrawl HTTPS – How many have made the switch?... HTTPS (22%) HTTP (78%) Has HTTPS reached critical mass? Our data says there’s a long way to go for the top one million sites. This figure is much lower than currently reported Mozcast – 68% of first page results have SSL encryption. Source: http://mozcast.com/features
- 47. @jondmyers @DeepCrawl HTTPS – Adoption across mobile configurations… Dynamically served sites appear to be particularly slow to adopt HTTPS. Surprisingly Separate Mobile sites leading the way for the switch to HTTPS.
- 48. @jondmyers @DeepCrawl HTTPS doesn’t end there… Watch out for non-secure form fields (we found a lot of them) as Chrome is now flagging these with security warnings. 12.7% incorrectly configured sites with links between protocols and some sites with mixed content e.g. a HTTPS page with reference to a script on an HTTP URL.
- 49. @jondmyers @DeepCrawl Carving up Configurations by Country…
- 50. @jondmyers @DeepCrawl Splitting the Majestic Million by TLDs… Percentage of sites per TLD
- 51. @jondmyers @DeepCrawl .edu & .gov are leading the way with responsive design…
- 52. @jondmyers @DeepCrawl …and .co.uk TLDs aren’t far behind… ResponsiveDynamic Separate
- 53. @jondmyers @DeepCrawl What next? Plenty of issues… Non Secure Form Fields Pages without Valid Canonical Tag Failed URLs Empty Pages Max Fetch Time Thin Pages Missing H1 Tags Non-301 Redirects Pages with Duplicate Body Duplicate Pages Malformed URLs High External Linking Max URL Length Missing Titles Pages with Duplicate Titles No Descriptions & No Snippets Unauthorised Pages 5xx Errors Max Links Broken Pages (4xx Errors) Max Description Length Short Titles Unlinked Paginated Pages Hreflang to Non-200 URLs Non-rel Alted AMP Pages All Broken Links Non-reciprocal Mobile/AM Duplicate Description Sets Max Content Size Uncategorised HTTP Response Codes Max Title Length Duplicate Body Sets Duplicate Page Sets Pages with Duplicate Descriptions Canonical to Non-200 Max Redirections Max HTML Size Short Descriptions Redirect Loops Duplicate Title Sets And more… Non-200 Mobile/AMP Excessive Redirects In (Admin Only) Mobile Links Out Mismatch Mobile Links In Mismatch Mobile Word Count Mismatch Mobile Content Mismatch Duplicate Pages including Primary
- 54. @jondmyers @DeepCrawl In a nutshell…
- 55. @jondmyers @DeepCrawl Separate Mobile sites are poorly configured
- 56. @jondmyers @DeepCrawl Dynamic sites need to check they have Vary HTTP response header…
- 57. @jondmyers @DeepCrawl HTTPS adoption relatively low amongst top million sites…
- 58. @jondmyers @DeepCrawl Fetch time is surprisingly low???…
- 59. @jondmyers @DeepCrawl Watch this space for an even deeper dive into the data… Source: http://dive-bohol.com/
- 60. @jondmyers @DeepCrawl But what does the future hold?...
- 61. @jondmyers @DeepCrawl Go beyond Responsive with PWA’s… Progressive Web Apps is: • Progressive • Responsive • App-like • And more... Progressive Web Apps can fall back to AMP. Hint… first load as AMP. You should think about it now. Progressively Web App is THE FUTURE.
- 62. @jondmyers @DeepCrawl Jon Myers DeepCrawl Chief Growth Officer