Webmaster level: All
We recently announced that our indexing system has been rendering web pages more like a typical modern browser, with CSS and JavaScript turned on. Today, we're updating one of our technical Webmaster Guidelines in light of this announcement.
For optimal rendering and indexing, our new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that your pages use. This provides you optimal rendering and indexing for your site. Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.
Historically, Google indexing systems resembled old text-only browsers, such as Lynx, and that’s what our Webmaster Guidelines said. Now, with indexing based on page rendering, it's no longer accurate to see our indexing systems as a text-only browser. Instead, a more accurate approximation is a modern web browser. With that new perspective, keep the following in mind:
In conjunction with the launch of our rendering-based indexing, we also updated the Fetch and Render as Google feature in Webmaster Tools so webmasters could see how our systems render the page. With it, you'll be able to identify a number of indexing issues: improper robots.txt restrictions, redirects that Googlebot cannot follow, and more.
And, as always, if you have any comments or questions, please ask in our Webmaster Help forum.
Webmaster level: intermediate-advanced
Submitting sitemaps can be an important part of optimizing websites. Sitemaps enable search engines to discover all pages on a site and to download them quickly when they change. This blog post explains which fields in sitemaps are important, when to use XML sitemaps and RSS/Atom feeds, and how to optimize them for Google.
Sitemaps can be in XML sitemap, RSS, or Atom formats. The important difference between these formats is that XML sitemaps describe the whole set of URLs within a site, while RSS/Atom feeds describe recent changes. This has important implications:
For optimal crawling, we recommend using both XML sitemaps and RSS/Atom feeds. XML sitemaps will give Google information about all of the pages on your site. RSS/Atom feeds will provide all updates on your site, helping Google to keep your content fresher in its index. Note that submitting sitemaps or feeds does not guarantee the indexing of those URLs.
Example of an XML sitemap:
<?xml version="1.0" encoding="utf-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://example.com/mypage</loc> <lastmod>2011-06-27T19:34:00+01:00</lastmod> <!-- optional additional tags --> </url> <url> ... </url> </urlset>
Example of an RSS feed:
<?xml version="1.0" encoding="utf-8"?> <rss> <channel> <!-- other tags --> <item> <!-- other tags --> <link>http://example.com/mypage</link> <pubDate>Mon, 27 Jun 2011 19:34:00 +0100</pubDate> </item> <item> ... </item> </channel> </rss>
Example of an Atom feed:
<?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> <!-- other tags --> <entry> <link href="http://example.com/mypage" /> <updated>2011-06-27T19:34:00+01:00</updated> <!-- other tags --> </entry> <entry> ... </entry> </feed>
“other tags” refer to both optional and required tags by their respective standards. We recommend that you specify the required tags for Atom/RSS as they will help you to appear on other properties that might use these feeds, in addition to Google Search.
XML sitemaps and RSS/Atom feeds, in their core, are lists of URLs with metadata attached to them. The two most important pieces of information for Google are the URL itself and its last modification time:
URLs in XML sitemaps and RSS/Atom feeds should adhere to the following guidelines:
Specify a last modification time for each URL in an XML sitemap and RSS/Atom feed. The last modification time should be the last time the content of the page changed meaningfully. If a change is meant to be visible in the search results, then the last modification time should be the time of this change.
<lastmod>
<pubDate>
<updated>
Be sure to set or update last modification time correctly:
XML sitemaps should contain URLs of all pages on your site. They are often large and update infrequently. Follow these guidelines:
RSS/Atom feeds should convey recent updates of your site. They are usually small and updated frequently. For these feeds, we recommend:
Generating both XML sitemaps and Atom/RSS feeds is a great way to optimize crawling of a site for Google and other search engines. The key information in these files is the canonical URL and the time of the last modification of pages within the website. Setting these properly, and notifying Google and other search engines through sitemaps pings and PubSubHubbub, will allow your website to be crawled optimally, and represented accordingly in search results.
If you have any questions, feel free to post them here, or to join other webmasters in the webmaster help forum section on sitemaps.
Meet my sister, Marnie, who owns a jewelry store and my cousin, Scott, who works as a realtor. Follow them as we talk about the big changes in the last decade, such as making sure your business can reach customers at work, home, or on-the-go using their mobile phones.
With the example of Scott, the realtor, you’ll learn about the marketing funnel, setting an online goal, and highlighting what makes your business special.
Marnie and Scott figure out their customers’ most common journeys to reach their business. We'll use their examples to brainstorm how you can reach customers on review sites, through search engines, maps apps, and social and professional networking sites.
The fundamentals and best practices to take your business from offline to online!
With Scott’s business as a realtor, see how to demonstrate that your local business is the best choice for customers by adding photos, videos, and getting reviews.
We'll end the series by showing how Scott makes sure his online presence sends a cohesive message to customers and answers all their common questions. :)
Webmaster level: advanced
Over the summer the Webmaster Tools team has been cooking up an update to the Webmaster Tools API. The new API is consistent with other Google APIs, makes it easier to authenticate for apps or web-services, and provides access to some of the main features of Webmaster Tools.
If you've used other Google APIs, getting started with the new Webmaster Tools API will be easy! We have examples for Python, Java, as well as OACurl (for fans of command lines).
This API allows you to:
We'd love to see what you're building with our APIs! Feel free to link to your projects in the comments below. Should you have any questions about the usage of the API, feel free to post in our help forum as well.
Update (16:30h CET, September 12th): We're noticing an enthusiastic uptick in the markup implementation after the initial announcement last week! Here are the two main issues we've observed so far, and what you need to do to fix them:
Everyone wants to use less bandwidth: hosts want lower bills, mobile users want to stay under their limits, and no one wants to wait for unnecessary bytes. The web is full of opportunities to save bandwidth: pages served without gzip, stylesheets and JavaScript served unminified, and unoptimized images, just to name a few.
So why isn't the web already optimized for bandwidth? If these savings are good for everyone then why haven't they been fixed yet? Mostly it's just been too much hassle. Web designers are encouraged to "save for web" when exporting their artwork, but they don't always remember. JavaScript programmers don't like working with minified code because it makes debugging harder. You can set up a custom pipeline that makes sure each of these optimizations is applied to your site every time as part of your development or deployment process, but that's a lot of work.
An easy solution for web users is to use an optimizing proxy, like Chrome's. When users opt into this service their HTTP traffic goes via Google's proxy, which optimizes their page loads and cuts bandwidth usage by 50%. While this is great for these users, it's limited to people using Chrome who turn the feature on and it can't optimize HTTPS traffic.
With Optimize for Bandwidth, the PageSpeed team is bringing this same technology to webmasters so that everyone can benefit: users of other browsers, secure sites, desktop users, and site owners who want to bring down their outbound traffic bills. Just install the PageSpeed module on your Apache or Nginx server [1], turn on Optimize for Bandwidth in your configuration, and PageSpeed will do the rest.
If you later decide you're interested in PageSpeed's more advanced optimizations, from cache extension and inlining to the more aggressive image lazyloading and defer JavaScript, it's just a matter of enabling them in your PageSpeed configuration.
Learn more about installing PageSpeed or enabling Optimize for Bandwidth.
Enter your email address:
Delivered by FeedBurner
