More information?
Lets' talk

SEO Tutorial

How to stop Google Search Engines to Crawling your Website?


Googlebot is Google’s web crawling bot (sometimes also called a “spider”). Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.

We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Googlebot’s crawl process begins with a list of webpage URLs, generated from previous crawl processes and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links (SRC and HREF) on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

For webmasters: Googlebot and your site

How Googlebot accesses your site

For most sites, Googlebot shouldn’t access your site more than once every few seconds on average. However, due to network delays, it’s possible that the rate will appear to be slightly higher over short periods.

Googlebot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites they’re indexing in the network. Therefore, your logs may show visits from several machines at, all with the user-agent Googlebot. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server’s bandwidth. Request a change in the crawl rate.

Blocking Googlebot from content on your site

It’s almost impossible to keep a web server secret by not publishing links to it. As soon as someone follows a link from your “secret” server to another web server, your “secret” URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log. Similarly, the web has many outdated and broken links. Whenever someone publishes an incorrect link to your site or fails to update links to reflect changes in your server, Googlebot will try to download an incorrect link from your site.

If you want to prevent Googlebot from crawling content on your site, you have a number of options, including using robots.txt to block access to files and directories on your server.

Once you’ve created your robots.txt file, there may be a small delay before Googlebot discovers your changes. If Googlebot is still crawling content you’ve blocked in robots.txt, check that the robots.txt is in the correct location. It must be in the top directory of the server (for example,; placing the file in a subdirectory won’t have any effect.

If you just want to prevent the “file not found” error messages in your web server log, you can create an empty file named robots.txt. If you want to prevent Googlebot from following any links on a page of your site, you can use the nofollow meta tag. To prevent Googlebot from following an individual link, add the rel="nofollow" attribute to the link itself.

Here are some additional tips:

  • Test that your robots.txt is working as expected. The Test robots.txt tool on the Blocked URLs page lets you see exactly how Googlebot will interpret the contents of your robots.txt file. The Google user-agent is (appropriately enough) Googlebot.
  • The Fetch as Google tool in Search Console helps you understand exactly how your site appears to Googlebot. This can be very useful when troubleshooting problems with your site’s content or discoverability in search results.

Making sure your site is crawlable

Googlebot discovers sites by following links from page to page. The Crawl errors page in Search Console lists any problems Googlebot found when crawling your site. We recommend reviewing these crawl errors regularly to identify any problems with your site.

If your robots.txt file is working as expected, but your site isn’t getting traffic, here are some possible reasons why your content is not performing well in search.

Problems with spammers and other user-agents

The IP addresses used by Googlebot change from time to time. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot). You can verify that a bot accessing your server really is Googlebot by using a reverse DNS lookup.

Googlebot and all respectable search engine bots will respect the directives in robots.txt, but some nogoodniks and spammers do not. Report spam to Google.

Google has several other user-agents, including Feedfetcher (user-agent Feedfetcher-Google). Since Feedfetcher requests come from explicit action by human users who have added the feeds to their Google home page and not from automated crawlers, Feedfetcher does not follow robots.txt guidelines. You can prevent Feedfetcher from crawling your site by configuring your server to serve a 404, 410, or other error status message to user-agent Feedfetcher-Google. More information about Feedfetcher.

Continue reading

How Do I Verify A Site with Google Webmaster Tools?

You have a number of options to verify that you own your website:

  • Add a meta tag to your home page (proving that you have access to the source files). To use this method, you must be able to edit the HTML code of your site’s pages.

  • Upload an HTML file with the name you specify to your server. To use this method, you must be able to upload new files to your server.

  • Verify via your domain name provider. To use this method, you must be able to sign in to your domain name provider (for example, or or hosting provider and add a new DNS record.

  • Add the Google Analytics code you use to track your site. To use this option, you must be an administrator on the Google Analytics account, and the tracking code must use the new asynchronous snippet.


How Do I Automatically Get New Web Pages Into Google?

You can ping Google (and other engines) using RSS or XML. Continue reading

Optimising For ‘The Long Click’

When it comes to rating user satisfaction, there are a few theories doing the rounds at the moment that I think are sensible. Google could be tracking user satisfaction by proxy. When a user uses Google to search for something, user behaviour from that point on can be a proxy of the relevance and relative quality of the actual SERP. Continue reading

User Experience’ Matters

Is User Experience A Ranking Factor?

User experience is mentioned 16 times in the main content of the quality raters guidelines (official PDF), but we have been told by Google it is not, per say, a classifiable ‘ranking factor‘ on desktop search, at least.

On mobile, sure, since UX is the base of the mobile friendly update. On desktop currently no. (Gary Illyes: Google, May 2015)

Continue reading

Balancing Conversions With Usability & User Satisfaction

Take pop-up windows  or pop-unders as an example:

According to usability expert Jakob Nielson, 95% of website visitors hated unexpected or unwanted pop-up windows, especially those that contain unsolicited advertising.

In fact, Pop-Ups have been consistently voted the Number 1 Most Hated Advertising Technique since they first appeared many years ago. Continue reading

Technical SEO

If you are doing a professional SEO audit for a real business, you are going to have to think like a Google Search Quality Rater AND a Google search engineer to provide real long term value to a client.

Google has a LONG list of technical requirements it advises you meet, on top of all the things it tells you NOT to do to optimise your website. Continue reading