Author

Analytics advocate

SEO expert

User Experience passionate

Follow me on
Category | SEO
Difficulty |

How to block crawler spam and other irrelevant bots.

Crawler Spam is a blackhat method used by some spammers to drive traffic to their website. They use a crawler to visit thousands of websites and leave a fake referrer/referral. This false data is not useful, and it only pollutes your Analytics.

These are some of the latest crawler spam:

  • keywords-monitoring-your-success.com
  • free-video-tool.com
  • uptime.com
  • top1-seo-service.com
  • responsive-test.net

The following methods will block any crawler whether it is spam or other irrelevant referrers, no matter the name it uses or which analytics platform you have Shopify, BigCommerce, Etsy, Piwik, etc.

Note: If you are using Google Analytics I recommend you to follow this guide. Most of the spam in GA never access your site (Ghost Spam) so all the server-side solutions mentioned below won't have any effect at all a difference with Crawler referral spam. If you are not sure what type of spam you are seeing, you can check this list.

Stop Crawler Spam with the .htaccess file - Apache

The .htaccess is a very powerful file that can restrict the traffic to your website between many other things.

You have to be extremely careful when editing this file because even one misplaced character can shut down your entire website. So it's important to make a backup first, just copy in a text file everything as it is. To access your .htaccess file, you should go to the root directory in your Cpanel "yourdomain.com/cpanel" (most of the times this file is hidden)

If you can't find your .htaccess file or don't feel comfortable editing it, you can ask your hosting company to help you.

Add the following lines to your htaccess file

## STOP Semalt.com Crawler Spam
RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} uptime\.com [NC,OR]
RewriteCond %{HTTP_REFERER} top1\-seo\-service\.com [NC,OR]
RewriteCond %{HTTP_REFERER} free\-video\-tool\.com [NC,OR]
RewriteCond %{HTTP_REFERER} responsive\-test\.net [NC,OR]
RewriteCond %{HTTP_REFERER} keywords\-monitoring\-your\-success\.com [NC]
RewriteRule .* - [F]

TIPS:

You can keep adding lines with crawlers you want to exclude. This file uses REGEX, so you have to follow a few rules

  • IMPORTANT: The last RewriteCond must have [NC] only at the end don't use [NC,OR] or you will block everything.
  • The dot and the hyphen are considered special characters and they need to be escaped using a backslash before them, for example, crawler-spam.com  will look like: crawler\-spam\.com
  • More about REGEX

Block Crawler Spam in IIS / Asp.Net using web.config file

If you are using IIS / Asp.Net you can use your web.config file to block the crawlers. As the .htaccess file, it is also important to make a backup or ask your hosting or website administrator.

To block crawler referral spam with the web.config file copy the following:

<rewrite>
  <rules>
     <rule name="abort crawler referer spam requests" stopProcessing="true">
       <match url=".*" />
         <conditions>
            <add input="{HTTP_REFERER}" pattern="semalt\.com" />
            <add input="{HTTP_REFERER}" pattern="uptime\.com" />
            <add input="{HTTP_REFERER}" pattern="top1\-seo\-service\.com" />
            <add input="{HTTP_REFERER}" pattern="free\-video\-tool\.com" />
            <add input="{HTTP_REFERER}" pattern="responsive\-test\.net" />
            <add input="{HTTP_REFERER}" pattern="keywords\-monitoring\-your\-success\.com" />
         </conditions>
         <action type="AbortRequest" />
    </rule>
    <!--The rest of your rules, if you have any-->
  </rules>
</rewrite>

TIPS:

Like in the htaccess file you can keep adding lines with crawlers you want to exclude. This file also uses REGEX, so you have to follow a few rules

  • The dot and the hyphen are considered special characters and they need to be escaped using a backslash before them, for example, crawler-spam.com  will look like: crawler\-spam\.com

Block Crawler Referrer Spam in WordPress with a Plugin

If you use WordPress and don't want to edit sensitive files, you can use any of this plugins; they do exactly what is explained in the first method, but without risking a misconfiguration.

You can use any of the following plugins to block Crawler Spam in WordPress:

The setup is pretty straight forward, simply install the plugin, enable it and use a URL to redirect it.

Author

Analytics advocate

SEO expert

User Experience passionate

Follow me on
Be the first to comment :)