How To Remove Semalt and Analytics Referral Spam

Ok Semalt, you have officially annoyed me and I’ve put you into my “bottom feeding scum, don’t touch with someone elses barge pole” file.

Why?

Today, once again, I used MY time to explain to a client that you (and your buddy Kambasoft) are NOT real visitors and are essentially worthless spam visits to their small local website.  I don’t appreciate it, so I am writing this blog post to save me time in future.

If you don’t want to read my rant and blurb, scroll down.  There’s instructions on how you can block Semalt and friends and/or ignore them through Google Analytics.

Semalt Referral Spam

My clients are small businesses often targetting a specific local area.  More often than not, they are trying to get their head around this “online stuff” and want to make it work for them.

One of the many things I’ll recommend is regular visits to their analytics.  Life is much easier when you know where your traffic is coming from, what is working and what is not.

A local business owner wears many hats.  Web marketing is a very small one and is often limited on time and budget.  Checking your analytics can help prioritise which social sites to spend time on, any ads are bringing traffic and the time and/or money spent is actually bringing a return.

Enter Semalt – showing up as a referrer in analytics.  Not once in a blue moon either…

Who are Semalt?

Semalt claim to be a

…professional webmaster analytics tool that opens the door to new opportunities for the market monitoring, yours and your competitors’ positions tracking and comprehensible analytics business information.

(Their words, no I’m not linking to their page)

For a “professional webmaster analytics tool” one has to wonder why they think it is acceptable to send their crawler as a referrer and not a standard bot visit.

They allegedly understand webmasters frustration at them totally screwing up your analytics with their referrer spam and invite you to remove your website from the seed list.

DON’T REQUEST TO REMOVE YOUR WEBSITE VIA SEMALT

I’ll show you why you shouldn’t use Semalt’s removal request in a moment.

Of course, to find where their removal request is you have to do some rummaging around the web.  The first you’ll hear of it is via comments on so many blogs complaining about Semalt or may be on Twitter.  They have a person that just goes around Tweeting and commenting trying to ease people’s concerns.  Semalt’s homepage gives you nothing but a sign up form.  No links to usual pages like contact, services, privacy or anything else.

Being the sweet and innocent person I am (stop laughing), I removed MyLocalBusinessOnline.co.uk from their seed list some months back in good faith.  They did honour it.

I no longer receive visits from the main Semalt crawler.

All was quiet for a week or two and then bombarded with visits from various semalt subdomains, kambasoft and savetubevideo.

The client who I was talking to earlier today did her own check on Semalt and also put in a removal request for her website a month or so ago.  She logged into analytics today to this…

Google Analytics screenshot – this is just a few of the Semalt referrals to a small local website in 1 month!

73% of her referral traffic (38% of total traffic) is from Semalt and friends.

I have no idea what Semalt’s game is, but those numbers certainly are NOT honouring a removal request.  Maybe in the Ukraine (where Semalt are based) “Remove” actually means, “Come, bring your friends! We have cake!”

What is a web crawler?

A web crawler is an automated bot that systematically crawls the World Wide Web.  Search engines use them to index your web pages so they can efficiently serve their search results.  There are other uses, both good and bad for crawlers.  Most of the time you wouldn’t be aware of their visits to your site.

Semalt claim that their crawler is no different than Google, Bing or Yahoo crawling your website.   None of the major search crawlers come in as a referrer.  Their bots (and others like them) pop along in the backgound and you wouldn’t see them screwing up your analytics pretending to be real visits.

The screenshot above is to a local website for a small business based in a village in south England.  It’s rarely updated.  Google, Bing, Yahoo and others do pop along regularly but not several times per day.  There’s simply no need to.

The long and short of the matter is Semalt’s crawlers do not act like legit web crawlers.

What are Semalt up to?

At first glance, this appears to be nothing more than a shady marketing technique to get curious webmasters to visit their site.  Go to any of the referral URLs for Semalt and you land on their main web page that currently invites you to try their software free for 7 days.

Look a bit closer, visit the Kambasoft referrals and you are redirected to random websites – perhaps it’s just a way of driving traffic?  Useless, untargeted traffic perhaps, but I am sure someone somewhere is fooled by big numbers.

It would appear or friends at Semalt are not content with spamming analytics in their own name.  Yo’ll also see them turning up as “best-seo-offer” and “best-seo-solution”.

Do more research and it gets a bit scary…

It would appear that Semalt are involved in more than shady referral tactics, going as far as actually infecting people’s computers with trojans to build their web of spambots.  Read more about that side of things at nabble.

How to block Semalt and friends

4 Ways To Block Semalt (and any other referral bot annoying you)

Semalt doesn’t appear to honour robot.txt.  They also have so many IP addresses, blocking by IP is impractical.  If you really want to try that route, you can find a list of IP addresses associated with Semalt here.

There are some alternative steps you can take.

1. Edit your .htaccess file

(if you have one and have access)

Since Semalt ignores robot.txt, you can block it’s crawler using your .htaccess file.  This file is very powerful and can break your website – so if you’re unsure then leave it be.  The semalt crawlers themselves don’t appear to be malicious at the moment, just a pain in the backside screwing stats and using resources (this can become a problem though!)

Add the following code to your .htaccess file

# Block fake traffic
RewriteEngine on
Options +FollowSymlinks
# Block all http and https referrals from "savetubevideo.com" and all subdomains of "savetubevideo.com"
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*savetubevideo\.com\ [NC,OR]
# Block all http and https referrals from "srecorder.com" and all subdomains of "srecorder.com"
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*srecorder\.com\ [NC,OR]
# Block all http and https referrals from semalt.com" and all subdomains of "semalt.com"
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*semalt\.com\ [NC,OR]
# Block all http and https referrals from "kambasoft.com" and all subdomains of "kambasoft.com"
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*kambasoft\.com\ [NC]
RewriteRule ^(.*)$ http://semalt.com/ [L]

The code basically says if a crawler comes in from any of the Semalt sites then turn it around and send it back to Semalt.  They can have their spam back, thank you.

If you’re not as annoyed as me and don’t feel comfortable sending their bot back to them, you can replace the last line with

RewriteRule .* - [F]

Update January 2015

The fake bot traffic battle increases with ilovevitaly and friends now joining in.  I’ve updated the code above on my htaccess to include ilovevitaly.com, ilovevitaly.co and their side-kicks priceg.com and econom.co.

Since I have had no issues or visits in my analytics from Semalt since adding the above code you also may have the same success.  A text file with the above Semalt code and extra ilovevitaly spam can be downloaded by right clicking here.  Copy and paste into your htaccess file after the WordPress commands.  This works for me, I take no responsibility for any errors etc etc. Use at your own risk.

Many thanks to Michael Martinez for the code on their post over at Marketing Pilgrim, “Tips for Blocking Semalt and Botnet Attacks“.

The original bit of code I was using is found at logorrhoea.net.

# block visitors referred from semalt.com
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [NC]
RewriteRule .* - [F]

This snippet only blocks Semalt, not subdomains or their friends at Kambasoft et. al.  Instead of redirecting the bot back to Semalt, it simply denies access.  It worked perfectly well until Semalt started adding more and more sources.  Since I reached 100’s of referrers between Semalt and Kambasoft, it was getting rather silly.  The first code above is a cleaner way of doing it.

You can add each referrer as you see one come in.

# block visitors referred from semalt.com
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} semalt\.semalt\.com [NC]
RewriteRule .* - [F]

WordPress recommend the following code added to .htaccess.

SetEnvIfNoCase Referer semalt.com spammer=yes
Order allow,deny
Allow from all
Deny from env=spammer

Again, this will only block the original Semalt bot and you’ll need to add each referrer as you see it.

ALWAYS take a copy of your original .htaccess file so you can change back if anything does go pear shaped.  I did not write any of this code (I’m not that clever!) I make no guarentees of suitability, fit for purpose or anything else.

2. Blocking via PHP

The nabble guys mentioned above who tracked down Semalt using malware added an update to their post:

Update / August 8 — We’ve created a simple PHP package to block referrer spammers such as Semalt from visiting your site: https://github.com/nabble/semalt-blocker

Far too technical for me – but may be useful to people running PHP based websites (or more likely, their developers!)

3. Remove Semalt from showing in Google Analytics

Thanks Malcolm Oakley from Southdowns Web for the nifty GIF tutorial. Click on the image to open a larger version in a new window.

• log into Google Analytics and select your website
• click on ADMIN in the top menu bar
• in the central PROPERTY column, click js TRACKING INFO then REFERRAL EXCLUSION LIST
• click the red ADD REFERRAL EXCLUSION button
• enter the referrer URL in the box and click the blue CREATE button

You will need to add each referrer individually.  Using this method won’t block Semalt but will stop them showing as a referrer in Google Analytics.

4. WordPress Plugin

Thanks RefuGeeks for bringing a new WordPress Plugin to my attention.  I’ve not tried the Semalt Blocker plugin out (yet), but it’s definately on the list of things to do!  Download free – search the plugins directory for Semalt Blocker or download direct from WordPress or Peadig.

“Unblockable” Google Analytics Referrer Spam

Since posting this blog in August 2014, the whole referral spam problem has ballooned.  There are several referrers showing in Google Analytics that never actually visit your website.  Examples are:

• Get-Free-Traffic-Now.com
• buy-cheap-online.info
• youporn-forum.ga

These bots spam Google Analytics directly, so blocking via .htaccess is ineffective.  Setting up and maintaining analytics filters for each referrer can be time consuming.

Analytics Edge have posted instructions on how to create a valid hosts filter in Google Analytics, which will stop all “ghost” referrals.  I’ve found it to be very effective, but do read the instructions carefully!

Conclusion

Semalt are obviously not a legit company.  Regardless of their tactics to get people to sign up for their service, a legit business would honour requests not to crawl.

The mess in analytics is not helpful particularly for small business owners who are pushed for time, resources and knowledge to make sense of them.

Other than making a mess in analytics, the Semalt crawlers don’t appear to be malicious.  Of course, the software they push people to download may well be, so I advise staying well clear.

Regardless of who is sending the referral spam, avoid clicking through to the website.  Much of the spam has the sole purpose of getting you to visit the website.  Unfortunately, visiting the spammer is also an open invitation to viruses and other malicious downloads.  A quick search for the referrer name and spam will show you if the referrer is a spam site or not.

Over to you…

Have you come across the Semalt crawler.  Did they honour your requests not to crawl?

Don’t forget to share this post if you found it useful!  Click the buttons below.