Comments on: How to defend against Yahoo! Slurp

By: Joyce

Joyce — Mon, 04 Jan 2010 11:35:26 +0000

I once had problem with a russian search engine (Yandex.ru). It is the biggest Russian Search engine and they where crawling my site at a rate of 5-10 pages per minute. What disturbed me more was that they were not following my robots.txt, so adding them to robots.txt was not an option. I finally had to add them to my iptables to block the crawler.

By: Niyaz PK

Niyaz PK — Sun, 08 Nov 2009 13:10:08 +0000

Of course all these depend on how much traffic the search engine brings you.

I think the search engines should consider this fact before crawling my server to death.

By: TheAnand

TheAnand — Mon, 12 Oct 2009 13:49:30 +0000

ah yahoo! it sends me as much as 80% traffic on one site and 0% on the other…guess it depends on the type of site audience you have.

In my case, women, kids based site gains a lot of traffic from yahoo for some reason.