Tagged: robots.txt

Due to a crawler that thinks sending hundreds of requests at a time is an acceptable practice, something that effectively results in a denial of service attack on the web server, Yahoo’s crawler has hereby been completely and totally banned...