Using the IIS 7 URL Rewrite Module to block crawlers

added by IDelectable
6/11/2010 3:36:08 AM

1 Kicks, 135 Views

Here’s an easy way to block the main web crawlers – Google Bing and Yahoo – from indexing any site across an entire server. This is really useful if you push all your beta builds to a public facing server, but don’t want them indexed yet by the search engines.