Using the IIS 7 URL Rewrite Module to block crawlers

Here’s an easy way to block the main web crawlers – Google Bing and Yahoo – from indexing any site across an entire server. This is really useful if you push all your beta builds to a public facing server, but don’t want them indexed yet by the search engines.