While working on soon-to-be-released projects there has often been a need to make a staging/testing website publicly accessible for client testing. This is a slippery slope if search engine spiders get in and index your site before the rest of the world is meant to see it (it happens more than you’d like to think) – If it happens to be a website you are building for something that the rest of the world shouldn’t see yet such as a product/service launch, having it leak too early can often make or break you. They have a word that describes this very fear of spiders – it’s called Arachnophobia.
The idea behind Arachnophobia is threefold:
- Make it dead easy to add/remove these things together (nuget)
- Act as a project for more of these features to grow (starting to collect spider user agents for hard block later)
- Enable you to add this functionality at the server level to all sites such as on a staging server ( if you take a look at the source there is a half completed installer in the works for this very purpose )