2 Kicks

Blocking not Robots.txt Compliant Crawlers(blog.sb2.fr)

submitted by ethilikethilik(1855) 3 years, 2 months ago

Today an article talking about blocking Crawlers which are not Robots.txt Compliant. Number of web sites grows exponentially day by day and crawlers too ! But there is a lot of Web Crawlers that don't follow Robots.txt Disallow Rules or totally ignore it ..

add a comment |category: |Views: 5

tags: another

new Add a live kick counter to your blog >> liveImage

You can even customize the image by choosing your own colors, and then clicking the button below to update the preview and the html code:

  • "Kick It" text
  • "Kick It" background
  • kick count text
  • kick count background
  • border

Simply copy and paste this HTML into your blog post.


Users who kicked this story:
leedumondleedumond(2139) - ethilikethilik(1855)
Comments:

No comments so far

information Login or create an account to comment on this story
Submit a Story
View upcoming stories
How else can I help?
Add our feeds to your site

Most Active Users

What's this?