[Boost-commit] svn:boost r69211 - website/public_html/live

Subject: [Boost-commit] svn:boost r69211 - website/public_html/live
From: grafikrobot_at_[hidden]
Date: 2011-02-23 11:58:55

Author: grafik
Date: 2011-02-23 11:58:53 EST (Wed, 23 Feb 2011)
New Revision: 69211
URL: http://svn.boost.org/trac/boost/changeset/69211

Log:
Add current robots exclusion settings.
Added:
   website/public_html/live/robots.txt   (contents, props changed)

Added: website/public_html/live/robots.txt
==============================================================================
--- (empty file)
+++ website/public_html/live/robots.txt	2011-02-23 11:58:53 EST (Wed, 23 Feb 2011)
@@ -0,0 +1,4 @@
+User-agent: *
+Disallow: /development          # Hide huge zip files
+Disallow: /doc                  # Hide huge zip files
+Crawl-delay: 10