|robot exclusion standard -->|
standard for robot exclusion
<world-wide web> A proposal to try to prevent the havoc wreaked by many of the early World-Wide Web robots when they retrieved documents too rapidly or retrieved documents that had side effects (such as voting). The proposed standard for robot exclusion offers a solution to these problems in the form of a file called "robots.txt" placed in the document root of the web site.
(01 Nov 2006)
|Bookmark with:||word visualiser||Go and visit our forums|