Robots
Name
Robots -- defines whether to use
robots.txtindexer.conf
Synopsis
Robots {yes | no}
Description
Robots defines whether to use
robots.txt and
<META NAME="robots"> exclusions.
Setting Robots to no can be useful when
running mnoGoSearch for
site validation purposes,
as well as when indexing your own Web site.
The default value is yes, for polite crawling purposes.
Scope
Robots can be used multiple times and
affects on all following Server
and Realm commands
until the end of the configuration file, or until the next
Robots command.