Package perl-WWW-RobotRules: Information
Source package: perl-WWW-RobotRules
Version: 6.01-alt1
Build time: Mar 21, 2011, 09:37 PM
Category: Development/Perl
Report package bugHome page: http://search.cpan.org/dist/WWW-RobotRules/
License: GPL or Artistic
Summary: Database of robots.txt-derived permissions
Description:
This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.
Maintainer: Alexey Tourbin