27 2002-10-25 22:34:12 +0000 checklink doesn't support the robots exclusion standard 2004-04-19 20:13:55 +0000 1 1 1 Unclassified LinkChecker checklink unspecified All All RESOLVED FIXED http://www.robotstxt.org/wc/exclusion.html P2 enhancement 4.0 1 ville.skytta ville.skytta oldest_to_newest 48 0 ville.skytta 2002-10-25 22:34:12 +0000 This one could be pretty trivial to implement just by changing W3C::UserAgent to inherit from LWP::RobotUA instead of LWP::UserAgent. What's needed: - Check if there are any side effects for the superclass change. - Provide an option to behave badly, ie. not respect robots.txt. - Make sure that the user understands whenever this causes 403's. 304 1 ville.skytta 2002-12-25 05:33:35 +0000 See also the thread at <http://lists.w3.org/Archives/Public/www-validator/2002Dec/0211.html> for additional discussion and resources. 1723 2 ville.skytta 2004-04-19 16:13:55 +0000 This is now implemented in CVS, and will be in the next version (3.9.3).