4985 2007-08-25 04:10:25 +0000 Link checker dies on links to particular url 2009-12-10 19:30:56 +0000 1 1 1 Unclassified LinkChecker checklink unspecified PC All RESOLVED FIXED http://validator.w3.org/checklink?uri=http%3A%2F%2Ffastcounter.bcentral.com%2Ffc-join&hide_type=all&depth=&check=Check P2 normal 4.5 1 robertgibsonx ville.skytta azerger gonzo1lee ot www-validator-cvs oldest_to_newest 16269 0 robertgibsonx 2007-08-25 04:10:25 +0000 There seems to be something that the link checker does not like about the following url. http://fastcounter.bcentral.com/fc-join Internet explorer times out, but the linkchecker does not (if the site being checked links to it), or gives a 500 error (if trying to check links from it). 16344 1 ville.skytta 2007-09-02 08:49:49 +0000 I can reproduce this locally - the error I get in the Apache error log is: [Sun Sep 02 10:56:13 2007] [warn] [client 127.0.0.1] Timeout waiting for output from CGI script /home/scop/cvs/w3c/LinkChecker/bin/checklink, referer: http://lo calhost/checklink However, I'm not sure what to do about this - it's Apache which gives up waiting for output from checklink, not something that is strictly speaking completely checklink's fault or under its control in my opinion. I've moved output of the initial HTTP headers so that they are written before the first document is fetched, but if the timeout happens and Apache gives up on our CGI, we'll still get missing results and no sane error message (or incomplete results if it kicks in later during the check) and invalid markup. One thing worth looking into would be to decrease link checker's timeout to something smaller - Apache defaults to 300 seconds (but the default in my Fedora setup's httpd.conf is 120 seconds) and the link checker uses 60 seconds by default. That doesn't explain why I see the timeout, but for example on qa-dev.w3.org the link checker reports a better error message which means Apache didn't kill it: http://qa-dev.w3.org/wlc/checklink?uri=http%3A%2F%2Ffastcounter.bcentral.com%2Ffc-join&hide_type=all&depth=&check=Check Error: 500 Can't connect to fastcounter.bcentral.com:80 (connect: timeout) Olivier, could you check what the httpd timeout is set to on validator.w3.org Apaches? Other ideas? 16349 2 ot 2007-09-03 01:22:39 +0000 (In reply to comment #1) > One thing worth looking into would be to decrease link checker's timeout to > something smaller I'm not sure that apache is giving up (on the link checker) before the link checker gives up (on the checked link). As I just checked, apache's timeout value is 120 on our servers, which is way above the link checker's. I'm suspecting that the actor abandoning the game is actually the browser, here, which I think typically has a 30 seconds timeout. therefore, I agree we could lower the link checker's default timeout value to, say, 20 or 30 seconds. 24044 3 ot 2009-03-04 14:04:09 +0000 (In reply to comment #2) > therefore, I agree we could lower the link checker's default timeout value to, > say, 20 or 30 seconds. In my test instance of the link checker, I have the default timeout set to 20 seconds. I just found a strange (not documented AFAICT) behavior of LWP. See this output: Checking link http://jtc1sc36.org/doc/36N1141.pdf HEAD http://jtc1sc36.org/doc/36N1141.pdf fetched in 41.01s Checking link http://jtc1sc36.org/doc/36N1142.pdf HEAD http://jtc1sc36.org/doc/36N1142.pdf fetched in 21.00s Both resources are later reported as timing out. Note that for the first resource, LWP waited twice the configured time before it timed out. This doubling would actually explain why apache (timeout 120) currently times out before the link checker (currenty timeout 60 in the production version) does. Browsers typically timeout at 300 seconds, so not a problem. I suggest we switch the default timeout value to 30s in the upcoming 4.5 release. Users checking very slow servers on the commandline can override that setup. 24052 4 ville.skytta 2009-03-04 17:26:07 +0000 I guess the reason for the first one taking double time is that under the hood, when accessing a host for the first time, link checker (LWP::RobotUA) tries to fetch /robots.txt which times out after 20 sec, then accessing the actual resource takes another 20. Subsequent resources on the same host during the same check no longer result in /robots.txt access. I agree with lowering the default timeout, it's 30 seconds in CVS now. 24129 5 ot 2009-03-10 12:41:57 +0000 *** Bug 6417 has been marked as a duplicate of this bug. *** 30066 6 ville.skytta 2009-12-10 19:30:56 +0000 It's 30 seconds in 4.5.