This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 19362 - robots.txt needs fixing!
Summary: robots.txt needs fixing!
Status: RESOLVED FIXED
Alias: None
Product: webplatform.org
Classification: Unclassified
Component: infrastructure (show other bugs)
Version: unspecified
Hardware: PC All
: P2 normal
Target Milestone: ---
Assignee: Doug Schepers
QA Contact: public-webplatform-bugs list
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-08 18:42 UTC by Chris Mills
Modified: 2012-10-08 18:46 UTC (History)
2 users (show)

See Also:

Attachments

Description Chris Mills 2012-10-08 18:42:00 UTC 
http://www1.webplatform.org/robots.txt currently disallows indexing of pages. "Don't you want sites indexing the docs?" one guy said. I'm assuming that we did this originally to stop content leaking on to google, but now we really ought to change this.
Comment 1 Ryan Lane 2012-10-08 18:46:49 UTC 
Fixed. Content is now:

#User-agent: *
#Disallow: /

Which means we're allowing everything. We'll need to watch for abusive bots.