W3C Systems Team

Archive: W3C System Status

Wikis upgrade Monday, 27 January

On Monday, 27 January, starting at 9:00 UTC, the W3C Systems Team will perform upgrades of the platform running the various W3C Wikis.
This process is expected to take less than one hour. While the upgrade is underway W3C wikis might be briefly unresponsive, and during the entire maintenance window it will not be possible to edit pages hosted on those wikis.

2013-11-25T1000Z

A disk failure on one of our servers has caused the Zakim and RRSAgent IRC bots to go off-line. The teleconference bridge itself is not affected. Service was restored around 16:15 UTC.

Bugzilla upgrade Wednesday, 30 October 2013

On Wednesday, 30 October, starting at 20:00 UTC, the W3C Systems Team will upgrade Bugzilla. This process is expected to take less than thirty minutes. While the upgrade is underway, it will not be possible to view or update bugs.

If you have questions or concerns about this maintenance, please write to the Systems Team at sysreq@w3.org.

Bugzilla upgrade Thursday, 5 September 2013

On Thursday, 5 September, starting at 18:30 UTC, the W3C Systems Team will upgrade Bugzilla to the latest version 4.4. This process is expected to take less than an hour. While the upgrade is underway, it will not be possible to view or update bugs.

If you have questions or concerns about this maintenance, please write to the Systems Team at sysreq@w3.org.

irc.w3.org downtime on 29 August 2013, 00:00 UTC

The W3C Systems Team will shut down irc.w3.org at 00:00 UTC, Thursday 29 August, for filesystem repairs. All IRC-related services will be unavailable while these repairs take place. The maintenance is expected to take less than one hour.

Planned downtime and reduced capacity for W3C services, 16 July 2013

On Tuesday, 16 July, the W3C Systems Team will make major system configuration changes to a number of servers hosted at MIT. Many W3C services will be running at reduced capacity during this time. This maintenance will begin at 00:00 UTC, and is expected to run between two and three hours.

These services will experience brief downtime (about 5 minutes) during the service window:

These services may experience intermittent performance degradation throughout the service window:

If you have questions about these service impacts, or notice problems after the maintenance is finished, please contact us by e-mail to sysreq@w3.org.

Planned downtime and reduced capacity for W3C services, 24 June 2013

On Monday, 24 June, the W3C Systems Team will perform maintenance on a storage array at MIT. Many W3C services will be running at reduced capacity during this time. This maintenance will begin at 22:00 UTC, and is expected to run between one and two hours.

These services will experience brief downtime (about 5 minutes) at the beginning of the service window:

For the duration of the maintenance window, mail delivery to @w3.org addresses may be delayed. All Validator services will run at reduced capacity, and may be slower than usual. The sites daml.org and iswc2003.semanticweb.org will be unreachable.

If you have questions about these service impacts, or notice problems after the maintenance is finished, please contact us by e-mail to sysreq@w3.org.

W3C Account authentication issues for accounts created or modified between April 9th and April 24th, 2013

We have been informed of authentication issues on several W3C services affecting accounts that were created or modified between April 9th and April 24th, 2013.

Upon investigation a bug was found in our code that lead to encrypt the password information incorrectly. The issue has been fixed but unfortunately 300+ accounts got impacted.

We are communicating with the affected users who are advised to reset their W3C Account password using the "Recover Password" form at http://www.w3.org/accounts/recover

Database upgrades on 13 June 2013

On Thursday, 13 June 2013, the W3C Systems Team will deploy the new publication system and perform upgrades to the underlying W3C database.

Applications relying on database will be unavailable for up to 15 minutes starting at approximately 12:00 noon UTC.

If you run into problems accessing http://www.w3.org/ after the database upgrade is done or if you have questions regarding this maintenance, please contact sysreq@w3.org.

Various server upgrades on 18 May 2013

The W3C systems team will perform software upgrades on several different servers on 18 May, from 18:00 to 22:00 UTC. A few services will be down briefly for system reboots:

Other services may be briefly unresponsive, but should generally remain available:

If you encounter problems using these services during this maintenance window, simply try again in a few minutes. If problems persist, please let us know by e-mail to sysreq@w3.org.

Various server upgrades on 12 March 2013

The W3C systems team will perform software upgrades on several different servers, starting at 22:00 UTC. Affected services include:

These services may be briefly unresponsive, but should generally remain available. If you encounter problems using them at this time, simply try again in a few minutes. If problems persist, please let us know by e-mail to sysreq@w3.org.

RDF Validator downtime, other service capacity loss on 8 March 2013

The W3C systems team will perform software upgrades on various servers beginning at 21:00 UTC on Friday, 8 March 2013. This maintenance will require brief server downtime, expected to last 15-30 minutes. While servers are down, the RDF Validator will be inaccessible, while other Validator services and mail delivery will operate with reduced capacity. If you have questions about this maintenance, please contact us at sysreq@w3.org.

IPv6 connectivity issues for CVS and IRC, 1 March 2013

W3C systems at MIT have lost their IPv6 connectivity due to a router software failure. Because of this, people on IPv6-capable networks that try to contact our CVS or IRC servers may experience slow connections. You may be able to work around the issue by expressly requesting a connection over IPv4 in your client software. We are working with network administrators at MIT to resolve this issue; they estimate that there will be a fix this weekend.

Bugzilla upgrade on 25 February 2013

On Monday, 25 February 2013, starting around 20:00 UTC, the W3C Systems Team will upgrade the Bugzilla software on both the Public and Member Bugzilla sites. If you have questions about these upgrades, or encounter any problems with Bugzilla, please contact us at sysreq@w3.org.

IRC server upgrade on 17 February 2013

The W3C's IRC server software will be upgraded on Sunday, 17 February 2013, around 15:00 UTC. The server will be down very briefly. If you have difficulties connecting around this time, please simply try again in a few minutes. If problems persist, please mail sysreq@w3.org.

IRC server upgrade on 6 January 2013

The W3C's IRC server software will be upgraded on Sunday, 6 January 2013, around 14:00 UTC. The server will be down very briefly. If you have difficulties connecting around this time, please simply try again in a few minutes. If problems persist, please mail sysreq@w3.org.

RDF Validator, GRDDL down as of 2 December 2012

On Thursday night, W3C's MIT host site suffered an unplanned power outage. The server that provides W3C's RDF Validator and GRDDL services was damaged in this incident and is currently unavailable. We are aware of the issue and working to restore service as quickly as possible. We apologize for the inconvenience to our users.

W3C database upgraded on 27 November 2012

On Tuesday, 27 November 2012, the W3C Systems Team deployed new pages related to account management and performed upgrades to the underlying W3C database.

Bugzilla upgraded on 20 November 2012

On Tuesday, 20 November 2012, the W3C Systems Team upgraded the Bugzilla software for both the Public and Member Bugzilla sites. If you have questions about these upgrades, or encounter any problems with Bugzilla, please contact us at sysreq@w3.org.

CVS server upgraded on 17 November 2012

On Saturday, 17 November 2012, the W3C Systems Team installed hardware and software upgrades on the W3C CVS servers, cvs.w3.org and dev.w3.org. Almost all checkouts should work as before, without any configuration changes required. Below we have documented known issues, and how they can be addressed.

"Bad root /w3ccvs" or "Bad root /sources/public" errors

W3C supports these two CVS root directories:

Previously, it was possible to work with a CVS checkout that used the “other” server name—i.e., using a root directory of dev.w3.org:/w3ccvs or cvs.w3.org:/sources/public. The new server is stricter and does not support these checkouts, aborting with a “Bad root” error.

If you don't have any local changes in your checkout, or you can easily save the changes another way, the easiest way to resolve this problem is to make a new checkout, using one of the root directories listed above, and then move over your changes. To do that with the official cvs command-line tool, run one of these commands:

Replace USERNAME with your W3C CVS account username, and MODULE with the name of the directory you want to check out.

If you have local changes in your checkout that are more substantial, it is possible to reconfigure your checkout in place by editing every CVS/Root file in the checkout to refer to one of the two root directories listed above. There are many ways to do this; if you would like assistance, please contact us with details about what operating system and CVS software you use, as well as what's in your checkout.

"Warning: the RSA host key for 'cvs.w3.org' differs from the key for the IP address '128.30.52.23'"

SSH uses “host keys” to verify the identity of the servers that you use. The host key for cvs.w3.org has not changed, but the IP address has changed to 128.30.52.23, and some people still have an old host key for that address in their databases.

You can safely stop this warning by deleting the host key for the address 128.30.52.23. SSH will still use the host key for cvs.w3.org to verify the server's identity. On Unix systems, run this command to clear the old host key:

ssh-keygen -R 128.30.52.23

If you want to reconfirm the host's identity yourself, the fingerprints for cvs.w3.org's host keys are:

Other issues

If you have any questions about these upgrades, or have a problem with your checkout that isn't discussed here, please contact us at sysreq@w3.org.

Planned database maintenance 25 October 2012, 21:00 UTC

At 21:00 UTC on Thursday, 25 October 2012, the W3C systems team will restart the W3C database for a necessary configuration change. While it's down, attempts to use other W3C services that rely on the database (account management, DBWG/IPP, Bugzilla, and others) may fail. This downtime should be very brief, so if you encounter a database failure around this time, simply wait a few minutes and try again.

If you have questions about this, please contact sysreq@w3.org.

New W3C IRC server up—passwords refused

W3C's new IRC server is up and running normally. We've received a few reports that W3C collaborators have been having trouble connecting to this public server. In each case, this has happened because their client was configured to send a server password. Passwords are not necessary to connect to this public IRC server. The old server software would ignore this extraneous password; the new server software rejects connections that provide it.

Public IRC server passwords are not encrypted in any way and can easily be sniffed by malicious third parties. Anyone who has an IRC client configured to connect to our public IRC server with a password should remove that password to avoid leaking sensitive information.

People with clients configured to send a W3C account password should change their W3C account password to help keep it secure.

As always, anyone with questions about this change should contact us at sysreq@w3.org.

W3C IRC server downtime Sunday, 14 October 2012

On Sunday, 14 October 2012, the W3C will deploy new IRC server software at irc.w3.org. This change should make it easier for W3C collaborators and team to connect to our IRC server. Unfortunately, the switchover requires a little downtime.

We will begin the migration at 17:00 UTC. Downtime should be brief—the new server is already running, so the service interruption should be limited to the time it takes DNS changes to propagate across the Internet. We will advertise the fact that that entry is due to change soon to encourage other DNS servers to update quickly. Our web-based IRC client will also be unavailable during this time.

Collaborators should not need to make any change to connect to the new IRC server. People who have long-lived connections should simply notice their connection drop, and then connect again when the new server is available.

If you have questions about this migration, whether beforehand or after the switch, please contact sysreq@w3.org.

people.w3.org software updates on July 5

Between 17:00 and 23:00 UTC on July 5, 2012, we will upgrade various software on people.w3.org. This upgrade will require a reboot, and may cause additional downtime for specific services running on the machine.

If you have any questions about this maintenance, please contact us at sysreq@w3.org.

Planned service outages on July 5, 2012

From 13:00 to 16:00 UTC on July 5, 2012, we will reboot four servers at MIT in order to improve our systems setup. These reboots will cause various W3C services to be unavailable for brief periods of time. Services that will be affected by this work include:

Expect all of these services to be unavailable once or twice for up to fifteen minutes between 13:00 and 16:00 UTC on July 5.

W3C DNS and mail servers will also be affected by this maintenance, but redundant servers should prevent any total service outage.

If you have any questions about this maintenance, please contact us at sysreq@w3.org.

9-10 June 2012: Planned network downtime

On the weekend of June 9-10, 2012, planned network maintenance at MIT will disrupt most W3C services, including our Web sites, mailing lists, and teleconference system. We currently anticipate three interruptions lasting up to two hours each starting approximately:

This schedule is approximate and subject to change. Network administrators will carry out upgrades as quickly as possible. For more details, please see CSAIL's network outage notice.

We will take advantage of this opportunity to perform minor hardware maintenance while the network is down. There is a small chance some services will still be unavailable after network service has been restored. Those should be restored within the two-hour outage window described above.

If you have questions about specific service impacts of this downtime, please contact us.

27 April 2012

Fri 27 Apr 2012 07:30Z. Our database server has been brought back online.

Fri 27 Apr 2012 02:30Z. Our database server is offline, again impacting many services. Due to the late hour Boston time it may not be repaired for several hours, possibly not until ~14:00Z.

Thu 26 Apr 2012 14:00Z. Our database server has been repaired; all subsystems should be functional again.

Thu 26 Apr 2012 12:00Z. A hardware failure in the machine hosting our database services is impacting access to many of our services and access-controlled parts of our Web site.

3 January 2012

Tue 3 Jan 2012 17:00Z. The Zakim teleconference bridge is back in service after replacing a failed power supply.

Tue 3 Jan 2012 11:30Z. The Zakim teleconference bridge is down; it appears that a power suppy has failed in the control computer system. No estimate at this time to recovery.

25 October 2010

Mon 25 Oct 2010 16:00Z, What appears to be a new, prolific application combined with a known browser bug is causing W3C's website a considerable amount of DTD requests. This is causing our servers to respond very slowly.

13 June 2010

Sun 13 Jun 2010 14:00Z, The France and UK Zakim bridge numbers are not correctly passing through the tones to enter conference codes. This appears to be a problem with the forwarding service we have been using. We are testing an alternative forwarding service now.

26 January 2010

Tue 26 Jan 2010 14:00Z, RRSAgent should be working again. Please report any unusual issues you may notice to sysreq@w3.org. Log access since 2100Z on 25-Jan may need to be manually fixed with ,access

Tue 26 Jan 2010 12:00Z, RRSAgent is not functioning due to a filesystem mount issue

11 September 2008

Thu 11 Sep 2008 ~21:00Z-22:19, database server was down and then had some data files that needed repair.

10 September 2008

Wed 10 Sep 2008 13:09Z, Zakim-bot is back in service after a network cable is repaired.

Wed 10 Sep 2008 12:00Z, Zakim-bot is currently unavailable. The cause is being investigated.

26 March 2008

Mon 31 Mar 2008 14:00Z, There have been no further errors logged for the Zakim telephone lines since 26 Mar 21:30Z. No satisfactory explanation has been found for the 12 minute outage that day. Extra monitoring is being done.

Wed Mar 26 13:30 UTC 2008, Zakim T1 lines are back. We have not identified the cause of the outage. The construction folk claim they are not the cause. Further investigation will be done if an outage recurs.

Wed Mar 26 13:15 UTC 2008, Zakim teleconference bridge lines are out, possibly due to construction outside the Stata Center.

19 January 2008

Sat Jan 19 18:04:03 UTC 2008, Cooling emergency in MIT machine room resolved.

Sat Jan 19 17:16:18 UTC 2008, Cooling emergency in MIT machine room, systems are being shut down.

21 September 2007

Fri Sep 21 15:47:32 UTC 2007, www.w3.org experienced a huge server resource demand.

17 September 2007

Tue Sep 18 07:42:45 UTC 2007, The lists.w3.org hardware migration is complete and list services have been resumed.

Tue Sep 18 04:43:09 UTC 2007, The lists.w3.org hardware migration continues and is expected to end soon. Queued mail will be delivered upon resumption of service, which is expected in the hour.

Mon Sep 17 19:11:20 UTC 2007, Beginning at 0100 UTC, 18 September 2007, we will be migrating lists.w3.org to new hardware. Mailing lists will still be served statically and sent mail will be queued for delivery upon service restoration. The migration is expected to take about 2 hours.

6 September 2007

Thu Sep 6 21:02:21 GMT 2007, www.w3.org was experiencing service delays. We hope to have that addressed at present.

Thu Sep 6 15:26:19 UTC 2007, www.w3.org is currently experiencing intermittent service delays. We are working diligently to resolve all outstanding issues.

Thu Sep 6 21:02:21 UTC 2007, full www.w3.org service restored at present.

29 June 2007

Fri Jun 29 21:43:43 UTC 2007. There seems to be a database issue that may be interfering with access to the Member and Team list archives on lists.w3.org. We are examining the issue now and will work toward a speedy resolution.

Fri Jun 29 22:07:52 UTC 2007. The underlying database issues have been resolved and Member and Team access is fully restored.

25 June 2007

On Monday 25 June 2007 at 13:00 GMT the W3C Systems team will perform a major upgrade of W3C main database server. This intervention is scheduled for 2 hours. Applications that rely on this service (eg: DBWG, WBS, Blogs...) will be unavailable during this outage.

25 June 15:50 GMT. This outage is taking much longer than expected, we will update this page when we have a better estimate on time to resolution.

25 June 16:45 GMT. Unfortunately we suffered a hardware failure during this upgrade process.
We are now moving this service to another server while using the latest data from the old hard drive.
Our sincere apologies for this additional downtime.

25 June around 19:00 GMT. All data backed services should be operational again.

26-28 March 2007

Wed, 28 March 1755Z. Zakim is back in service, though not at full capacity yet.

Wed, 28 March 1700Z. Zakim is rebooting; expected back in service shortly.

Wed, 28 March 1500Z. Repair parts have arrived and are being installed. Optimism is appropriate.

Tue, 27 March 2007 1200Z. During the night of Monday 26 March we had a power supply failure in the Zakim teleconference bridge hardware.

The bridge is currently down and callers hear a busy signal. Repair parts are expected to arrive on Wednesday, 28 March. Further status will be reported when available.

Chairs and those with W3C Member authentication are encouraged to consult Zakim Teleconference Bridge Status for current status and details on an interim alternative for scheduling teleconferences.

We apologize for the inconvenience.

17 November 2006:Friday, 17 November 2006. Previous mail delivery issues with lists.w3.org and the W3C mailhubs, have been reolved.

14-17 November 2006: Tuesday - Friday, 14-17 November 2006, lists.w3.org, and the majority of W3C mailhubs, have experienced significant mail delivery delays due to a combination of factors that is being aggressively examined at this time.
We will apprises all Members and Team staff upon confirmed resolution of all outstanding issues. If you have urgent e-mail pertaining to W3C business, publishing, or activity issues, be apprised of potential delays in response (delay times vary widely).

22 September 2006: Friday, 22 September 2006, MIT, where we host much of W3C's global infrastructure, is experiencing networking issues. Mail, dns and our main website (www.w3.org, less some services) will remain operational in the meantime. MIT networking administrators are aware of the problem and working towards correcting it.

5 September 2006: On Tuesday 5 September 2006, W3C experienced a database service failure that has since been repaired. Please report any service updates that may need our attention if you experienced service failure in any transactions during the brief period of downtime (approximately 1550-1700 UTC).

21 August 2006: On Monday 21 August 2006, W3C again experienced problems with the Zakim teleconferencing bridge. We believe the trouble has been identified and repaired.

21 August 2006: On Monday 21 August 2006, W3C again is experiencing a component failure on the Zakim teleconferencing bridge. Callers to Zakim will not hear the usual greeting and prompt for conference code. It does appear, however, that the code will be accepted if entered without the voice prompt and conferences may proceed. We are investigating the hardware issue.

15 August 2006: On Tuesday 15 August 2006, W3C began experiencing a component failure on the Zakim teleconferencing bridge. The impact is that all currently scheduled teleconferences cannot be conducted until the failure is resolved.

Urgent efforts are being made to get the bridge back online as soon as possible. We will update the status as updates become available. Chairs will receive updates via the system-notices mailing list.

We apologize for the inconvenience.

3 August 2006: On Thursday 3 August 2006, W3C began experiencing mirror overloads, slowing down access to the Web site. The W3C Systems team is in the process of resolving the traffic issues. In the meantime, access to the W3C Web site may include delays and timeouts. We apologize for the inconvenience.

14 February 2006: On Tuesday, 14 February 2006, starting around 5pm EST we will be adding some needed hardware to our existing servers.
There will be an interruption to accessing our CVS repositories cvs.w3.org:/w3ccvs, subsequently jigedit.w3.org, and dev.w3.org:/sources/public and our database. The CVS interruption will likely be more pronounced of the two, perhaps an hour or two in duration. The database interruption will affect numerous services including changing acls, access to authentication required lists, WBS and other data backed services.

30 January 2005: Some scheduled, intentionally non-interruptive work was performed by our CSAIL network managers Sunday, 30 Jan 2005:1700-2000EST, but minor problems were encountered, and some disruptions seem to have occurred. All services have since been restored.

7 August 2004: The server, lists.w3.org, was moved to new hardware.

25 May 2004: The public interface (pserver) to W3C CVS source repository is temporarily closed. The Web interface remains available.

13 March 2004: W3C systems @MIT will be moved during the weekend of 13-15 March 2004. Many services will be effected for Public, Member, and Staff users of the site. Please see the announcement for details.

27 January 2004: W3C's mailing lists may be slow due to a new mass-mailing virus known as W32/Novarg.A, W32/Shimg, or W32/Mydoom.
For more information about this virus see the CERT Incident Note IN-2004-01.

3 September 2003: Some of W3C's mailing list archives may be slow to update due to problems introduced during a recent upgrade of our mail software. As the mail archives are rebuilt, Web Accessibility Initiative (WAI) improvements are being implemented.

26 August 2003: The Archive Approval system is offline subsequent to mail infrastructure changes. Queued mail pending approval will be kept and expiration dates will be extended if necessary to allow adequate response time for all posts.

25 August 2003: W3C email and mailing list services have returned to normal. There are a few remaining cases of 1-2 hour delivery delays.

23 August 2003: All new and backlog email is expected to reach its destinations within 24 hours or so. In some cases, email may be delivered out of chronological order.

22 August 2003: Email delivery to W3C mailing lists remains delayed. Archives are functioning but lists.w3.org is under heavy load and visitors may see occasional connection failures.

21 August 2003: W3C email and mailing lists archives are slowed due to inbound email computer worms. The W3C Systems Team is working to restore service. We apologize for the inconvenience and expect normal delivery to resume soon.

28 December 2002: MIT scheduled power outage

9-10 August 2002: MIT scheduled power outage

31 May - 1 June 2002: MIT scheduled power outage

21-23 December 2001: MIT scheduled power outage

7-8 August 2001: MIT power outage

31 March 2001: MIT scheduled 7 April power outage cancelled


System notices archive (Member-only link)


Webmaster
Last updated: $Date: 2014-01-27 18:26:27 $