Migrating info.cern.ch to a multiple-host platform

Abstract

The main World-Wide Web server at CERN had become too overloaded to provide an acceptable service. To solve this, Its services were split up onto several machines in the W3 Organization's name domain. Further, the http service was put on two machines, serving files stored with AFS, in the method pioneered by NCSA. Initial performance shows a marked improvement, and future expansion should be much easier.

The Problem

The computer popularly known as info.cern.ch is a very well-known machine.

In the early days of the World-Wide-Web project, info was the original NeXT machine sitting on Tim Berners-Lee's desk. It was the central repository of all WWW code, documentation, and "promotional material" (such as the Virtual Library). It also served as an FTP server for people climbing onto the web, and it offered a service where people could telnet to the machine to use the simple line-mode browser. It hosted an HTTP-to-WAIS Gateway for early web clients which did not support WAIS directly. It hosted several mailing lists for people involved in the project.

There were several results of this. There were many URLs with the hostname info.cern.ch created, all over the web. Early clients had the address of this machine hardcoded in as the location of the WAIS gateway. The line-mode browser had a hardcoded into it a default initial URL pointing to that machine. The standard bootstrapping instructions for people new to the Web were,

"Telnet to info.cern.ch and go from there."

As the web exploded in popularity, this NeXT machine became extremely overloaded. In November 1993, a new Sun, named www0.cern.ch was purchased as a replacement and the name info.cern.ch became an alias for that machine. This machine was still the central repository and server, and it additionally became an NFS server for other local development machines.

By Summer 1994, info.cern.ch was again overloaded. During peak load times, roughly corresponding to normal business hours in America, the peak average request rate would approach 5 Hz, resulting in a data flow of 50 kilobytes per second. The load average on www0.cern.ch would climb to anywhere between 50 and 100, NFS would shut down, and the machine became unusable. Not only did this impede code development at CERN, having an unresponsive info.cern.ch meant that a large central portion of the web would be down.

Initial decision

It was decided that the most cost-effective way of solving this overload problem was to split the services onto more than one machine.

*.w3.org

One easy, and rather common, solution is to separate the services onto differently named machines, i.e., one name for telnet, one for ftp, one for http, etc. These names can be created as aliases of one or more machines, and can be moved around transparently as the load dictates. This solution was adopted. Because of naming restrictions at CERN, and because the responsibility for these core services was at this time moving to the new W3 Organization, the new names were created in the Organization's DNS domain, w3.org.

The names chosen were:

       Service     Name
       -------     ----
       http       www.w3.org
       ftp        ftp.w3.org
       gopher     gopher.w3.org
       WAIS g/w   www.w3.org
       telnet     telnet.w3.org
       mail       mail.w3.org

It was accepted that the name info.cern.ch would have to continue to exist. There was some debate over whether it should be synonymous with www.w3.org, to preserve all of the URLs, or telnet.w3.org, to preserve the bootstrap instructions. It was decided administratively that info.cern.ch should be the same as www.w3.org.

AFS and round-robin DNS

As most of the load on info.cern.ch was http traffic, and as this traffic continues to grow exponentially, it was decided to offer the http service from multiple platforms. A method for doing this was pioneered at NCSA; it was decided to implement the same solution here. Basically, the solution consists of making the filsystem of information available over AFS to more than one identically-configured server machines, and using multiple DNS address records to point to the set of machines.

An additional benefit of this approach is that when the W3O gets going more fully, we can provide multiple identical servers at members' sites scattered around the world. With the new multihome response-seeking code added to the library, this should take advantage of the AFS cache to provide a rapid delivery information, while keeping coordination tasks simple.

Hardware

The old single-point host, www0.cern.ch, is a Sun SparcServer 10 (sun4m sparc) running SunOS 5.3, with 128 Mb of memory. This machine is (at the time of writing) the host for telnet.w3.org, gopher.w3.org, ftp.w3.org, and mail.w3.org. There are two machines, canonically named www6.cern.ch and www7.cern.ch, providing the www.w3.org (a.k.a info.cern.ch) service. They are both Sun SparcStation 20s (sun4m sparc) running SunOS 5.3, with 128 Mb and 96 Mb of memory, respectively. The machines' disks were reorganized, to provide for 400 Mb of AFS cache each.

Implementation

AFS

AFS is a supported service at CERN. This made the setup of the W3O filespace very simple. In particular, there is a locally written command, sysctl, which takes simplifies most administrative tasks. The commands given below are sysctl commands.

W3O was allocated 2 Gb on each of two CERN AFS servers. The W3O's AFS filespace is globally accessible at /afs/cern.ch/w3.org/. Under this directory one finds the directories ftp, w3, and www. These correspond to the anonymous ftp directory, http://info.cern.ch/, and http://info.cern.ch/hypertext/WWW. The latter split reflects the original partition of the URL space onto www0's two 1Gb disks. There is a stub var directory, and softlinks WWW-keyser -> www/www-name/WWW-keyser, WWW -> www, and W3 -> w3. The first was a result of a slight rationalization of the directory structure; the latter two are necessary because many pathnames refer to the capitalized directory names, but a restriction requires that all AFS volume mount points begin with a lowercase letter.

After analyzing the existing structure of the directories, it was decided to place the WWW, WWW-name, Administration, and History directories on one server; and the ftp, W3/hypertext, and DataSources directories on the other. The commands used to create the system follow:

www6% /afs/usr/local/bin/sysctl -h afs4 -s
Sysctld (Version 1) on afs4.cern.ch
sysctl> help

Primitives:
        vos-addsite       
        vos-create        
        vos-move          
        vos-release      or 
        vos-rename       
        vos-remove        
 
        fs-mkmount       
        fs-rmmount      
        fs-setacl         
        fs-setquota      
 
Project volumes:
        pv-list [{-s | -a}]
 
sysctl> pv-list -s
ACL        Mount point / Pattern                    Server(s)
===        ===== =====   =======                    =========
w3         /afs/.cern.ch/w3.org/[a-z]               afs3/j afs4/j
         - p\.w3\.


sysctl> vos-create afs4 j p.w3.org
sysctl> fs-mkmount /afs/.cern.ch/w3.org p.w3.org
sysctl> fs-setquota p.w3.org 5000

sysctl> vos-create afs4 j p.w3.ftp
sysctl> fs-mkmount /afs/.cern.ch/w3.org/ftp p.w3.ftp
sysctl> fs-setquota p.w3.ftp 500000

sysctl> vos-create afs4 j p.w3.hypertext
sysctl> !mkdir /afs/cern.ch/w3.org/w3
sysctl> fs-mkmount /afs/.cern.ch/w3.org/w3/hypertext p.w3.hypertext
sysctl> fs-setquota p.w3.hypertext 600000

sysctl> vos-create afs4 j p.w3.DataSources
sysctl> fs-mkmount /afs/.cern.ch/w3.org/w3/hypertext/DataSources p.w3.DataSources
sysctl> fs-setquota p.w3.DataSources 200000

sysctl> vos-create afs3 j p.w3.WWW
sysctl> fs-mkmount /afs/.cern.ch/w3.org/www p.w3.WWW
sysctl> fs-setquota p.w3.WWW 850000

sysctl> vos-create afs3 j p.w3.WWW-name
sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/www-name p.w3.WWW-name
sysctl> fs-setquota p.w3.WWW-name 350000

sysctl> vos-create afs3 j p.w3.Administration
sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/Administration p.w3.Administration
sysctl> fs-setquota p.w3.Administration 500000

sysctl> vos-create afs3 j p.w3.History
sysctl> fs-mkmount /afs/.cern.ch/w3.org/www/History p.w3.History
sysctl> fs-setquota p.w3.History 500000

sysctl> vos-create afs3 j p.w3.var.adm
sysctl> !mkdir /afs/cern.ch/w3.org/var
sysctl> fs-mkmount /afs/.cern.ch/w3.org/var/adm p.w3.var.adm
sysctl> fs-setquota p.w3.var.adm 400000

sysctl> exit
www6% 

Put in the ACL stuff here...

Data Transfer

Data was transferred from the disks of www0 to the AFS volumes using tar and rcp. The files were read as root, to overcome permissions; they were created as the new AFS user wwwteam. Unfortunately, chown and chgrp are not supported under AFS, and as a result wwwteam ended up owning everything. However, all team members, as well as wwwteam, are full members of the wwwteam:users group and have full permissions on these files.

The initial data copy took approximately two days (part of the delay was a result of the standard overload on www0.cern.ch). No attempt was made at this time to ensure data consistancy; this was deliberately merely a snapshot.

Service and testing

At this point, http daemons were installed on www6.cern.ch and www7.cern.ch; with identical configurations based on the www0.cern.ch original. Some sub-webs on info.cern.ch are actually NFS-mounted from other machines (notably, dxcern.cern.ch); arrangements had to be made to automount these directories.

As the name info.cern.ch will point to these new machines, but the non-http services will not be supported on them, pseudo-servers were installed on the telnet, ftp, and gopher ports. These "servers" merely consist of the following entries in /etc/inetd.conf:

ftp     stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/ftp.message
telnet  stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/telnet.message
70      stream  tcp     nowait  nobody  /usr/bin/cat    cat /etc/inet/gopher.message
where the message files contain pointers to the new service names.

There are a few web services on info.cern.ch which accept input and modify files (notably WIT and the New Server Registration Form). The CGI scripts implementing these services had to be modified to authenticate themselves; the modifications are described in the CERN AFS User's Guide. Another AFS account, wwwcron, was created for these automated scripts: this account only has the bare AFS permissions needed for the scripts to run.

One concern was whether this AFS-based service could keep up with the request rate: it would be expected if file access over AFS were slower than access to local disks; the question was whether or not the overhead would significantly impact performance. Therefore, a small multithreaded program was created which would read one of www0.cern.ch's own httpd logfiles, and rapidly fire off requests to the new servers simulating actual usage patterns. The number of threads could be chosen; varying this number varied the number of concurrent requests and allowed this variable to be factored out of the results. This program, with a couple dozen threads, could run through the logfile at about five times the actual logged rate. After going over the new servers' logfiles, it was concluded that AFS did not create a bottleneck that would slow the new machines below the response time of the original.

A request was made to the CERN firewall folks to allow http access to the two new machines; to do this, the machines had to be renumbered. They are now 128.141.202.119 and 128.141.202.120.

Final copy and official switchover

Following these tests, a find was done on the entire file hierarchy of www0.cern.ch, to find files modified since the time they were transferred to AFS. A moratorium was placed on modifiying files, these files were moved across, and then the AFS versions were officially announced as being the official masters. Development continued using AFS, greatly relieving the problems created when www0.cern.ch was too loaded to serve NFS requests.

At this time, http requests to info.cern.ch were of course still going to www0.cern.ch. The http server on www0.cern.ch was re-configured to use the AFS versions. For historical reasons, www0.cern.ch did not have the full AFS software installed upon it; rather, it used an NFS/AFS gateway service. This created two problems: the NFS/AFS gateway only provides read-only access, breaking the CGI scripts mentioned above; second, the NFS client software on www0.cern.ch could not stand up to the intense load, and the machine repeatedly crashed. We relented, and reverted to using the local disks.

DNS change

The next step was to change the DNS address records to point to the new machines. Typical procedure is merely to create multiple address records in the DNS configuration files, like this:

Before

$ORIGIN cern.ch.
www0             IN      A       128.141.201.214
info             IN      CNAME   www0.cern.ch.

After

$ORIGIN cern.ch.
www0             IN      A       128.141.201.214
www6             IN      A       128.141.202.119
www7             IN      A       128.141.202.120
info             IN      A       128.141.202.119
                 IN      A       128.141.202.120
$ORIGIN w3.org.
telnet           IN      CNAME   www0.cern.ch.
gopher           IN      CNAME   www0.cern.ch.
ftp              IN      CNAME   www0.cern.ch.
mail             IN      CNAME   www0.cern.ch.
www              IN      A       128.141.202.119
                 IN      A       128.141.202.120

Note: the w3.org domain is actually maintained at MIT.

Unfortunately, things aren't so simple at CERN. There is a large Oracle database listing every computer on site, with all kinds of data: its hardware address, its physical location, the people responsible for it, its name, what networks it is on, what networking protocols it supports, and its network addresses. When this database was created, a constraint was placed on it, requiring a one-to-one mapping between machine names and IP addresses. Operations diagnostic tools rely on these constraints, so they can't merely be lifted. It was estimated by the responsible people that it would take a couple weeks to lift this restriction (which had already been requested by other people, as well, for similar purposes). One possibility we proposed was to alias info.cern.ch to the name www.w3.org; the w3.org domain is maintained by MIT, and does not suffer this limitation. However, after we suggested this possibility, a work-around was found. This work-around will bridge the gap until the modified database system is ready.

The change was made the afternoon of Monday, the 17th of October. Initially the load to the two new machines was light, as the new DNS information had not yet propagated to out to refill the caches. However, by Wednesday virtually all requests were being sent to the new servers.

The initial response has been very good. The load average of all three machines has stayed quite low, and neither machine serving http has crashed. Response time seems to be better, although there are anomalous failures tenatively connected with touching a file not in the AFS cache.

Next steps

There are still a few things that should be attended to:

Conclusions

The services once all hosted on info.cern.ch have been successfully separated onto multiple machines. The method used allows further splits between services, if needed, and establishes a parallel, distributed http service that can be easily extended in the future.

The immediate problems of overload have been alleviated. Also, the main filesystem has been moved to a separate service, making it more independant of our overloaded machines. While we will probably always be overloaded, we believe this change will reduce the resultant problems.