TheW3Book -- /WWW WORLD-WIDE WEB Documentation on the World Wide Web is normally picked up by browsing with a W3 browser. If you need a printed copy, it is also available in "laTeX" and "Postscript" formats by anonymous FTP from node info.cern.ch. This document introduces the "World Wide Web Book", a paper document derived from the hypertext about the project. The book contains General informaion about the project, people and history; A list of things to be done, including how YOU can put data onto the web; A technical discussion of the design issues in projects such as WWW; Actual details of the implementation of the WWW project Such low-level details as software architecures and coding standards The text of the book has been automatically generated from the hypertext, so it may seem strange in places due to links in the hypertext which are not there in the printed copy. The authors of the material are general members of the W3 team at CERN, except where otherwise noted. WorldWideWeb - Summary The WWW project merges the techniques of information retrieval and hypertext to make an easy but powerful global information system. The project is based on the philosophy that much academic information should be freely available to anyone. It aims to allow information sharing within internationally dispersed teams, and the dissemination of information by support groups. Originally aimed at the High Energy Physics community, it has spread to other areas and attracted much interest in user support, resource discovery and collaborative work areas. Reader view The WWW world consists of documents, and links. Indexes are special documents which, rather than being read, may be searched. The result of such a search is another ("virtual") document containing links to the documents found. A simple protocol (" HTTP ") is used to allow a browser program to request a keyword search by a remote information server. The web contains documents in many formats. Those documents which are hypertext, (real or virtual) contain links to other documents, or places within documents. All documents, whether real, virtual or indexes, look similar to the reader and are contained within the same addressing scheme. To follow a link, a reader clicks with a mouse (or types in a number if he or she has no mouse). To search and index, a reader gives keywords (or other search criteria). These are the only TheW3Book -- /WWW (126/4980) operations necessary to access the entire world of data. Information provider view The WWW browsers can access many existing data systems via existing protocols (FTP, NNTP) or via HTTP and a gateway. In this way, the critical mass of data is quickly exceeded, and the increasing use of the system by readers and information suppliers encourage each other. Making a web is as simple as writing a few SGML files which point to your existing data. Making it public involves running the FTP or HTTP daemon , and making at least one link into your web from another. In fact, any file available by anonymous FTP can be immediately linked into a web. The very small start-up effort is designed to allow small contributions. At the other end of the scale, large information providers may provide an HTTP server with full text or keyword indexing. This may allow access to a large existing database without changing the way that database is managed. Such gateways have already been made into Digital's VMS/Help, Technical Univerity of Graz's "Hyper-G", and Thinking Machine's "W.A.I.S." systems. The WWW model gets over the frustrating incompatibilities of data format between suppliers and reader by allowing negotiation of format between a smart browser and a smart server. This should provide a basis for extension into multimedia, and allow those who share application standards to make full use of them across the web. This summary does not describe the many exciting possibilities opened up by the WWW project, such as efficient document caching. the reduction of redundant out-of-date copies, and the use of knowledge daemons. There is more information in the online project documentation, including some background on hypertext and many technical notes. Try it You can try the simple line mode browser by telnetting to info.cern.ch (no user or password. From UK JANET, use the Gateway ). You can also find out more about WWW in this way. This is the least sophisticated browser -- remember that the window-oriented ones are much smarter! It is much more efficient to install a browser on your own machine. The line mode browser is currently available in source form by anonymous FTP from node info.cern.ch [currently 128.141.201.74] if you take both files /pub/www/src/WWWLibrary_v.vv.tar.Z. /pub/www/src/WWWLineMode_v.vv.tar.Z. (v.vv is the version number - take the latest.) Also available is a hypertext editor for the NeXT (in /pub/www/bin/next), the MidasWWW and ViolaWWW browsers for X11, an alpha-test Mac browser, and and a basic server (/pub/www/src/WWWDaemon_v.vv.tar.Z). Documentation, including the latest list of software available , is readable using www. A plain text version of the installation instructions is included in the tar file! Printable (postscript) documentation and articles are in /pub/www/doc. _________________________________________________________________ Tim BL WWW people This is a list of some of those who have contributed to the WWW project , and whose work is linked into this web. Unless otherwise stated they are at CERN, Phone +41(22)767 plus the extension given below or look them up in the phone book . Address: 1211 Geneva 23, Switzerland. See also: Wizards at SLAC . EELCO VAN ASPEREN Ported the line-mode browser the PC under PC-NFS; developed a curses version. Email: evas@cs.few.eur.nl. CARL BARKER Carl was at CERN for a six month period during his degree course at Brunel University, UK. Carl worked on the server side, on client authentication and multiple format handling. TIM BERNERS-LEE Currently in CN division. Before comming to CERN, Tim worked on, among other things, document production and text processing. He developped his first hypertext system, "Enquire", in 1980 for his own use (although unaware of the existence of the term HyperText). With a background in text processing, real-time software and communications, Tim decided that high energy physics needed a networked hypertext system and CERN was an ideal site for the development of wide-area hypertext ideas. Tim started the WorldWideWeb project at CERN in 1989. He wrote the application on the NeXT along with most of the communications software. Phone: 3755, Email: timbl@info.cern.ch ROBERT CAILLIAU Formerly in programming language design and compiler construction, Robert has been interested in document production since 1975, when he designed and implemented a widely used document markup and formatting system. He ran CERN's Office Computing Systems group from 87 to 89. He is a long-time user of Hypercard, which he used to such diverse ends as writing trip reports, games, bookkeeping software, and budget preparation forms. When he is not doing WWW's public relations, Robert is contributing browser software for the Macintosh platform, and analysing the needs of physics experiments for online data access. Phone: +41 (22) 767 50 05, Email: cailliau@cernnext.cern.ch DAN CONNOLLY An early follower of the project, Dan wrote a private X-Windows editor for his company, and encouraged the use of proper SGML and MIME in the future. He wrote a DTD for HTML and an HTML legalizer for old files. The "SGML cop" himself, Dan has put a lot of work in on the HTML specs. Email: connolly@pixel.convex.com. PETER DOBBERSTEIN TheW3Book -- /WWW (253/4980) While at the DESY lab in Hamburg (DE), Peter did the port of the line-mode browser onto MVS and, indirectly, VM/CMS. These were the most difficult of the ports to date. He also overcame many incidental problems in making a large amount of information in the DESY database available. "ERWISE" TEAM Kim Nyberg, Teemu Rantanen, Kati Suominen and Kari Syd{nmaanlakka ('{' is 'a' with two dots above it.. we must get some character set description into HTML!) (under the supervision of Ari Lemmke) are "Erwise". At Helsinki Technical University, they are writing a Motif-based WWW browser (editor? we can hope...) for their undergraduate final year project. The team can be reached as erwise@cs.hut.fi and Ari as arl@cs.hut.fi. ALAIN FAVRE Alain is an undergraduate working with ECP/PT on a browser for Windows on PCs. Phone: 8265, no email yet. In CERN mostly in the afternoons. DAVID FOSTER With wide experience in networking, and a current conviction information systems and PC/Windows being the way of the future, Dave is having a go at a MS-Windows browser/editor. Dave also has a strong interest in server technology and intelligent information retrieval algorithms. JEAN-FRANCOIS GROFF Provided some useful input in the "design issues". During his stay at CERN as "cooperant", J-F joined the project in September 1991. He wrote the gateway to the VMS Help system , worked on the new modular browser architecure, and helped support and present WWW at all levels. He is now porting the communications code to DECnet in order to set up servers for physics experiments. JF now works for NeXT Europe but continues his interest in the web...( Contact ) TONY JOHNSON Tel: (415) 926 2278, TONYJ@scs.slac.stanford.edu. Designer of MidasWWW . Boston University, collaborating with SLAC, SSC, etc. A SLAC server expert and a WWWizard . PAUL KUNZ Paul took the W3 word across to SLAC, installed the clients and inspired the setting up of servers by the WWWizards . Paul spreads enthusiasm for all sort of good ideas such as OO programming, NeXTs, etc... WILLEM VAN LEEUWEN at NIKHEF, WIllem put up many servers and has provided much useful feedback about the w3 browser code. NICOLA PELLOW With the project from November 1990 to August 1991, and October 1992 to ??. A graduate of Leicester Polytechnic, UK, Nicola wrote the original line mode browser . ( More ) Nicola is now (Oct 92) working on the Mac browser . BERND POLLERMANN Bernd is responsible for the "XFIND" indexes on the CERNVM node, for their operation and, largely, their contents. He is also the editor of the Computer Newsletter (CNL), and has experience in managing large databases of information. Bernd is in the AS group of CN division. He has contributed code for the FIND server which allows hypertext access to this large store of information. Phone: 2407, Office: 513-1-16, Email: bernd@cernvm.cern.ch A R THUR SECRET A student at CERN during August and September 1992, Arthur wrote the first W3-Oracle gateway . Arthur will be back at CERN from 1 march to 31 October 1993. JONTHAN STREETS Online Support group, FNAL. Jonathan put up a VMS server using DCL and later C. He helped debug the Mac browser. PEI WEI Pei is the author of " Viola", a hypertext browser, and the ViolaWWW variant which is a WWW browser. He was at the University of California at Berkeley, Experimental Computing Facility, now full time with O'Reilly and Associates, Sebastopol, CA, USA. Email: wei@xcf.berkeley.edu BEBO WHITE one of the WWWizards at SLAC, Bebo enthusiastically spreads the word. During a short stay at CERN in summer '92, Bebo put up a number of servers for information from the Aleph experiment. JAMES WHITESCARVER New Jersey Institute of Technology. jim@eies2.njit.edu Policy This outlines the policy of the W3 project at CERN. Whilst not legally binding, this attempts to explain my understanding of the CERN rules and the desires of the team at CERN. AIM The basic aim of the project is to promote communication and information availability for the High Energy Physics (HEP) community. The project is based at CERN, whose budget is provided by contributions of taxpayer's money from the European member states. It is in the interests of HEP, CERN, and the project itself that it should interwork with systems and information in many other fields, and so active collaboration with other groups is essential. To produce an information system isolating HEP from the rest of the world would be counter-productive, so the aim can be seen as furthering a global web of information. The WWW team are all enthusiastic that information of all types should be available as widely as possible. TheW3Book -- /WWW (380/4980) COLLABORATION We encourage collaboration by academic or commercial parties. There are always many things to be done, ports to be made to different environments, new browsers to be written, and additional data to be incorporated into the "web". There have already been many contributions in these terms, and also with hardware support from manufacturers. If you may be interested in extending the web or the software, please mail or phone us. CODE DISTRIBUTION Code written at CERN is covered by the CERN copyright. In practice the interpretation of this in the case of the W3 project is that the programs are freely available to academic bodies. To commercial organizations who are not reselling it, but are using it to participate in global information exchange, the charge is generally waived in order to cut administrative costs. Code is of course shared freely with all collaborators. Commercial organizations wishing to sell software based on W3 code should contact CERN. We are in the process of getting agreement to release certain parts of the WWW project code with the General Public License (and GP Library License). Where CERN code is included in otherwise public domain code, that CERN code becomes also public domain. Code not originating at CERN is of course covered by terms set by the copyright holder involved. PROTOCOLS AND DATA FORMATS The definition of protocols such as HTTP and data formats such as HTML are in the public domain and may be freely used by anyone. Tim BL WORLDWIDEWEB DISTRIBUTED CODE See the CERN copyright . This is the README file which you get when you unwrap one of our tar files. These files contain information about hypertext, hypertext systems, and the WorldWideWeb project. If you have taken this with a .tar file, you will have only a subset of the files. THIS FILE IS A VERY ABRIDGED VERSION OF THE INFORMATION AVAILABLE ON THE WEB. IF IN DOUBT, READ THE WEB DIRECTLY. If you have not got any browser installed, do this by telnet to info.cern.ch (no username or password). Archive Directory structure Under /pub/www, besides this README file, you'll find bin, src and doc directories. The main archives are as follows: src/WWWLibrary_v.vv.tar.Z The W3 Library. All source, and Makefiles for selected systems. src/WWWLineMode_v.vv.tar.Z The Line mode browser - all source, and Makefiles for selected systems. Requires the Library . src/WWW/LineModeDefaults_v.vv.tar.Z The on-line doucmnetation for the line mode browser, which you may still want if you get the program as binary. A subset of the full source above. src/WWWNextStepEditor.tar.Z The Hypertext Browser/editor for the NeXT -- source and binary. src/WWWDaemon_v.vv.tar.Z The HTTP daemon, and WWW-WAIS gateway programs. Source. doc/WWWBook.tar.Z A snapshot of our internal documentation - we prefer you to access this on line -- see warnings below. bin/xxx/bbbb Executable binaries of program bbbb for system xxx. Check what's there before you bother compiling. (Note HP is called "snake") Generated Directory structure The tar files are all designed to be unwrapped in the same (this) directory. They create different parts of a common directory tree under that directory. There may be some duplication. They also generate a few files in this directory: README.*, Copyright.*, and some installation instructions (.txt). NeXTStep Browser/Editor The browser for the NeXT is those files contained in the application directory WWW/Next/Implementation/WorldWideWeb.app and is compiled.Whe you install the app, you may want to configure the default page, WorldWideWeb.app/default.html. These must point to some useful information! You should keep it up to date with pointers to info on your site and elsewhere. If you use the CERN home page note there is a link at the bottom to the master copy on our server. See Installation instructions. Line Mode browser Binaries of this for some systems are available in /pub/www/bin/ . . If the binary exists for your system, take that and also the /pub/www/WWWLineModeDefaults.tar.Z. Unwrap the documentation, and put (link) its directory into /usr/local/lib/WWW on your machine. Put the www executable into your path somewhere, and away you go. If no binary exists, procede as follows. Take the source tar file WWWLineMode_v.vv.tar.Z , uncompress and untar it. You will then find the line Mode browser in WWW/LineMode/Implementation/... (See Installation notes ). Do the same thing (in the same directory) to the WWWLibrary_v.cc.tar.Z file to get the common library. You will have an ASCII printable manual in the file WWW/LineMode/Defaults/line-mode-guide.txt which you can print out at this stage. This is a frozen copy of some of the online documentation. Subdirectories to WWW/LineMode contain Makefiles for systems to which we have already ported. If your system is not among them, make a new subdirectory with the system name, and copy the Makefile from an existing one. Change the directory names as needed. PLEASE INFORM US OF THE CHANGES WHEN YOU HAVE DONE THE PORT. This is a condition of your use of this code, and will save others repeating your work, and save you repeating it in future releases. Whe you install the browsers, you should configure the default page. This is /usr/local/lib/WWW/default.html for the line mode browser. This must point to some useful information! You should keep it up to date with pointers to info on your site and elsewhere. If you use the CERN home page note there is a link at the bottom to the master copy on our server. Some basic documentation on the browser is delivered with the home page in the directory WWW/LineMode/Defaults. A separate tar file of that directory (WWWLineModeDefaults.tar.Z) is available if you just want to update that. The rest of the documentation is in hypertext, and so wil be readable most easily with a browser. We suggest that after installing the browser, you browse through the basic documentation so that you are aware of the options and customisation possibilities for example. Viola browser for X11 Viola is an X11 application for reading global hypertext. If a binary is available from your machine, in /pub/www/bin/.../viola*, then take that and also the Viola "apps" tar file which contains the scripts you will need. To generate this from source, you will need both the W3 library and the Viola source files. There is an Imakefile with the viola source directory. You will need to generate the XPA and XPM libraries and the W3 library befere you make viola itself. Documentation In the /pub/www/doc directory are a number articles, preprints and guides on the web. See the online WWW bibliography for a list of these and other articles, books, etc. The archive /pub/www/doc/WWWBook.tar.Z is an extract of the text from the WorldWideWeb (WWW) project documentation. This is a snapshot of a changing hypertext system. The text is provided as example hypertext only, not for general distribution. The accuracy of any information is not guaranteed, and no responsibility will be accepted by the authors for any loss or damage due to inaccuracy or omission. A copy of the documentation is inevitably out of date, and may be inconsistent. There are links to information which is not provided in that tar file. If any of these facts cause a problem, you should access the original master data over the network using www, or mail us. Servers TheW3Book -- /WWW (570/4980) The Daemon tar file contains (in this release) the code for the basic HTTP daemon for serving files, and also for the WWW-WAIS gateway. To compile the WAIS gateway, you will need [a link to] a WAIS distribution at the same level as the WWW directory. General Your comments will of course be most appreciated, on code, or information on the web which is out of date or misleading. If you write your own hypertext and make it available by anonymous ftp or using a server, tell us and we'll put some pointers to it in ours. Thus spreads the web... Tim Berners-Lee WorldWideWeb project CERN, 1211 Geneva 23, Switzerland Tel: +41 22 767 3755; Fax: +41 22 767 7155; email: timbl@info.cern.ch COPYRIGHT CERN 1990-1992 The information (of all forms) in these directories is the intellectua l property of the European Particle Physics Laboratory (known as CERN ). It is freely availble for non-commercial use in collaborating non-m ilitary academic institutes. Commercial organisations wishing to use this code should apply to CERN for conditions. Any modifications, imp rovements or extensions made to this code, or ports to other systems, must be made available under the same terms. No guarantee whatsoever is provided by CERN. No liability whatsoever i s accepted for any loss or damage of any kind resulting from any defec t or inaccuracy in this information or code. See also: distribution conditions of WWW code Tim Berners-Lee CERN 1211 Geneva 23, Switzerland Tel +41(22)767 3755, Fax +41(22)767 7155, Email: tbl@cernvax.cern.ch ___________________________________________ ______________________ Tim BL FREQUENTLY ASKED QUESTIONS ON W3 An FAQ list is really a cop-out from managed information. You should be able to find everything you want to know by browsing from the WWW project page, as everything should be arranged in a logical way. Here though are things which maybe didn't fit into the structure, with pointers to the answers which maybe did. Its an experiment, started May 92. The questioners are anonymous. I am just starting: how do I find out more? How does www keep track of the available servers? How does W3 compare with WAIS and Gopher ? How do I create my own server ? Can I get W3 documents if I'm not on the internet ? TheW3Book -- /WWW (634/4980) Tim BL How does www keep track of the available servers? Q How does www keep track of the available servers? How does a user know where to go to get a specific piece of information? According to the description of the http protocol, when a user wants to do a search, the corresponding UDI specifies, among other things, the server's address. How does the user find out about the server's address? Or from the server's perspective, how does a server announce its existence? THE RESOURCE DISCOVERY PROBLEM 14 May 1992 This is what people seem to call this problem in general. As a physical sever can serve many different types of information from different servers, we talk about finding documents and indexes, as that is what the user sees. To the reader, the web is a continuum. When a new server appears, it may serve many databases of data from different sources and on different subjects. The new data must be incorporated into the web. This means putting links to data on the new server (especially to a general overview document for the server if there is one) from existing documents which interested readers might be reading, or putting it into an index which people might search. The person publishing the data must go through the same process as the person searching for it. When (s)he has found an overview page which (s)he feels ought to refer to the new data, (s)he can ask the author of that document (who ought to have signed it with a link to his or her mail address) to put in a link. There may be several links from different documents: there is not one master list. Of course, some servers are put up for internal use only, and links are only made from local documents. I only find out about these servers by word of mouth, but they exist. Currently, there are three parallel trees in the web for finding data starting from scratch. The most interesting one is a classification by subject. I've got an "Other subjects" link from Cern's home page to a master page of information by subject . From that I have links to individual servers of all kinds (W3, WAIS and Gopher), and in cases where there are a lot like physics and biology, a link to a page about one specific subject. In this way you can browse the web by subject like a library. I am looking for people in other disciplines to take over the subtrees for those disciplines as the load gets heavier (I may have candidates for some). The tree tends to be ought of date, and its authors rely on feedback to put in things which are missing. The other trees are by organization and by server type. The list by server type is easy, because the people responsible for each protocol keep a list of the servers using it. That is, there is a tree of gophers, and there is an index of WAIS indexes. There is the W3/WAIS/Archie server for FTP sites. This tree isn't so useful unless you know what sort of a server you are looking for, but it tends to be more up-to-date than the subject index. It also has things in which aren't just about subjects. The third tree was going to be a geographic tree of organizations, but that isn't at all up-to-date. By the way, it would be easy in principle for a third party to run over these trees and make indexes of what they find. Its just that noone has done it as far as I know because there isn't yet an indexer which runs over the web directly. As you can see, the web is sufficiently flexible to allow a number of ways of finding infomation. In the end, I think a typical resource discovery session will involve someone starting on their "home" document, following one to two links to an index, then doing a search, and following several links from what they have found. In some cases, there will be more than one index search involved, such as at first for an organization, and having found that, a search within it for a person or document. We need to keep this flexibility, as the available information in diffferent places has such different characteristics. In the long term, when there is a really large mass of data out there, with deep interconnections, then there is some really exciting work to be done on automatic algorithms to make multi-level searches. Tim BL W3 vs WAIS and Gopher QUESTION What's the difference betwen W3 and WAIS? What's the difference between W3 and Gopher? Why invent yet another system? Which one should I use? THE DATA MODEL W3 is comparable to both WAIS and Gopher , in that it is a client-server information system running over the internet. There is a difference in the data models. The W3 model is that everything (document, menu, index etc) is represented to the user as a hypertext (hypermedia) object. There two navigation operations are available to the user: to follow a link or to send a query to a server. Only certain documents are flagged as having a search facility, and not all documents have links, but some documents have both. That's a pretty simple model, and results in a pretty simple user interface. Two neat things fall out of this model. One is that it turns out that almost all other information systems can be represented in terms of W3 documents. A W3 user can interrogate WAIS indexes (example) and Gopher servers (example). This comes from the flexibility of the W3 model to describe other structures. A WAIS database is a searchable document. The hit-list returned by a WAIS server (or any other query engine) is a hypertext document with links to the documents found. Gopher menus (or any other hierarchical menu system, including a file system) are represented as lists of items linked to other objects. The W3 system has an open addressing scheme allowing links to be made to any objects on W3, WAIS, Gopher, FTP, NFS, or Network News servers. This flexibility has allowed lots of different kinds of data to be put on-line by writing a simple script to generate a hypertext "view" of the database. The hypertext model, then, is flexible. It is also powerful as a communications medium. To author a document in hypertext is to communicate better. It allows one to put in a link whenever the reader might need background information . WAIS WITHOUT LINKS You miss the links in WAIS in two ways. One is when you are looking for an index. You can't follow links from an overview page to "browse" through different indexes. You can only use a master index (the directory of sources) to find indexes. The other way is that when you have retrieved something, whether part of the FORTRAN manual or part of a mail discussion, you get it in isolation. You can't follow links from that document to related documents. GOPHER WITHOUT TEXT A Gopher menu is a dry list of items. Each line has 80 characters in which to describe an option. In practice, to communicate with the reader, one needs the full power of text formatting in a number of styles. A plain list turns out to be relatively infrequently used when the author or the program generating the document has a choice. Note that the "Panda" project adds some plain text to Gopher menus, but this is only a small step toward the flexible blending of links and text which is hypertext. GROUP WORK The second big difference is that W3 is designed to include collaborative authoring (CSCW) so that groups can share information, rather than simply individuals disseminate it. We only have a first stab at this on the NeXT platform, as we were overtaken by the web's success in dissemination mode. DEPLOYMENT LEVELS The W3 software is not (May 92) as deeply deployed as WAIS and Gopher software. This is basically because it takes more time to write a hypertext client than a menu or query client. (Also, because the initial W3 instigators are paid to work for the world of High-Energy Physics primarily). However, the W3 world is growing very fast. There is widespread recognition that hypertext is essential for the next generation. It is planned to merge the W3 and Gopher systems, and there is no reason (apart from server simplicity and, perhaps, response time) why both of these systems could not use the WAIS protocol when it settles down. THE CHOICE Bear in mind: A W3 client can read data from any other system. If you run a W3 server you can upgrade certain parts of the documentation to hypertext later. Hypertext is neat for representing existing data easily. So install W3 clients, and W3 servers. If you want to install a Gopher or WAIS server, fine: the W3 clients will access it. If you install a WAIS server, then you could install the W3-WAIS gateway locally to save bandwidth. _________________________________________________________________ How to create a W3 server (Warning: page under construction) QUESTIONS What's the difference between a server and a gateway ? Which one do I need ? What's the easiest way to create a server ? How do I customize the distributed software to my needs ? How can I write an index server ? BASICS Since this FAQ came up, a lot more documentation about starting servers has come onto the web, so it might pay you too browse a bit more. There are so many ways, that it might seem complicated, but in fact most of the methods are very easy. The server that we distribute as WWWDaemon is in two parts: a common HTDaemon program taking care of the communications, and a variable HTRetrieve function called by HTDaemon with the significant part of the address (i.e. after host) split as argument and keywords (look at the code). The default HTRetrieve function, located in HTRetrieve.c, only knows about finding files and sending them. To make an index server, you need an HTRetrieve that knows how to query a database, or otherwise find info, from keywords supplied after the '?' in the address. It can be as simple as a 'grep' in a series of files. We provide 3 example index servers built this way: VMSHelpGate, which gives access to VMS help, FindGate which gives access to a mainframe search engine called XFind, and WAISGate, which speaks the WAIS protocol to contact their search engines. You can read about all this in the Web... Oh yes, the ISINDEX tag should be returned by the server to notify the client that it accepts searches. Page under construction. Tim BL No internet connection You are not on the internet? You know you aren't because none of the useual commands are available to you like telnet and ftp. Don't worry: all is not lost. There are two possibilities. IF YOU ARE ON DECNET If you are on a DECnet, then you can use DECnet versions of the W3 software. You can get infromation from outside the DECnet you are on so long as somebody somewheer runs a gateway into the internet ona a machine which is connected to both. BY MAIL If you have electronic mail, then it possible (though slow!) to get W3 information by mail. try sending a mail to listserv@info.cern.ch with a line in it saying just HELP to get back instructions . Your mail system must have a mail gataeway onto internet mail, but hat is quite likely. You might have to take the internet address above and ask your friendly system manager how to convert it into the equivalent mail address on the system you are using. See more about the robot . Tim BL Getting Started Q So where can I find information about W3? GETTING STARTED USING TELNET All the information about W3 is on ther web. So you do you get started? A number of ways. You can browse through all that information by just telnetting to one of the addreses below, or you can pick up information using anonymous FTP. If you want to use telnet, try some of the following. (Log in as www if asked for a user name) telnet info.cern.ch (or telnet 128.141.201.74) The simplest line mode browser. This server is in Geneva, Switzerland. telnet eies2.njit.edu (or telnet 128.235.1.43) Log in as www. A full-screen browser in New Jersey Institute of Technology. USA. telnet vms.huji.ac.il (or telnet 128.139.4.3). A dual-language Hebrew/English database, with links to the rest of the world. The line mode browser, plus extra features. Log in as www. Hebrew Uiversity of Jerusalem, Israel. telnet info.funet.fi (or telnet 128.214.6.100) Not available when tested in O29-Oct-92. (FINLAND) Cornel Law school (address?) USING FTP Alternatively, you can pick up some information in plain text or postscript form from the anonymous FTP archive on info.cern.ch. Just FTP to info.cern.ch (or 128.141.201.74) and log in as "anonymous" with for password your mail address user@host. Change directory (cd) to pub/www/doc, and see what there is (ls command). See also: the W3 bibliography , about W3 distributed code , using FTP . Tim BL History to date A few steps to date in the WWW project history are as follows: March 1989 First project proposal written and circulated for comment (TBL) . Paper "HyperText and CERN" (in ASCII or WriteNow format) produced as background. October 1990 Project proposal reformulated with encouragement from CN and ECP divisional management. RC is co-author. November 1990 Initial WorldWideWeb prototype developed on the NeXT (TBL) . November 1990 Nicola Pellow joins and starts work on the line-mode browser . Bernd Pollermann helps get interface to CERNVM "FIND" index running. TBL gives a colloquium on hypertext in general. Christmas 1990 Line mode and NeXTStep browsers demonstrable. Acces is possible to hypertext files, CERNVM "FIND", and internet news articles. Febraury 1991 workplan for the purposes of ECP division. 26 February 1991 Presentation of the project to the ECP group. March 1991 Line mode browser (www) released to limited audience on priam vax, rs6000, sun4. May 1991 Workplan produced for CN/AS group 17 May 1991 Presentation to C5 committee. General release of www on central CERN machines. 12 June 1991 CERN Computer Seminar on WWW. August 1991 Files available on the net, posted on alt.hypertext (6, 16, 19th Aug), comp.sys.next (20th), comp.text.sgml and comp.mail.multi-media (22nd). Jean-Francois Groff joins the project. October 1991 VMS/HELP and WAIS gateways installed. Mailing lists www-interest (now www-announce) and www-talk@info.cern.ch started. One year status report. Anonymous telnet service started. December 1991 Presented poster and demonstration at HT91 . W3 browser installed on VM/CMS. CERN computer newsletter announces W3 to the HEP world. 15 January 1992 Line mode browser release 1.1 available by anonymous FTP. See news . Presentation to AIHEP'92 at La Londe. 12 February 1992 Line mode v 1.2 annouced on alt.hypertext, comp.infosystems, comp.mail.multi-media, cern.sting, comp.archives.admin, and mailing lists. May 1992 Presentation and demo at JENC3 (Innsbruck). Carl Barker joins the project. June 1992 Presentation and demo at HEPVM (Lyon). July 1992 Distribution of WWW through CernLib, including Viola. WWW library code ported to DECnet. Report to the Advisory Board on Computing. August 1992 introduction of CVS for code management. Jan 1993 Midas and Viola browsers available for X; Mac browser and XMosaic released as alpha. Around 50 known HTTP servers. Not up to date.... check the W3 news. HOW CAN I HELP? There are lots of ways you can help if you are interested in seeing the web grow and be even more useful... Put up some data There are many ways of doing this. The web needs both raw data -- fresh hypertext or old plain text files, or smart servers giving views of existing databases. See more details , etiquette . Suggest someone else does Maybe you know a system which it would be neat to have on the web. How about suggesting to the person involved that they put up a W3 server? Manage a subject area If you know something of what's going on in a particular field, organization or country, would you like to keep up-to-date an overview of online data? Write some software We have a big list of things to be done. Help yourself -- all contributions gatefully received! see the list . Send us suggestions We love to get mail... www-bug@info.cern.ch Tell your friends Install/get installed the client software on your site. Quote things by their W3 address to allow w3 users to pick them straight up. _________________________________________________________________ Tim BL Information Provider There are many ways of making your new or existing data available on t he "web" . The best method depends on what sort of data you have. (If you have any questions, mail the www team at www-bug@info.cern.ch.). S ee also: Web etiquette . How can I help ? YOU HAVE PLAIN TEXT FILES If you have some plain text files then you can easily write, or genera te using a script, a small hypertext file which points to them. To m ake them accessible you can use either anonymous FTP , or the HTTP dae mon . YOU HAVE A NEXT You can use our prototype hypertext editor to create a web of hypertex t, linking it to existing files. This is not YET available for X11 wor kstations. This is a fast way of making online documentation, as well as performing the hyper-librarian job of making sure all your informat ion can be found. USING A SHELL SCRIPT An HTTP daemon is such a simple thing that a simple shell script will often suffice. This is great for bits of information available locally through other programs, which you would like to publish. More deatils on writing servers using shell scripts under unix , or in DCL under V MS . YOU HAVE MANY FILES In this case, for speed of access, the HTTP daemon will probably be be st. You can write a tree of hypertext in HTML linking the text files, or you can even generate the tree automatically from your directory tr ee. If you want to generate a full-text index, then you could use the public domain WAIS software - your data will then be accessible (as p lain text, not hypertext) through the WAIS gateway . YOU HAVE AN EXISTING INFORMATION BASE If you have a maintained base of information, don't rush into changing the way you manage it. A "gateway" W3 server can run on top of your existing system, making the information in it available to the world. This is how it works: Menus map onto sets of hypertext links Different search options map onto different "index" document addresses (even if they use the same index underneath in your system). Procedures used by those who contribute and manage information stay unaltered. If your database is WAIS, VMS/HELP, XFIND, or Hyper-G, a gateway exis ts already. These gateway servers did not take long to write. You can pick up a skeleton server in C from our distribution . You can also wr ite one from scratch, for example in perl or sh . An advantage of a g ateway is that you can maintain your existing procedures for creating text and managing the database. For more information, see: W3 Server software A case study of one system . Making a W3 server for existing data . Allowing multiple selections How to set up a telnet server Tim BL Etiquette There are a few conventions which will make for a more useable, less confusing, web. SIGN IT! An important aspect of information which helps keep it up to date is that one can trace its author. Doing this with hypertext is easy -- all you have to do is put a link to a page about the author (or simply to the author's phone book entry). Make a page for yourself with your mail address and phone number. At the bottom of files for which you are responsible, put a small note -- say just your initials -- and link it to that page. The address style (right justified) is useful for this. Your author page is also a convenient place to put and disclaimers, copyright noitices, etc which law or convention require. It saves cluttering up the mesages themselves with a long signature. If you are using the NeXT hypertext editor, then you can put this link from your default blank page so that it turns up on the bottom of each new document. GIVE THE STATUS OF THE INFORMATION Some information is definitive, some is hastily put together and incomplete. Both are useful to readers, so do not be shy to put information up which is incomplete or out of date -- it may be the best there is. However, do remember to state what the status is. When was it last updated? Is it complete? What is its scope? For a phone book for example, what set of people are in it? REFER BACK You may create some data as part of an information tree, but others may may make links to it from other places. Don't make assumptions about what people have just read. Make links from your data back to more general data, so that if people have jumped in there, and at first they don't undertand what it's all about, they can pick up some background information to get the context. A ROOT PAGE FOR OUTSIDERS You don't have to have any particular structure to the data you publish: you can let it evolve as you think best. However, it is neat to have a document on each host which others can use to get a quick idea (with pointers) of what information is available there. I suggest you put a "map" line into your daemon rule file to map the document name "/" onto such a document. As well as a summary of what is available at your host, pointers to related hosts are a good idea. AN ALIAS FOR YOUR SERVER If you have a serious server then it may last longer than the machine on which it runs. Ask your internet domain name manager to make an alias for it so that you can refer to it, instead of as "mysun12.dom.edu" as "info.dom.edu" for example, or "www.dom.edu". This will mean that when you change machines, you move the alias, and people's links to your data will still work. AN ALIAS FOR YOURSELF You should make a mail alias "webmaster" on the server machine so that people who have problems with your server can mail you about it easily. This is similar to the "postmaster" alias for people who have mail problems with your machine. Tim BL Using Anonymous FTP Anonymous FTP access means using the FTP protocol with username "anon ymous" and a password which is the reader's mail address. It is a conv entional way of publishing material on the internet. It allows anyone on the internet (or gatewayed network) to access a S UBSET of your files. On a unix system, this involves setting up a user "ftp" in whse home directory the public files will be. An external us er sees a file system whose root is that home directory. Look at the m an page for ftpd on your system. Normally, the public files are kept in /pub or a subdirectory of that. Any file with which is accesible using anonymous ftp is accesible as a W3 document with address file://node.domain/path/path/path for (imaginary) example file://info.cern.ch/pub/README.txt Tim BL Things to be done There are many of these ...if you have a moment, take your pick! Some of these would make good projects, or would be a way your organization could contribute to the building of the web. There are also special lists of little things to do for each existing project such as the line mode browser , the NeXT browser , the common library , and the server . CLIENT SIDE More clients Clients exist for many platforms, but not all. Editors only exist on the NeXT, but will be really useful for sourcing info and group work. (Group editor?) Search engines Now the web of data and indexes exists, some really smart intelligent algorithms ("knowbots?") could run on it. Recursive index and link tracing, Just think... Text from hypertext We need a quick way to print a book from the web. (A simple method using TeX exists) TheW3Book -- /WWW (1265/4980) SERVER SIDE Server upgrade Easier to install, port. Export directories as hypertext. Run shell scripts embedded in the directory for virtual documents and searches. More Servers See the list of things we have thought of or been pointed to. WAIS integration WAIS protocol extensions to allow hypertext; HTML data type, docids to be conforming UDIs. Integrate WAIS in client too. Integrate client and server A client which generates HTML becomes a general purpose gateway. Especially useful for sites where general access to news, external internet, etc, is limited. OTHER SOFTWARE Gateways JANET and DECnet for example. Real need. HTTP enhancements Format conversion, authorization, better logging information for statistics. Mail manager A hypertext view of a mail archive with lists of messages by author, topic, with links between messages. A friendly face on a mail archive would be a great project management tool. HTTP enhancements Format conversion, authorization, better logging information for statistics. Form processing If you can edit hypertext, you edit a hypertext form and return it. To be able to submit a form back to the server would allow special search patterns, administrative processing, electronic voting, ... Graphic overview Display the web from any document in a graphical form. Phone-line protocol There is a need for a point-point low bandwidth protocol designed for beating the hell out of a phone line. One that will keep the phone line occpied in a very inteligent way with look-ahaed fetches of related documents and lists or parts of them so that a home user with a big disk can explore with optimised ease when he is paying by the minute. DOCUMENTATION Tutorials Hypertext leading a new or prospective user through W3. Canned demos A hypertext path through some good represenattive places to visit. Maybe a script to make a timed presentation (with sound?). Videos Would save us giving so many talks! Policy documents Statements on NIR: collect from institutes, provide background for organizations making NIR policy decisions. _________________________________________________________________ Tim BL TECHNICAL DETAILS (See also: W3 project ) How to provide data How can I make my own data available on the web? Protocols A description of the network protocols used. HTML format A description of the markup language used for some documents and for search hit-lists. Addressing The syntax of W3 document addresses. Design Issues Discussions of decisions to be made when designing or selecting a hypertext/IR system. See also related products . Working notes Work in progress at the drawing board. Data sources Sources of data on the web itself, including W3 servers . News Some internet/usenet newsgroups of possible interest to the WorldWideWeb project. Notation Explanation of notation used in formal descriptions. Implementation guidelines Rules to be followed by implementers. Coding standards A basic style guide for W3 code contributors. If you write code, read this! DESIGN ISSUES This lists decisions to be made in the design or selection of a hypermedia information system. It assumes familiarity with the concept of hypertext. A summary of the uses of hypertext systems is followed by a list of features which may or may not be available. Some of the points appear in the Comms ACM July 88 articles on various hypertext systems. Some points were discussed also at ECHT90 . Tentative answers to some design decisions from the CERN perspective are included. Here are the criteria and features to be considered: Intended uses of the system. Availability on which platforms? TheW3Book -- /WWW (1392/4980) Navigational techniques and tools: browsing, indexing, maps, resource discovery, etc Keeping track of previous versions of nodes and their relationships Multiuser access: protection, editing and locking, annotation. Notifying readers of new material available The topology of the web of links The types of links which can express different relationships between nodes These are the three important issues which require agreement betwen systems which can work together Naming and Addressing of documents Protocols The format in which node content is stored and transferred Implementation and optimisation - Caching , smart browsers, knowbots etc., format conversion , gateways. Intended Uses Here are some of the many areas in which hypertext is used. Each area has its specific requirements in the way of features required. General reference data - encyclopaedia, etc. Completely centralized publishing - online help, documentation, tutorial etc More or less centralized dissemination of news which has a limited life Collaborative authoring Collaborative design of something other than the hypertext itself Personal notebook The CERN requirement has a mixture of many of these uses, except that there is not a requirement for distribution of fixed hypertext on hard media such as optical disk. Evidently, the system will have to be networked, though databases may start life at least as personal notebooks. For looking up data bases, the user should be able to refer to already prepared complex queries by simple UDIs. A moe advanced user should also be able to prepare a complex query himself, store it (interpreted language!) in his local filing space, and use it through a simple UDI. [The (paper) document "HyperText and CERN" describes the problem to be solved at CERN, and the requirements of a system which solves them. Availability on various platforms TheW3Book -- /WWW (1456/4980) The system is to be available (at CERN) on many sorts of machine, but priorities must be decided. A list comprises: A unix or VMS workstation with X-windows An 80 character terminal attached to a unix or VMS machine, or an MSDOS PC An 80 character terminal attached to an IBM mainframe running VM/CMS A Macintosh A unix workstation with NextStep An MS-DOS/Windows PC The order above does not imply a priority. It may be that the implementation on one system will lead more easily to an implementation on one of the others, and this would in practice change the order of porting. The requirement for 80 column terminals to be useable (emphasized by M. Goossens) follows from low budgets of many of our users. The order of implementation of special browsers at CERN is a function Navigational Techniques and Tools TBL There are a number of ways of accessing the data one is looking for. Navigational access (i.e., following links) is the essence of hypertext, but this can be enhanced with a number of facilities to make life more efficient and less confusing. DEFINED STRUCTURE It is sometimes nice for a reader to be able to reference a document structure built specifically to enhance his understanding, by the document author. This is especially important when the structure is part of the information the author wishes to convery. See a separate discussion of this point . GRAPHIC OVERVIEW A Graphic overview is useful and could be built automatically. Should it be made by the author, server, browser or an independent daemon? Can one provide an overview with less granularity than the basic web by grouping nodes in some way? The user could select from link types used to imply the tree structure. (JFG) I think this depends on how long it will take. It might be interesting to experiment with daemons which will independently make and update maps of the web. This is not essential for a first pilot model. HISTORY MECHANISM This allows users to retrace their steps. Typical functions provided can be interpreted in a hypertext web as follows: TheW3Book -- /WWW (1520/4980) Home Go to initial node Back Go to the node visited before this one in chronological order. Modify the history to remove the current node. Next When the current node is one of several nodes linked to the ªbackº node, go to the next of those nodes. Leave the ªBackº node unchanged. Modify the history to remove the current node and replace it with the "next" (new current) node. Previous When the current node is one of several nodes linked to the ªbackº node, go to the preceding one of those nodes. In many hypertext systems, a tree structure is forcibly imposed on the data, and these functions are interpreted only with respect to the links in the tree. However, the reader as he browses defines a tree, and it may be more relevant to him to use that tree as a basis for these functions. I would therefore suggest that an explicit tree structure not be enforced. (If a default tree is needed by the system for some reason, then we can always use the creation order: when a node is created it is always created with a link to an existing node. Such links, whatever their type, may be used to define a tree. If they are deleted, an alternative link must be chosen to become a tree link.) If authors want to write a tree structure into their documents, then the words "after", "before" and "above" could be used to mean a static structure. INDEX An Index helps new readers of a large database quickly find an obscure node. Keyword schemes I include in the general topic of indexes. The index must, like a graphic overview, be built either by the author, or automatically by one of the server, browser, or a daemon. The index entries may be taken from the titles, a keyword list, or the node content or a combination of these. Note that keywords, if they are specifically created rather than random words, map onto hypertext ªconceptº nodes, or nodes of special type ªkeywordº. It is interesting to establish an identity relationship between keywords in two different databases -- this may lead a searcher from one database into another. Index schemes are important but indexes or keywords should look like normal hypertext nodes. The particular special operation one can do with a good keyword index system which one can't do with a normal hypertext system is to do a fast search on multiple keywords. This must to be provided as an extension to the hypertext navigation scheme. However, it is in fact analogous to a trace starting with more than one node, which is a valid hypertext tracing operation. The difference is that the tracing would normally be done by a browser, but the indexed search done by the server. When many nodes in a web represent different indexes, then a query search can chain between them (See " Web of indexes "). TheW3Book -- /WWW (1584/4980) See also: HyperText and Information Retrieval NODE NAMES These allow faster access if one knows the name. They allow people to give references to hypertext nodes in other documents, over the telephone, etc. This is very useful. However, in Notecards, where the naming of nodes was enforced, it was found that thinking up names for nodes was a bore for users. KMS thought that being able to jump to a named node was important. The node name allows a command line interface to be used to add new nodes. I think that naming a node should be optional: perhaps by default the system could provide a number which can be used instead of a name.The system should certainly support the naming of nodes, and access by name. MENU OF LINKS Regular linkwise navigation may be done with ªhotspotsº (highlighted anchors) or may be done with a menu. It may be useful to have a menu of all the links from a given node as an alternative way of navigating. Enquire, for example, offers a menu of references as the only way WEB OF INDEXES In WWW , an index is a document like any other. An index may be built to cover a certain domain of information. For example, at CERN there is a CERN computer center document index . There is a separate functional telephone book index . Indexes may be built by the original information provider, or by a third party as a value-added service. Indexes may point to other indexes. An index search on one index may turn up another index in the result hit list. In this case, the following algorithm seems appropriate. Index context Most index searches nowadays, though some look like intelligent semantically aware searches, are basically associative keyword searches. That is, a document matches a search if there is a large correlation (with or without boolean operations) between the set of words it or its abstract contains and the set of words specified in the search. Let us consider extending these searches to linked indexes. Each index has a certain context. This may be represented by a set of keywords which may be considered to apply implicitly to everything indexed. For example, in the CERN computer center documentation index, one may imagine that everything in it will be considered as pertaining to the CERN computer center. We might represent the context by the keyword list "CERN computer center documentation physics support". Context narrowing Suppose we search a general physics index with the keywords "CERN NEWSLETTER". That index may contain an entry with keyword "CERN" pointing to the CERN index. Therefore, a search on the first index will turn up the CERN index. We should then search the CERN index, but looking only for the keyword "NEWSLETTER". The keyword "CERN" is discarded, as it is assumed by the new context. In this simple model, we can assume that the contextwords could be used directly as the keywords for the index itself. A simple algorithm, then, would be for the server to discard from a search list any keywords matching the index's context -- but is this really what we want to do? Perhaps those keywords have a more refined meaning within the context. For example, if I am looking for documents about document storage schemes at CERN, I might search the index with the keyword "documents". I don't want this to be discarded because it is in the context: I am looking for documents about documents. It is understood that we are already within the context of computer center documentation, so to ask about documentation in this context implies more than that I am looking for a document. A more refined approach would therefore be to strip from the search those keywords which were used in order to find the index. The keyword list for the entry of one index within anotherthen reflects the change in context. Context Broadening We have discussed here only a narrowing of context, not a broadening. One can imagine also a reference to a broader context index. In this case, perhaps one should add to the search some keywords which come from the original context but were not expressed. This would be dangerous, and people would not like it as they often feel that they are expressing their request in absolute terms even when they are not. Also, they may have been trying to escape from too restricing a context. One should also consider a search which traces hypertext links as well as using indexes. See also: Navigational techniques , Hypertext and IR , _______________________________________________________________ __ Tim BL Tracing Links A form of search in a hypertext base involves tracing the links between given nodes. For example, to find a module suitable for connecting a decstation to SCSI, one might try finding paths between a document on decstations and a document on SCSI. This is similar to relevance feedback in index searching. Tracing is made more powerful by using typed links. In that case, one could perform semantic searches for all document written by people who were part of the same organisation as the author of this one, for example. This can use node typing as well. When using link tracing, documents take over from keywords. See Scott Preece's vision. _________________________________________________________________ Tim BL Versioning Definition: The storage and management of previous copies of a piece of information, for security, diagnostics, and interest. Do you want version control? Can you reference a version only? If you refer to a particular place in a node, how does one follow it in a new version, if that place ceases to exist? (ªPeter Aiken is the expert in this areaº - Tim Oren, Apple) Yes, at CERN we will want versioning. Very often one wants to correct a news item, even one of limited life, without reissuing it. This is a problem with VAX/NOTES for example. I would suggest that the text for the current version is stored, and separately those modifications necessary to backtrack to previous versions. I would expect previous versions to be regenerated only on the fly, as needed. (Apparaently SCCS stores the original file and the differences. This system does allow you to ditsribute the differences when updating copies.) If full differences (deltas) are kept, the first version is just the first delta from a null document. The latest version is not available without regenerating it from all the deltas. For speed, it is obviously useful to keep a copy of the latest version (see caching ). Versioning is necessary for accountability (--David Durand, dgd@cs.bu.edu). If an author is to be accountable for information published, it should be possible to demonstrate later what he wrote, even if he has later changed it. A WWW server may provide versioning, by allowing links between a document version and its previou and succesive versions. This would be a good use of link typing . Multiuser considerations Multiuser access is made easier with a client/server model.We obviously want this. We also need simultaneous reading and writing of the same database. This is done by locking parts or all of the database while they are updated. One has to decide on the unit of data to be locked. I ( TBL ) imagine that it would be a node, not a database. There is a specific problem which all distributed hypertext systems have had to tackle, in linking to living documents . ANNOTATION Annotation is the linking of a new commentary node to someone else's existing node. It is the essence of a collaborative hypertext. An annotation does not modify the text necessarily: one can separate protection against writing and annotation. PROTECTION Protection against unauthorized reading and writing is provided by servers. We use the word ªdomainº to describe a set of data which has the same protection. Life is simple if the domain is the database, or all the data administered by a given server. One can also add author-based protection to the contents of a node, or links, which have author information stored about them. There is a problem illustrated by the following example. One might want to make a private annotation to something which is visible world-wide but unwritable. The annotation would be invisible to another reader: it would be stored in a private domain. The node itself is visible everywhere: it is stored in a public domain. This is a general problem of links being in a different domain to nodes. PRIVATE OVERLAID WEB A possible solution to this is to have, in the private domain, a partial copy of the public web, so that link information can be added to it. The copy of the net could also be used to tag on local cached copies of the contents of the remote nodes. The writer would have to be aware of the domain into which he was writing. One could use a server per domain, but could imagine the need for more than one server per domain, or more than one domain per server. See also: Generic Linking LOCKING AND MODIFYING Modification of text in a multiuser environment requires in principle some sort of atomic locking feature, so that two users do not update the same text at the same time. In fact some systems do not have this and still survive quite well: it depends a lot on the human environment. Practically, the HTTP protocol must contain a lock/unlock command, and some way of recovering from a lock left on by a vanished user. The actual implementation will depend on the server or gateway. In the case of files, then a number of possibilitie exist: One can write-protect the file temporarily. This unfortunately levaes no clue as to who has locked it, when and why. It is also indistinguishable from a genuine protection to a document which should not be modified One can create a lock file containing information about who/when/why, whose name is derived from the name of the file in question. ANNOTATION Annotation is the linking of a new commentary node to someone else's existing node. It is the essence of a collaborative hypertext. (See Multiuser considerations ) There is a problem when the web includes more than one authentication domain. See ªprotectionº. Notification of new material Does one need to bring it to a reader's attention when new unread material is added? Asynchronously (e.g. by mail) when the update is made? Synchronously when he browses or starts the application? Under the control of the modifying author? (i.e. can I say whether my change is a notifiable change? - Yes) How do you express interest - in a domain, in a node, in things near a node, in anything you have read already, etc? A separate web which is stored locally, and logically overlay the public web? There are two ways to make the connection between the modified material, and an interested person. One is, at the time of modification, to trace the interested parties. The other is, at some later time, for a daemon program (or a browser) to make a search for new things of interest to a given reader. Topology Here are a few questions about the underlying connectivity of a hypertext web. ARE LINKS TWO- OR MULTI-ENDED? The term "link" normally indeicates with two ends. Variations of this are liks with multiple sources and/or multiple destinations, and constructs which relate more than two anchors. The latter map onto logic description systems, predicate calculus, etc. See the "Aquanet" system from Xerox PARC - paper at HT91). This is a natural step from hypertext whose the links are typed with semantic content. For example, the relation "Document A is a basis for document B given argument C". From now on however, let us restrict ourselves to links in the conventional sense, that is, with two ends. SHOULD THE LINKS BE MONODIRECTIONAL OR BIDIRECTIONAL? If they are bidirectional, a link always exists in the reverse direction. A disadvantage of this being enforced is that it might constrain the author of a hypertext - he might want to constrain the reader. However, an advantage is that often, when a link is made between two nodes, it is made in one direction in the mind of its author, but another reader may be more interested in the reverse link. Put another way, bidirectional linking allows the system to deduce the inverse relationship, that if A includes B, for example, that B is part of A. This effectively adds information for free. This is important when a critical parameter of the system is how long it takes someone to create a link. KMS and hypercard have one-way links; Enquire has two-way links. There is a question of how one can make a two-way link to a protected database. The automatic addition of the reverse link is very useful for enhancing the information content of the database. See also: Private overlaid web , Generic Links . It may be useful to have bidirectional links from the point of view of managing data. For example: if a document is destroyed or moved, one is aware of what dangling links will be created, and can possibly fix them. A compromise that links be one-way in the data model, but that a reverse link is created when any link is made, so long as this can be done without infringing protection. An alternative is for the reverse links to be gathered by a background process operating on a basically monodirectionally linked web. See Building Back-links. SHOULD ANCHORS HAVE MORE THAN ONE LINK? There is a design issue in whether one anchor may lead to many links, and/or on link have many anchors. It seems reasonable for many anchors to lead to the same reference. If one source anchor leads to more than one destination anchor, then there will be ambiguity if the anchor is clicked on with a mouse. This could be resolved by providing a menu to the user, but I feel this would complicate it too much. I therefore suggest a many-to-one mapping. JFG disagrees and would like to see a small menu presented to the user if the link was ambiguous. Microcosm does this. SHOULD LINKS BE TYPED? A typed link carries some semantic information, which allows the system to manage data more efficiently on behalf of the user. A default type ("untyped") normally exists in some form when types are implemented. See also a list of some types . (Should a link be allowed to have many types? (- JFG ) I don't think so: that should be represented by more than one link.(- TBL )) Link typing helps with the generation of graphical overviews , and with automatic tracing . SHOULD LINKS CONTAIN ANCILLARY INFORMATION? Does the system allow dating, versioning, authorship, comment text on a link? If so, how is it displayed and accessed? This sort of information complicates the issue, in that readable information is no longer carried within node contents only. Pretty soon, following this path leads to a link becoming a node in itself, annotatable and all. This perverts the data model significantly, and I cannot see that that is a good idea. Information about the link can always be put in the source node, or in an intermediate node, for example an annotation. However, this makes tracing more difficult. It is certainly nice to be able to put a comment on a link. Perhaps one should make a link annotatable. I think not. SHOULD A LINK CONTAIN PREVIEW INFORMATION? This is information stored at the source to allow the reader to check whether he wants to follow a link before he goes. I feel that the system may cache some data (such as the target node title), or the writer of the node may include some descriptive material in the highlighted spot, but it is not necessary to include preview information just because access may be slow. Caching should be done instead of corrupting the user interface. If you have a fast graphic overview , this could Link Types See discussion of whether links should be typed . Descriptive (normal) link types are mainly for the benefit of users and tracing, and graphics representation algorithms. Some link types for example express relationships between the things described by two nodes. A Is part of B / B includes A TheW3Book -- /WWW (1963/4980) A Made B / B is made by A A Uses B / B is used by A A refers to B / B is referred to by A MAGIC LINK TYPES These have a significance known to the system, and may be treated in special ways. Many of these relate whole nodes, rather than particular anchors within them. (See also multiended links and predicate logic) Suggestions: UseIndex The destination is the related index for a search by a user reading this document who asks for an index search function. A document may have any number of index links, causing several indexes top be searched in a client-defined manner. UseGlossary The destination of the link is an index which should be used to resiolve glossary queries in the document. (Typically, a double-clik on a word which is not within an anchor). A document may have any number of glossary links. Annotation The information in the destination node is additional to that in the source node, and may be viewed at the same time. It may be filtered out (as a function of author?). Annotation is used by one person to write the equivalent of "margin notes" or other criticism on another's document, for example. Tracing may ignore annotations when generating trees or sequences. Embedded information If this link is followed, the node at the end of it is embedded into the display of the source node. This is supported by Guide, but not many other systems. It is used, in effect, by those systems (VAX/notes under Decwindows, Microsoft Word) which allow "Outlining" -- expanding a tree bit by bit. The browser has a more difficult job to do if this is supported. person described by node A is author of node B This information can be used for protection, and informing authors of interest, for sending mail to authors, etc. person described by node A is interested in node B This information can be used for informing readers of changes. Node A is in fact a previous version of node B Node A is in fact a set of differences between B and its previous version. This information will probably not be stored as nodes, but be generated from regular diff files. or some other delta method. Document Naming This is probably the most crucial aspect of design and standardization in an open hypertext system. It concerns the syntax of a name by which a document or part of a document (an anchor) is referenced from anywhere else in the world. As many protocols are currently used for information retrieval, the address must be capable of encompassing many protocols, access methods or, indeed, naming schemes. The WWW scheme uses a prefix to give the addressing sub-scheme, and then a syntax dependent on the prefix used, in order to be open to any new naming systems. NAME OR ADDRESS, OR IDENTIFIER? Conventionally, a "name" has tended to mean a logical way of referring to an object in some abstract name space, while the term "address" has been used for something which specifies the physical location. The term "unique identifier" generally referred to a name which was guaranteed to be unique but had little significance as regards the logical name or physical address. A name server was used to convert names or unique identifiers into addresses. With wide-area distributed systems, this distinction blurs. Locally, things which at first look like physical addresses develop more and more levels of translation, so that they cease to give the actual location of the object. At the same time, a logical name or a unique identifier must contain some information which allows the name server to know where to start looking. In a global context, for example "1237159242346244234232342342423468762342368" might well be unique, but it contains insufficient (apparent) structure for a name server to look it up. The name "info.cern.ch" has a structure which allows a search to be made in several stages. In fact, practical systems using unique identifiers generally hide within them some clues for the name server, such as a node name. A hypertext link to a document ought to be specified using the most logical name as opposed to a physical address. This is (almost) the only way of getting over the problem of documents being physically moved. As the naming scheme becomes more abstract, resolving the name becomes less of a simple look-up and more of a search. One expects in practice the translation of a document name taking several stages as the name becomes less abstract and more physical. HINTS Some document reference formats contain "hints" to the reader about the document, such as server availability, copyright status, last known physical address and data formats. It is very important not to confuse these with the document's name, as they have a shorter lifetime than the document. X500 The X500 directory service protocol defines an abstract name space which is hierarchical. It allows objects such as organizations, people, and documents to be arranged in a tree. Whereas the hierarchical structure might make it difficult to decide in which of two locations to put an object (it's not hypertext), this does allow a unique name to be given for anything in the tree. X500 functionally seems to meet the needs of the logical name space in a wide-area hypertext system. Implementations are somewhat rare at the moment of writing, so it cannot be assumed as a general infrastructure. If this direction is chosen for naming, it still leaves open the question of the format of the address into which a document name will be translated. This must also be left as open-ended as the set of protocols. _________________________________________________________________ Tim BL Document formats The question of the format of the contents of a node is independent of the format of all the management information (except for the format of the anchor position within the node content). Therefore, the hypertext system can be largely defined without specifying the node format. However, agreement must be reached between client and server about how they exchange content information. Many hypertext systems qualify as ªhypermediaº systems because they handle media other than plain text. Examples are graphics, video and sound clips, object-oriented graphics definitions, marked-up text, etc. FORMAT NEGOTIATION Most hypermedia systems on the market today have the same application program responsible for the hypertext navigation and for the browsing. It would be safer to separate these features as much as possible: otherwise, in defining a universal hypertext system, one is burdened with defining a universal multimedia browser. This would certainly not stand the test of time. Node content must be left free to evolve. This implies that format conversion facilities must be available to allow simple browsers to access data which is stored in a sophisticated format. Such conversion facilities tend to exist in many applications, though not, in general, in hypertext applications. The format of the content of a node should be as flexible as possible. Having more than one format is not useful from the user's point of view -- only from the point of view of an evolving system. I suggest the following rules: 1. BASIC FORMATS There is a set of formats which every client must be able to handle. These include 80-column text and basic hypertext ( HTML ). 2. CONVERSION A server providing a format which is not in the basic set of formats required for a client must have the possibility of generating some sort of conversion of the text (even if necessary an apology for non-conversion in the case of graphics to text) for a client which cannot handle it. This ensures universal readability world over. 3. NEGOTIATION For every format, there must be a set of other possible formats which the server can convert it into, and the most desirable format is selected by negotiation between the two parties. The negotiation must take into account: the expected translation time, including current load factors the expected data degradation the expected transmission time (?!!) The times one could assume will be roughly proportional to the length of the document, or at least linear in it. Application-specific node formats (e.g. physics event) would allow specialized browsers to perform local processing. This is a natural extension of the hierarchy of node formats. I would suggest one stick to the rule that a server providing such a type of data must provide some default conversion to a standardized view. An index or a keyword could be a specific node format which would be manageable by a browser. EXAMPLES Examples of rich text formats which exist already at CERN are as follows, with, in brackets after each, other formats into which it might be convertible: SGML ( Tex , Postscript, plain text) Bookmaster (Postscript, I3812, plain text) TeX (DVI, plain text) DVI (IBM 3812, Postscript, etc) Microsoft RTF (postscript, plain text, Next ªWriteNowº) - See Specs Postscript, Editable Postscript (IBM 3812 bitmap) plain text When a server (or browser) is obliged to perform a conversion from one format to another, one imagines that the result would be cached so that, if the same conversion were needed later, it would be available more rapidly. Format conversion, like notification of new material, is something which can be triggered either by the writer or by the browser. In many cases, a conversion from, say, SGML into Postscript or plain text would be made immediately on entry of the new material, and kept until the source has been updated (See caching , design issues ). Document caching Three operations in the retrieval of a document may take significant time: Format conversion by the server, including version regeneration Data transmission across the network Format conversion by the browser At each stage, the server (in the first case) or browser (in the other cases) may decide to keep a temporary copy of the result. This copy should ideally be common to many browsers. Automatic caching relieves the user of having to explicitly save things which may be referred to again. It also relieves the system of keeping multiple copies (one for each user who has read the document). It allows local disk space to used optimally. Cache management takes into account such factors as expiry date file size time taken to get the file frequency of access time since access EXPIRY DATE As a guide to help a cache program optimise the data it caches, it is useful if a document is transmitted with an estimate by the server of the lengt of time the data may be kept for. This allows fast changing documents to be flushed from the system, preventing readers from being mislead. (I would not propose any notification of document changes to be distributed to cache managers automatically). For example, an RFC may be cached for years, while the state of the alarm system may be marked as valid for only one minute. Window-oriented browsers effectively cache documents when they keep several at a time in memory, in different windows. In this case, for very volatile data, it may be useful to have the browser automatically refresh the window when its data expires. ( design issues )_________________________________ Scott Preece on retrieval 3 Oct 91 (See tracing, Navigation) My own "vision" of information retrieval models the whole database as a network of objects. Some of the objects are words, some of them are index terms (from a controlled vocabulary), some of them are documents some of them are pieces of documents, some of them are authors, etc. There are also typed links between nodes in the graph -- words are connected to documents by occurrence links, words are tied to words by dictionary links, document pieces are tied to documents by "is section of" links, etc. Searching then becomes a process of turning some of the nodes on, then turning on the nodes attached to them by certain kinds of links, and so forth. So a dictionary expansion of the query works by activating a set of terms and then following all the dictionary links from those terms to other terms; a "search" works by activating a set of terms, then following all the occurrence links to the documents they appear in; relevance feedback works by starting with a set of activated documents and following the links back to the terms that occur in them. If you use appropriate rules for calculating the level of activation of a node you can implement many of the similarity functions that have been reported in the literature and do a pretty effective job of seaching. For instance, suppose you have a term node which is activated with a weight of 1. Suppose the spreading rule is that the weight is split among all the occurrence links leading from it to documents and the combining rule is that all weights coming into a node are summed. Then after one spreading cycle each active document will have a weight equal to the sum of the inverse frequency of the terms in contains, which is a pretty reasonable search strategy. One enhancement is to have each link also weighted -- for term occurrence links it makes sense for that weight to be the number of occurrences of the term in the document. It is true that doing this effectively requires doubly inverting the database, so that each document points to all its terms as well as vice versa, although you can finesse that by encoding the document as a list of terms rather than as Ascii text, with a slightly higher cost of rebuilding the text when you need to display the document. [My dissertation, describing this in excruciating detail, is *A Spreading Activation Model for Information Retrieval*, University of Illinois, 1981. You might be able to get it from University Microfilms if you're really interested. If you're at Thinking Machines, Dave Waltz had a copy once, but may well have shed it in the last decade. The machine readable form, alas, no longer exists (it lived on a long-dead PDP10)] scott -- scott preece motorola/mcg urbana design center 1101 e. university, urbana, il 61801 uucp: uunet!uiucuxc!udc!preece, arpa: preece@urbana.mcd.mot. com phone: 217-384-8589 fax: 217-384-8550 RELEVANT PROTOCOLS The WorldWideWeb system can pick up information from many information sources, using existing protocols. Among these are file and news transfer protocols. HyperText Transfer Protocol (HTTP) WWW's own protocol, a faster search and retrieve protocol. It is "HTTP" not because it is only for transfering hypertext, but because it operates in a fast, stateless way as is needed for hypertext jumps. For more details: TheW3Book -- /WWW (2342/4980) The protocol as implemented (1991) The protocol as proposed (1992) RFC 1341: Mutippurpose Internet Mail Extensions and a summary below . File Transfer Protocol (FTP) The file transfer protocol currently most used for accessing fairly stable public information over a wide area is "Anonymous FTP". This means the use of the internet File Transfer Protocol without authentication. As the WWW project currently operates for the sake of public information, anonymous FTP is quite appropriate, and WWW can pick up any information provided by anonymous FTP. FTP is defined in RFC 959 which includes material from many previous RFCs. (See also: file address syntax ). Directories are browsed as hypertext. The browser will notice references to files which are in fact accessible as locally mounted (or on DECnet on VMS systems) and use direct access instead. See also the prospero project and the shift project, for more powerful file access systems. Network News The "Network News Transfer Protocol" (NNTP) is defined in RFC 977 by Kantor and Lampsley. This allows transient news information in the USENET news format to be exchanged over the internet. The format of news articles is defined in RFC 850, Standard for Interchange of USENET Messages by Mark Horton. This in turn refers to the standard RFC 822 which defines the format of internet mail messages. News articles make good examples of hypertext, as articles contain references to other articles and news groups. News groups appear like directories, but more informative. HTTP Search and Retrieve The WWW project defines its own protocol for information transfer, called HTTP, for HyperText Transfer Protocol . HTTP is a very simple internet protocol, similar in implementation to FTP and NNTP. The HTTP client sends a document identifier with or without search words, and the server responds with hypertext or plain text. Extensions are foreseen to allow for more elaborate queries and format negotiation . The protocol runs over TCP, using one connection per document request. The browser acts as a pipeline, so that as data arrives from the server, it is presented to the reader on the fly. Whilst the HTTP protocol provides a simple keyword search function, a famous search-and-retrieve protocol is Z39.50, and the version of it used by the WAIS project. (See also the WAIS-WWW gateway ). Gopher The Gopher distributed information system uses a lightweight protocol very similar to HTTP. Therefore, it is now included in every WWW client, so that the Gopher world can be browsed as part of the Web. Gopher menus are easily mapped onto hypertext links. Chances are that future versions of the Gopher and HTTP protocols will converge. HTTP as implemented in WWW This document defines the Hypertext Transfer protocol (HTTP) as currently implemented by the WorldWideWeb initaitive software. This is a subset of the proposed full HTTP protocol. No client profile information is transferred with the query. Future HTTP protocols will be back-compatible with this protocol. The definition of this protocol is in the public domain (see policy ). The protocol uses the normal internet-style telnet protocol style on a TCP-IP link. The following describes how a client acquires a (hypertext) document from an HTTP server, given an HTTP document address . CONNECTION The client makes a TCP-IP connection to the host using the domain name or IP number , and the port number given in the address. If the port number is not specified, 80 is always assumed for HTTP. The server accepts the connection. Note: HTTP currently runs over TCP, but could run over any connection-oriented service. The interpretation of the protocol below in the case of a sequenced packet service (such as DECnet(TM) or ISO TP4) is that that the request should be one TPDU, but the response may be many. REQUEST The client sends a document request consisting of a line of ASCII characters terminated by a CR LF (carriage return, line feed) pair. A well-behaved server will not require the carriage return character. This request consists of the word "GET", a space, the document address , omitting the "http:, host and port parts when they are the coordinates just used to make the connection. (If a gateway is being used, then a full document address may be given specifying a different naming scheme). The search functionality of the protocol lies in the ability of the addressing syntax to describe a search on a named index . A search should only be requested by a client when the index document itself has been descibed as an index using the ISINDEX tag . RESPONSE The response to a simple GET request is a message in hypertext mark-up language ( HTML ). This is a byte stream of ASCII characters. Lines shall be delimited by an optional carriage return followed by a mandatory line feed chararcter. The client should not assume that the carriage return will be present. Lines may be of any length. Well-behaved servers should retrict line length to 80 characters excluding the CR LF pair. The format of the message is HTML - that is, a trimmed SGML document. Note that this format allows for menus and hit lists to be returned as hypertext. It also allows for plain ASCII text to be returned following the PLAINTEXT tag . The message is terminated by the closing of the connection by the server. Well-behaved clients will read the entire document as fast as possible. The client shall not wait for user action (output paging for example) before reading the whole of the document. The server may impose a timeout of the order of 15 seconds on inactivity. Error responses are supplied in human readable text in HTML syntax. There is no way to distinguish an error response from a satisfactory response except for the content of the text. DISCONNECTION The TCP-IP connection is broken by the server when the whole document has been transferred. The client may abort the transfer by breaking the connection before this, in which case the server shall not record any error condition. Requests are idempotent . The server need not store any information about the request after disconnection. _________________________________________________________________ Tim BL HyperText Transfer Protocol Requirements These are dsicussions of requirements for HTTP. See also: The HTTP2 specification Why a new protocol? , Other protocols used the HTTP protocol as currently implemented Protocol design issues . The definition of this protocol is in the public domain (see policy ). UNDERLYING PROTOCOL Current HTTP uses ASCII transmission over a telnet-style internet pr otocol, to make it simple to program, so that it will catch on: conver sion to run over an OSI stack will be simple as the structure of the m essages is well defined. HTTP2 similiarly runs ove an ASCII telnet-style link. the fiedls are represened in RFC-xxxx style mail message format, and as far as possi ble are taken from the equivalent mail header names. IDEMPOTENT This protocol is stateless, in that no state is kept by the server on behalf of the client. (This does not rule out caching by the server i nternally). REQUEST: INFORMATION TRANSFERRED FROM CLIENT Parameters below, however represented on the network, are given in up per case, with parameter names in lower case. This set assumes a model of format negociation in which in which the client says what he can t ake, and the server decides what to give him. One imagines that each f unction would return a status, as well as information specified below. When running over a byte stream protocol, SGML would be an encoding po ssibility (as well as ASN/1 etc). GET document_name HTRQ Please transfer a named document back. Transfer the results back in a standard format or one which I have said I can accept. SEARCH keywords Please search the given index document for all items with the given word combination, and transfer the results back as marked up hypertext. This could elaborate to an SQL query. There are many advantages in making the search criterion just a subset of the document name space. SINCE datetime For a search, refer to documents only dated on or after this date. Used typically for building a journal, or for incremental update of indexes and maps of the web. BEFORE datetime For a search, refer to documents before this date only. ACCEPT format penalty I can accept the given formats . The penalty is a set of numbers giving an estimate of the data degradation and elapsed time penalty which would be suffered at the CLIENT end by data being received in this way. Gateways may add or modify these fields. PORT See the RFC959 PORT command. We could change the default so that if the port command is NOT specified, then data must be sent back down the same link. In an idempotent world, this information would be included in the GET command. HEAD doc Like GET, but get only header information. One would have to decide whether the header should be in SGML or in protocol format (e.g. RPC parameters or internet mail header format). The function of this would be to allow overviews and simple indexes to be built without having to retrieve the whole document. See the RFC977 HEAD command. The process of generation of the header of a document from the source (if that is how it is derived) is subject to the same possibilties (caching, etc) as a format convertion from the source. TheW3Book -- /WWW (2595/4980) USER id The user name for logging purposes, preferably a mail address. Not for authentication unless no other authentication is given. AUTHORITY authentication A string to be passed across transparently. The protocol is open to the authentication system used. HOST The calling host name - useful when the calling host is not properly registered with a name server. Client Software For interest only, the application name and version number of the client software. These values should be preserved by gateways. RESPONSE Suppose the response is an SGML document, with the document type a fun ction of the status. ( Example ) Status A status is required in machine-readable format. See the 3-figure status codes of FTP for example. Bad status codes should be accompanied by an explantory document, possible conianing links to futher information. A possibility would be to make an error response a special SGML document type. Some special status codes are mentioned below . Format The format selected by the server Document The document in that format STATUS CODES Success Accompanied by format and document. Forward Accompanied by new address. The server indicates a new address to be used by the client for finding the document. the document may have moved, or the server may be a name server. Need Authorisation The authorisation is not sufficient. Accompanied by the address prefix for which authorisation is required. The browser should obtain authoisation, and use it every time a request is made for a document name matching that prefix. Refused Access has been refused. Sending (more) authorization won't help. Bad document name The document name did not refer to a valid document. Server failure Not the client's fault. Accompanied by a natural language explanation. Not available now Temporary problem - trying at a later time might help. This does not i,ply anything about the document name and authorisation being valid. Accompaned by a natural language explaination. Search fail Accompanied by a HTML hit-list without any hits, but possibly containing a natural explanation. _________________________________________________________________ Tim BL RULES FOR PENALTY CALCULATION There are two sorts of transformation: (a) conversion from one format into another, for reasons of presentation, and whereby information generally is lost, and (b) encoding, for reasons of speed (compaction), security (encryption), or transmission, and whereby the information and format remain untouched. There are two questions to consider when deciding on different possible transfer formats between servers and clients: Information degradation and elapsed time. Degradation When information is converted from one format to another, it may be degraded. For example, when a postscript file is rendered into bitmap, it loses its potentially infinite resolution; when a TeX file is rendered into pure ASCII, it loses its structure and formatting. This degradation is difficult to guess from simply the file type. and for a given file it is quite subjective. Any attempt to estimate a penalty will therefore be very aproximate, and only useful for distinguishing widely differing cases. A suitable unit would be the proportion, between 0 and 1, of the information which is not lost. Let's call it the degradation coefficient. One would hope that these coefficiemnts are multiplicative, that is that the process of converting a document into one format with degradation coeficient c1 and then further converting the result of that with coeficient c2 would in all be a process with coeffcient c1*c2. This is not, in fact, necessarily the case in practice but is a reasonable guess when we know no better. Elapsed time The elapsed time is another penalty of conversion. As an aproximation one might assume this to be linear in the size of the file. It is not easy to say whether the constant part or the size-proportional part is going to be the most important. The server, of course, knows the size of the file. It can in fact as a result of experience make improving guesses as to the conversion time. The conversion time will be a function also of local load. For particular files, it may be affected by the caching of final or intermediate steps in a conversion process. Given a model in which the server makes the decision on the basis of information supplied by the client, this information could include, for each type, both the constant part (seconds) and the size-related part (seconds per byte). _________________________________________________________________ TheW3Book -- /WWW (2722/4980) Tim BL, RC Why a new protocol? Existing protocols cover a number of different tasks. Mail protocols allow the transfer of transient messages from a single author to a small number of recipients, at the request of the author. File transfer protocols allow the transfer of data at the request of either the sender or receiver, but allow little processing of the data at the responding side. News protocols allow the broadcast of transient data to a wide audience. Search and Retrieve protocols allow index searches to be made, and allow document access. Few exist: Z39.50 is one and could be extended for our needs. The protocol we need for information access ( HTTP ) must provide A subset of the file transfer functionality The ability to request an index search Automatic format negotiation. The ability to refer the client to another server _________________________________________________________________ Tim BL HTTP Request fields These header lines are sent by the client in a HTTP protocol transacti on. All lines are RFC822 format headers. FROM: In Internet mail format, this gives the name of the requesting user. T his field may be used for logging purposes and an insecure form of acc ess protection. The interpretation of this field is that the request is being performed on behalf of the person given, who accepts responsa bility for the method performed. The Internet mail address in this field does not have to correspond to the internet host which issued the request. (For example, when a requ est is passed through a agateway, then the original issuer's address s hould be used). The mail address should, if possible, be a valid mail address, whether or not it is in fact an internet mail address or the internet mail re presentation of an address on some other mail system. ACCEPT: This field contains a comma-separated list of representation schemes ( MIME compatible content-type values) which will be accepted in the res ponse to this request. The set given may of course vary from request to request from the same user. This field may be extended onto several lines. The format of each ent ry in the list is = [ (

, , )]

= = = = It is assumed that there See the appendix on the negotiation algorith m as a function and penalty model. Example Accept: text/plain, text/html ACCEPT-ENCODING: Similar to Accept, but lists the Content-Encoding types which are acc eptable in the repsonse. USERAGENT: This line if present gives the software user agent used by the origina l client. This is for statistical purposes and teh tracing of protocol violations. It should be included. Format TBS. (Separate basic software name from version number in a st andard way). REFEREE URI: This optional header field allows te client to specify, for the server 's benefit, the address (URI) of the document (or element within the document) from which the URI in the request was obtained. This allows a server to generate lists of back-links to documents, for interest, logging, etc. If a partial URI is given, then it should be parsed relative to the UR L of the object of the request. AUTHORIZATION: This line is present contains authorization information. The format is TBS. The format of this field must be in extensible form. The first word is a specification of the authorisation system in use. CHARGETO : This line is present contains account information for the costs of the application of the method requested. The format is TBS. The format of this field must be in extensible form. The first word starts with a specification of the namespace in which the account is . (This is simi lar to extensible URL definition.) No namespaces are currently defined . Namespaces will be registered with the registration authority . The format of the rest of the line is a function of the charging syste m, but it is recommeded that this include a maximum cost whose payment is authorized by the client for this transaction, and a cost unit. Ti m BL TBL Object Headers The header fields given with or in relation to objects in HTTP are as follows. All are optional. The order of header lines withing the HTTP header has no significance. (TBS in more detail) ALLOWED: *METHOD Lists the set of requests which the requesting user is allowed to issue for this URL. If this header line is omitted, the deafault allowed methods are "GET HEAD" Example of use: Allow: GET HEAD PUT PUBLIC: *METHOD As "Allow" but lists those requests which anyone may use. If omitted, the default is "GET" only. Example of use: Public: GET HEAD TEXTSEARCH LENGTH: INT Implies that the body is binary and should be read directly from the communications link, without parsing lines, etc. @@@ This should be part of the MIME header, as it applies to any binary encoded part. Note HTML is the first internet protocol to allow MIME "binary" encoding.. Use Content-Length only curently OK in external message type! EXPIRES: DATE Gives the date (in GMT, format TBS) after which the information given ceases to be valid and should be retrieved again. This allows control of caching mechanisms, and also allows for the periodic refreshing of displays of volatile data. Format as RFC850. This does NOT imply that the original object will caese to exist. (Is this a semantic clash with RFC850 ? TBL) LAST-MODIFIED: DATE Last time object was modified. Format as for Date: and Expires:. DATE: DATE Creation date of object. (or last modified, and separately have a Created: field?) Format as in RFC850 . MESSAGE-ID: URI A unique identifier for the message. As in RFC850 , except that the unlimited lifetime of HTTP objects requires that the Message-ID be unique in all time, not just in two years. A document may only have one Message-ID. No two documents, even if different versions of the same live document, may have the same Message-id. DEAD-URI: 1*URI This gives a URI with which the object may be found. There is no guarantee that if the object is retrieved usingthe URI specified. However, it is guaranteed that if an object is successfully retrieved using that uri it will be the same unmodified object as this one. Multiple occurencies of this field give alternative access names or addresses for the live document. LIVE-URI: 1*URI This gives a URI with which the most recent version of an object, may be found. There is no guarantee that if the object is retrieved using the URI specified. However, it is guarranteed that if an object is successfully retrieved using that uri that it will be the same object or a more recent version of the same object. Multiple occurencies of this field give alternatives which should refer to the same live object. COST: TBS The cost of retrieving the object is given. This is the cost of access of a copyright work. Format of units to be specified. Currently refers to an unspecified charging scheme to be agreed out of band between parties. Last modified 22/09/92 by CTB HTRESP RESPONSE TYPES The syntax of the responses to HTTP requests are as follows: Error RESPONSE="ERROR" HTTPV=VersionNumber CrLf STATUS=String REASON=String [IDENTIFIER=String] CrLf [body section] Forward RESPONSE="FORWARD" HTTPV=VersionNumber CrLf UDI=String CrLf [body section] Redirection RESPONSE="REDIRECTION" HTTPV=VersionNumber CrLf UDI=String CrLf [body section] Query RESPONSE="QUERY" HTTPV=VersionNumber CrLf SERVER=String CrLf [body section] Algorithm Index RESPONSE="INDEX" HTTPV=VersionNumber CrLf [body section] Data RESPONSE="DATA" HTTPV=VersionNumber CrLf [body section] Confirmation RESPONSE="CONFIRMATION" HTTPV=VersionNumber CrLf [body section] BODY SECTION The body section has the following syntax: NOTATION=String TOEOF|BYTES=Number [EDITABLE] CrLf [PROFILE KEY=String [DURATION=String]] CrLf [FETCHED=TimeStamp CrLf] ENDHEADER CrLf [data] where the NOTATION string is a list of format transformations which must be applied to the 'data' in order to reach the final format in the list, e.g.: NOTATION="UUDECODE,UNCOMPRESS,PLAINTEXT" BYTES=1000 means that if the following1000 bytes of data are UUDECODEd and then UNCOMPRESSed, the resulting data will be in PLAINTEXT format. Therefore the 'data' section may not necessarily be in ASCII. The PROFILE KEY is an indication by the server that it has cached the client's profile (sent as part of the request), and that the client need only sent this key, instead of the complete profile. The DURATION is optionally sent by the server to indicate how long the key will be valid for. This will be an estimated relative time. Note that the presence of the TOEOF indicator means that the data section is of indeterminate length, i.e. the length could not be calculated, either because it is the result of some translation process which does not allow the server to calculate the length, or because the data is being produced by a data sourcing process which may run for an indeterminate amount of time. The FETCHED field is the data and time that the server retrieved the document. When the browser attempts to PUT a document, this is returned to prevent the erasure of any changes made to the document in the time between the document being retrieved and being returned. The format of the TimeStamp is DD MM YY HH:MM:SS, and is the local time of the server. Where the RESPONSE is an ERROR, the STATUS and REASON fields may have a number of values, depending upon the situation encountered by the server. These are described below: The following STATUS values are allowed: SERVER_ERR Fatal error at the server end, no retry (of the same request) possible. CLIENT_ERR Fatal error caused by the client, no retry (of the same request) possible. TEMP_ERR Temporary error at server end, retry of same request possible after some delay. The meanings of the various REASON values change depending on the value of STATUS. Where the STATUS is CLIENT_ERR, the meanings are: AUTH_FAIL The user ID and password supplied do not match any pair used by the server. [Should always use AUTH_REFUSED - CTB] AUTH_REFUSED The user does not have authorisation to carry out that operation on that document. This is only used where a password and user ID have been supplied. PWD_REQ The operation can only be carried out by an authorised user, and no such authorisation was supplied by the client. PROFILEKEY_FAIL The profile key supplied is not valid or no longer valid for this server. DOC_UNAVAIL The UDI supplied does not match any file available to the server (note that the file could exist, but not be available to the server). VERSION_ERR The HTTP version number supplied by the client is higher than that being used by the server, and the server cannot handle that request. INVALID_COMMAND The user supplied a command which the server does not recognise (where the client's protocol version is higher than the server's, a VERSION_ERR should be sent instead) FMT_FAIL The document cannot be supplied in any of the formats supplied by the user. Where the STATUS is SERVER_ERR, the meanings are: AUTH_FAIL The server was unable to authorise the user, even though a password and user ID have been supplied. AUTH_REFUSED The server has insufficient privilege to carry out that operation, even though the client does (this should only happen if the server has been incorrectly set up). VERSION_ERR The HTTP version number supplied by the client is lower than that being used by the server, and the server cannot handle that request. INVALID_COMMAND The client has sent the server a command which it cannot handle, even though it is specified in the protocol version that the server is using. FWD_FAIL The server cannot access the information necessary to return the forwarding information. Where the STATUS is TEMP_ERR, the meanings are: AUTH_FAIL The server was unable to carry out the authorisation operation at the present time due to a temporary failure (such as password file being updated by someone else). PROFILE_FAIL The server was unable to access the stored profile information at the moment. DOC_UNAVAIL The document requested is temporarily unavailable. SERVER_BUSY The server is unable to process the request at the present time. Other combinations of STATUS/REASON are undefined at the moment. The field IDENTIFER may be included to identify the scope over which the error ranges. For example, for an authorisation error, the server may return a mask for all those files which will generate such an error.Implementation Note: Note that in theory the body of an error message may be in ANY acceptable format. However, in order to make the implementation of the new browser as simple as possible, it will assume only HTML will be contained in these messages. But since most of these will be generated by the server anyway, this isn't such a restriction. Note also that the browser will not allow any links in an error document to be followed. Last modified 7-Dec-92 TBL Reason fields The values of the reason field in HTTP error responses are as follows. Where the HTTP RESPONSE is an ERROR, the W3-Status and Reason fields may have a number of values, depending upon the situation encountered by the server. These are described below. The following status groups are recognised: 4xx = SERVER_ERR Fatal error at the server end, no retry (of the same request) possible. 5xx = CLIENT_ERR Fatal error caused by the client, no retry (of the same request) possible. TheW3Book -- /WWW (3164/4980) TEMP_ERR Temporary error at server end, retry of same request possible after some delay. The meanings of the various REASON values change depending on the value of STATUS. CLIENT ERRORS Where the STATUS is CLIENT_ERR, the meanings are: AUTH_REFUSED The authorzation given was not sufficient to allow the request to be carried out. This is only used where a authorization field was given in the request. AUTH_REQUIRED The operation can only be carried out by an authorised user, and no such authorisation was supplied by the client. The parameter to this message gives a specification of authorization schemes which are acceptable. The client should retry the request with a suitable Authorization header. PAY_REQUIRED The operation can only be carried out if an acceptable ChageTo header is supplied. The parameter to this message gives a specification of charging schemes acceptable. The client may retry the request with a suitable ChargeTo header. The body of the reply must contain header fields for the object includingthe Cost: field. PAY_REFUSED An error occured with the ChargeTo field in the request. (For example, payment was refused, the cost limit was too low, PROFILEKEY_FAIL The profile key supplied is not valid or no longer valid for this server. DOC_UNAVAIL The UDI supplied does not match any file available to the server (note that the file could exist, but not be available to the server). VERSION_ERR The HTTP version number supplied by the client is higher than that being used by the server, and the server cannot handle that request. INVALID_COMMAND The user supplied a command which the server does not recognise (where the client's protocol version is higher than the server's, a VERSION_ERR should be sent instead) FMT_FAIL The document cannot be supplied in any of the formats supplied by the user. SERVER ERRORS Where the STATUS is SERVER_ERR, the meanings are: 500 = INCOMPREHENSIBLE The format of the request is not recognised. TheW3Book -- /WWW (3228/4980) 501 = AUTH_FAIL The server was unable to authorise the user, even though a password and user ID have been supplied. 502 = AUTH_REFUSED The server has insufficient privilege to carry out that operation, even though the client does (this should only happen if the server has been incorrectly set up). 503 = VERSION_ERR The HTTP version number supplied by the client is lower than that being used by the server, and the server cannot handle that request. 504 = INVALID_METHOD The client has sent the server a method name which does not apply to the object in question. 505 = INTERNAL The server cannot satisfy the request due to some condition internal to the server. TEMPORARY ERRORS Where the STATUS is TEMP_ERR, the meanings are: 451 = AUTH_FAIL The server was unable to carry out the authorisation operation at the present time due to a temporary failure (such as password file being updated by someone else). 552 = PROFILE_FAIL The server was unable to access the stored profile information at the moment. 553 = OBJECT_UNAVAILABLE The document requested is temporarily unavailable. 554 = SERVER_BUSY The server is unable to process the request at the present time. The resaon line may give suitabl etimes to contact the server. Other status code values are undefined at the moment. Experimental values may be used if they have the form xx9. RC , TBL HTTP Negotiation algoritm This note defines the significance of the d, a and b values optionally sent in the Accept: field of the HTTP protocol request message. It is assumed that there is a certain value of the presentation of the document, optimally rendered using all the information available in its original source. It is further assumed that one can allocate a number between 0 and 1 to represent the loss of value which occurs when a document is rendered intop a represenmtation with loss of information. Whilst this is a very subjective measurement, and in fact largely a function of the document in question, the approximation is made that one can define this "degradation" figure as a function of merely the representation involved. TheW3Book -- /WWW (3292/4980) The next assumption is that there the other cost to the user of viewing the doument is a function of the time taken for presentation. We first assume that the cost is linear in time, and then assume that the time is linear in the size of the message. The final net value to the user can therefore be written presented_value = initial_value * total-degradation - a - b * size for a document in a given incoming represenattion. Suppose we normalize the initial value of the document to be 1. The total degradation is the product of any degradation due to conversions internal to the server, and the degradation "d" sent in the Accept field. The values of a and b are also sent by the protocol. If not sent, they default to no cost (d=1, a=b=0). The server may optimize the presented value for the user when deciding what to return. The hope is that fine decisions will not have to be made, as in most cases the results for different formats will be very different, and there will be a clear winner. See also: Design issues discussions around this point. Tim BL Note: The cost of retrieval time The assuption that the cost to the user associated with a certain retrieval time is linear in that time is wildly innaccurate. The real fucntion could be very dependednt on circumstances (like go to infinity at a deadline). A better general approximation might be logarithmic for large time delays, and linear for small ones, like a*log(b*t-1) which has two parameters. Tim BL Note: Client tolerance of bad servers In order that clients using the HTTP protocol should be able tocommunicate with servers using the protocol originally implemented in the W3 data model, clients should tolerate responses which do not start with a numeric version number and response codes. In this case, they should assume that the rest of the response is a document in type text/html. Tim BL Note: Server tolerance of bad clients Whilst it is seen apropriate for testing parsers to check full conformance to this specification, it is recommended that operational parsers by tolerant of deviations. In particular, lines should be regarded as terminated by the Line Feed, and the preceeding Carriage Return character ignored. Any HTTP Header Field Name which is not recognised should be ignored in operational parsers. TheW3Book -- /WWW (3356/4980) For backward compatibility, a GET method with a URL containing a search operator ("?") should be supported by any server supporting TEXT SEARCH. Tim BL Registration Authority The HTTP Registration Authority is responsible for maintaining lists of: Charge account name spaces Authorization schemes Data format names Data encoding names It is proposed that the Internet Assigned Numbers Authority or their successors take this role. Unregistered values may be used for experimental purposes if they are start with "X-". W3 NAMING SCHEMES (See also: a discussion of design issues involved , BNF syntax , W3 ba ckground) The format of a hypertext name consists of the name of the naming sub- scheme to be used, then a name in a format particular to that subschem e, then an optional anchor identifier within the document. For example , the format is for all internet-based access methods: scheme : // host.domain:port / path / path # anchor A suffix # anchor id allows one to refer to a particular anchor within a document. A suffix ? followed by words separated by + signs allows one to seach an index (see details ). References from one document to another with a similar name may be abb reviated to a relative name . This imposes certain restrictions on the way that the "path" is represented. A special format is used to represent a search on an index . See also: the full BNF description , about escaping illegal characters . Examples file://cernvax.cern.ch/usr/lib/WWW/defaut.html#123 This is a fully qualified file name, referring to a document in the fi le name space of the given internet node, and an imaginary anchor 123 within it. #greg This refers to anchor "greg" in the same document as that in which the name appears. Naming sub-schemes Different schemes usually use different protocols on the network. The format of the address after the scheme name is a function of the parti cular scheme. In practice, all internet-based schemes have a common fo rmat for the node name and port. Schemes currently defined are as fo llows, with links to more details. file Access is provided to files, using whatever means the browser and/or gateways have to reach files on obscure machines. news Access is provided to news articles, and newsgroups, normally using the NNTP protocol. http Access is provided to any other information using the HTTP search and retrieve protocol . The internal addressing of the information system is mapped onto a W3 path. telnet Access is provided by an interactive telnet session. This is provided ONLY as an interface to other existing online systems which cannot or have not been mapped onto the W3 space. gopher Access is provided using the "gopher" protocol. The gopher protocol is similar to HTTP but uses separate concepts of menus and text files rather than hypertext. wais Access is provided using the WAIS adaptaion of the Z39.50 protocol. x500 Format to be defined. Systems (such as WAIS) which are not currently accessed directly be W3 servers may be accessed though gateways, in which case the document a ddress is encoded within the http address of the document in the gatew ay. Browsers which do not have the ability to use certain protocols m ay (in principle) be configured to automaticaly use certain gateways f or certain addressing schemes. This will allow, for example, simple PC-based clients to follow links Address for an index Search If a given hypertext node is an index, or the server has an index asso ciated with it, then a search may be done on that index by suffixing t he name of the index with a list of keywords, after a question mark: address_of_index ? keywordlist The address of the index is a normal hypertext address. In the keyword list, multiple keywords are separated by plus signs (+) . (See BNF sy ntax description .) The resulting string still does not contain any s paces. It may be considered to be the hypertext address of a document which is the result of making the keyword search on the index. Normall y, if the search was successful, the document returned will contain an chors leading to other documents which match the selection criteria. The search method, and the logical and lexical functions, weights, etc applied to the keywords will depend on the index address. One actual index may have several hypertext addresses, which when searched on w ill behave in different ways. For example, one may allow a search on a uthor-given keywords only, while another may be a full text search. Th ese things particular to an index should be descibed in the hypertext page for the index node itself (or in linked documents). For example, a server may allow specific boolean search combinations may be represe nted by the words "and", "or" and "not". EXAMPLE: http://cernvm/FIND/?sgml+cms indicates the result of perfoming a search for keywords "sgml" and W3 addresses of files The format of a hypertext reference to a file is an extension of the u nix naming system. The full explicit format is: file : // node / directories / name The actual protocols used by the client depend on the implementation o f the browser and the environment. Typically, the browser will check t o see whether the node is the local node, or a node for which files a re available mounted in some form of distributed file system. If neit her of these are the case, then the browser may try rpc, anonymous FTP or other protocols. EXAMPLES file://cernvax.cern.ch/usr/lib/WWW/defaut.html This is a fully qualified file name. fred.html This relative name , used within a file, will refer to a file of the s ame node and directory as that file, but the name fred.html. IMPROVEMENTS : DIRECTORY ACCESS The final file name should be optional. If the address ends with a '/' , the browser should retrieve the contents of the specified directory and generate a page of virtual hypertext pointing to its contents. In addition, it could display an information file contained in that direc tory, if any is present. Suggested file names to search for in order : README.html, *README*.html, README, *README*, *readme*. Hypertext address for net News The format of a hypertext reference to information in the internet/usenet news system can take any of the following forms: news: newsgroup This refers to a list of articles currently available in the given newsgroup. The newsgroup is a series of alphanumeric characters and dots. news:* This refers to a list of valid newsgroups. TheW3Book -- /WWW (3546/4980) news: message_id This refers to a given article explicitly. The message_id is optionally surrounded by angle brackets, and must contain an @ sign. Possible extensions to this are more generous wildcarding for the list of newsgroups. It takes too long to load the whole list, and it would be more useful to be able to browse through a set of newsgroups. There is no way of referring to "unread" articles. Keeping track of this is the job of the browser. EXAMPLES news:<12345678@cernvax.cern.ch> news:12345678@cernvax.cern.ch These addresses both refer to the same (imaginary!) article by its unique message-id. (Note the hostname in the message-id is just part of the message id generated by the sender of the message - it is not a news server address). news:comp.sys.next.announce This refers to a list of articles in the newsgroup comp.sys.next.announce. Relative naming The address of a hypertext document is normally given within the context of another hypertext document. Where the addresses of the two documents are the similar, this allows only the difference between the two names to be given, saving space. An example is the address of the destination of a hypertext link , which is specified relative to the source document address. (A futher practical advantage is that a group of documents may be transmitted without internal changes, or accessed using more than one address.) This implies that certain characters ("/", "..") have a significance reserved for representing a hierarchical space, and must be recognized as such by both clients and servers. In the WWW address format , the rules for relative naming are: If the " scheme " parts are different, the whole absolute address must be given. Other wise, the scheme is omitted, and: If the "host" and/or "port" parts are different, the host name and all the rest of the address must be given. The host name may be given using internet hostname conventions, ie domains may be omitted where different. This is not very well defined: one tends to assume that if any dot is present, then the full domain name is being given, up to the root (.) domain, while if there are no dots, the domain is the same as that of the hostname part of the the base address. If the access and host parts are the same, then the path may be given with the unix convention, including the use of ".." to mean indicate deletion of a path element. Within the path: If a leading slash is present, the path is absolute. Otherwise: The last part of the path of the base address (e.g. the filename of the current document) is removed, and the given relative address appended in its place. Within the result, all occurences "xxx/.." or "/." are recursively removed, where xxx is one path element (directory). The use of the slash "/" and double dot ".." in this case must be respected by all servers. If necessary, this may mean converting their local representations in order that these characters should not appear within path elements (see "escaping" ). HTTP Addressing With an access code of http:, a protocol introduced for the WWW initiative is used to acquire data from a server. This is the "Hypertext Transfer protocol", HTTP , a simple search and retrieve (S and R) protocol. The syntax of an http address is, with [] indicating optional parts (see BNF description ), http : // hostname [ : port ] / path [ ? searchwords ] for example, the following are valid addresses: http://info.cern.ch/hypertext/WWW/TheProject.html http://crnvmc.cern.ch/FIND?sgml+examples HTTP addresses conform to the WWW conventions, including the possibility of using the search format . The significance of the items in the path part of the document name is completely up to the server. Different paths may be used to select different databases, different views of the same database, etc. hostname This is the name of the server in internet form. A numeric form (e.g. 128.141.201.74) may be used, by the domain name form (e.g. info.cern.ch) is preferred. The hostname is mandatory. port This is a numeric port number. If a non-numeric string is used, it must be a defined service name. Note that as there is no central repository for service names (they are defined locaaly for each host), a service name is NOT an appropriate way to specify a port number for a hypertext address. If the port number is omitted the preceding colon must also be omitted. In this case, port number 2784 is assumed [This may change!]. See also: WWW addressing in general , HTTP protocol . _________________________________________________________________ Tim BL Telnet addressing A telnet address is a spcecial case of a W3 address. When a telnet address is used, information can only be rertrieved using an interactive telnet session. This has the disadvantage that information cannot be indexed, searched, etc automatically, nor can it be gatewayed into other systems. The telnet addressing form is used to allow a pointer to information systems such as library information systems which have not been gatewayed into the web properly yet. The syntax is, with [] indicating optional parts (see full BNF) telnet : / / [ user @ ] host [ : port ] There should be no spaces. For example, the following are valid telnet addresses: telnet://www@info.cern.ch:23 telnet://www@info.cern.ch telnet://info.cern.ch user is the optional name of the user to be used for login. If the username is omitted, then so must be the "@" sign. This is equivalent to the argument used with the -l option on the ucb telnet command. When the username is omitted, some access servers will prompt for a username and password. host This is the name of the server in internet form. A numeric form (e.g. 128.141.201.74) may be used, by the domain name form (e.g. info.cern.ch) is preferred. The host is mandatory. port This is a numeric port number. If a non-numeric string is used, it must be a defined service name. Note that as there is no central repository for service names (they are defined locaaly for each host), a service name is NOT an appropriate way to specify a port number for a hypertext address. If the port number is omitted the preceding colon must also be omitted. In this case, port number 23 is assumed. _________________________________________________________________ Tim BL W3 address syntax: BNF This is a BNF-like description of the W3 addressing syntax . We use a vertical line "|" to indicate alternatives, and [brackets] to indicate optional parts. Spaces are representational only: no spaces are actually allowed within a W3 address. Single letters stand for single letters. All words of more than one letter below are entites described elsewhere in the syntax description. (Entity names are here linked to their definitions, probably making this difficult to read with the line mode browser.) An absolute address specified in a link is an anchoraddress . The address which is passed to a server is a docaddress . TheW3Book -- /WWW (3736/4980) anchoraddress docaddress [ # anchor ] docaddress httpaddress | fileaddress | newsaddress | telnetaddress | prosperoaddress | gopheraddress | waisaddress httpaddress h t t p : / / hostport [ / path ] [ ? search ] prosperoaddress p r o s p e r o : / / hostport / path fileaddress f i l e : / / host / path newsaddress n e w s : groupart waisaddress waisindex | waisdoc waisindex w a i s : / / hostport / database [ ? search ] waisdoc w a i s : / / hostport / database / wtype / digits / path groupart * | group | article group ialpha [ . group ] article xalphas @ host database xalphas wtype xalphas telnetaddress t e l n e t : / / [ user @ ] hostport gopheraddress g o p h e r : / / hostport [/ gtype [ / selector ] ] [ ? search ] hostport host [ : port ] host hostname | hostnumber hostname ialpha [ . hostname ] hostnumber digits . digits . digits . digits port digits selector path path void | xalphas [ / path ] search xalphas [ + search ] user xalphas anchor xalphas gtype xalpha xalpha alpha | $ | _ | @ | ! | % | ^ | & | * | ( | ) | . | digit xalphas xalpha [ xalphas ] ialpha alpha [ xalphas ] alpha a | b | c | d | e | f | g | h | i | j | k | l | m | n | o | p | q | r | s | t | u | v | w | x | y | z | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z digit 0 |1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 digits digit [ digits ] alphanum alpha | digit alphanums alphanum [ alphanums ] void See also: General description of this syntax, Escaping conventions. _________________________________________________________________ Tim BL Escaping illegal characters The W3 address syntax allows a path to contain most printable ASCII characters, but some are inevitably used for punctuation are excluded. W3 addresses are sometimes used to represent addresses in some other space. This happens when an HTTP server, for example, uses file names as its document names, or when addresses from some other protocol (Gopher, WAIS, etc) are mapped into the W3 web. In these cases, a convention is normally used to map illegal characters in these "foreign" names onto the allowed set. In the case of an HTTP server, any mapping may be used. A suitable convention is that a percent sign (%) followed by two hexadecimal digits (0-9 or a-f) stands for the single character with ASCII hexadecimal code represented by those two digits (Most significant digit first). A percent sign itself must therefore be represented by %25, as 25 hex is the ASCII code for "%". _________________________________________________________________ Tim BL Gopher addressing Gopher addresses indicate that the gopher protocol should be used to access the information. The Gopher protocol is a simple internet protocol similar to HTTP . It allows the transfer of menus or plain text files. (HTTP expresses both menus and plain text files as special cases of hypertext files). See the gopher protocol notes . The syntax is, with [] indicating optional parts (see BNF ) gopher:// hostname [: port ] [/gtype/ [selector] ] [ ? search ] There should be no spaces. For example, the following are valid addresses: gopher://gopher.micro.umn.edu:70 gopher://gopher.micro.umn.edu:70/1/ gopher://gopher.micro.umn.edu:70 The W3 address for a gopher item may be derived from the fields of a gopher menu line which has the format host This is the name of the server in internet form. A numeric form (e.g. 128.141.201.74) may be used, by the domain name form (e.g. info.cern.ch) is preferred. The hostname is mandatory. port This is a numeric port number. If a non-numeric string is used, it must be a defined service name. Note that as there is no central repository for service names (they are defined locaaly for each host), a service name is NOT an appropriate way to specify a port number for a hypertext address. If the port number is omitted the preceding colon must also be omitted. In this case, port number 70 is assumed. gtype This is a gopher item type number, a (hopefully printable!) ASCII character. Currently these types are all ASCII decimal digit characters. Character "0" (hex 30) signifies a plain text file. Character "1" signifies a Menu. Character "7" signifies a searchable index. Character "8" should not be used in a W3 address: use telnet addressing instead. In general W3 terms, the type is the first part of the path. The rest of the path is the gopher selector string. The type field is a hint to the client as to how to represent the anchor, and how to follow it. selector This is the string to be sent to the gopher server to identify the information required. _________________________________________________________________ Tim BL W3 addresses for WAIS servers Servers using the WAIS ("Wide Area Information Systems") protocols from Thinking Machines may be accessed as part of the web using addresses of the form (see BNF description) w a i s : / / hostport / database ... Access (currently) goes through a gateway which stores the "source" files which contain the descriptions of WAIS servers. This address corresponds to the address of an index. To this may optionally be appended either a search string or a document identifier. TheW3Book -- /WWW (3926/4980) Note that changes have been proposed to WAIS document id format, so this representation of them may have to change with that format. Currently the WAIS document address necessary for retrieval by a client requires the following information, which is orginally provided by the server in the hit list. Document format This is normally "TEXT" but other formats such as PS, GIF, exist. Document length This is needed by the client who must loop to retrie the whole document in slices. Document identifier This is an entity consisting of numerically tagged fields. the binary representation used by WAIS is transformed for readability into a sequence of fields each consisting of a decimal tag, an equals sign (=) , the field value, and a semicolon. Within the field value, hex escaping is used for otherwise illegal characters. See also: Other W3 address formats, BNF definition. _________________________________________________________________ HTML The WWW system uses marked up text to represent a hypertext document for transmision over the network. The hypertext markup language is an SGML format. To find out how to write HTML, or to write a program to generate it, read the following sections:- Text and Markup An introduction to SGML The elements A list of the tags used in HTML with their significance. Entities Special characters are represented by SGML entities HTML Specification Guidance for parser implementors DTD The SGML document type definition for HTML. (Hypertext version ) The following do not form part of the specifciation. Example A file containing a variety of tags used for test purposes, and its source text . See also finding examples on the web . Future directions Changes suggested for HTML improvements Constraints Design constraints for HTML which might explain some of its properties. HTML Elements This is a list of elements used in the HTML language. (See also informal About Tags ). Documents should (but need not absolutely) contain an intial HEAD element followed by a BODY element. Old style documents may contain a just the contents of the normal HEAD and BODY elements, in any order. This is deprecated but must be supported by parsers. See also: Status of elements PROPERTIES OF THE WHOLE DOCUMENT Properties of the whole document are defined by the following elements. They should appear within the HEAD element. Their order is not significant. TITLE The title of the document ISINDEX Send by a server in a searchable document NEXTID A parameter used by editors to generate unique identifiers LINK Relationship between this document and another. See also the Anchor element, Relationships. TEXT FORMATTING These are element which occur withing the BODY element of a document. Their order is the order in which the elements should be rendered on the output device. Headings Several levels of heading are supported Anchors Sections of text which form the beginning and/or end of hypertext links are called "anchors" and defined by the A tag. Paragraph marks The P element marks the break between two paragraphs Address style An ADDRESS element is diaplayed in a particular style. Blockquote style A block of text quoted from another source Lists Bulleted lists, glossaries, etc. Preformatted text Sections in fixed-width font for preformatted text. Character highlighting Formatting elements which do not cause paragraph breaks. OTHER ELEMENTS The following elements are either obsolete or not yet available in all implementations. Each is marked with its status. Base Address Status: Not in use. May be introduced. TheW3Book -- /WWW (4053/4980) Anchors specify addresses of other documents, in a from relative to the address of the current document. Normally, the address of a document is known to the browser because it was used to access the document. However, is a document is mailed, or is somehow visible with more than one address (for example, via its filename and also via its library name server catalogue number), then the browser needs to know the base address in order to correctly deduce external document addresses. The format of this tag is not yet specified. NOT CURRENTLY USED Plaintext Status: Deprecated. Will be made obsolete by the new HTTP protocol. The empty PLAINTEXT tag terminates the HTML entity. What follows is not SGML. In stead, there's an HTTP convention that what follows is an ASCII (MIME "text/plain") body. An example if its use is:

0001 This is line one of a ling listing 0002 file from <any@host.inc.com> which is sen t This tag allows the rest of a file to be read efficiently without parsing. Its presence is an optimisation. There is no closing tag. The rest of the data is not in SGML. Example sections Status: Deprecated. This are in use and must be recognised by browers. New servers should use <PRE> instead. These styles allow text of fixed-width characters to be embedded absolutely as is into the document. The format is: <LISTING> ... </LISTING> The text between these tags is to be portrayed in a fixed width font, so that any formatting done by character spacing on successive lines will be maintained. Between the opening and closing tags: The text may contain any ISO Latin printable characters, but not the end tag opener. (See Historical note ) Line boundaries are significant, except any occuring immediately after the opening tag or before the closing tag. and are to be rendered as a move to the start of a new line. The ASCII Horizontal Tab (HT) character must be interpreted as the smallest positive nonzero number of spaces which will leave the number of characters so far on the line as a multiple of 8. Its use is not recommended however. The LISTING element is portrayed so that at least 132 characters will fit on a line. The XMP elemnt is portrayed in a font so that at least 80 characters will fit on a line but is otherwise identical to LISTING. TheW3Book -- /WWW (4118/4980) Highlighted Phrase HP1 etc Status: Obsolete . These tags like all others should be ignored if not implemented. Replaced will more meaningful elements -- see highlighting . Examples of use: <HP1>...</HP1> <HP2>... </HP2> etc. Comment element Status: Deprecated. A comment element used for bracketing off unneed text and comment has been intriduced in some browsers but will be replaced by the SGML command feature in new implementations. SGML ISO 8879:1986, Information Processing -- Text and Office Systems -- Standard Generalized Markup Language (SGML) This is an ISO standardised derivative of an earlier IBM "GML". It allows the structure of a document to be defined, and the logical relationship of its parts. This structure can be checked for validity against a " Document Type Definition ", or DTD. The SGML standard defines the syntax for the document, and the syntax and semantics of the DTD. See books -- Eric van Herwijnen's "Practical SGML" and Charles Goldfarb's "SGML Handbook". Some of the points generally broght up in (frequent) discussions of SGML follow. HIGH LEVEL MARKUP An SGML document is marked up in a way which says nothing about the representation of the document on paper or a screen. A presentation program must marge the document with style information in order to produce a printed copy. This is invaluable when it comes to interchange of documents between different systems, providing different views of a document, extracting information about it, and for machine processing in general. However, some authors feel that the act of communication includes the entire design of the document, and if this is done correctly the formatting is an essential part of authoring. They resist any attempts to change the representation used for their documents. SYNTAX The SGML syntax is sufficient for its needs, but few would say that it is particularly beautiful. The language shows its origins in systems where text was the principle content and markup was the exception, so a document which contains a lot of SGML is clumsy. There is always, of course, an element of personal taste to syntax. TOOLS For many years, SGML was generated by hand, by people editing the source. This has lead to a hatred of SGML among those who prefer their own mark-up language which may have been quicker, more powerful, or more familiar. The advent of WYSIWYG editors and solid SGML applications should improve that facet of SGML. ARCHIVE TheW3Book -- /WWW (4182/4980) There are a number of SGML archive sites. In Genermany, there the Darmstadt archive . See also: HyTime , HTML , Hypertext Document formats . Davenport. group . A public domain parser, SGMLS, exists. There is an archive of SGML related information at ifi.uio.no Tim BL AAP AAP stands for the American Asociation of Publishers, one of the first groups to fix on a common SGML DTD. Design Constraints When designing the HTML document type, consideration was given to a ce rtain simplicity in order to allow many browsers and hopefully editors to be developed on many platforms. LACK OF NESTING Many text editing systems (Microsoft Word, The NeXT text object, the M ac text object, etc) handle text in a variety of styles but do not hav e any concept of nestable structure in the SGML sense. The constraint here is therefore that HTML be able to be mapped into a sequence of paragraphs of styled text, and that if that text is edite d that the editor should be able to map the sequence of styles back on to a sequence of elements in a well-defined way. This allows some limi ted trivial nesting (eg LI within UL) but no general nesting, as a fin ite and small set of styles is used. In particular, the styles are not parameterized by the nesting level. LOGICAL MARKUP It is required that HTML be a common language between all platforms. T his implies no device-specific markup, or anything which requires cont rol over fonts or colors, for example. This is in keeping with BROWSER SOFTWARE ARCHITECTURE The WWW browsers share a common architrecure, and a certain amount of common code. (See also: Browser operation , and utility modules which are used throughout, and using the common library.). In the contol flow diagram , common code is to the right of the grey l ine. Application This module is the main program, and is window-system-dependent. In the line mode browser , it is HTBrowse . The application is called by the operating system, and manages the overall running of the program. It asks the navigation module to load the default page. Navigation The module which acually loads documents is based in HTAccess.c. This uses all the protocol modules . Given an anchor ID to jump to, it asks the anchor object for the address in order to load it. History This module records and replays on request the documents which the user vists. Format manager The format manager uses the parser modules to load the document as appropriate. It can also decide on the format of a file from its name. Anchor object The HTAnchor module takes care of creating anchors, managing the links between them and their attributes. This module is independent of the type of graphics object (text, line drawing etc). It stores hypertext addresses of anchors, and ensures that anchors with the same address are the same anchor. ( More ) Protocol modules A protocol module is invoked by the navigation module in order to acce ss a document. Each protocol module is responible for extracting info rmation from a local file or remote server using a particular protocol . Depending on the protocol, the protocol module either builds a grap hic object (e.g. hypertext) itself, or it passes a socket descriptor t o the format manager for parsing by one of the parser modules. File access HTFile.c provides access to files, using HTFTP.c for remote access. The latter uses HTTCP for common TCP routines. HTTP access The HTTP module handles document search and retrieve using the HTTP protocol. News access The NNTP internet news protocol is handled by HTNews which builds a hypertext. Gopher access The internet gopher access to menus and flat files (and links to telnet nodes etc) is handled by HTGopher . WAIS access is implemented in a separate gateway program . Parser modules The parser modules allow different formats to be used to generate grap hic objects. A parser is invoked by the format manager. Currently we only parse HTML and plain text, but obviously other formats can be add ed. HTML Basic hypertext parsing is done by HTML.c which uses the simple SGML engine SGML.c as a basic tokeniser and element stack manager. Plain text This is built directly by the format manager as it is so simple. Graphic objects A graphic object is a (complex) displayable entity. It is built by a p rotocol module directly or using a parser. Graphic objects are in gen eral necessarily coded differently on diferent window systems. The gra phic object is resposible for displaying istelf, catching mouse clicks , and calling the navigation object in order to follow links. We use t he more common term "document" to describe the logical entity which a graphics object represents and displays. Hypertext This object is window-system dependent. In the line mode browser, the GridText module is the hypertext object, providing the generic functionality of HText.h Tim BL Anchors Anchors represent parts of graphic objects which may be the sources or destinations of links. Here follows a general description of thir imp lementation in the WWW architecure. (See definition) An anchor be the source of no, one, or many links. It has one "main" l ink for the (common) case in which it is the source for one link. (In the w3 software, only the main link is normally used - jan 92) An anchor may be the destination of no, one, or many links. The ancho r module stores all links known by the program, and so in fact manages a copy of a small part of the web. There are two types of anchors: Parenet anchors and child anchors. PARENT ANCHORS These represent whole documents. Every graphic object has an associat ed parent anchor. Associated with a parent anchor is data including: The title of the associated document, if known. This allows the document's title to be displayed in lists of previous nodes visited, etc., even when the document itself has been freed. A flag as to whether the document is an index. The address of the document. When a new anchor is created, the code ensures that if an anchor with that address already exists, that that anchor is returned instead, so no duplicates can exist. A list of children CHILD ANCHORS These represent parts of documents. The graphic object stores the cor relation between the id of the anchor and the actual space (time) sha pe which is referred to. Child anchors contain A pointer to the parent. _________________________________________________________________ Utility modules HTChunk A module which manipulates flexibly arrays in memory. See header file . HTAtom A module which generates unique 32-bit pointers representeing strings, for rapid comparison. See header file . HTList A container object for other objects. See header file . HTFont Definition of a font object. See header file . No code. Needs to be changed for different font systems. HTTCP General TCP-IP routines to be used in combination of the socket library. See header file . CBT Browser Operation The WWW browsers operate, in general, in the following manner. DATA FLOW The data flow is demonstrated in a separate diagram . The applicatio n gives the navigation module the home page address. The anchor is gen erated (using the Anchor object ) and passed to the relevant protocol module for loading. In the case of HTTP or a file, a character stream of data is passed to the parser , which build a hypertext object. In the case of news, the protocol module builds the hypertext object itse lf. The hypertext object is built as a stream of text interspersed with st yle changes and anchor start/end points. The parsers create the anchor s, giving their addresses, and just pass the id of the object to the g raphic object. Events such as mouse clicks are picked up by the application, and pass ed to the hypertext object. This may determine that a link should be f ollowed, in which case it invokes the navigation module again, passing the anchor object's id. The naviagation module asks the anchor object for its address, and so loads the next document. Tim BL Authorisaztion in the server The authorization for a HTTP file server running on a unix system uses the underlying file protection scheme. The user and password registration tools are those of the system, so t wo parallel systems do not have to learned by administrators. The daemon may be run under a non-root uid so that it is less prone to pose a security problem due to an obscure programming bug. The daemon will therefore only have access to certain files by virtue of the uid under which it runs. This will provide a certain security. The daemon will itself voluntarily refuse access if the rule file deni es access. The daemon is running in secure mode it will require a user/password p air for any file which does not have public read access. It will che ck that the password is valid, then that the relevant user would have access to the file. Therefore the file must be accessible by BOTH the uid under which the daemon runs, and the authorized user. TheW3Book -- /WWW (4435/4980) Authorized useds may be dummy (non login) users representing groups of people. Administration Every class of user having privileged access will have to be given a ( dummy) user id on the system using the normal tools. Every set of documents requiring a different pattern of access rights should be given a corresponding groups id using the normal tools. The many-many mapping of groups to users (traditionally, in the "/etc/ groups" file) may be used to describe the mapping of the document sets onto the dummy users. Browser The browser maintains a list of server/user/password for each protecte d server that the user has accessed during the session. (Note that the first access of a protected document must necessarily fail, and lead the browser to putting up a panel requesting the user/password pair. A s one wants to send the user/password pair only to the correct server, the browser should not be so constructed as to contain or ask for the triples before actual access).tbl ctb rc CODING STANDARDS This document describes a coding style for C code (and therefore large ly for C++ and Objective-C code). The style is used by the W3 project used so that:- Code is portable and maintainable. Code is easily readable by other project members. If you have suggestions, do send them. (We do not include points desig ned to allow automatic processing of code by parsers with an incomplet e awareness of C syntax.). IF YOU DO NOT ABIDE BY THIS GUIDE YOUR CO DE WILL ANNOY OTHER TEAM MEMBERS AND MAY NOT PORT. The style guide is divided into sections on Language features , Macros , Module header , Modules in straight C , Function header , Code styl e , Identifiers , Code management: the use of CVS , Include files , Di rectory structure , special marks . (See also pointers to some public domain styles ). ___________________ ______________________________________________ Language features Code to be common shared code must (unfortunately!) be written in C, r ather than any objective C or C++, to ensure maximum portability. This section does not apply to code written for specific platforms. C code must compile under either a conforming ANSI C compiler OR an or iginal Kernighan Ritchie C compiler. Therefore, the __STDC__ macro mus t be used to select alternative code where necessary.. ( example ) Code should compile without warnings under an ANSI C compiler such as gcc with all warnings enabled. Parameters and Arguments The PARAMS(()) macro is used to give a format parameter list in a declataion so that it will be suppressed if the compiler is not standard C - see example . The ARGS1 macro is for the declaration of the implementation, taking first the type then the argument name. For n arguments, macros ARGn exists taking 2n arguments each. #endif Do put the ending condition in a comment. Don't put it as code - it won't pass all compilers. #elif Don't use it. Basic cpp doesn't know it. #preprocessor statements DON'T indent any preprocessor statements. They must (for some compilers) begin in column 1. const This keyword does not exist in K C, so use the macro CONST which expands to "const" under standard C and nothing otherwise. -- See HTUtils.h (part of: style guide ) Tim BL Module Header The module header is the comment at the top of a .h or .c file. Inform ation need not (except for the title) be repeated in both the .c and . h files. Of course History sections are separate. See a dummy example . Note:- Heading To make it easy to spot the file in a long listing, put a header and te file name in the top right-hand corner. Authors Just a list to make the initials intelligible. Use initials in the history or in comments in the file. History A list of major changes of the file. You do not need to repeat information carried by a code management system or in an accompanying hypertext file. Section headings Sections in the file such as public data, private module-wide data, etc should be made visible. Two blank lines and a heading are useful for this. Tim BL /* Foo Bar Module foobar .c ** ============== ** ** Authors: ** JB J. Bloggs, ACME Widget Company, TX, USA ** JD J. Doe, University of Tamahalahula, Samabaria ** ** History: ** Jan 1983 First wriiten as a widget sorter (JB) ** 23 Nov 1986 Converted into mangle worzle sorter (JD) ** 38 Dec 1992 Bug fix: Used to carsh on null worzles. (JD, J B) ** ** Copyright: ** CERN copyright -- See Copyright.html */ /* Global Data ** ----------- */ Tim BL Function Headings This style concerns the comments, and so is not essential to compilati on. However, it helps readability of code written by a number of peopl e. Some of these conventions may be arbitrary, but are none the less u seful for that. FORMAT See a sample procedure heading . Note:- White space of two lines separating functions. The identifier of the function right-justified to make it easy to find when flicking through a listing The separate definitions for standard and old C. The macros PUBLIC and PRIVATE (in HTUtils.h ) expand to null and to "static" respectively. They show that one has thought about whether visibility is required outside the module, and they get over the overloading of the keyword "static" in C. Use one or the other. (Use for top level variables too). ENTRY AND EXIT CONDIDTIONS It is most important to document the function as seen by the rest of t he world (especially the caller). The most important aspects of the ap pearance of the function to the caller are the pre- and post-condition s. The pre conditions include the value of the parameters and structures they point to. Both include any requirements on or changes to global data, the screen, disk files, etc. _________________________________________________________________ Tim BL FUNCTION HEADING: DUMMY EXAMPLE } /* previous_function() */ /* Scan a line scan_l ine() ** ----------- ** On entry, ** l points to the zero-terminated line to be scann ed ** On exit, ** *l The line has null termintors inserted after ea ch ** word found. ** return value is the number of words found, or -1 if error. ** lines This global value is incremented. */ PRIVATE int scan_line ARGS1(const char *, l); { /* Code here */ } /* scan_line() */ Tim BL Function body layout With the body of functions, this is the way we aim to do it...we're not religious about it, but consistency helps. If you think your way is smarter, you may be right but this is the way we do it. Whatever you do, NEVER make global changes to the indentation etc in a file without agreement from me first. (It screws up diffs and cde management as well as people). INDENTATION Put opening { at the end of the same line as the if, while, etc which affects the block; Align the closing brace with the START of that opening line; Indent everything between { and } by an extra 4 (FOUR) spaces. Never indent preprocessor instructions (#ifdef, etc), nor increase the indentation level on account of preprocessor instructions. Block comments should start with /* in column 1, end end with */ in column 1. In between, having ** in column 1 is conventional but not necessary for very large comments. Comment the closing braces of conditionals and other blocks with the type of block, including the correct sense of the condition of the block being closed if there was an "else", of the function name. For example, if (cb[k]==0) { /* if black */ foo = bar; } else { /* if white */ foo = foobar; } /* if white */ } /* switch on character */ } /* loop on lines */ } /* scan_lines() */ _________________________________________________________________ Identifiers When chosing identifier names, TheW3Book -- /WWW (4688/4980) Macros should be un upper case entirely unless they mimic and replace a genuine function. External names should be prefixed with HT to avoid confusion with other projects' code. Wthin the rest of the identifier, we use initial capitals a la Objective-C (e.g. HTSendBuffer). The macro SHORT_NAMES is defined on systems in which external names must be unique to within 8 characters (case insesitive). If your names would clash, at the top of the .h file for a module you should include macros defining distinct short names: #ifdef SHORT_NAMES #define HTSendBufferHeader HTSeBuHe #define HTSendBuffer HTSeBuff #endif (back to Overview) ___________________________________________________________ Directory structure This is an outline of the directory structure used to support multiple platforms. All code is under a subdirectory "Implementation" at the appropriate point in the tree. All object files are in a subdirectory Implementation/xxx where xxx is the machine name. See for example WWW/LineMode/Implementation/*. Makefiles in the system-specific directories incldue a CommonMakefile which is in the parent Implementation directory (..). Include Files MODULE INCLUDE FILES Every module in the project should have a C #incldue file defining its interface, and a .c source file (of the same name apart from the suffix) containing the implementation. The .c file should #include its own .h file. A .h file should be protected so that no errors occur if it is #included twice. An interface which relies on other interfaces should #include those interface files. An implemention file which uses other modules should #include the .h file if it not already #included by its own .h file. COMMON INCLUDE FILES These are all in the WWW/Implementation directory. HTUtils.h Definitions of macros like PUBLIC and PRIVATE and YES and NO. For use in all .c files. TheW3Book -- /WWW (4752/4980) tcp.h All machine-dependent code for accesing TCP/IP channels and files. Also defines some machine-dependent bits like SHORT_NAMES. WWW.h Project-wide definition of constants, etc. (See also: Style in general , directory structure ) Macros used in W3 code These are macros which make it less tedious to program in portable C. They are defined in HTUtils.h . See Language features . ARGS1 Generates the format parameter list for a function heading. Expands to either an ANSI or non-ANSI argument declaration list. Note in order to do this, it has to take one argument for the type and one for the argument name. This is messy but necessary. See example, also NOARGS, ARGSn, PARAMS . ARGSN As ARGS1 , macros ARGS2, ARGS3 etc up to ARGS9 generate arguments lists for function headings with different numbers of arguments. See also NOARGS, PARAMS . These should be used for private function definitions. If the function is PUBLIC (not static) then there will already be a function prototype in the header file, so a K&R style function definition may be used instead of ARG CONST This is used instead of the keyword "const" which does not exist in K&R C. It expands to const when using ANSI C, otherwise nothing. NOARGS Generates the argument list for a function header with no arguments.. See also ARGSn, NOPARAMS. NOPARAMS This is used to generate the formal parameter list in a function declaration having no parameters. It expands to (void) for ANSI C and () for non-ANSI. See also: PARAMS PARAMS Define formal parameter list in function declaration. The normal ANSI function header is given enclosed in DOUBLE parentheses. This causes the whole list to be replaced with just () when a non-ANSI compiler is used. See also: NOPARAMS PRIVATE This expands to "static". It is used in a function or variable declaration to signify that the function is private to the module, rather than public (the default). Avoids the confusion in C between "static" used in this sense, and used to declare storage variables within blocks as static as opposed to automatic. PUBLIC The alternative to PRIVATE. expands to nothing. One or other of PRIVATE or PUBLIC should be used to ensure that the programmer has thought about it. This prevents unnecessry accidental public definitions which may later clash with other people's routines. STRALLOCCOPY This is a safe string assignment function. It takes as arguments two pointers. If the first is non-zero, it frees the block it points to. It then allocates a block of size sufficient for the zero-terminated string pointed to by the second, and assigns the first pointer to point to that block. Implemented nowadays in terms of a function. Note that unlike the function, StrAllocCopy takes its first pointer by name not address (p, not ). (Part of style guide ) Tim BL Rules for straight ANSI C: ASSUMPTIONS Object oriented programming classes are not supported. A PROGRAM A program's executable code is made up of global variables, knowable to all code. several modules of routines and corresponding private global variables, some of the routines and/or variables may be exported to other modules. a main routine, which starts the program and normally exports nothing. SOURCE CODE FILES For maintenance, the source code is written as: one or more ".d" files containing definitions (no variables nor routines!) A ".d" file never generates code. The set of ".d" files implements a hierarchical tree of dependencies between the definitions. one or more modules, each consisting of a ".h" file and a ".c" file. A ".c" file always generates code. A ".h" file defines only what the module exports to others, and never generates any code. Its definitions may however depend on ".d" files. one main routine file, which is ".c" and possibly has no ".h" file (it normally exports nothing). SIMULATING MODULES WITH INCLUDE FILES We try to simulate modules with include files. As stated above, the so urce of a module consists of: a ".h" file with the exported constants, variables and routine headers, a ".c" file with the private constants and variables and the code of all routines. The ".h" file is included by other modules, which must see the variabl e as extern during their compilation. The exporting module also needs the declarations, but without the "extern" keyword. Therefore either t he declarations occur again inside the module's ".c" file or we find a mechanism of re-using the ".h" file. Writing the declarations twice m eans a maintenance problem when changes happen. The following mechanism is proposed for including ".h" files: a declaration of an exported variable is preceded by the word "PROVIDE". every ".h" file uses #include <Importation.d> in order to define "PROVIDE" to be either null or "extern" depending o n whether the word "IMPORT" is defined. If a variable is initialised, then its initial value is given as INITIALLY(initial value); (no equal sign!). Example for initialising exported integer i to 5: PROVIDE int i INITIALLY(5); The file Importation.d contains: #undef PROVIDE #ifdef IMPORT #define PROVIDE extern #define INITIALLY(s) #else #define PROVIDE #define INITIALLY(s) = s #endif Thus, in a module A which imports from a module B, the ".c" file starts with: #define IMPORT #include "B.h" ... #undef IMPORT But module B's ".c" file has the line #include "B.h" outside of "#define IMPORT" / "#undef IMPORT" brackets. The ".h" file of a module never imports from other modules (but it can of course use #include for definitions). TheW3Book -- /WWW (4942/4980) GLOBAL VARIABLES Because they are instantiated as code, they are written as a module. Because there are only variables, the ".h" file contains all the declarations and the ".c" file is empty apart from an include for generation of the code. The ".c" file is simply: #include "Globals.h" THE MAIN ROUTINE The main routine's ".c" file looks like: #include <... .h> #include <... .h> #define IMPORT #include "Definitions.d" #include "Globals.h" #include "A.h" #include "B.h" #include "... ".h"" #undef IMPORT void main() { ... ... } Use of CVS HISTORY of introducing CVS in WWW code management SHORT DESCRIPTION Code management in WWW is done with tools to be found in the directory hypertext/WWW/CodeManagexec. CVS (Concurrent Versions System) is installed there, with all its sources. CVS keeps all versions and all branches of a particular source file "f.c" in a single file "f.c,v". This file has a structure which uses the differences from version to version, so that it takes up much less space than the corresponding set of versions. Access to versions is through a command "check out" and a new version is incorporated into this file through a command "check in". The set of all ",v" files used for an application is called its "repository". The repository is structured as a hierarchy of directories, just like the sources for the application would be. You never touch the repository except through commands given to CVS. Normally you will start work by checking out of the repository a complete set of source files into your private work space. CVS will extract the sources and create the necessary directory structure in your space. Development can then go on independently of any other person working on the same version: each developer has his own private copies. When you have work ready, you will check in your files: CVS will incorporate the changes of all modified files into the repository as a new version. More important: when a different developer has checked in some work (e.g. a bug fix), you can incorporate that into your current work *without* having to check in, by the "update" operation. Developers synchronise once in a while through updates. There are also various means of identifying important versions. WWW INSTALLATION The directory CVSROOT is where CVS stores its bookkeeping files. The directory CVSRepository is where CVS stores the sources in the RCS format, i.e. one file for all versions and branches of each source file. Please do not touch CVSROOT nor CVSRepository, they are to be maintained by use of CVS commands only. CVS MANUAL The CVS and RCS manuals are in the web. (Note: Terry Hung introduced it at SLAC, but I believe we got it from the "standard" sources: prep.ai.mit.edu.)