This document is also available in these non-normative formats: XML.
This document discusses the deployment of protocols and URI schemes on the World Wide Web. Implications for correct configuration of user agents, servers and proxy gateways are discussed. Guidelines are also provided for deciding whether a new protocol or scheme is merited, and for maximizing interoperability of new protocols with those that are already deployed.
This document is an editors' copy that has no official standing.
This document has been produced by the W3C Technical Architecture Group (TAG) to address TAG issue schemeProtocols-49. An earlier draft of this finding was circulated for discussion in June of 2005. This draft is intended to incorporate insights gained from those discussions, and to deal with some of the concerns raised.
This document builds on and complements information from [RFC 2717] and [RFC 2718]. At the time of this writing, an Internet Draft has been submitted that would revise and subsume both of those RFCs (see [RFC 2717bis]). The IETF has approved that Internet Draft for publication as an RFC, which should happen before this finding is formalized by the TAG. Accordingly, it is anticipated that this finding will reference the new RFC.
The terms MUST, SHOULD, and SHOULD NOT are used in this document in accordance with [RFC 2119].
3.1 Scenario: an existing http resource adds video capability
3.2 Scenario: Accessing http resources using a peer-to-peer protocol
3.3 Scenario: Accessing https resources using a peer-to-peer protocol
3.4 Scenario: Configuring browsers to use the right protocol
3.5 Scenario: A new scheme to launch a media player
3.6 Scenario: Configuring gateway proxies
3.7 Scenario: Web services gateway scenario
4 Rules and Guidelines
4.1 Rules for Servers
4.1.1 R1. Serve using any protocol
4.1.2 R2. Serve faithfully
4.1.3 R3. URI scheme sets expectations for integrity of access
4.2 Guidelines for Servers
4.2.1 G4. Serve using the protocol associated with the URI scheme
4.2.2 G5. Deploy new protocols using existing schemes
4.2.3 G6. Provide protocol support for the http scheme
4.2.4 Discussion of server guidelines
4.3 Rules for User Agents
4.3.1 R7. Try any protocol for any resource
4.4 Guidelines for User Agents
4.4.1 G8. Access using the protocol associated with the URI scheme
4.4.2 G9. Don't attempt access with untrustworthy protocols
5.1 The relationship of operations to schemes and protocols
5.2 Consistency of operations across protocols
5.3 Protocol operations vs. API operations
The purpose of this finding is to clarify the relationship between URI schemes and network protocols on the World Wide Web. A short list of rules and guidelines is provided for servers that support resources, and for the user agents that access those resources. Also included is a section containing scenarios which illustrate the application of the rules in practical situations.
Of the many ways in which the Web can be extended, the provision for new URI schemes may be the most fundamental. Establishment of a new scheme can enable access to information that was not previously available to the Web, or can be used to suggest use of a particular network protocol for manipulating resources. Precisely because so many aspects of resource naming and interaction are subject to change through introduction of URI schemes, new schemes can undermine the interoperability of the Web. [AWWW] explains why unnecessary proliferation of URI schemes must be avoided.
Some URI schemes, such as http and ftp, are associated with network protocols. Originally the associations between such schemes and protocols were 1-to-1: the http scheme was for use primarily with the HTTP protocol, the ftp scheme with the FTP protocol, and so on. Indeed, such associations are the basis for the common practice in which user agents such as browsers select protocols based on the scheme name. Unfortunately, there are circumstances in which adherence to this 1-to-1 rule would unduly limit the growth or utility of the Web, and Web Architecture does not in all cases mandate such a strict correspondence. Indeed, it is already common practice to use the http scheme with two related protocols, HTTP 1.0 and HTTP 1.1. Accordingly, the purpose of this finding is to give guidance on the deployment of schemes and protocols, the application of scheme names when a given family of protocols is revised (e.g. from HTTP 1.0 to HTTP 1.1), use of scheme names where the same resource is to be accessible using multiple protocols, and conventions for using gateways to relay information from one protocol to another.
[RFC 2718] sets out guidelines and caveats for the creation of new URI schemes. This finding provides complementary information relating to the deployment of resources using such schemes, the design of protocols, the choice of operations to be supported, and the suggested behavior of user agents. Indeed, this finding for the most part avoids restating principles already covered in [RFC 2717] and [RFC 2718]; readers are encouraged to become familiar with them before proceeding.
This finding is mainly concerned with the relationship between URI schemes and protocols. In the examples used, scheme names are lowercase, and protocol names are uppercase. Thus, http is a scheme and HTTP refers to (either version of the) HTTP protocol.
This finding is intended to cover a broad range of information formats and protocols including client/server, peer-to-peer, streaming multimedia, multicast, etc. For convenience, the following client/server-oriented terminology is used except in cases where the extension to other protocols is unlikely to be clear:
In the case of HTTP, state is exchanged in the form of media-typed resource representations, but other protocols may implement other abstractions or use different formats or typing systems.
This finding frequently refers to the set resources named by URIs from a particular scheme such as http. For brevity, such resources are collectively referenced by their schemes. The resources named by http://example.org/res1 and http://example.org/res2 are thus both referred to as "http resources". Also for brevity, the distinction between a resource's state and representations of that state is ignored where not significant. Thus, this finding speaks informally of "retrieving the state" of a resource, when in fact what is retrieved is a representation of that state.
These sections describe some hypothetical scenarios to illustrate particular issues or challenges involving URI schemes and protocols. Good practice for each scenario is explained, along with cross references to the pertinent rules and guidelines provided later in this finding.
An http resource named http://example.org/todaysNews is deployed using HTTP. It is a popular source of news, and is widely bookmarked by users of the Web. HTTP is used to provide representations of the news reports in one or more traditional media types such as application/xhtml+xml, text/plain, application/pdf, etc. The video industry creates a new VIDEO protocol for managing high quality video on the Web. It provides streaming and frame-rate control capabilities beyond those available with HTTP, and is in any case a completely different protocol. The owner of http://example.org/todaysNews wishes to provide a high quality video feed of the news resource, while continuing to support HTTP access for the traditional media types. At least three approaches are licensed by this finding:
Approach 1: a new "video" scheme is created by the inventors of the VIDEO protocol. The owner of http://example.org/todaysNews creates a second URI such as video://example.org/todaysNews, which is used for references to the video feed (see G4. Serve using the protocol associated with the URI scheme). Browsers follow the common practice of choosing protocol handlers based on the scheme (see G8. Access using the protocol associated with the URI scheme), correctly choosing HTTP for the original URI and the VIDEO protocol for the new one. Unfortunately, existing bookmarks do not successfully access the video feed, emails pointing to the news feed might require two links, semantic web statements about one resource do not in general apply to the other, and so on.
Approach 2: the VIDEO protocol supports operations on existing URI schemes including http (see G5. Deploy new protocols using existing schemes and G6. Provide protocol support for the http scheme). Browsers have the option to dereference the existing bookmarks using either protocol, but no standard means is established for choosing which protocol to try first (see R7. Try any protocol for any resource). A given browser might set a fixed choice using configuration options, might provide a per-link selection in the user interface, or might after an initial successful interaction default to using the same protocol for subsequent references. This approach has the advantage that there is only one URI for the resource, so existing bookmarks and Semantic Web statements work. It has the disadvantage of complicating the configuration of browsers, and of failing to honor the tradition that URI schemes designate protocols or protocol families.
Approach 3: an initial request is in all cases made using the HTTP protocol. The result is either a representation of the resource in some traditional media type such as application/xhtml+xml, or a video deployment descriptor in some new media type such as application/videoDeploymentDescriptor+xml. As is customary, the browser triggers a handler based on the media type, and the handler for application/videoDeploymentDescriptor+xml in turn accesses the video stream itself using the VIDEO protocol. (This approach is discussed in G6. Provide protocol support for the http scheme). Like approach 2, this one also has the key advantage that existing links work and each resource can be referenced with a single URI. It has the further advantage that the means by which protocols are activated is covered by normative specifications, rather than by unspecified logic that might vary from one user agent to another, and it preserves the convention that the http scheme is directly associated only with protocols in the HTTP family (currently HTTP 1.0 and HTTP 1.1). The principle drawbacks of approach 3 are that it requires the availability of an HTTP implementation even to access the video stream, and more importantly, that it requires a round-trip to the HTTP server before even attempting to access the VIDEO server.
The rules and guidelines in this finding license all three approaches. In examples such as the one described here, approach 3 is preferred whenever the need for HTTP access and the associated round-trip delays are acceptable. Approach 1 may be desirable in situations where the uses of the new protocol and the old mostly for distinct resources. In such cases, approach 1 preserves the tradition that both user agents and users can infer the protocol from the URI scheme.
A government committee is appointed to investigate a leading political figure, and the committee's report is to be published on the web as an XHTML document named with a URI in the http scheme. Past experience suggests that the servers will be subject to millions of "hits" during the first few hours after publication, a load beyond the capability of the available servers using HTTP. Accordingly, the document is also to be published using a scalable peer-to-peer protocol.
Rule R1. Serve using any protocol makes clear that an http resource can in principle be published using protocols other than HTTP, but rule R2. Serve faithfully requires that good faith efforts be made to serve resources accurately and reliably. In other words, the publisher has the responsibility to ensure that the protocols used will in fact serve correct representations of the intended resource. Many peer-to-peer protocols have the characteristic that the content they serve is based on a distributed agreement among those using the network. For example, in some networks the representations served for a resource may be determined by voting among all representations that have been offered by any user of the network. In the case of a controversial document such as the investigative report, there might be a risk that erroneous or "spoofed" copies would be inserted to corrupt the system. Thus, the P2P protocol is appropriate only to the extent that the publisher can in good faith rely on it to faithfully serve the resource.
Consider the scenario just discussed, but modified so that the investigative report is to be named with an https URI. To a significant degree, the considerations remain the same, but R3. URI scheme sets expectations for integrity of access explains that the scheme chosen can set expectations for the integrity of access to resources. In this regard, the two scenarios are different.
The https scheme is a URI space in which resolutions must be authenticable (is this a word??), typically using Public Key Infrastructure. The P2P protocol is thus suitable only insofar as it enforces such integrity checking in a manner that is true to the spirit of https resource resolution. In principle, the same problem exists for the http-based scenario, but the expectations for http integrity are lower. Note that in some cases, checksums or other out-of-band mechanisms can be used to ensure the integrity of accesses to resources even over otherwise unreliable networks; the checksums do not guarantee that correct results will be received, but they can greatly reduce the chances that incorrect information will be trusted.
The scenarios above make clear that multiple protocols may on occasion be used to access resources from a given URI scheme. This scenario involves configuring a browser or other user agent to use the correct protocol(s) for accessing resources.
Q.If the scheme does not uniquely determine a protocol, then how is it that user agents know what protocol to use?
A.Guideline G4. Serve using the protocol associated with the URI scheme makes clear that the tradition of associating a scheme with at most one protocol, or failing that with a set of versions of the same protocol, is important. Thus G8. Access using the protocol associated with the URI scheme establishes the heuristic that, unless otherwise specified, user agents should use (a suitable version of) the protocol associated with a given scheme. Thus, the default for accessing an ftp resource should be to use the FTP protocol, and so on.
The means by which a user agent selects among multiple versions from the same family is not specified by Web architecture. Indeed it is common, if not entirely appealing, to find expert-level user agent switches to control whether HTTP 1.0 or HTTP 1.1 should be used for http resources. Similarly, as discussed above, configuration controls are likely to be needed in cases where protocols from different families are used to access resources in the same scheme.
A provider of multimedia on the web wishes to use its own software for "playing" content in user agents. The content is delivered using the HTTP protocol, but named using a new media: scheme. Browser plugins are registered to handle the new scheme, to render the desired user interface, and to use HTTP to retrieve content as necessary.
This is an inappropriate use of a new scheme. As discussed in [AWWW], scheme names, especially those that are short, easy to remember, and easy to type, are a scarce resource on the web. Having been allocated for the convenience of the example media player, the media: scheme is unavailable for any other use. Schemes are properly introduced to enable Web access to new information spaces and sometimes to enable new network protocols; new schemes should not be allocated merely to facilitate invocation of user-agent software. As discussed in 4.2.3 G6. Provide protocol support for the http scheme, a better approach would be to name the media resources using the http scheme, and to return either the media files or descriptors of them using an identifiable media type such as application/mediaDescriptor+xml. User agent software is generally capable of dispatching handlers based on the media type returned from an HTTP GET.
Although the HTTP protocol is primarily used for http resources, it can be and often is used with others as well. For example, HTTP is commonly used as a gateway protocol to access ftp resources. Often, an intermediate server known as a gateway proxy is used to convert between the two protocols:
Q.Should the ftp URI be used as the HTTP Request-URI, or should an http-scheme URI be created for each ftp resource to be accessed?
A.R1. Serve using any protocol makes clear that an ftp resource can in principle be served using HTTP, and indeed HTTP provides for ftp- or other-scheme URIs to be carried as Request-URIs. Using the same URI on both "hops" avoids the need to create two names for the same resource, but with the drawback that user agents may require special configuration to access ftp resources using HTTP. Rule R7. Try any protocol for any resource establishes that user agents have permission to access an ftp resource using HTTP, but there is no standard means of establishing when to do so.
In contrast, allocating a second http URI for each resource preserves the illusion of an "all http" world for the many software tools supporting that scheme, and in particular user agents can use the traditional heuristic of choosing a protocol based on the URI scheme (see G8. Access using the protocol associated with the URI scheme).
The gateway can thus be implemented using either of these conventions. In general, the [AWWW] admonition in to avoid multiple names is strong, suggesting that when in doubt the first option should be chosen.
[To be supplied - this will be another gateway scenario, similar in physical arrangement to the one above, but the role of ftp will be played by a robust queuing system such as WebSphere MQ. We assume for the scenario that the queuing service has its own scheme queue: for the resources that it manages. A web service (on the far right) will "mint" an Endpoint Reference to a resource stored in the server, but clients on the left will send messages to that resource using HTTP. The Web Services gateway in the middle will translate from http to the queuing system. The scenario will explore the question of whether the [address] property in the endpoint reference MAY/MUST/SHOULD/MUSTNOT be in the queue scheme. In practice, the reasoning is no different than for the FTP gateway, but the applicability to Web Services may be interesting. I'll add it later if the rest of the finding meets with approval -- Noah]
This section sets out rules and guidelines for using scheme names and protocols on the Web. Rules are fundamental principles that apply in all cases. Guidelines represent what is often good practice, but may not be universally applicable. Indeed, in some cases the guidelines are to some extent contradictory in attempting to simultaneously achieve desirable but mutually inconsistent characteristics on the Web. In such cases, the tradeoffs involved are discussed.
The following rules apply to servers:
A server MAY offer representations or operations on a resource using any protocol, regardless of URI scheme. For example, a server might choose to respond to HTTP GET requests for an ftp resource. Of course, this is possible only for protocols that allow references to URIs in the given scheme.
A server MUST serve resources faithfully. Regardless of the protocol used, the server is responsible for ensuring that the correct resource is accessed, that operations are correctly implemented according to the specifications for the protocol, and thus that the correct resource state is either retrieved or updated. In the example above, serving the ftp resource using HTTP is appropriate only insofar as the operations of HTTP can in good faith be supported on that particular resource. The server must have access to the state of the resource, using an FTP gateway or by some other means, and it must ensure that all supported HTTP operations properly retrieve or update the state of the resource.
For many combinations of scheme name and protocol, such faithful serving is by definition impossible. For example, there is probably no faithful manipulation of a mailto: resource possible using the FTP protocol. Thus, this rule prohibits use of many combinations of schemes and protocols.
Servers MUST (SHOULD??) use protocols that meet the integrity expectations established by the scheme.
For example, the http and https schemes represent URI spaces which are similar in structure, but which differ in expectations for the robustness with which resources are accessed. Thus, it is typically inappropriate to serve an https resource using the HTTP protocol. Although all the required operations are supported, HTTP is vulnerable to DNS cache polution and other sources of misconfiguration, whereas HTTPS typically uses Public Key Infrastructure to verify the authenticity of the server and thus indirectly of the resource. Thus an https resource should be served only using protocols that support the PKI-based integrity expectations established by the scheme.
Rules R1. Serve using any protocol, R2. Serve faithfully and R3. URI scheme sets expectations for integrity of access establish that a resource may be served using any protocol as long as the appropriate integrity expectations can be supported. This guideline (G4) establishes a strong preference for honoring the traditional associations between schemes and protocols. Where practical, ftp resources should be served using the FTP protocol, http resources using (a version of) the HTTP protocol, and so on.
Although new protocols have traditionally been associated with new URI schemes, there are also advantages to supporting new protocols using existing schemes. In particular, the URIs of existing resources may be widely known, may have been bookmarked or otherwise recorded, or may have been the subject of semantic web assertions. One alternative for using new protocols with such resources would be to generate new URI names for each of them, but Web Architecture [AWWW] explains that there are serious drawbacks to naming the same resource with more than one URI. By supporting existing schemes it becomes possible to use new protocols with existing resources, or in general to decouple the choice of protocol from the naming of a resource.
When new protocols are created, serious consideration should be given to supporting the http scheme, either directly in the new protocol, or by providing for use of HTTP to retrieve a media-typed descriptor that in turn signals use of the new protocol for further interactions. By supporting the http scheme, new protocols can in principle be used to serve the hundreds of millions of resources that are already named within that scheme.
The above guidelines are to some degree contradictory, which is among the reasons they are not stated as firm rules. The traditional practice of using a new scheme for each protocol greatly facilitates the selection of protocol handlers by user agents, and it provides in the URI a strong hint as to how the resource may be accessed. For the reasons stated above, supporting existing schemes with new protocols facilitates the use of those protocols with existing resource, but requires that user agents must find other means (not specified here) to choose protocols. Those creating new protocols and those who administer servers using such protocols are encouraged to weigh these conflicting considerations in deciding which schemes to support and which to use with particular resources.
The following rule applies to user agents
A user agent MAY to attempt to access any resource using any protocol. Insofar as networks are free from misconfiguration and tampering, any response received is by definition authoritative, regardless of whether the scheme is one traditionally associated with the protocol. (Note that R2. Serve faithfully enjoins servers from responding except on behalf of resources that are indeed served authoritatively.)
No standard means is provided for determining when a new protocol is available for use wiht a resource named with a scheme usually associated with another protocol. As noted in G5. Deploy new protocols using existing schemes, one effective approach is to retrieve a descriptor using the older protocol (typically HTTP — see G6. Provide protocol support for the http scheme ) and to use the media-type or content of that descriptor to trigger use of the new protocol for further interactions.
The following are guidelines for user agents:
Although R7 allows for any protocol to be used with any resource, G4. Serve using the protocol associated with the URI scheme makes clear that many resources will be deployed using the protocol(s) associated with their schemes. Thus, user agents SHOULD in most cases use the scheme name as the basis for selecting a protocol.
Where one scheme is associated with multiple protocols, Web Archicture mandates no fixed means by which user agents choose among them. For example, most browsers provide configuration options to determine whether HTTP 1.0 or HTTP 1.1 should by default be used to access http resources.
Note that rule R7 and guideline G8 are not contradictory. The rule specifies that it is safe for a user agent to attempt to manipulate resources using any protocol that has appropriate characteristics; the guideline suggests that the common practice of associating schemes with protocols should be exploited wherever practical.
Rule R7. Try any protocol for any resource allows user agents to attempt access to any URI using any protocol, but there is an important caveat to be considered: integrity guarantees are enforced by the protocol, not the scheme. Although R3 (see R3. URI scheme sets expectations for integrity of access) ensures that a properly configured server will not in general support access to an https resource using HTTP, for example, a misconfigured or compromised system may direct HTTP requests to an incorrect or malicious site. Accordingly, it is usually inappropriate for a user agent to request access to a resource using a protocol that does not provide integrity guarantees consistent with those mandated by the scheme. This is not stated as a rule, because the user agent is presumably aware of any such risks, and is not prohibited from attempting such an access if the guarantees are not in fact desired, or if the network used is known to be trustworthy.
Protocols typically support specific operations. HTTP, for example, provides the familiar GET/PUT/POST/DELETE/HEAD, etc. The following sections discuss the relationship between such operations and the naming of resources with URIs. A subsequent section gives guidance on designing operations for new protocols.
User agents often are called upon to provide consistent user interfaces for resources accessed using a variety of protocols. Although each protocol can in principle support its own operations, consistency is easier to maintain when protocols support the same or similar operations. It is highly desirable, for example, that any new protocols used for information retrieval support operations as close as possible to HTTP GET. This is an example of the Principle of Orthogonality (see [AWWW]). So, a high definition streaming video protocol may not support exactly an HTTP "Get", but if it supports something similar then a browser can provide a consistent navigation experience as one goes from a web page to a movie and back. Similarly, if a peer-to-peer protocol supports retrieval of media typed octet streams, then browsers can use existing renderers, caches, etc.
In HTTP, GET is visible both as a browser operation and on the wire. When a link is clicked, the browser typically invokes a local operation that results in an HTTP "GET" message being transmitted to the server. The distinction between local and wire-level operations becomes important when new sorts of protocols are integrated into the Web. Many peer-to-peer networks support information retrieval that is very close to the service provided by HTTP GET, but the resulting wire-level messages are often very different. In a peer-to-peer network, the messages resulting from such a retrieval may be sent to multiple servers, and may or may not directly relate to the information being retrieved. For example, in some peer systems, the messages will have to do with negotiating caching policies, bandwidth commitments, etc. The data actually received may multiplex information from multiple resources. Thus, in considering the compatibility goal set out in 5.2 Consistency of operations across protocols, it is often the abstract API-level operations for which consistency is important. In most cases, the messages transmitted "on the wire" are private to the protocol.