W3C WD-http-pep-960820

HTTP/1.2 Extension Protocol (PEP)

W3C Working Draft 20-Aug-96

This version:
http://www.w3.org/pub/WWW/TR/WD-http-pep-960820.html (To be submitted as Internet Draft: Text, HTML)
Previous versions:
http://www.w3.org/pub/WWW/TR/WD-http-pep-960819.html (Internet Draft: Text, HTML)
http://www.w3.org/pub/WWW/TR/WD-http-pep-960222.html (Internet Draft: Text, PostScript)
Latest version:
Rohit Khare <khare@w3.org>

Status of this memo

This is a W3C Working Draft for review by W3C members and other interested parties. It is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress". A list of current W3C working drafts can be found at: http://www.w3.org/pub/WWW/TR

Note: since working drafts are subject to frequent change, you are advised to reference the above URL, rather than the URLs for working drafts themselves.


PEP is a system for HTTP clients, servers, and proxies to reliably reason about custom extensions to HTTP. Traditionally, HTTP agents offer extended behavior by private agreement using additional message headers. PEP has features for expressing the scope, strength, and ordering of such extensions, as well as which extensions are available.

PEP as described here is substantially simpler than previous drafts. The title has also been changed, since PEP addresses a chartered HTTP WG work item.

1. Introduction

HTTP [3] messages, like most applications of RFC 822 [4], can be extended with additional header fields and content formats. Traditionally, the standards update process itself extends the protocol for major, global new features. However, this approach is problematic for deploying casual experiments which could scale up from pairwise to universal deployment, especially as HTTP agents become more modular, dynamic, and diverse. Casual extension of HTTP is hindered by several challenges:

  1. Naming. To guarantee interoperability, header names, encoding tokens, types, response codes, etc. must be centrally arbitrated by appealing to IETF standards-track documents or the Internet Assigned Numbers Authority (IANA). This requires more effort from implementors than an extension may warrant.

  2. Importance. There is no way to indicate the consequences of ignoring an extension: can an agent silently ignore it, may it warn the user, or must it return an error message?

  3. Scoping. There is no reliable way to indicate who is to process an extension: the next HTTP agent (proxy), or only the origin agent (client or server)? 1.1's Connection is only a partial solution, since it cannot convey the criticality of stripped headers.

  4. Ordering. There is no way to indicate in what order to process extensions. If several are applied, it can be critical to 'unpack' the message in the correct order.

  5. Initiation. Short of unilaterally utilizing an extension, there is no way to request the other party to initiate it.

  6. Advertising. Though it is possible to enumerate content variants and the headers they differ upon, it is not possible to describe available extensions in detail.

  7. Inquiring. There is no way to inquire if an extension is supported before attempting to use it. This is critical for selecting among related extensions and upgrading to new versions.

PEP introduces the concept of protocol extensions to systematically address these issues at a level of abstraction above header names and content-types (hence, `Protocol Extension Protocol'). With PEP, HTTP agents can interoperate correctly with known and unknown protocol extensions, select protocol extensions available to both sides, and query partners for specific capabilities. PEP adds to HTTP the extensibility lessons learned from ESMTP (extension naming) [10], IPv6 (unknown-option disposition) [5], and Telnet (option negotiation) [12].

In addition to reliably describing statically extended HTTP servers and clients, PEP will work with dynamically extended agents. Indeed, the authors expect that PEP will drive the deployment of a new generation of exensible agents (such as W3C's Jigsaw server and libWWW reference library).

NOTE: PEP has been developed over the last year by many interested parties, including the W3C/CommerceNet Joint Electronic Payments Initiative (JEPI). Two strongly related drafts are also available from the IETF and W3C: Don Eastlake's Universal Payment Preamble [6] and ``Selecting Payment Mechanisms Over HTTP (or, seven examples of UPP over PEP)'' [8].

1.1 Example: PICS

Consider the example presented by the first deployed PEP application, the Platform for Internet Content Selection (PICS) [11]. PICS describes a format for labeling almost any content. Since it's intended to be a fairly ubiquitous addition to various Internet protocols, PICS appropriates the PICS-Label header for 822 applications, including mail, news, and web content. The spec could not use, say, Label, since there's no way to effectively arbitrate between conflicting uses of header names (PICS was not developed under the IETF standards track).

Even with an eventual dispensation from IANA to use this new header, new issues arise: If a server attaches this new header to label an HTTP response, will a caching server pass it along? Will it act upon the label at all? What kind of feedback will the end-user get when this ``unknown'' header appears? How can the user specify which rating systems to report?

The direct fix to the first few problems is to add a machine-understandable statement about the extension and its new headers. Protocol headers describe the name of the extension, the scope, importance, and any associated data headers. Moving further, though, additional information about the protocol can be added to that directive: which PICS services were employed, etc.

Using PEP in this way also solves the next logical problem: how can a client request a server to initiate vending PICS labels? The same structure for a Protocol can be reused in a Protocol-Request asking the counterparty to start using the specified extension.

In addition, PEP affords additional capabilities, like using Protocol-Query to test if PICS is available and Protocol-Info to advertise it. In all cases, the PICS protocol extension can specify additional parameters, such as which rating systems, services, which kinds of labels, and which other URLs are labelled.

2. The PEP Model

Protocol extensions can modify HTTP messages, most commonly by adding headers, but also by modifying the contents, much like a proxy server might do today. There are four points at which PEP processing occurs in a basic HTTP transaction (see Section 3 for usage descriptions at each step):

  1. Sending an HTTP Request

  2. Receiving an HTTP Request

  3. Sending an HTTP Reply

  4. Receiving an HTTP Reply

During a conversation, either side may invoke a Protocol, issue a Protocol-Request, advertise some Protocol-Info, or test a Protocol-Query. PEP-compliant recipients decode extended messages and reply to each Protocol-Request and Protocol-Query.

Protocol extensions are attributes of a resource rather than an entire server or connection, which was the model in 1.1's Connection and Upgrade fields, and in previous extension proposals [9]. This means each PEP statement is applied to the single resource mentioned in an HTTP message (though there is a facility to hint at which other resources the same statement applies for).

An HTTP agent can begin using a protocol extension (or query for one) because:

Another aspect of the PEP model is that protocol extensions can have negotiable parameters. Unlike previous proposals which model extensions as binary have/don't have features, PEP expects protocol extensions will be able to handle different versions of a particular protocol, invocation of ``compatible protocols'', and the selection of compatible modes from several that might be advertised.

To summarize, PEP as proposed here will allow HTTP/1.2 implementors to make the following kinds of statements:

3. PEP Usage

Before presenting specifics about each of the four PEP directives and the four points where they are processed, here are a few common points:

A bag is an attribute-value list, where bags are allowed recursively as values as well. In PEP, each directive begins with values describing the protocol (making the extension name the top attribute). Some of those values are common to all directives: the strength (str), scope (scope), for list (for) , extension-specific parameters (params), and other headers associated with this extension (headers). The formal BNF is included in Section 3.5.1.

Strength and scope are important common concepts. Any directive can be ignored, but strength specifies whether an error should be generated as a result. Similarly, scope defines which agents should handle the directive at all; either the immediate (conn) or final (origin) recipient. The combination of the two concepts makes PEP powerful enough to deploy features that in the past required a minor-version number bump. With PEP, for example, Content-Transfer-Encoding could be described as {str req} {scope conn} for rapid experimentation with compressed headers, sticky headers, etc.

A word about the for list: Since each PEP directive applies only to the resource at hand (rather than, say, the entire server or all resources of a given type), a hint as to which other resources the directive applies to will prevent negotiation round-trips each time. Thus, many PEP applications could include {for /*} to apply to an entire server. Remember, there already are examples in HTTP/1.1 of features which only apply to particular resources, such as Range.

When fetching a URL, a client will typically search through its database of hints to find all applicable directives before sending an HTTP request. Of course, an agent which does not want to use hints can ignore them: if the for list includes the URI of the current message, just throw away the list; if it does not, just throw away the entire directive.

Finally, note that the Protocol-Query / Protocol-Info pair describes hypothetical capabilities of agents and resources but that the Protocol-Request / Protocol pair has immediate consequences for the current HTTP transaction. Each pair hangs together and must be implemented together, though only the latter pair is mandatory.

3.1 Protocol-Query

The first statement in a conversation might be to ask if the counterparty supports an extension at all. The Protocol-Query header lets one agent ask the other if a particular extension is supported; the response comes back in a subsequent Protocol-Info.

A query directive specifies its scope, but not a strength, which defaults to opt; that's because a query always asks optionally for more information, but can neither demand an answer (since query support is optional), nor refuse one.

An agent can query if an extension is supported for an entire list of URIs. This means that an answer is expected when the replying agent sends a message about any resource on the list; and that an answer should reveal as much as can be said about extension support for that URI set. This way, a server can ask for a client to be prepared to use an extension in a part of his site (e.g. UPP in /Catalog/*); or a client can ask a server if an extension applies to all its documents of the same type (e.g. indexed range retreival for *.pdf).

3.1.1 Sending an HTTP Request

If a client believes an extension may be supported, it can attach a Protocol-Query to an outgoing request to test if an extension is supported for that URI (empty for list), an entire server ({for /*}) or something in between.

A client can wait to piggyback its queries on other HTTP transactions or it can explictly issue an OPTIONS request with its queries.

Note that a protocol extension might be defined to have side-effects when it is queried. For example, the UPP payment selection protocol defines that the answer to any queries for UPP should include additional information about particular payment mechanisms. Thus, in a dynamically extended agent an agent should call back the extension to handle a query.

3.1.2 Receiving an HTTP Request

When an agent receives a query, it should respond whether the stated configuration is acceptable or refused for any of the URI(s) specified. Upon receipt of a query, an agent should test if the extension is available and, if so, its response to the proposition. The server should be prepared to send the resulting information in its reply.

3.1.3 Sending an HTTP Reply

In preparing an HTTP reply, a server must respond to queries in the HTTP request, as well as issue its own queries on the reply. For the responses, see Section 3.2 below.

3.1.4 Receiving an HTTP Reply

The answers returned to the client are in the form of Protocol-Info; see Section 3.2. The server may have attached queries of its own; the client should prepare an answer and attach it to the next HTTP request it makes for a URI which matches the realm of the query. Here, too, a dynamically extended client should immediately call back an extension, since an extension's response to Protocol-Query may result in immediate side effects (e.g. a payment selection interface appearing with highlighted possible selections).

3.2 Protocol-Info

The second statement in a conversation might be to declare that one party requires/allows/forbids a given protocol extension, either unilaterally or in response to an earlier query. The Protocol-Info header lets one agent tell the other if a particular extension is supported; there is no required response. Remember, though, that Protocol-Info itself is the required response to Protocol-Query (see Section 3.1).

An informational response states if the protocol is required, allowed, or forbidden for all matching URIs.

3.2.1 Sending an HTTP Request

A client will add Protocol-Info directives if required by a previous server query (Section 3.1.4) or if it has any unilateral declarations to advertise.

3.2.2 Receiving an HTTP Request

A server should take note of any available information; knowledge of a client's capabilities is important for step #3 (Section 3.3.3 and 3.4.3).

3.2.3 Sending an HTTP Reply

A server will add Protocol-Info directives if required by a previous client query (Section 3.1.1) or if it has any unilateral declarations to make.

3.2.4 Receiving an HTTP Reply

A client should take note of any available information; knowledge of a server's capabilities is important for step #1 (Section 3.3.1 and 3.4.1). The response can be to either ignore this information, request initiation, or directly initiate the extension on the next request.

3.3 Protocol-Request

The third statement in a conversation might be to directly demand or forbid that the counterparty begin using an extension. Protocol-Request lets one agent tell the other to initiate a particular extension; the response requires a subsequent Protocol directive for each request.

3.3.1 Sending an HTTP Request

When a client wants to tell the server to begin using some extension, it should check if it knows that the extension will work because of stored Protocol-Info. The Protocol-Request describes the desired protocol; it may include a range of parameters for the respondent to pick and choose from.

3.3.2 Receiving an HTTP Request

A server must test if a request will succeed. If so, the server must begin using the specified extension. If it cannot, the preferred mechanism is to echo back the same Protocol-Request but with {str ref}. In addition, if the original request was {str req}, then indicate that the request failed with a 42x client or 52x server error status code (Section 3.5.2).

3.3.3 Sending an HTTP Reply

A server must reply to the client's request to require, accept or forbid an extension by either initiating or preventing use of the specified protocol or a compatible equivalent (Section 3.4.3). At this point, a server may also attach its own requests for a client to initiate extensions when accessing this URI or others.

Note that if a server Protocol-Request later fails to elicit a Protocol response from the client, nothing can be assumed about the client's ability to do PEP, since it may have ``forgotten'' the request in the meantime.

3.3.4 Receiving an HTTP Reply

The client must evaluate each server Protocol-Request and, if valid, be prepared to use the specified protocol extension when accessing any URI matching the request's for list (Section 3.4.1). If it is not willing or able to use the specified protocol, it must echo back an error on the next access as in Section 3.3.2.

Note that if a client Protocol-Request fails to elicit a Protocol or PEP error response (assuming the response URI is in the request's realm) at, the client must conclude the origin server is not PEP-compliant.

3.4 Protocol

The final statement in a conversation might be to use an extension directly. Protocol lets one agent declare that the particular extension is in use for the current message; the response may be to continue using the extension. An agent can apply an extension at any time; the preceeding three conversational steps of testing and requesting are only an example.

This directive packages together any added header lines and any reprocessing of the message body. Since HTTP/1.1 eliminates many earlier ambiguities about the ordering of headers, we can state categorically that protocol directives must be dispatched in the order they are listed in the Protocol header. [The Content-Encoding pipelining mechanism of earlier proposals has been scrapped].

It is worth noting that the application of a protocol extension can affect the cacheablity of the response. This document does not define a cache control directive, but expects it to be feasible within the 1.1 model.

Connection-scope extensions can remain backward-compatible by listing every associated header and the name of the extension in the Connection header.

3.4.1 Sending an HTTP Request

The client must decide which extensions to apply to the current request: it must respond to each applicable Protocol-Request, it can continue using extensions the server has already used for the request-URI, and it may initiate others. As each extension is applied, added headers are referenced from the Protocol line in order. If some Protocol-Request is satisfied by some other protocol extension compatible with the one requested, the client will issue two protocol directives, one for the extension used, and another indicating the extension requested is being used via the other.

3.4.2 Receiving an HTTP Request

The server evaluates each protocol extension which is in scope according to the order it is listed in the Protocol header.

As the server evaluates each protocol, it can interrupt and return HTTP error status codes: if a client encoding error is detected (42x), the server can't or won't decode properly (52x), or if a protocol is used inappropriately (it's refused, not allowed in conjunction with certain other protocols, missing, etc.).

Proxy servers operate similarly: they can simply remove unknown optional connection-scope directives but must report errors for required directives.

3.4.3 Sending an HTTP Reply

The server must also choose protocols it will use from the sets of those requested by the client, used by the client, and those it will initiate independently. If it cannot construct a valid reply due to contradictions among those requirements, it must return HTTP error status 520 and enumerate the server's constraints in Protocol-Request headers of the error message.

3.4.4 Receiving an HTTP Reply

The client evaluates each protocol extension which is in scope according to the order it is listed in the Protocol header.

As the client evaluates each protocol, it could fail for many reasons. One recovery technique is to reissue its request with the offending protocol refused in a Protocol-Request header. Errors from executing required extensions should be reported to a user or log.

3.5 Syntax

PEP-related syntax is specified here relative to the definitions and rules of the HTTP/1.0, HTTP/1.1, and the relative URL specification.

3.5.1 PEP Headers

PEP defines two new required general header fields, Protocol and Protocol-Request, as well as two new optional general header fields, Protocol-Query and Protocol-Info.

/* Added to General Header rule, Sec 4.3 of HTTP/1.1 */ Protocol = "Protocol" ":" 1#bag Protocol-Request = "Protocol-Request" ":" 1#bag Protocol-Query = "Protocol-Query" ":" 1#bag Protocol-Info = "Protocol-Info" ":" 1#bag /* Following rules are copied from Section 2.2 of HTTP/1.1 */ bag = "{" bagname 1*LWS *bagitem "}" bagname = token ¦ URI bagitem = bag ¦ token ¦ quoted-string word = token ¦ quoted-string token = 1*<any CHAR except CTLs or tspecials> tspecials = "(" ¦ ")" ¦ "<" ¦ ">" ¦ "@" ¦ "," ¦ ";" ¦ ":" ¦ "\" ¦ <"> ¦ "/" ¦ "[" ¦ "]" ¦ "?" ¦ "=" ¦ "{" ¦ "}" ¦ SP ¦ HT quoted-string = ( <"> *(qdtext) <"> ) qdtext = <any CHAR except <"> and CTLs but including LWS>

Each bag has several top-level reserved attributes defined:

{<protocol-identifier> 	{scope (origin ¦ conn)}  
                        {str (opt ¦ req ¦ ref)} 
                        {headers *<token>}
                        {params ...} 
                        {for *<uri>}
                        {via <protocol-identifier>}}

The well-known protocol extension URI or an IANA-registered token.

scope is either of "conn" or "origin" ; the default is "origin". It defines which agent must recognize this directive: the immediate recipient or the origin client/server.

strength is one of "req", "ref" or "opt" ; the default is "opt". In particular, a protocol directive only allows req or opt; request and informational directives allow all three; and queries only allow opt.

A list of associated message headers; the default is the empty set. If a header token ends in "*", it must be used to refer to all message headers matching that prefix. Each directive enumerates its own associated headers to disambiguate `ownership'. This information is critical when 1) reporting an error, 2) stripping an extension off of a message (if possible), and 3) allowing multiple applications of an extension.

A list of parameters configuring the extension, according to the protocol specification; the default is empty.

a list of relative or absolute URIs [1,2,7] which a protocol request is binding upon; the default when the for list is empty, is the request-URI or Content-Location response header. If a reference ends in "*", it should be understood as a wild-card character (particular implementations may choose to only recognize restricted applications of "*", e.g. `only in the last component of http: URLs').

When responding to a request addressed to one protocol extension (a query or request directive), an agent may answer via another (more specific, upgraded, compatible) extension (token or URI). The latter extension is expected to be used elsewhere in the same message. This is a hint and only allowed in the Protocol and Protocol-Info directives. The default is empty.

Note: Multiple occurrences of PEP-defined attribute names in a list ("{str req}...{str ref}") will yield undefined behavior.

3.5.2 PEP Status Codes

PEP defines several new status codes for HTTP replies. Note that the HTTP/1.1 specification states in Section 6.1.1:

The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role.

Informally, PEP distinguishes x2z response codes:

200 Class
220 Uses Protocol Extensions
400 Class
420 Bad Protocol Extension Request
421 Protocol Extension Unknown

422 Protocol Extension Refused
423 Bad Protocol Extension Parameters
500 Class
520 Protocol Extension Error
521 Protocol Extension Not Implemented

522 Protocol Extension Parameters Not Acceptable

Each of 400 and 500 class responses may include entity bodies with an explanation of the error, and an indication of whether the problem is temporary or permanent. A 220 response should be used only when some PEP feature is required for correct handling -- it affects the cacheability of the response.

4. Conclusions

This is a discussion draft, not a complete specification. In the meantime, this will be the basis for all further JEPI work.

For additional details and explorations of various issues (e.g. security concerns), readers are encouraged to study the earlier drafts, available at http://www.w3.org/pub/WWW/TR/WD-http-pep.

5. References

Berners-Lee, T., "Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web", RFC 1630, CERN, June 1994.

Berners-Lee, T., Masinter, L., and M. McCahill, Editors, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox Corporation, University of Minnesota, December 1994.

R. Fielding, J. Gettys, J. C. Mogul, H. Frystyk, T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1". Internet Draft UC Irvine, DEC, DEC, W3C/MIT, W3C/MIT, August 1996 (Work in Progress).

D. H. Crocker. "Standard for the Format of ARPA Internet Text Messages." STD 11, RFC 822, UDEL, August 1982.

S. Deering, R. Hinden, Editors, "Internet Protocol, Version 6 (IPv6) Specification", RFC 1883, December 1995.

D. Eastlake, "Universal Payment Preamble", Internet Draft CyberCash, March 1996 (Work in Progress).

R. Fielding. "Relative Uniform Resource Locators." RFC 1808 , UC Irvine, June 1995.

JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft, August 1996 (Work in Progress). [Also available as http://www.w3.org/pub/WWW/Payments/JEPI/draft-jepi-uppflow-00.html]

D. Kristol, "A Proposed Extension Mechanism for HTTP", Internet Draft, January 1995 (Work in Progress, Expired).

J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker. "SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach Consulting, Network Management Associates, Brandenburg Consulting, November 1995.

J. Miller. "Label Syntax and Communication Protocols (Version 1.1)." W3C Proposed Recommendation PR-PICS-services , W3 Consortium/MIT, August 1996.

J. Postel, J. Reynolds, "Telnet Protocol specification." STD 8, RFC 854, USC/ISI, May 1983

6. Author's Address

[Note: a great many people have participated heavily in the research, design, development, and description of the ideas presented here. Future revisions will sort out the contributions of various folks, likely adding co-authors &c. In the meantime, I will be the single contact point for this draft. Many, many thanks to these folks -- you know who you are!]

Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org
Web: http://www.w3.org/People/Khare