Note: since working drafts are subject to frequent change, you are advised to reference the above URL, rather than the URLs for working drafts themselves.
PEP is a system for HTTP clients, servers, and proxies to reliably reason about custom extensions to HTTP. Traditionally, HTTP agents offer extended behavior by private agreement using additional message headers. PEP has features for expressing the scope, strength, and ordering of such extensions, as well as which extensions are available.
PEP as described here is substantially simpler than previous drafts. The title has also been changed, since PEP addresses a chartered HTTP WG work item.
HTTP [3] messages, like most applications of RFC 822 [4], can be extended with additional header fields and content formats. Traditionally, the standards update process itself extends the protocol for major, global new features. However, this approach is problematic for deploying casual experiments which could scale up from pairwise to universal deployment, especially as HTTP agents become more modular, dynamic, and diverse. Casual extension of HTTP is hindered by several challenges:
PEP introduces the concept of protocol extensions to systematically address these issues at a level of abstraction above header names and content-types (hence, `Protocol Extension Protocol'). With PEP, HTTP agents can interoperate correctly with known and unknown protocol extensions, select protocol extensions available to both sides, and query partners for specific capabilities. PEP adds to HTTP the extensibility lessons learned from ESMTP (extension naming) [10], IPv6 (unknown-option disposition) [5], and Telnet (option negotiation) [12].
In addition to reliably describing statically extended HTTP servers and clients, PEP will work with dynamically extended agents. Indeed, the authors expect that PEP will drive the deployment of a new generation of exensible agents (such as W3C's Jigsaw server and libWWW reference library).
NOTE: PEP has been developed over the last year by many interested parties, including the W3C/CommerceNet Joint Electronic Payments Initiative (JEPI). Two strongly related drafts are also available from the IETF and W3C: Don Eastlake's Universal Payment Preamble [6] and ``Selecting Payment Mechanisms Over HTTP (or, seven examples of UPP over PEP)'' [8].
Consider the example presented by the first deployed PEP application, the Platform for Internet Content Selection (PICS) [11]. PICS describes a format for labeling almost any content. Since it's intended to be a fairly ubiquitous addition to various Internet protocols, PICS appropriates the PICS-Label header for 822 applications, including mail, news, and web content. The spec could not use, say, Label, since there's no way to effectively arbitrate between conflicting uses of header names (PICS was not developed under the IETF standards track).
Even with an eventual dispensation from IANA to use this new header, new issues arise: If a server attaches this new header to label an HTTP response, will a caching server pass it along? Will it act upon the label at all? What kind of feedback will the end-user get when this ``unknown'' header appears? How can the user specify which rating systems to report?
The direct fix to the first few problems is to add a machine-understandable statement about the extension and its new headers. Protocol headers describe the name of the extension, the scope, importance, and any associated data headers. Moving further, though, additional information about the protocol can be added to that directive: which PICS services were employed, etc.
Using PEP in this way also solves the next logical problem: how can a client request a server to initiate vending PICS labels? The same structure for a Protocol can be reused in a Protocol-Request asking the counterparty to start using the specified extension.
In addition, PEP affords additional capabilities, like using Protocol-Query to test if PICS is available and Protocol-Info to advertise it. In all cases, the PICS protocol extension can specify additional parameters, such as which rating systems, services, which kinds of labels, and which other URLs are labelled.
Protocol extensions can modify HTTP messages, most commonly by adding headers, but also by modifying the contents, much like a proxy server might do today. There are four points at which PEP processing occurs in a basic HTTP transaction (see Section 3 for usage descriptions at each step):
During a conversation, either side may invoke a Protocol, issue a Protocol-Request, advertise some Protocol-Info, or test a Protocol-Query. PEP-compliant recipients decode extended messages and reply to each Protocol-Request and Protocol-Query.
Protocol extensions are attributes of a resource rather than an entire server or connection, which was the model in 1.1's Connection and Upgrade fields, and in previous extension proposals [9]. This means each PEP statement is applied to the single resource mentioned in an HTTP message (though there is a facility to hint at which other resources the same statement applies for).
An HTTP agent can begin using a protocol extension (or query for one) because:
Another aspect of the PEP model is that protocol extensions can have negotiable parameters. Unlike previous proposals which model extensions as binary have/don't have features, PEP expects protocol extensions will be able to handle different versions of a particular protocol, invocation of ``compatible protocols'', and the selection of compatible modes from several that might be advertised.
To summarize, PEP as proposed here will allow HTTP/1.2 implementors to make the following kinds of statements:
Before presenting specifics about each of the four PEP directives and the four points where they are processed, here are a few common points:
A bag is an attribute-value list, where bags are allowed recursively as values as well. In PEP, each directive begins with values describing the protocol (making the extension name the top attribute). Some of those values are common to all directives: the strength (str), scope (scope), for list (for) , extension-specific parameters (params), and other headers associated with this extension (headers). The formal BNF is included in Section 3.5.1.
Strength and scope are important common concepts. Any directive can be ignored, but strength specifies whether an error should be generated as a result. Similarly, scope defines which agents should handle the directive at all; either the immediate (conn) or final (origin) recipient. The combination of the two concepts makes PEP powerful enough to deploy features that in the past required a minor-version number bump. With PEP, for example, Content-Transfer-Encoding could be described as {str req} {scope conn} for rapid experimentation with compressed headers, sticky headers, etc.
A word about the for list: Since each PEP directive applies only to the resource at hand (rather than, say, the entire server or all resources of a given type), a hint as to which other resources the directive applies to will prevent negotiation round-trips each time. Thus, many PEP applications could include {for /*} to apply to an entire server. Remember, there already are examples in HTTP/1.1 of features which only apply to particular resources, such as Range.
When fetching a URL, a client will typically search through its database of hints to find all applicable directives before sending an HTTP request. Of course, an agent which does not want to use hints can ignore them: if the for list includes the URI of the current message, just throw away the list; if it does not, just throw away the entire directive.
Finally, note that the Protocol-Query / Protocol-Info pair describes hypothetical capabilities of agents and resources but that the Protocol-Request / Protocol pair has immediate consequences for the current HTTP transaction. Each pair hangs together and must be implemented together, though only the latter pair is mandatory.
The first statement in a conversation might be to ask if the counterparty supports an extension at all. The Protocol-Query header lets one agent ask the other if a particular extension is supported; the response comes back in a subsequent Protocol-Info.
A query directive specifies its scope, but not a strength, which defaults to opt; that's because a query always asks optionally for more information, but can neither demand an answer (since query support is optional), nor refuse one.
An agent can query if an extension is supported for an entire list of URIs. This means that an answer is expected when the replying agent sends a message about any resource on the list; and that an answer should reveal as much as can be said about extension support for that URI set. This way, a server can ask for a client to be prepared to use an extension in a part of his site (e.g. UPP in /Catalog/*); or a client can ask a server if an extension applies to all its documents of the same type (e.g. indexed range retreival for *.pdf).
If a client believes an extension may be supported, it can attach a Protocol-Query to an outgoing request to test if an extension is supported for that URI (empty for list), an entire server ({for /*}) or something in between.
A client can wait to piggyback its queries on other HTTP transactions or it can explictly issue an OPTIONS request with its queries.
Note that a protocol extension might be defined to have side-effects when it is queried. For example, the UPP payment selection protocol defines that the answer to any queries for UPP should include additional information about particular payment mechanisms. Thus, in a dynamically extended agent an agent should call back the extension to handle a query.
When an agent receives a query, it should respond whether the stated configuration is acceptable or refused for any of the URI(s) specified. Upon receipt of a query, an agent should test if the extension is available and, if so, its response to the proposition. The server should be prepared to send the resulting information in its reply.
In preparing an HTTP reply, a server must respond to queries in the HTTP request, as well as issue its own queries on the reply. For the responses, see Section 3.2 below.
The answers returned to the client are in the form of Protocol-Info; see Section 3.2. The server may have attached queries of its own; the client should prepare an answer and attach it to the next HTTP request it makes for a URI which matches the realm of the query. Here, too, a dynamically extended client should immediately call back an extension, since an extension's response to Protocol-Query may result in immediate side effects (e.g. a payment selection interface appearing with highlighted possible selections).
The second statement in a conversation might be to declare that one party requires/allows/forbids a given protocol extension, either unilaterally or in response to an earlier query. The Protocol-Info header lets one agent tell the other if a particular extension is supported; there is no required response. Remember, though, that Protocol-Info itself is the required response to Protocol-Query (see Section 3.1).
An informational response states if the protocol is required, allowed, or forbidden for all matching URIs.
A client will add Protocol-Info directives if required by a previous server query (Section 3.1.4) or if it has any unilateral declarations to advertise.
A server should take note of any available information; knowledge of a client's capabilities is important for step #3 (Section 3.3.3 and 3.4.3).
A server will add Protocol-Info directives if required by a previous client query (Section 3.1.1) or if it has any unilateral declarations to make.
A client should take note of any available information; knowledge of a server's capabilities is important for step #1 (Section 3.3.1 and 3.4.1). The response can be to either ignore this information, request initiation, or directly initiate the extension on the next request.
The third statement in a conversation might be to directly demand or forbid that the counterparty begin using an extension. Protocol-Request lets one agent tell the other to initiate a particular extension; the response requires a subsequent Protocol directive for each request.
When a client wants to tell the server to begin using some extension, it should check if it knows that the extension will work because of stored Protocol-Info. The Protocol-Request describes the desired protocol; it may include a range of parameters for the respondent to pick and choose from.
A server must test if a request will succeed. If so, the server must begin using the specified extension. If it cannot, the preferred mechanism is to echo back the same Protocol-Request but with {str ref}. In addition, if the original request was {str req}, then indicate that the request failed with a 42x client or 52x server error status code (Section 3.5.2).
A server must reply to the client's request to require, accept or forbid an extension by either initiating or preventing use of the specified protocol or a compatible equivalent (Section 3.4.3). At this point, a server may also attach its own requests for a client to initiate extensions when accessing this URI or others.
Note that if a server Protocol-Request later fails to elicit a Protocol response from the client, nothing can be assumed about the client's ability to do PEP, since it may have ``forgotten'' the request in the meantime.
The client must evaluate each server Protocol-Request and, if valid, be prepared to use the specified protocol extension when accessing any URI matching the request's for list (Section 3.4.1). If it is not willing or able to use the specified protocol, it must echo back an error on the next access as in Section 3.3.2.
Note that if a client Protocol-Request fails to elicit a Protocol or PEP error response (assuming the response URI is in the request's realm) at, the client must conclude the origin server is not PEP-compliant.
The final statement in a conversation might be to use an extension directly. Protocol lets one agent declare that the particular extension is in use for the current message; the response may be to continue using the extension. An agent can apply an extension at any time; the preceeding three conversational steps of testing and requesting are only an example.
This directive packages together any added header lines and any reprocessing of the message body. Since HTTP/1.1 eliminates many earlier ambiguities about the ordering of headers, we can state categorically that protocol directives must be dispatched in the order they are listed in the Protocol header. [The Content-Encoding pipelining mechanism of earlier proposals has been scrapped].
It is worth noting that the application of a protocol extension can affect the cacheablity of the response. This document does not define a cache control directive, but expects it to be feasible within the 1.1 model.
Connection-scope extensions can remain backward-compatible by listing every associated header and the name of the extension in the Connection header.
The client must decide which extensions to apply to the current request: it must respond to each applicable Protocol-Request, it can continue using extensions the server has already used for the request-URI, and it may initiate others. As each extension is applied, added headers are referenced from the Protocol line in order. If some Protocol-Request is satisfied by some other protocol extension compatible with the one requested, the client will issue two protocol directives, one for the extension used, and another indicating the extension requested is being used via the other.
The server evaluates each protocol extension which is in scope according to the order it is listed in the Protocol header.
As the server evaluates each protocol, it can interrupt and return HTTP error status codes: if a client encoding error is detected (42x), the server can't or won't decode properly (52x), or if a protocol is used inappropriately (it's refused, not allowed in conjunction with certain other protocols, missing, etc.).
Proxy servers operate similarly: they can simply remove unknown optional connection-scope directives but must report errors for required directives.
The server must also choose protocols it will use from the sets of those requested by the client, used by the client, and those it will initiate independently. If it cannot construct a valid reply due to contradictions among those requirements, it must return HTTP error status 520 and enumerate the server's constraints in Protocol-Request headers of the error message.
The client evaluates each protocol extension which is in scope according to the order it is listed in the Protocol header.
As the client evaluates each protocol, it could fail for many reasons. One recovery technique is to reissue its request with the offending protocol refused in a Protocol-Request header. Errors from executing required extensions should be reported to a user or log.
PEP-related syntax is specified here relative to the definitions and rules of the HTTP/1.0, HTTP/1.1, and the relative URL specification.
PEP defines two new required general header fields, Protocol and Protocol-Request, as well as two new optional general header fields, Protocol-Query and Protocol-Info.
Each bag has several top-level reserved attributes defined:/* Added to General Header rule, Sec 4.3 of HTTP/1.1 */ Protocol = "Protocol" ":" 1#bag Protocol-Request = "Protocol-Request" ":" 1#bag Protocol-Query = "Protocol-Query" ":" 1#bag Protocol-Info = "Protocol-Info" ":" 1#bag /* Following rules are copied from Section 2.2 of HTTP/1.1 */ bag = "{" bagname 1*LWS *bagitem "}" bagname = token ¦ URI bagitem = bag ¦ token ¦ quoted-string word = token ¦ quoted-string token = 1*<any CHAR except CTLs or tspecials> tspecials = "(" ¦ ")" ¦ "<" ¦ ">" ¦ "@" ¦ "," ¦ ";" ¦ ":" ¦ "\" ¦ <"> ¦ "/" ¦ "[" ¦ "]" ¦ "?" ¦ "=" ¦ "{" ¦ "}" ¦ SP ¦ HT quoted-string = ( <"> *(qdtext) <"> ) qdtext = <any CHAR except <"> and CTLs but including LWS>
{<protocol-identifier> {scope (origin ¦ conn)} {str (opt ¦ req ¦ ref)} {headers *<token>} {params ...} {for *<uri>} {via <protocol-identifier>}}
Note: Multiple occurrences of PEP-defined attribute names in a list ("{str req}...{str ref}") will yield undefined behavior.
PEP defines several new status codes for HTTP replies. Note that the HTTP/1.1 specification states in Section 6.1.1:
The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role.
Informally, PEP distinguishes x2z response codes:
Each of 400 and 500 class responses may include entity bodies with an explanation of the error, and an indication of whether the problem is temporary or permanent. A 220 response should be used only when some PEP feature is required for correct handling -- it affects the cacheability of the response.
This is a discussion draft, not a complete specification. In the meantime, this will be the basis for all further JEPI work.
For additional details and explorations of various issues (e.g. security concerns), readers are encouraged to study the earlier drafts, available at http://www.w3.org/pub/WWW/TR/WD-http-pep.
[Note: a great many people have participated heavily in the research, design, development, and description of the ideas presented here. Future revisions will sort out the contributions of various folks, likely adding co-authors &c. In the meantime, I will be the single contact point for this draft. Many, many thanks to these folks -- you know who you are!]
Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org
Web: http://www.w3.org/People/Khare