Note: since working drafts are subject to frequent change, you are advised to reference the above URL, rather than the URLs for working drafts themselves.
PEP is a system for HTTP clients, servers, and proxies to reliably reason about custom extensions to HTTP. Traditionally, HTTP agents offer extended behavior by private agreement using additional message headers. PEP has features for expressing the scope, strength, and ordering of such extensions, as well as which extensions are available.
PEP as described here is substantially simpler than previous drafts. The title has also been changed, since PEP addresses a chartered HTTP WG work item.
HTTP [3] messages, like most applications of RFC 822 [4], can be extended with additional header fields and content formats. Traditionally, the standards update process itself extends the protocol for major, global new features. However, this approach is problematic for deploying casual experiments which could scale up from pairwise to universal deployment, especially as HTTP agents become more modular, dynamic, and diverse. Casual extension of HTTP is hindered by several challenges:
PEP introduces the concept of protocol extensions to systematically address these issues at a level of abstraction above header names and content-types (hence, `Protocol Extension Protocol'). With PEP, HTTP agents can interoperate correctly with known and unknown protocol extensions, select protocol extensions available to both sides, and query partners for specific capabilities. PEP adds to HTTP the extensibility lessons learned from ESMTP (extension naming) [10], IPv6 (unknown-option disposition) [5], and Telnet (option negotiation) [12].
In addition to reliably describing statically extended HTTP servers and clients, PEP will work with dynamically extended agents. Indeed, the authors expect that PEP will drive the deployment of a new generation of exensible agents (such as W3C's Jigsaw server and libWWW reference library).
NOTE: PEP has been developed over the last year by many interested parties, including the W3C/CommerceNet Joint Electronic Payments Initiative (JEPI). Two strongly related drafts are also available from the IETF and W3C: Don Eastlake's Universal Payment Preamble [6] and ``Selecting Payment Mechanisms Over HTTP (or, seven examples of UPP over PEP)'' [8].
Consider the example presented by the first deployed PEP application, the Platform for Internet Content Selection (PICS) [11]. PICS describes a format for labeling almost any content. Since it's intended to be a fairly ubiquitous addition to various Internet protocols, PICS appropriates the PICS-Label: header for 822 applications, including mail, news, and web content. The spec could not use, say, Label:, since there's no way to effectively arbitrate between conflicting uses of header names (PICS was not developed under the IETF standards track).
Even with an eventual dispensation from IANA to use this new header, new issues arise: If a server attaches this new header to label an HTTP response, will a caching server pass it along? Will it act upon the label at all? What kind of feedback will the end-user get when this ``unknown'' header appears? How can the user specify which rating systems to report?
The direct fix to the first few problems is to add a machine-understandable statement about the extension and its new headers. Protocol: headers describe the name of the extension, the scope, importance, and any associated data headers. Moving further, though, additional information about the protocol can be added to that directive: which PICS services were employed, etc.
Using PEP in this way also solves the next logical problem: how can a client request a server to initiate vending PICS labels? The same structure for a Protocol: can be reused in a Protocol-Request: asking the counterparty to start using the specified extension.
In addition, PEP affords additional capabilities, like using Protocol-Query: to test if PICS is available and Protocol-Info: to advertise it. In all cases, the PICS protocol extension can specify additional parameters, such as which rating systems, services, which kinds of labels, and which other URLs are labelled.
Protocol extensions can modify HTTP messages, most commonly by adding headers, but also by modifying the contents, much like a proxy server might do today. There are four points at which PEP processing occurs in a basic HTTP transaction (see Section 3 for usage descriptions at each step):
During a conversation, either side may invoke a Protocol:, issue a Protocol-Request:, advertise some Protocol-Info:, or test a Protocol-Query:. PEP-compliant recipients decode extended messages and reply to each Protocol-Request: and Protocol-Query:.
Unlike some previous extension proposals [9], and unlike some existing HTTP/1.1 features (Upgrade: and Connection: tokens), extensions are attributes of a resource rather than a server. This means all PEP statements are applied to the single resource mentioned in any HTTP message (though there is a facility to hint at which other resources the same statement applies to).
An HTTP agent can begin using a protocol extension (or query for one) because:
Another aspect of the PEP model is that protocol extensions can have negotiable parameters. Unlike previous proposals which model extensions as binary have/don't have features, PEP expects protocol extensions will be able to handle different versions of a particular protocol, invocation of ``compatible protocols'', and the selection of compatible modes from several that might be advertised.
To summarize, PEP as proposed here will allow HTTP/1.2 implementors to make the following kinds of statements:
Before presenting specifics about each of the four PEP directives and the four points where they are processed, here are a few common points:
A bag is an attribute-value list, where bags are allowed recursively as values as well. In PEP, each directive begins with values describing the protocol (making the extension name the top attribute). Some of those values are common to all directives: the strength (str), scope (scope), for list (for) , extension-specific parameters (params), and other headers associated with this extension (headers). The formal BNF is included in Section 3.5.1.
Strength and scope are important common concepts. Any directive can be ignored, but strength specifies whether an error should be generated as a result. Similarly, scope defines which agents should handle the directive at all; either the immediate (conn) or final (origin) recipient. The combination of the two concepts makes PEP powerful enough to deploy features that in the past required a minor-version number bump. With PEP, for example, Content-Transfer-Encoding: could be described as {str req} {scope conn} for rapid experimentation with compressed headers, sticky headers, etc.
A word about the for list: Since each PEP directive applies only to the resource at hand (rather than, say, the entire server or all resources of a given type), a hint as to which other resources the directive applies to will prevent negotiation round-trips each time. Thus, many PEP applications could include {for /*} to apply to an entire server. Remember, there already are examples in HTTP/1.1 of features which only apply to particular resources, such as Range:.
When fetching a URL, a client will typically search through its database of hints to find all applicable directives before sending an HTTP request. Of course, an agent which does not want to use hints can ignore them: if the for list includes the URI of the current message, just throw away the list; if it does not, just throw away the entire directive.
Finally, note that the Protocol-Query: / Protocol-Info: pair describes hypothetical capabilities of agents and resources but that the Protocol-Request: / Protocol: pair has immediate consequences for the current HTTP transaction. Each pair hangs together and must be implemented together, though only the latter pair is mandatory.
The first statement in a conversation might be to ask if the counterparty supports an extension at all. The Protocol-Query: header lets one agent ask the other if a particular extension is supported; the response comes back in a subsequent Protocol-Info:.
A query directive specifies its scope, but not a strength, which defaults to opt; that's because a query always asks optionally for more information, but can neither demand an answer (since query support is optional), nor refuse one.
An agent can query if an extension is supported for an entire list of URIs. This means that an answer is expected when the replying agent sends a message about any resource on the list; and that an answer should reveal as much as can be said about extension support for that URI set. This way, a server can ask for a client to be prepared to use an extension in a part of his site (e.g. UPP in /Catalog/*); or a client can ask a server if an extension applies to all its documents of the same type (e.g. indexed range retreival for *.pdf).
If a client believes an extension may be supported, it can attach a Protocol-Query: to an outgoing request to test if an extension is supported for that URI (empty for list), an entire server ({for /*}) or something in between.
A client can wait to piggyback its queries on other HTTP transactions or it can explictly issue an OPTIONS request with its queries.
Note that a protocol extension might be defined to have side-effects when it is queried. For example, the UPP payment selection protocol defines that the answer to any queries for UPP should include additional information about particular payment mechanisms. Thus, in a dynamically extended agent an agent should call back the extension to handle a query.
When an agent receives a query, it should respond whether the stated configuration is acceptable or refused for any of the URI(s) specified. Upon receipt of a query, an agent should test if the extension is available and, if so, its response to the proposition. The server should be prepared to send the resulting information in its reply.
In preparing an HTTP reply, a server must respond to queries in the HTTP request, as well as issue its own queries on the reply. For the responses, see Section 3.2 below.
The answers returned to the client are in the form of Protocol-Info:; see Section 3.2. The server may have attached queries of its own; the client should prepare an answer and attach it to the next HTTP request it makes for a URI which matches the realm of the query. Here, too, a dynamically extended client should immediately call back an extension, since an extension's response to Protocol-Query: may result in immediate side effects (e.g. a payment selection interface appearing with highlighted possible selections).
The second statement in a conversation might be to declare that one party requires/allows/forbids a given protocol extension, either unilaterally or in response to an earlier query. The Protocol-Info: header lets one agent tell the other if a particular extension is supported; there is no required response. Remember, though, that Protocol-Info: itself is the required response to Protocol-Query: (see Section 3.1).
An informational response states if the protocol is required, allowed, or forbidden for all matching URIs.
A client will add Protocol-Info: directives if required by a previous server query (Section 3.1.4) or if it has any unilateral declarations to advertise.
A server should take note of any available information; knowledge of a client's capabilities is important for step #3 (Section 3.3.3 and 3.4.3).
A server will add Protocol-Info: directives if required by a previous client query (Section 3.1.1) or if it has any unilateral declarations to make.
A client should take note of any available information; knowledge of a server's capabilities is important for step #1 (Section 3.3.1 and 3.4.1). The response can be to either ignore this information, request initiation, or directly initiate the extension on the next request.
The third statement in a conversation might be to directly demand or forbid that the counterparty begin using an extension. Protocol-Request: lets one agent tell the other to initiate a particular extension; the response requires a subsequent Protocol: directive for each request.
When a client wants to tell the server to begin using some extension, it should check if it knows that the extension will work because of stored Protocol-Info:. The Protocol-Request: describes the desired protocol; it may include a range of parameters for the respondent to pick and choose from.
A server must test if a request will succeed. If so, the server must begin using the specified extension. If it cannot, the preferred mechanism is to echo back the same Protocol-Request: but with {str ref}. In addition, if the original request was {str req}, then indicate that the request failed with a 42x client or 52x server error status code (Section 3.5.2).
A server must reply to the client's request to require, accept or forbid an extension by either initiating or preventing use of the specified protocol or a compatible equivalent (Section 3.4.3). At this point, a server may also attach its own requests for a client to initiate extensions when accessing this URI or others.
Note that if a server Protocol-Request: later fails to elicit a Protocol: response from the client, nothing can be assumed about the client's ability to do PEP, since it may have ``forgotten'' the request in the meantime.
The client must evaluate each server Protocol-Request: and, if valid, be prepared to use the specified protocol extension when accessing any URI matching the request's for list (Section 3.4.1). If it is not willing or able to use the specified protocol, it must echo back an error on the next access as in Section 3.3.2.
Note that if a client Protocol-Request: fails to elicit a Protocol: or PEP error response (assuming the response URI is in the request's realm) at, the client must conclude the origin server is not PEP-compliant.
The final statement in a conversation might be to use an extension directly. Protocol: lets one agent declare that the particular extension is in use for the current message; the response may be to continue using the extension. An agent can apply an extension at any time; the preceeding three conversational steps of testing and requesting are only an example.
This directive packages together any added header lines and any reprocessing of the message body. Since HTTP/1.1 eliminates many earlier ambiguities about the ordering of headers, we can state categorically that protocol directives must be dispatched in the order they are listed in the Protocol: header. [The Content-Encoding: pipelining mechanism of earlier proposals has been scrapped].
It is worth noting that the application of a protocol extension can affect the cacheablity of the response. This document does not define a cache control directive, but expects it to be feasible within the 1.1 model.
Connection-scope extensions can remain backward-compatible by listing every associated header and the name of the extension in the Connection: header.
The client must decide which extensions to apply to the current request: it must respond to each applicable Protocol-Request:, it can continue using extensions the server has already used for the request-URI, and it may initiate others. As each extension is applied, added headers are referenced from the Protocol: line in order. If some Protocol-Request: is satisfied by some other protocol extension compatible with the one requested, the client will issue two protocol directives, one for the extension used, and another indicating the extension requested is being used via the other.
The server evaluates each protocol extension which is in scope according to the order it is listed in the Protocol: header.
As the server evaluates each protocol, it can interrupt and return HTTP error status codes: if a client encoding error is detected (42x), the server can't or won't decode properly (52x), or if a protocol is used inappropriately (it's refused, not allowed in conjunction with certain other protocols, missing, etc.).
Proxy servers operate similarly: they can simply remove unknown optional connection-scope directives but must report errors for required directives.
The server must also choose protocols it will use from the sets of those requested by the client, used by the client, and those it will initiate independently. If it cannot construct a valid reply due to contradictions among those requirements, it must return HTTP error status 520 and enumerate the server's constraints in Protocol-Request: headers of the error message.
The client evaluates each protocol extension which is in scope according to the order it is listed in the Protocol: header.
As the client evaluates each protocol, it could fail for many reasons. One recovery technique is to reissue its request with the offending protocol refused in a Protocol-Request: header. Errors from executing required extensions should be reported to a user or log.
PEP-related syntax is specified here relative to the definitions and rules of the HTTP/1.0, HTTP/1.1, and the relative URL specification.
PEP defines two new required general header fields, Protocol: and Protocol-Request:, as well as two new optional general header fields, Protocol-Query: and Protocol-Info:.
Each bag has several top-level reserved attributes defined:/* Added to General Header rule, Sec 4.3 of HTTP/1.1 */ Protocol = "Protocol" ":" 1#bag Protocol-Request = "Protocol-Request" ":" 1#bag Protocol-Query = "Protocol-Query" ":" 1#bag Protocol-Info = "Protocol-Info" ":" 1#bag /* Following rules are copied from Section 2.2 of HTTP/1.1 */ bag = "{" bagname 1*LWS *bagitem "}" bagname = token ¦ URI bagitem = bag ¦ token ¦ quoted-string word = token ¦ quoted-string token = 1*<any CHAR except CTLs or tspecials> tspecials = "(" ¦ ")" ¦ "<" ¦ ">" ¦ "@" ¦ "," ¦ ";" ¦ ":" ¦ "\" ¦ <"> ¦ "/" ¦ "[" ¦ "]" ¦ "?" ¦ "=" ¦ "{" ¦ "}" ¦ SP ¦ HT quoted-string = ( <"> *(qdtext) <"> ) qdtext = <any CHAR except <"> and CTLs but including LWS>
{<protocol-identifier> {scope (origin ¦ conn)} {str (opt ¦ req ¦ ref)} {headers *<token>} {params ...} {for *<uri>} {via <protocol-identifier>}}
Note: Multiple occurrences of PEP-defined attribute names in a list ("{str req}...{str ref}") will yield undefined behavior.
PEP defines several new status codes for HTTP replies. Note that the HTTP/1.1 specification states in Section 6.1.1:
The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role.
Informally, PEP distinguishes x2z response codes:
This is a discussion draft, not a complete specification. In the meantime, this will be the basis for all further JEPI work.
For additional details and explorations of various issues (e.g. security concerns), readers are encouraged to study the earlier drafts, available at http://www.w3.org/pub/WWW/TR/WD-http-pep.
Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org
Web: http://www.w3.org/People/Khare