W3C: WD-pep-970131

PEP: an Extension Mechanism for HTTP

W3C Working Draft 31 Jan 1997

Dan Connolly
This version:
$Id: WD-http-pep-970131.html,v 1.2 2000/09/08 16:13:46 mirror Exp $
Latest Version:
Previous Version:
http://www.w3.org/pub/WWW/TR/WD-http-pep-960820 http://www.w3.org/pub/WWW/TR/WD-http-pep-960819 http://www.w3.org/pub/WWW/TR/WD-http-pep-960222 http://www.w3.org/pub/WWW/TR/WD-http-pep-951122

Status of this Document

This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress".

To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast).

Distribution of this document is unlimited. Please send comments to the HTTP working group at http-wg@cuckoo.hpl.hp.com. Discussions of the working group are archived at http://www.ics.uci.edu/pub/ietf/http/.

This document is also available as W3C Working Draft WD-http-pep-970131. The most up-to-date current version is available at http://www.w3.org/pub/WWW/TR/WD-http-pep.

The contribution of W3C staff time to the HTTP workin group is part of the W3C HTTP Activity.


HTTP is an extensible protocol. PEP is an extension mechanism designed to address the tension between private agreement and public specification and to accomodate extension of HTTP clients and servers by software components.

The mechanism is to associate each extension with a URL, and use a few new header fields to carry the extension identifier and related information from HTTP clients, thru proxies and intermediaries, to servers, and back again.


2. Introduction

This document presents PEP, and extension mechanism for HTTP. The PEP design is the result of analyzing a variety of HTTP extensions and extension mechanisms, and the motivation behind them. This is discussed in requirements section (Section 2.1).

The section on extension identifiers discusses the mechanism itself, which is to associate each extension with a URL, and use a new header field, Protocol: to carry the extension identifier and related information from HTTP clients, thru proxies and intermediaries, to servers, and back again.

The next section, Notification, provides information providers with a mechanism to inform clients of extension policies, that is, which extensions should or should not be used to access resources.

The Extension Header Fields section addresses the use of new HTTP headers in extensions.

After Security Considerations, and a Syntax Reference, appendices discuss Considerations for Defining Extensions and the use of PEP in and with software components.


HTTP is being used for an increasing number of applications involving distributed authoring, collaboration, printing, and various RPC like protocols. Often these extensions are deployed dynamically, extending existing applications. They motivate the need to independently introduce extensions and new features to HTTP in such a way that 1) They are orthogonal to other extensions. 2) They can be deployed automatically and dynamically.

This requires:

This form for extensibility is not supported by HTTP/1.1. PEP is a framework to satisfy these requirements.

The current design does not meet all the requirements. See Future Work for details.

Related Work

HTTP is an extensible protocol; applications have exploited its extensibility along a number of degrees of freedom:

URL path
The URL path allows different functionality in different parts of the URL space[URL]. This has been exploited in [CGI], and HTML forms [HTML2.0] for example. Since then, it has been combined with software component technology (such as shared libraries, DLLs, etc.) for use in [NSAPI], [ISAPI], [Apache], [OM], [Spy95].
media type
The request and response payload data is typed; new internet media types can be introduced. A host of web extensions are based on the extenion of user agents to handle internet media types [MAILCAP].
method names
New method names can be added. (@@Cite BROWSE, MKDIR in aolserver)
header fields
New headers may be introduced: entity headers, request headers, or response headers. For example, [STATE]

Using the media type and/or URL to extend the web is an extension within, rather than beyond, the HTTP protocol. On the other hand, using new request header fields is a change to the HTTP protocol itself ([HTTP] section 5.3 Request Header Fields).

3. Extension Identifiers

The agents in an HTTP transaction are a client and a server, which send HTTP messages to each other However, semantically, an HTTP trasacion is between a client party (for example, the referent of the From: header field) and a the principle responsible for the publication of a given resource.

The publishing party is basically the one responsible for the mapping between the URI and any representation of the resource to which it refers. Exactly who takes this role is out of the scope discusion of this document; but for example, it may be the writer of a document or the server administrator or the organization running the server.

The HTTP specification, which codifies the agreement between these parties, is subject to thorough community review. While any extension can be defined and used by private agreement, the web provides a medium to deploy extensions to the global community without centralized control.

PEP exploits this aspect of the web, and uses URLs to identify extensions. This allows parties to learn about extensions and decide which ones to participate in by dereferncing their URL. This learning may be done by humans, or it may be done my machines aquiring software components.

See Considerations for Defining Extensions for details.

The Protocol Header Field

Each protocol extension has an extension identifier, which is a URL[URL]. The extensions used in a message are declared using the Protocol request/response header field.

Along with the extension identifier, an extension may define any number of parameters. See also, Extension Header Fields and Extension Encodings.

The syntax is:

       Protocol         = "Protocol" ":" 1#extension-decl
       extension-decl   = "{" extension-id *ext-info "}"
       ext-info         = params | headers | enc
       params           = "{" "params" *bagitem "}"
       headers          = "{" "headers" 1*token "}"
       enc              = "{" "enc" 1*token "}"

       bag              = "{" bagname 1*LWS *bagitem "}"
       bagname          = token  URI
       bagitem          = bag  token  quoted-string

For example:

GET /a-document HTTP/1.1
Protocol: {http://some.org/an-extension}

HTTP/1.1 200 OK
Protocol: {http://some.org/an-extension}
Vary: Protocol
Content-Type: text/plain

Glad you're using an-extension!

Note the use of the Vary header to notify proxies that responses to GET /a-document depend on the Protocol header fields used in the request.

Hop-by-hop Extensions

Extensions declared with the Protocol header field are end-to-end extensions, transparent to intermediaries. Hop-by-hop extensions are declared with the C-Protocol header field, in conjunction with the Connection header ([HTTP}, section @@).

The syntax is essentially the same as the Protocol header field.

       C-Protocol       = "C-Protocol" ":" 1#extension-decl

Considerations for Defining Extensions

While the protocol extension definition should be published at the address of the extension identifier, this is not strictly necessary. The only absolute requirement is that distinct names be used for distinct semantics.

For example, one way to achive this is to use an mid:, cid:, or uuid: URL. The association between the extension identifier and the specification might be made by distributing a specification which references the extension identifier. Care must be taken not to distribute conflicting specifications which reference the same name.

Even when the web is used to publish extension specifications, care must be taken that the specification made available at that address does not change significantly over time. One agent may associate the identifier with the old semantics, and another might associate it with the new semantics.

Bootstrapping and Dynamic Loading

The extension definition may be made available in different representations. For example, a software component that implements the specification may reside at the same address as a human-readable specification (distinguished by content negotation).

The human-readable representation serves to document the extension and encourage deployment, while the software component to allows clients and servers to be dynamically extended.

Caching and Connections

For each aspect of an extension, the interaction with other aspects of HTTP/1.1 must be fully specified, and the issues of compatibility should be discussed. For example, any extension headers which are not strictly entity headers require careful consideration. See [HTTP] sections 13.5.1. (@@more references were lost in an editing disaster)


Some extensions are used spontaneously by participating clients; for example, a client may be configured to use and extension, or a user interface option may trigger the use of an extension.

But in many cases, the origin server dictates the use of one or more extensions. In this case, it is useful for the server to communicate its policies to the client.

The server may notify the client that some resources should be accessed using one or more extensions with the Protocol-Info header field. The resources are specified by a relative or absolute URI, with an optional wildcard flag indicating that the notification applies to all URIs containing the specified URI as a prefix.

The strength of the policy for an extension for the resources is one of req, ref, or opt.

Required. The resource must be accessed using the extension.
Optional. The resource may be accessed using the extension or not using the extension.
Refused. The resource may not be accessed using the extension.

The syntax is:

       Protocol-Info    = "Protocol-Info" ":" 1#policy-decl
       policy-decl      = "{" extension-id *policy-info "}"
       policy-info      = str | params | headers | scope | for
       str              = "{" "str" ("req" | "ref" | "opt" ) "}"
       scope            = "{" "scope" ( "conn" | "origin" ) "}"
       for              = "{" "for" URI [ wildcard ] "}"
       wildcard         = "*"

Note that a Protocol-Info with a for parameter may give information about a different resource from the resource described by the other header fields in the same message. Nonetheless, the freshness of the information in the Protocol-Info header field is the same as the rest of the header fields (which see [HTTP] section 13.2, "Expiration Model").

The notice is strictly advisory. The client should heed the notice on its next request to the relavent server, unless the delay between receiving the notice and that next request far exceeds the freshness of the reply containing the Protocol-Info header.

For example, consider the case of an HTML form, where the associated ACTION resource requires a payment extension. In the response that provides the form, the server may notify the client about the ACTION resource:

HTTP/1.1 200 OK
Content-Type: text/html
Protocol-Info: {http://some.org/payment-thingy {for /cgi-bin/buy *} {str req}}

<form action="/cgi-bin/buy">

420: Bad Extensions

A server policy may require (or refuse) the use of some extensions in some circumstances. If a request fails to fulfill the policy, the server should respond with a 420 status code (Bad Extensions) and specify the policy using the Protocol-Info header field.

Implementors may note the similarity to the way authentication challenges are issued with the 401 (Unauthorized) status code.

Extension Header Fields

Each HTTP extension that uses the PEP mechanism may define one or more extension header fields.

Note that params in extension declarations provide the same facility with less complexity, and provide a syntactic structure that closely resembles the semantic structure. This mechanism is redundant, but provided for the case where the use of header fields is essential.

Each extension header field present in a message is associated with exactly one protocol extension identifier in a Protocol or C-Protocol header field.

It is an error (400 Bad Request) to include the same header field name in two different extension-decls in the same message, and it is an error if a header field name matches wildcard prefixes in more than one extension-decl.

Wildcard matching is as follows: A header field name N matches a prefix P-* iff N is the concatenation of Q- and any string S, where P and Q are the same except for differences in the case of letters.

For example:

GET /newsletter.html HTTP/1.1
   Protocol: {http//www.someschool.edu/HTTP/MicroPay {headers micropay} }
   Micropay: USD $0.003 creds:lw3jlkwj3lkw3ljk

or using a wildcard prefix:

GET /newsletter.html HTTP/1.1
   Protocol: {http//www.someschool.edu/HTTP/MicroPay {headers M-*} }
   M-micropay: USD $0.003 creds:lw3jlkwj3lkw3ljk

Header Field Name Collisions

It is possible that two extensions specify the use of the same header field name. If two such extensions are used in the same message, the name collision must be resolved, either by prefixing or replacing the header names.

The header field names in the message can be replaced with arbitrary names; the header fields must be given a distinguished order in the protocol extension definition. This order can be used to associate the replacement names with the original semantics.

For example, consider extensions E1 and E2. E1 involves headers Tax and Price, and E2 involves Price and Color.

These might be combined in the same message as:

Protocol: {E1 {headers price tax}}
Price: $2.99
Tax: 8.25%
Protocol: {E2 {headers AB12-price color }}
AB12-Price: $2.99
Color: red

Since the first extension header specified in E2 is Price, the semantics of the AB12-price header are clear.

Header prefixing is similar; if the name in the protocol extension specification is N, and the distinguishing prefix is P-, then the name used in the message is P-N. For example:

Protocol: {E1 {headers price tax}}
Price: $2.99
Tax: 8.25%
Protocol: {E2 {headers AB12-*}}
AB12-Price: $2.99
AB12-Color: red

6. Extension Encodings

Each HTTP extension that uses the PEP mechanism may define one or more extension content encodings. Each extension content encoding in the Content-Encoding header field present in a message is associated with exactly one protocol extension identifier in a Protocol header field.

The association is either by name (case insensitive) or by wildcard prefix, as above. Name collisions must be resolved by prefixing.

For example:

GET /sparse-document HTTP/1.1
Protocol: {http://some.org/special-encoding {enc special}}

HTTP/1.1 200 OK
Protocol: {http://some.org/special-encoding {enc special}}
Content-Encoding: special
Content-Type: application/sparse-data

... sparse data encoded with "special" encoding ...

Security Considerations

The for parameter allows one party to give information about the extensions used by another party's resources. The parties may provide resources on different servers, or at different addresses on the same server. While there is no reasonable way for clients to distinguish between the parties responsible for different resources at the same server, clients should ignore information given by one server about another unless they have reason to trust it, or reason to believe that trusting it will have no significant negative consequences.

Future Work

Further design and implementation work is necessary to completely meet the requirements for PEP.

Binding Extensions

This design does not completely meet the requirement that one party can require another party to participate in an extension. An earlier draft specified a new version number and the use of {str req} in extension-declarations. But this will have no impact on HTTP 1.1 clients and servers, and hence does not meet the requirement.

One possibility is a change to the syntax of methods in HTTP request for the purpose of expressing binding extensions. For example:

BINDING PUT /a-resource HTTP/1.2
Protocol: {http://some.org/rights-management {str req}
   {params {copyright-remains-with-client}
            {nonexclusive-right-to-redistribute} }
Content-Type: text/html

<!doctype html ...

Unfortunately, this does not accomodate the case of a binding end-to-end extension that passes through a proxy.

Mult-Transaction Negotiation

An earlier draft of PEP included a mechanism for multi-transaction negotiation. Implementation experience showed the need to identify clients across transactions, which the mechanism did not provide.

It is possible, within the design specified here, to do multi-transaction negotiation within an extension (for example, by putting information to disambiguate conversation threads in the params).

Other possibilities under consideration include the use of state management "cookies" to disambiguate clients, or the use of an analagous PEP-specific mechanism.

Appendix: Considerations for the Design of a PEP Software Component Interface

This section got blown away in an editing disaster. The editor will attempt to include it in a future draft.

Normative References

T. Berners-Lee and L. Masinter and M. McCahill, RFC 1738: Uniform Resource Locators (URL), pp. 25, Dec 1994.
R. Fielding, J. Gettys, J. C. Mogul, H. Frystyk, T. Berners-Lee, Hypertext Transfer Protocol -- HTTP/1.1. RFC 2068 UC Irvine, DEC, DEC, W3C/MIT, W3C/MIT, January 1997

Bibliography: Informative References

D. Robinson The WWW Common Gateway Interface Version 1.1, work in progress 15 February 1996
T. Berners-Lee & D. Connolly, November 1995. Hypertext Markup Language - 2.0 RFC 1866
Netscape server API documentation, 1995
ISAPI documentation, Microsoft Corporation, in ActiveX Alpha SDK, http://www.msn.com/download/sdk/msactivedk.zip, 1996
Thau, Robert, Design considerations for the Apache Server API, Fifth International World Wide Web Conference, May 6-10, 1996, Paris, France
OpenMarket server technical overview sometime in 1996.
Spyglass Server Application Development Interface Spyglass, Inc. version 1.17 1995/09/11
N. Borenstein, RFC 1524: A User Agent Configuration Mechanism For Multimedia Mail Format Information, pp. 12, Sep 1993.
D. Kristol, L. Montulli, 22 Nov 1996. "HTTP State Management Mechanism",
RFC xxxx. Proposed Standard Approved by the IESG, not yet assigned an RFC.
David M. Kristol, A Proposed Extension Mechanism for HTTP, Jan 1995. D. Kristol, A Proposed Extension Mechanism for HTTP, Internet Draft, January 1995 (Work in Progress, Expired).
D. H. Crocker. Standard for the Format of ARPA Internet Text Messages. STD 11, RFC 822, UDEL, August 1982.

D. Eastlake, "Universal Payment Preamble", Internet Draft CyberCash, March 1996 (Work in Progress).

JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft, August 1996 (Work in Progress). [Also available as http://www.w3.org/pub/WWW/Payments/JEPI/draft-jepi-uppflow-00.html]
J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker. "SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach Consulting, Network Management Associates, Brandenburg Consulting, November 1995.
J. Miller. PICS Label Syntax and Communication Protocols (Version 1.1).; W3C Proposed Recommendation PR-PICS-services , W3C/MIT, August 1996.
Spyglass Client Software Development Kit
Electronic Commerce Standards for the WWW
WN server documentation, 1995
Spinner server technical overview, http://spinner.infovav.se/overview.html, 1995


This draft of PEP is the product of a substantial amount of investigation and collaboration. Dave Kristol did some of the first writing on HTTP extension mechanisms. [Kristol95]. Jim Miller and Dave Raggett sketched out an initial design, which Rohit Khare wrote up in a number of drafts.

This draft is a direct reflection of some implementation work: a client implementation Henrik Frystyk Nielsen et. al. (see the HTPEP module in libwww) and a server implementation by Eui Suk Chung and Anit Chakraborty for the JEPI project.

Tim Berners-Lee contributed significantly to the requirements section, and Daniel Dardailler provided extensive review ocmments.

Author's Addresses

Dan Connolly
Architecture Domain Lead, W3C
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (512) 310 2971
Email: connolly@w3.org

Rohit Khare
Technical Staff, W3C
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org

Henrik Frystyk Nielsen
Technical Staff, W3C
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 8143
Fax: +1 (617) 258 5999
Email: frystyk@w3.org