W3C: WD-pep-970321

PEP: an Extension Mechanism for HTTP

W3C Working Draft 21 Mar 1997

Dan Connolly
This version:
$Id: pep-spec.html,v 1.20 1997/03/21 20:59:19 connolly Exp $
Latest Released Version:
Previous Version:

Status of this Document

This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress".

To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast).

This document is also available as a W3C Working Draft. The most recent release is available at http://www.w3.org/pub/WWW/TR/WD-http-pep.

Distribution of this document is unlimited. Please send comments to the HTTP working group at http-wg@cuckoo.hpl.hp.com. Discussions of the working group are archived at http://www.ics.uci.edu/pub/ietf/http/. The editor maintains background information about PEP at http://www.w3.org/pub/WWW/Protocols/PEP/

The contribution of W3 Consortium staff time to the HTTP working group is part of the W3C HTTP Activity.


HTTP is an extensible protocol. PEP is an extension mechanism designed to address the tension between private agreement and public specification and to accommodate extension of HTTP clients and servers by software components.

The PEP mechanism is to associate each extension with a URI, and use a few new header fields to carry the extension identifier and related information from HTTP clients, thru proxies and intermediaries, to servers, and back again.

PEP relies on some HTTP 1.1 features, but is intended to be compatible with all versions of HTTP from 1.1 on, and to be compatible with HTTP/1.0 inasmuch as HTTP 1.1 is compatible with HTTP/1.0.

The kinds of extension protocol capable of being introduced by PEP are envisioned as ranging from:



HTTP is a generic request-response protocol, designed to accommodate a variety of applications, from network information retrieval and searching to file transfer and repository access to query and forms processing.

HTTP is used increasingly in applications that need more facilities than the standard version of the protocol provides, from distributed authoring, collaboration and printing, to various remote procedure call mechanisms.

Many of these applications do not require agreement across the whole Internet about the extended facilities; rather, it suffices:

This document defines PEP, an extension mechanism for HTTP. The PEP design is the result of analyzing a variety of HTTP extensions and extension mechanisms, and the motivation behind them.

PEP relies on some HTTP 1.1 features, but is intended to be compatible with all versions of HTTP from 1.1 on, and to be compatible with HTTP/1.0 inasmuch as HTTP 1.1 is compatible with HTTP/1.0. See section Considerations for Defining Extensions.

Operational Overview

PEP is intended to be used as follows:

  1. Some party designs and specifies an extension to HTTP; the party assigns the extension an identifier which is a URI, and they make the specification of the protocol available at that address.
  2. Extended clients and servers are implemented per the HTTP specification as extended by the extension specification.
  3. Requests and responses declare the use of the extension by reference to its URI.
  4. If the extension becomes ubiquitous, a new version of the HTTP specification can include the extension, and messages can declare use of the new HTTP version instead of the extension.
Editor's note: I have used the term URI throughout this specification, since PEP relies only on the identification of resources, not on location nor resolution. URNs are expected to serve just as well as URLs for use as PEP extension identifiers. Hence, I cite the URL syntax draft in progress.

Note that, at the cost of some extra bytes to spell out the URI in full, the use of a central registry of extension names is avoided.

See Considerations for Defining Extensions for more on defining extensions.

The PEP mechanism is designed to accommodate extension of clients, servers, and proxies by software components as follows:

  1. Clients and servers are implemented with software component interfaces that allow dynamic installation of extension facilities.
  2. An extension is assigned a URI; in addition to a human-readable specification of an extension, a machine-readable implementation or description of the extension is published at that address.
  3. If a message that refers to an extension is received by a party that has no awareness of the extension, the receiver can dereference the extension's identifier and dynamically load support for the extended facility.

Strength, Scope and Semantics of Extended Transactions

The agents in an HTTP transaction are a client and a server, which send HTTP messages to each other, with intermediaries between them in some cases. However, semantically, an HTTP transaction is between a client party (for example, the referent of the From: header field) and a the principle responsible for the publication of a given resource.

The publishing party is basically the one responsible for the service provided at any particular URI, for example, the mapping between the URI and any representation of the resource to which it refers. Exactly who takes this role is beyond the scope this document, but for example, it may be the writer of a document, the server administrator, the organization running the server, or a combination of these.

PEP extensions MAY be used to extend the end-to-end transaction semantics, or, using the Connection header field (see [HTTP] section 14.10 Connection), they MAY be used to extend the hop-by-hop transaction semantics. See The Protocol Header Field and Hop-by-hop Extensions for details.

A PEP extension declaration has a strength which MAY be either required or optional. If it is optional, the transaction may succeed even if some of the parties within its scope do not participate. If it is reuired, it MUST be understood and responded to by all parties within its scope. See The Protocol Header Field and Binding Request for details.

The Protocol Header Field

The extensions used in a message are declared using the Protocol request/response header field.

The syntax is:

          Protocol         = "Protocol" ":" 1#extension-decl
                           | "C-Protocol" ":" 1#extension-decl
          extension-decl   = "{" extension-id 1*LWS *ext-info "}"
          extension-id     = URI
          ext-info         = str | enc | params
          params           = "{" "params" *bagitem "}"
          str              = "{" "str" ("req" | "opt" ) "}"
          enc              = "enc"
          bag              = "{" bagname *bagitem "}"
          bagname          = token ¦ URI
          bagitem          = bag ¦ token ¦ quoted-string
          URI              = 1*<any char except CTLs or space>

Each extension-decl:

Note that, since URIs may contain { and } characters, a space is required after the extension-id.

For example:

   GET /a-document HTTP/1.1
   Host: a.host
   Protocol: {http://some.org/an-extension }
   HTTP/1.1 200 OK
   Protocol: {http://some.org/an-extension }
   Vary: Protocol
   Content-Type: text/plain
   Glad you're using an-extension!

Note the use of the Vary header to notify proxies that responses to GET /a-document depend on the Protocol header fields used in the request. See [HTTP], section 14.43 Vary.


Each extension-decl declares the use of the extension to be one of:

Required: {str req}
Ignorance of the extension or failure to process the extension declaration by the relevant agent(s) is an error that results in failure of the entire transaction, and SHOULD be reported using the 420 Bad Extension status code. See also: Binding Request.
Optional: {str opt}
The success of the transaction does not depend on any agents recognizing or processing the extension declaration. This is the default.

If any of the extension declarations in a request is required, then the request is a binding request. See: Binding Request.

The issue of "relevant agents" is defined in section Hop-by-hop Extensions and summarized in section Summary of Protocol Interactions.

Hop-by-hop Extensions

Extensions declared with the Protocol header field are end-to-end extensions. Hop-by-hop extensions are declared with the C-Protocol header field, in conjunction with the Connection header ([HTTP], section 13.5.1 and 14.10).

The relevant agents in a hop-by-hop extension are the agents at the ends of the connection.

The relevant agents in an end-to-end extension are the origin client and server, and depending on the extension in question any intermediaries acting on behalf of the origin client and server in ccordance with the extension specification and any private agreements.

Binding Request

A request with {str req} in any of its Protocol header fields is a binding request -- the transaction cannot be succeed without consulting and adhering to the relevant extension specification(s).

Because legacy HTTP agents MAY ignore all protocol header fields, the {str req} is not sufficient to evoke the correct behaviour from HTTP agents.

The method name of all binding request MUST be prefixed by BINDING-. Legacy HTTP agents (i.e. agents implemented without consulting this specification) SHOULD respond with 501 (Not Implemented) (see [HTTP] section 5.1.1, Method). Other agents MUST process the request resulting from removing the BINDING- from the method name and leaving the rest of the request (request URI, version, header fields, body) as is.

NOTE: All method names beginning with BINDING- are reserved for this use.

For example, a client might express the binding rights-management constraints on its request as follows:

   BINDING-PUT /a-resource HTTP/1.2
   Protocol: {http://some.org/rights-management {str req}
      {params {copyright-remains-with-client}
               {nonexclusive-right-to-redistribute} }
   Host: some.hose
   Content-Length: 1203
   Content-Type: text/html
   <!doctype html ...

Summary of Protocol Interactions

The processing of PEP extensions is subject to a number of considerations:

Proxy or Origin server?
Proxies need not process end-to-end extensions, but they do Connection header processing.
Extension supported? PEP supported?
Some agents will have no support for PEP (in particular, the BINDING- syntax). Some agents will have support for the relevant extension, and some will not. Even those that support the extension may elect not to process it in some cases; this is indistinguishable from lack of support.
Hop-by-hop or End-to-end?
The Connection header is used to be sure that C-Protocol and C-Protocol-Info headers are processed by exactly one receiving agent.
Optional or Required?
Is the extension essential to the transaction?

The following table summarizes the outcome in each case:

The proxy MUST pass the extension declaration along to the next agent.
The proxy MUST strip the extension declaration out and pass the remainder along to the next agent.
extended processing
The agent MUST process the request in conformance with the extesion speification.
standard processing
The agent MUST process the request in the standard, unextended fashion.
420 Bad Extension
The agent MUST report a 420 Bad Extension error
Editor's note: I need to make a plain-text equivalent of the following table. Until then, please see the HTML version.
Strength / Scope PEP Summary
  Hop-by-hop End-to-end
Optional Required Optional Required
Proxy PEP not supported


501 not implemented


501 not implemented

Extension not supported


420 Bad Extensions



Extension supported

extended processing

extended processing

extended processing

extended processing

Origin Server PEP not supported

standard processing

501 not implemented

standard processing

501 not implemented

Extension not supported

standard processing


standard processing


Extension supported

extended processing

extended processing

extended processing

extended processing

*1: HTTP/1.0 proxies might not do Connection: processing, so they might pass such extension declarations along.

Extension Encodings

An extension declaration MAY use the enc to signal that it is an extension encoding, that is, an extension that involves encoding the body of the message.

For example:

   GET /sparse-document HTTP/1.1
   Host: a.host
   Protocol: {http://some.org/special-encoding enc}
   HTTP/1.1 200 OK
   Protocol: {http://some.org/special-encoding enc}
   Content-Type: application/sparse-data
   ... sparse data encoded with special-encoding ...

Encodings are applied in the order that they occur in the head of the message. For example:

   HTTP/1.1 200 OK
   Protocol: {http://some.org/inner-encoding enc},
    {http://some.org/outer-encoding enc}
   Content-Type: text/plain
   ... text encoded with inner-encoding, then outer-encoding...

While the order of Protocol header fields is guaranteed to be preserved across proxy boundaries, the order of Protocol header fields with respect to Content-Encoding header fields is not ([HTTP] section 4.2 Message Headers); hence the use of Content-Encoding is prohibited in messages with extension encodings.

Extension Policy Information

Some extensions are used spontaneously by participating agents; for example, a client may be configured to use and extension, or a user interface option may trigger the use of an extension.

But in many cases, a server dictates the use of one or more extensions. In this case, it is useful for the server to communicate its policies to clients.

The server MAY notify the client that some resources SHOULD be accessed using one or more extensions with the Protocol-Info entity header field. The resources are specified by a relative or absolute URI, with an optional wildcard flag indicating that the notification applies to all URIs containing the specified URI as a prefix.

The syntax is:

          Protocol-Info    = "Protocol-Info" ":" 1#policy-decl
                           | "C-Protocol-Info" ":" 1#policy-decl
          policy-decl      = "{" extension-id 1*LWS *policy-info "}"
          policy-info      = policy-str | params | for
          policy-str       = "{" "str" ("req" | "ref" | "opt" ) "}"
          for              = "{" "for" URI [ wildcard ] "}"
          wildcard         = "*"

The for syntax specifies the URI (or set of URIs) to which the policy declaration applies. A URI followed by a wildcard represents the set of URIs that contain the given URI as a prefix. The default, in the case that the for syntax does not appear, is the request URI of the transaction.

Note that A policy-decl with a for parameter MAY give information about a different resource from the resource described by the other header fields in the same message. Nonetheless, the freshness of the information in the Protocol-Info header field is the same as the rest of the header fields (which see [HTTP] section 13.2, "Expiration Model").

The policy-decl is strictly advisory. The client SHOULD heed the policy-decl on its next request to the relevant server, unless the delay between receiving the policy-decl and that next request far exceeds the freshness of the reply containing the Protocol-Info header.

The strength of the policy for an extension for the resources MUST be one of req, ref, or opt.

Required. The resource MUST be accessed using the extension; that is, the server expects any attempt to access the resource without using the extension to fail.
Optional. The resource MAY be accessed using the extension or not using the extension.
Refused. The resource MUST NOT be accessed using the extension; that is, any attempt to access the resource using the extension is expected to fail.

For example, consider the case of an HTML form, where the associated ACTION resource requires a payment extension. In the response that provides the form, the server may notify the client about the ACTION resource:

   HTTP/1.1 200 OK
   Content-Type: text/html
   Protocol-Info: {http://some.org/payment-thingy {for /cgi-bin/buy *} {str req}}
   <form action="/cgi-bin/buy">

Hop-by-hop Policies

The C-Protocol-Info header field provides hop-by-hop policies; that is, it allows a server to express policy(ies) to an agent at the other end of an HTTP connection, rather than to all parties in an HTTP transaction. Other than scope, its semantics are the same as the Protocol-Info header field; the name is distinct so that the Connection header field can distinguish between hop-by-hop and end-to-end protocol information notifications.

For example, consider a server whos policy is to access cache usage statistics from clients that connect to it. In response from a client, it might advertise its policy as follows:

   HTTP/1.1 200 OK
   C-Protocol-Info: {http://some.org/provide-stats {for / * }}
   Connection: C-Protocol-Info
   Content-Type: text/plain
   some content

The next time that client makes a request to this server, it may provide statistics as follows:

   GET /some-resource HTTP/1.1
   Host: some.org
   C-Protocol: {http://some.org/provide-stats {params {hits 10}}}
   Connection: C-Protocol

420: Bad Extensions

A server policy MAY require (or refuse) the use of some extensions in some circumstances. If a request fails to fulfill the policy, the server SHOULD respond with a 420 status code (Bad Extensions) and specify the policy using the Protocol-Info header field.

Implementors may note the similarity to the way authentication challenges are issued with the 401 (Unauthorized) status code.

Considerations for Defining Extensions

While the protocol extension definition SHOULD be published at the address of the extension identifier, this is not strictly necessary. The only absolute requirement is that distinct names be used for distinct semantics.

For example, one way to achieve this is to use an mid:, cid:, or uuid: URI. The association between the extension identifier and the specification might be made by distributing a specification which references the extension identifier. Care should be taken not to distribute conflicting specifications which reference the same name.

Even when the web is used to publish extension specifications, care must be taken that the specification made available at that address does not change significantly over time. One agent may associate the identifier with the old semantics, and another might associate it with the new semantics.

Interaction with Caching, Connections, etc.

For each aspect of an extension, the interaction with other aspects of HTTP/1.1 SHOULD be fully specified, and the issues of compatibility SHOULD be discussed. For example, any extension headers which are not strictly entity headers require careful consideration.

In particular:

Cache-Control, [HTTP] section 14.9
Any interactions with existing and extended cache-control directives MUST be specified
[HTTP] 19.7 Compatibility with Previous Versions
Interaction with HTTP 1.0 clients and servers SHOULD be clarified.
Upgrade (14.41)
Extensions which define transitions to new application-layer protocols SHOULD discuss interaction with the Upgrade header, at least to mention that PEP is being used instead of the Upgrade header.
Content-Encoding (14.12)
Extension encodings SHOULD mention the prohibition against mixing Content-Encoding with extension encodings, among other interactions.

Bootstrapping and Dynamic Loading

The extension definition MAY be made available in different representations. For example, a software component that implements the specification MAY reside at the same address as a human-readable specification (distinguished by content negotiation).

The human-readable representation serves to document the extension and encourage deployment, while the software component to allows clients and servers to be dynamically extended.

Security Considerations

Future Work

The design of some aspects of earlier drafts of this specification are still pending implementation experience.

Multi-Transaction Negotiation

An earlier draft of PEP included a mechanism for multi-transaction negotiation. Implementation experience showed the need to identify clients across transactions, which the mechanism did not provide.

It is possible, within the design specified here, to do multi-transaction negotiation within an extension (for example, by putting information to disambiguate conversation threads in the params).

Other possibilities under consideration include the use of state management "cookies" to disambiguate clients, or the use of an analogous PEP-specific mechanism.

Appendix: Considerations for the Design of a PEP Software Component Interface

This section got blown away in an editing disaster. If requested, the editor will attempt to include it in a future draft.

Normative References

T. Berners-Lee, R. Fielding, L. Masinter, Uniform Resource Locators (URL), draft-fielding-url-syntax-03 , MIT/LCS, U.C. Irvine, Xerox Corporation, December 1996, work in progress
R. Fielding, J. Gettys, J. C. Mogul, H. Frystyk, T. Berners-Lee, Hypertext Transfer Protocol -- HTTP/1.1. RFC 2068 U.C. Irvine, DEC, DEC, W3C/MIT, W3C/MIT, January 1997
@@may/must/should RFC

Bibliography: Informative References

D. Robinson The WWW Common Gateway Interface Version 1.1, work in progress 15 February 1996
Netscape server API documentation, 1995
ISAPI documentation, Microsoft Corporation, in ActiveX Alpha SDK, http://www.msn.com/download/sdk/msactivedk.zip, 1996
Thau, Robert, Design considerations for the Apache Server API, Fifth International World Wide Web Conference, May 6-10, 1996, Paris, France
OpenMarket server technical overview sometime in 1996.
Spyglass Server Application Development Interface Spyglass, Inc. version 1.17 1995/09/11
N. Borenstein, RFC 1524: A User Agent Configuration Mechanism For Multimedia Mail Format Information, pp. 12, Sep 1993.
D. Kristol, L. Montulli, 22 Nov 1996. "HTTP State Management Mechanism",
RFC xxxx. Proposed Standard Approved by the IESG, not yet assigned an RFC.
David M. Kristol, A Proposed Extension Mechanism for HTTP, Jan 1995. D. Kristol, A Proposed Extension Mechanism for HTTP, Internet Draft, January 1995 (Work in Progress, Expired).
D. H. Crocker. Standard for the Format of ARPA Internet Text Messages. STD 11, RFC 822, UDEL, August 1982.

D. Eastlake, "Universal Payment Preamble", Internet Draft CyberCash, March 1996 (Work in Progress).

JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft, August 1996 (Work in Progress). [Also available as http://www.w3.org/pub/WWW/Payments/JEPI/draft-jepi-uppflow-00.html]
J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker. "SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach Consulting, Network Management Associates, Brandenburg Consulting, November 1995.
J. Miller. PICS Label Syntax and Communication Protocols (Version 1.1). 31 October 1996
Spyglass Client Software Development Kit
Electronic Commerce Standards for the WWW
WN server documentation, 1995
Spinner server technical overview, http://spinner.infovav.se/overview.html, 1995


This draft of PEP is the product of a substantial amount of investigation and collaboration. Dave Kristol did some of the first writing on HTTP extension mechanisms. [Kristol95]. Jim Miller and Dave Raggett sketched out an initial design, which Rohit Khare wrote up in a number of drafts. Rohit also coined the term "PEP."

This draft is a direct reflection of some implementation work: a client implementation Henrik Frystyk Nielsen et. al. (see the HTPEP module in libwww) and a server implementation by Eui Suk Chung and Anit Chakraborty for the JEPI project.

Tim Berners-Lee contributed significantly to the requirements section, and Daniel Dardailler provided extensive review comments.

Authors Addresses

Dan Connolly
Architecture Domain Lead, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (512) 310 2971
Email: connolly@w3.org

Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org

Henrik Frystyk Nielsen
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 8143
Fax: +1 (617) 258 5999
Email: frystyk@w3.org