W3C: WD-pep-970313

PEP: an Extension Mechanism for HTTP

W3C Working Draft 13 Mar 1997

Dan Connolly
This version:
$Id: pep-spec.html,v 1.16 1997/03/13 05:37:56 connolly Exp $
Latest Released Version:
Previous Version:

Status of this Document

This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress".

To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast).

This document is also available as a W3C Working Draft. The most recent release is available at /TR/WD-http-pep.

Distribution of this document is unlimited. Please send comments to the HTTP working group at http-wg@cuckoo.hpl.hp.com. Discussions of the working group are archived at http://www.ics.uci.edu/pub/ietf/http/. The editor maintains background information about PEP at /Protocols/PEP/

The contribution of W3 Consortium staff time to the HTTP working group is part of the W3C HTTP Activity.


HTTP is an extensible protocol. PEP is an extension mechanism designed to address the tension between private agreement and public specification and to accomodate extension of HTTP clients and servers by software components.

The PEP mechanism is to associate each extension with a URL, and use a few new header fields to carry the extension identifier and related information from HTTP clients, thru proxies and intermediaries, to servers, and back again.

PEP relies on some HTTP 1.1 features, but is intended to be compatible with all versions of HTTP from 1.1 on, and to be compatible with HTTP/1.0 inasmuch as HTTP 1.1 is compatible with HTTP/1.0.


1. Introduction

HTTP is a generic request-response protocol, designed to accomodate a variety of applications, from network information retrieval and searching to file transfer and repository access to query and forms processing.

HTTP is used increasingly in applications that need more facilities than the standard version of the protocol provides, from distributed authoring, collaboration and printing, to various remote procedure call mechanisms.

Many of these applications do not require agreement across the whole Internet about the extended facilities; rather, it suffices:

This document defines PEP, an extension mechanism for HTTP. The PEP design is the result of analyzing a variety of HTTP extensions and extension mechanisms, and the motivation behind them.

PEP relies on some HTTP 1.1 features, but is intended to be compatible with all versions of HTTP from 1.1 on, and to be compatible with HTTP/1.0 inasmuch as HTTP 1.1 is compatible with HTTP/1.0. See section Considerations for Defining Extensions.

1.1. Operational Overview

PEP is intended to be used as follows:

  1. Some party designs and specifies an extension to HTTP; the party assigns the extension an identifier which is a URL, and they make the specification of the protocol available at that URL.
  2. Extended clients and servers are implemented per the HTTP specification as extended by the extension specification.
  3. Requests and responses declare the use of the extension by reference to its URL.
  4. If the extension becomes ubiquitous, a new version of the HTTP specification can include the extension, and messages can declare use of the new HTTP version in stead of the extension.

Editor's note: I have used the term URL throughout this specification, following the popular jargon from the NCSA Mosaic documentation and and the standards track documents RFC1738 and RFC1808. The HTTP 1.1 spec uses URL in some places and URI in others. I don't understand the distinction, and I suspect that none was intended. PEP does not exploit any distinctions between URIs and URLs, and so I would prefer to use only one of the terms.

Note that, at the cost of some extra bytes to spell out the URL in full, the use of a central registry of extension names is avoided.

See Considerations for Defining Extensions for more on defining extensions.

The PEP mechanism is designed to accomodate extension of clients, servers, and proxies by software components as follows:

  1. Clients and servers are implemented with software component interfaces that allow dynamic installation of extension facilities.
  2. An extension is assigned a URL; in addition to a human-readable specification of an extension, a machine-readable implementation or description of the extension is published at that address.
  3. If a message that refers to an extension is received by a party that has no awareness of the extension, the receiver can dereference the extension's identifier and dynamically load support for the extended facility.

1.2. Strength, Scope and Semantics of Extended Transactions

The agents in an HTTP transaction are a client and a server, which send HTTP messages to each other, with intermediaries between them in some cases. However, semantically, an HTTP trasacion is between a client party (for example, the referent of the From: header field) and a the principle responsible for the publication of a given resource.

The publishing party is basically the one responsible for the service provided at any particular URL, for example, the mapping between the URL and any representation of the resource to which it refers. Exactly who takes this role is beyond the scope this document, but for example, it may be the writer of a document, the server administrator, the organization running the server, or a combination of these.

PEP extensions may be used to extend the end-to-end transaction semantics, or, using the Connection header field (see [HTTP] section 14.10 Connection), they may be used to extend the hop-by-hop transaction semantics. See The Protocol Header Field and Hop-by-hop Extensions (Section 2.2) for details.

PEP extensions often define inessential, non-binding facilities; that is, the transaction can still succeed even if some parties do not participate in the extension. The distinction between binding and non-binding uses of extensions is syntactically evident in requests and responses. See The Protocol Header Field and Binding Extensions Request for details.

2. The Protocol Header Field

The extensions used in a message are declared using the Protocol request/response header field.

The syntax is:

           Protocol         = "Protocol" ":" 1#extension-decl
                            | "C-Protocol" ":" 1#extension-decl
           extension-decl   = "{" extension-id *ext-info "}"
           extension-id     = URL
           ext-info         = str | enc | params
           params           = "{" "params" *bagitem "}"
           str              = "{" "str" ("req" | "ref" | "opt" ) "}"
           enc              = "enc"
           bag              = "{" bagname 1*LWS *bagitem "}"
           bagname          = token  URL
           bagitem          = bag  token  quoted-string
           URL              = 1*<any char except CTLs or space>

Each extension-decl:

Note that, since URLs may contain { and } characters, a space is required after the bagname.

For example:

    GET /a-document HTTP/1.1
    Host: a.host
    Protocol: {http://some.org/an-extension }
    HTTP/1.1 200 OK
    Protocol: {http://some.org/an-extension }
    Vary: Protocol
    Content-Type: text/plain
    Glad you're using an-extension!

Note the use of the Vary header to notify proxies that responses to GET /a-document depend on the Protocol header fields used in the request. See [HTTP], section 14.43 Vary.

2.1. Strength

Each extension-decl declares the use of the extension to be one of:

Required: {str req}
Ignorance of the extension or failure to process the extension declaration by the relavent agent(s) is an error that results in failure of the entire transaction, and should be reported using the 420 Bad Extension status code. Agents should be prepared for a 500 Not implemented response as well. See: Binding Extensions Request.
Optional: {str opt}
The success of the transaction does not depend on any agents recognizing or processing the extension declaration. This is the default.

If any of the extension declarations in a request is required, then the request is a binding extension request. See: Binding Extensions Request.

The issue of "relavent agents" is defined in section Hop-by-hop Extensions and summarized in section Summary of Protocol Interactions.

2.2. Hop-by-hop Extensions

Extensions declared with the Protocol header field are end-to-end extensions. Hop-by-hop extensions are declared with the C-Protocol header field, in conjunction with the Connection header ([HTTP], section 13.5.1 and 14.10).

The relavent agents in a hop-by-hop extension are the agents at the ends of the connection.

The relavent agents in an end-to-end extension are the origin client and server.

Editor's note: Are proxies ever relavent to end-to-end extensions?

3. Binding Extensions Request

A request with {str req} in any of its Protocol header fields is a binding extension request -- the transaction cannot be succeed without consulting and adhering to the relavent extension specification(s).

Because legacy HTTP agents may ignore all protocol header fields, the {str req} is not sufficient to evoke the correct behaviour from HTTP agents.

The method name of all binding extension request must be prefixed by BINDING-. Legacy HTTP agents (i.e. agents implemented without consulting this specification) should respond with 501 (Not Implemented) (see [HTTP] section 5.1.1, Method). Other agents must process the request resulting from removing the BINDING- from the method name and leaving the rest of the request (request URL, version, header fields, body) as is.

NOTE: All method names beginning with BINDING- are reserved for this use.

For example, a client might express the binding rights-management constraints on its request as follows:

    BINDING-PUT /a-resource HTTP/1.2
    Protocol: {http://some.org/rights-management {str req}
       {params {copyright-remains-with-client}
                {nonexclusive-right-to-redistribute} }
    Host: some.hose
    Content-Length: 1203
    Content-Type: text/html
    <!doctype html ...

4. Summary of Protocol Interactions

The processing of PEP extensions is subject to a number of considerations:

Proxy or Origin server?
Proxies need not process end-to-end extensions, but they do Connection header processing.
Extension supported? PEP supported?
Some agents will have no support for PEP (in particular, the BINDING- syntax). Some agents will have support for the relavent extension, and some will not. Even those that support the extension may elect not to process it in some cases; this is indistinguishable from lack of support.
Hop-by-hop or End-to-end?
The Connection header is used to be sure that C-Protocol and C-Protocol-Info headers are processed by exactly one receiving agent.
Optional or Required?
Is the extension essential to the transaction?

The following table summarizes the outcome in each case:

The proxy will pass the extension declaration along to the next agent.
The proxy will strip the extension declaration out and pass the remainder along to the next agent.
extended processing
The agent will process the request in the extended fashion.
standard processing
The agent will process the request in the standard, unextended fashion.
420 Bad Extension
The agent will report a 420 Bad Extension error

Editor's note: I need to make a plain-text equivalent of the following table. Until then, please see the HTML version.

Strength / Scope PEP Summary
  Hop-by-hop End-to-end
Optional Required Optional Required
Proxy PEP not supported


501 not implemented


501 not implemented

Extension not supported


420 Bad Extensions



Extension supported

extended processing

extended processing

extended processing

extended processing

Origin Server PEP not supported

standard processing

501 not implemented

standard processing

501 not implemented

Extension not supported

standard processing


standard processing


Extension supported

extended processing

extended processing

extended processing

extended processing

*1: HTTP/1.0 proxies might not do Connection: processing, so they might pass such extension declarations along.

5. Extension Encodings

An extension declaration may use the enc to signal that it is an extension encoding, that is, an extension that involves encoding the body of the message.

For example:

    GET /sparse-document HTTP/1.1
    Host: a.host
    Protocol: {http://some.org/special-encoding enc}
    HTTP/1.1 200 OK
    Protocol: {http://some.org/special-encoding enc}
    Content-Type: application/sparse-data
    ... sparse data encoded with special-encoding ...

Encodings are applied in the order that they occur in the head of the message. For example:

    HTTP/1.1 200 OK
    Protocol: {http://some.org/inner-encoding enc}
    Protocol: {http://some.org/outer-encoding enc}
    Content-Type: text/plain
    ... text encoded with inner-encoding, then outer-encoding...

While the order of Protocol header fields is guaranteed to be preserved accross proxy boundaries, the order of Protocol header fields with respect to Content-Encoding header fields is not ([HTTP] section 4.2 Message Headers); hence the use of Content-Encoding is prohibited in messages with extension encodings.

6. Extension Policy Information

Some extensions are used spontaneously by participating agents; for example, a client may be configured to use and extension, or a user interface option may trigger the use of an extension.

But in many cases, a server dictates the use of one or more extensions. In this case, it is useful for the server to communicate its policies to clients.

The server may notify the client that some resources should be accessed using one or more extensions with the Protocol-Info entity header field. The resources are specified by a relative or absolute URL, with an optional wildcard flag indicating that the notification applies to all URLs containing the specified URL as a prefix.

The syntax is:

           Protocol-Info    = "Protocol-Info" ":" 1#policy-decl
                            | "C-Protocol-Info" ":" 1#policy-decl
           policy-decl      = "{" extension-id *policy-info "}"
           policy-info      = policy-str | params | for
           policy-str       = "{" "str" ("req" | "ref" | "opt" ) "}"

Note that a Protocol-Info with a for parameter may give information about a different resource from the resource described by the other header fields in the same message. Nonetheless, the freshness of the information in the Protocol-Info header field is the same as the rest of the header fields (which see [HTTP] section 13.2, "Expiration Model").

The notice is strictly advisory. The client should heed the notice on its next request to the relavent server, unless the delay between receiving the notice and that next request far exceeds the freshness of the reply containing the Protocol-Info header.

The strength of the policy for an extension for the resources is one of req, ref, or opt.

Required. The resource must be accessed using the extension; that is, any attempt to access the resource without using the extension is expected to fail.
Optional. The resource may be accessed using the extension or not using the extension.
Refused. The resource may not be accessed using the extension; that is, any attempt to access the resource using the extension is expected to fail.

For example, consider the case of an HTML form, where the associated ACTION resource requires a payment extension. In the response that provides the form, the server may notify the client about the ACTION resource:

    HTTP/1.1 200 OK
    Content-Type: text/html
    Protocol-Info: {http://some.org/payment-thingy {for /cgi-bin/buy *} {str req}}
    <form action="/cgi-bin/buy">

6.1. Hop-by-hop Policies

The C-Protocol-Info header field provides hop-by-hop policies; that is, it allows a server to express policy(ies) to an agent at the other end of an HTTP connection, rather than to all parties in an HTTP transaction. Other than scope, its semantics are the same as the Protocol-Info header field; the name is distinct so that the Connection header field can distinguish between hop-by-hop and end-to-end protocol information notifications.

For example, consider a server whos policy is to access cache usage statistics from clients that connect to it. In response from a client, it might advertise its policy as follows:

    HTTP/1.1 200 OK
    C-Protocol-Info: {http://some.org/provide-stats {for / * }}
    Connection: C-Protocol-Info
    Content-Type: text/plain
    some content

The next time that client makes a request to this server, it may provide statistics as follows:

    GET /some-resource HTTP/1.1
    Host: some.org
    C-Protocol: {http://some.org/provide-stats {params {hits 10}}}
    Connection: C-Protocol

7. 420: Bad Extensions

A server policy may require (or refuse) the use of some extensions in some circumstances. If a request fails to fulfill the policy, the server should respond with a 420 status code (Bad Extensions) and specify the policy using the Protocol-Info header field.

Implementors may note the similarity to the way authentication challenges are issued with the 401 (Unauthorized) status code.

8. Considerations for Defining Extensions

While the protocol extension definition should be published at the address of the extension identifier, this is not strictly necessary. The only absolute requirement is that distinct names be used for distinct semantics.

For example, one way to achive this is to use an mid:, cid:, or uuid: URL. The association between the extension identifier and the specification might be made by distributing a specification which references the extension identifier. Care must be taken not to distribute conflicting specifications which reference the same name.

Even when the web is used to publish extension specifications, care must be taken that the specification made available at that address does not change significantly over time. One agent may associate the identifier with the old semantics, and another might associate it with the new semantics.

8.1. Interaction with Caching, Connections, etc.

For each aspect of an extension, the interaction with other aspects of HTTP/1.1 must be fully specified, and the issues of compatibility should be discussed. For example, any extension headers which are not strictly entity headers require careful consideration.

In particular:

Cache-Control, [HTTP] section 14.9
Any interactions with existing and extended cache-control directives must be specified
[HTTP] 19.7 Compatibility with Previous Versions
Interaction with HTTP 1.0 clients and servers should be clarified.
Upgrade (14.41)
Extensions which define transitions to new application-layer protocols should discuss interaction with the Upgrade header, at least to mention that PEP is being used in stead of the Upgrade header.
Content-Encoding (14.12)
Extension encodings should mention the prohibition against mixing Content-Encoding with extension encodings, among other interactions.

8.2. Bootstrapping and Dynamic Loading

The extension definition may be made available in different representations. For example, a software component that implements the specification may reside at the same address as a human-readable specification (distinguished by content negotation).

The human-readable representation serves to document the extension and encourage deployment, while the software component to allows clients and servers to be dynamically extended.

9. Security Considerations

10. Future Work

The design of some aspects of earlier drafts of this specification are still pending implementation experience.

10.1. Mult-Transaction Negotiation

An earlier draft of PEP included a mechanism for multi-transaction negotiation. Implementation experience showed the need to identify clients across transactions, which the mechanism did not provide.

It is possible, within the design specified here, to do multi-transaction negotiation within an extension (for example, by putting information to disambiguate conversation threads in the params).

Other possibilities under consideration include the use of state management "cookies" to disambiguate clients, or the use of an analagous PEP-specific mechanism.

11. Appendix: Considerations for the Design of a PEP Software Component Interface

This section got blown away in an editing disaster. If requested, the editor will attempt to include it in a future draft.

12. Normative References

T. Berners-Lee and L. Masinter and M. McCahill, RFC 1738: Uniform Resource Locators (URL), pp. 25, Dec 1994.
R. Fielding, J. Gettys, J. C. Mogul, H. Frystyk, T. Berners-Lee, Hypertext Transfer Protocol -- HTTP/1.1. RFC 2068 UC Irvine, DEC, DEC, W3C/MIT, W3C/MIT, January 1997

13. Bibliography: Informative References

D. Robinson The WWW Common Gateway Interface Version 1.1, work in progress 15 February 1996
T. Berners-Lee & D. Connolly, November 1995. Hypertext Markup Language - 2.0 RFC 1866
Netscape server API documentation, 1995
ISAPI documentation, Microsoft Corporation, in ActiveX Alpha SDK, http://www.msn.com/download/sdk/msactivedk.zip, 1996
Thau, Robert, Design considerations for the Apache Server API, Fifth International World Wide Web Conference, May 6-10, 1996, Paris, France
OpenMarket server technical overview sometime in 1996.
Spyglass Server Application Development Interface Spyglass, Inc. version 1.17 1995/09/11
N. Borenstein, RFC 1524: A User Agent Configuration Mechanism For Multimedia Mail Format Information, pp. 12, Sep 1993.
D. Kristol, L. Montulli, 22 Nov 1996. "HTTP State Management Mechanism",
RFC xxxx. Proposed Standard Approved by the IESG, not yet assigned an RFC.
David M. Kristol, A Proposed Extension Mechanism for HTTP, Jan 1995. D. Kristol, A Proposed Extension Mechanism for HTTP, Internet Draft, January 1995 (Work in Progress, Expired).
D. H. Crocker. Standard for the Format of ARPA Internet Text Messages. STD 11, RFC 822, UDEL, August 1982.

D. Eastlake, "Universal Payment Preamble", Internet Draft CyberCash, March 1996 (Work in Progress).

JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft, August 1996 (Work in Progress). [Also available as /Payments/JEPI/draft-jepi-uppflow-00.html]
J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker. "SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach Consulting, Network Management Associates, Brandenburg Consulting, November 1995.
J. Miller. PICS Label Syntax and Communication Protocols (Version 1.1). 31 October 1996
Spyglass Client Software Development Kit
Electronic Commerce Standards for the WWW
WN server documentation, 1995
Spinner server technical overview, http://spinner.infovav.se/overview.html, 1995

14. Acknowledgements

This draft of PEP is the product of a substantial amount of investigation and collaboration. Dave Kristol did some of the first writing on HTTP extension mechanisms. [Kristol95]. Jim Miller and Dave Raggett sketched out an initial design, which Rohit Khare wrote up in a number of drafts. Rohit also coined the term "PEP."

This draft is a direct reflection of some implementation work: a client implementation Henrik Frystyk Nielsen et. al. (see the HTPEP module in libwww) and a server implementation by Eui Suk Chung and Anit Chakraborty for the JEPI project.

Tim Berners-Lee contributed significantly to the requirements section, and Daniel Dardailler provided extensive review comments.

15. Authors Addresses

Dan Connolly
Architecture Domain Lead, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (512) 310 2971
Email: connolly@w3.org

Rohit Khare
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 5884
Fax: +1 (617) 258 5999
Email: khare@w3.org

Henrik Frystyk Nielsen
Technical Staff, W3 Consortium
MIT Laboratory for Computer Science
545 Technology Square
Cambridge, MA 02139, U.S.A.
Tel: +1 (617) 253 8143
Fax: +1 (617) 258 5999
Email: frystyk@w3.org