Client handling of MIME headers

TAG Finding 05 May 2003

This version:
Latest version:
http://www.w3.org/2001/tag/doc/mime-respect (XML)
Previous version:
5 Apr 2003 Draft
Ian Jacobs, W3C


The architecture of the Web depends on applications making dispatching and security decisions for resources based on their Internet Media Types and other MIME headers. In this finding, we review the architectural design choice that MIME headers be authoritative. We also examine why client behavior that misrepresents the user or server is harmful. Finally, we consider how specification authors should incorporate these points into their work.

Status of this Document

This is an early draft of this finding. The TAG has had an opportunity to review the previous version of this finding, but this draft does not represent consensus within the TAG. This draft takes into account some comments by Stuart Williams.

This document has been developed for discussion by the W3C Technical Architecture Group. This finding addresses issue contentTypeOverride-24 and partly addresses issue errorHandling-20. The TAG finding "Internet Media Type registration, consistency of use" also includes material related to this issue.

Publication of this finding does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time.

Additional TAG findings, both approved and in draft state, may also be available. The TAG expects to incorporate this and other findings into a Web Architecture Document that will be published according to the process of the W3C Recommendation Track.

Please send comments on this finding to the publicly archived TAG mailing list www-tag@w3.org (archive).

Table of Contents

1 Summary of Key Points
2 Scenarios
3 Why MIME headers are authoritative
4 Why user agent behavior that misrepresents the user is harmful
5 Hints in specifications
6 Conclusion
7 References
8 Acknowledgments

1 Summary of Key Points

The following are the key architectural points of this finding:

Section 2 presents scenarios where these principles/points have been ignored and pose the question of what has been ignored and by whom. Section 3 discusses the motivation for a Web architecture where MIME headers are authorative. Section 4 examines the potential harm caused by user agents that misrepresent the user or silently disregard authoritative headers. Section 5 discusses the interaction between metadata hints in format specifications and protocol headers.

2 Scenarios

Scenario 1: Stuart runs his own Web server at http://www.example.org/. He creates an HTML page but misconfigures the server so that the content is served via HTTP/1.1 [RFC2616] as text/plain rather than as text/html. Tim's browser looks inside the page, detects some markup that suggests that this is an HTML document (e.g., a <!DOCTYPE declaration or <title> element), and quietly renders it according to the HTML specification rather than as plain text. Janet's browser displays the content as plain text.

Which party has neglected a principle of Web architecture: Stuart for the server configuration, Tim's browser for silently overriding the server's headers, or Janet's browser for not detecting that the content looked like HTML?

Answer 1: By silently overriding authoritative headers, Tim's browser did not respect Web architecture principles that promote shared understanding.

Scenario 2: Norm publishes an XHTML document that includes:

<link href="cool-style" type="text/css" rel="stylesheet"/>

Norm's "cool-style" is an XSLT style sheet, but Norm has set type to text/css. Stuart has configured the server so that "cool-style" is served via HTTP/1.1 as application/xslt+xml. With a user agent that understands XSLT but not CSS, Janet requests the content that includes this link element. As it loads the page, Janet's user agent reads the type hint and does not fetch "cool-style."

Which party is responsible for the fact that Janet did not receive content she should have: Stuart for the server configuration, Norm for stating that the "cool-style" sheet is served as text/css when in fact it's served with a different content type, or Janet's user agent for not double-checking the content type with the server?

Answer 2: Though not a violation of principles of Web architecture, Norm's mislabeling of content deprived Janet of content she should have received.

In the sections below, we explore these answers in more detail.

3 Why MIME headers are authoritative

Successful communication between two parties about a piece of information relies on shared understanding of the meaning of the information. On the Web, thousands of independent parties can identify and communicate about a Web resource. To give these parties the confidence that they are all talking about the same thing when they refer to "the resource identified by the following URI ..." the design choice for the Web is, in general, that the owner of a resource assigns its authoritative meaning. In terms of Web architecture, the authoritative meaning of a resource is communicated as follows:

  1. Parties on the Web agree to use URIs to identify resources.
  2. URI schemes may specify how names are allocated (e.g., the HTTP URI scheme [RFC2616] relies on the Internet domain name registry). For these schemes, the party that creates a URI has the right to establish the authoritative meaning of the resource.
  3. The resource owner communicates the meaning of the resource with bits; the resource owner's server exchanges these bits with the user's clients.
  4. The user's clients interpret the bits according to specification.

This model does not imply that a given set of bits can only be interpreted as the author intended. The model is designed to enable global understanding by having parties agree to follow a small set of rules for interpreting bits. Parties may reach local agreements independently, but they are not part of the Web architecture.

Generally the interpretation of bits is governed by a protocol specification (e.g., HTTP/1.1 and FTP) or a format specification (e.g., XHTML, CSS, PNG, XLink, RDF/XML, and SMIL animation). A "bag of bits" that is self-describing, i.e., that includes enough information to allow two parties to figure out how to interpret it, has the advantage of allowing the same interpretation by clients in different contexts, without requiring additional guidance.

There are also benefits to allowing different interpretations of a bag of bits depending on context. For flexibility, some protocols allow resource owners to direct the interpretation of a bag of bits by sending metadata along with the bits. In HTTP/1.1 for example, a response from the server can include a bag of bits (the "entity body") and metadata about those bits (the "entity headers", including Content-Type, Content-Language, and Content-Encoding). In Web architecture terms, a bag of bits plus metadata is called a representation of a resource. In practice, the MIME mechanism defined in RFC2046 is used to associate a bag of bits with metadata. MIME headers are key to understanding the resource owner's authoritative meaning for a bag of bits.

A particularly important piece of metadata is the content type header, which instructs a client on which specification to follow first in order to interpret a bag of bits; that specification may invoke others recursively. For convenience, the MIME mechanism includes a registry of content type/specification bindings maintained by the Internet Assigned Numbers Authority [IANA]. For instance, in the IANA registry, the content type text/html is associated with [RFC2854], which in turn states that:

The text/html media type is now defined by W3C Recommendations; the latest published version is [HTML401].

Thus, by serving a bag of bits with content type text/html, the resource owner declares that the HTML 4.01 Recommendation governs the authoritative interpretation. By serving a bag of bits (even HTML bits) with content type text/plain, the resource owner declares that [RFC2046] and [RFC2646] govern the authoritative interpretation. This is the first piece of explaining why Tim's browser in scenario 1 is the culprit.

4 Why user agent behavior that misrepresents the user is harmful

A user agent represents the user for interactions with servers. User agent behavior that misrepresents the user or misrepresents the server ultimately undermines trust on the Web and is thus considered harmful. Misrepresentation may lead to violations of privacy, security holes, and just plain confusion. A client that ignores authoritative server headers without informing the user undermines the goal of creating a shared information space.

In scenario 1, in terms of Web architecture, Stuart is innocent; misconfiguration of the server is not an architectural error. Instead, Tim's browser is the culprit since it misrepresents the server by ignoring the authoritative headers, without informing Tim. Janet's browser respected the text/plain header, and by doing so, helps Janet and Stuart detect a server misconfiguration.

Examples of inconsistencies between headers and a bag of bits that have been observed on the Web include:

Clients should detect such inconsistencies but should not resolve them without involving the user (e.g., by securing permission or at least providing notification).

Another form of inconsistency is when the client expects a MIME header and the server doesn't send one. For instance, HTTP/1.1 [RFC2616], section 7.2.1 describes client behavior in the case when the server sends no content type header:

Any HTTP/1.1 message containing an entity-body SHOULD include a Content-Type header field defining the media type of that body. If and only if the media type is not given by a Content-Type field, the recipient MAY attempt to guess the media type via inspection of its content and/or the name extension(s) of the URI used to identify the resource. If the media type remains unknown, the recipient SHOULD treat it as type "application/octet-stream".

This excerpt is consistent with the principle that the content type header, when present, is authoritative. HTTP/1.1 allows a client to guess when not content type is present; in this case, content that is self-describing is likely to lead to a coherent interpretation. A flexible client would do even more than merely guess and silently proceed. For instance, in different configurations the client could:

In Scenario 2, Norm is responsible for Janet not having access to content she was meant to receive. The HTML 4.01 Recommendation states that "Authors who use [the type] attribute take responsibility to manage the risk that it may become inconsistent with the content available at the link target address." Janet's client could have done more than merely read the type hint and decide to skip the "cool-style." Users benefit from clients that allow different configurations for handling hints, including:

It is not a violation of Web architecture when a client overrides server headers and processes a bag of bits in a non-authoritative manner, as long as the client is not misrepresenting the user or server. For instance, an application does not violate Web architecture when it receives a content header of text/html and, rather than following the HTML 4.01 Recommendation, provides the service of validating the HTML, detecting broken links, converting it to another format, or rendering it as plain text. The problem arises when the user agent engages in non-authoritative behavior without the user's awareness or consent.

5 Hints in specifications

Some format specifications allow authors to include in content "hints" for servers and clients. For instance, the http-equiv attribute of the HTML meta element is intended for servers (not clients). In HTML 2.0 [RFC1866], section 5.2.5, the attribute is specified as follows:

HTTP servers may read the content of the document <head> to generate header fields corresponding to any elements defining a value for the attribute HTTP-EQUIV.

The HTML 4.01 attribute type for the link element (used in Scenario 2) gives clients a hint about what the content type of the linked resource is likely to be.

A format specification that includes hints for clients should make clear that when these hints interact with server headings, they are advisory only. Format specifications should not include requirements for clients to override server headers without user consent. An architecturally sound description of an advisory attribute might read:

A URI reference MAY be accompanied by a media type that provides a hint to the client about the likely media type of representations of the designated resource. Although the client MUST treat headers from the server (including those provided by the file system) as authoritative, the client MAY use the hint in a number of ways, including as input to a decision to retrieve a representation, or to recover from a misconfigured server. However, when the client does override the server's headers (by using the hint or any other mechanism), the client MUST inform the user and SHOULD allow the user to control processing.

6 Conclusion

  1. Client behavior that misrepresents the user or server is harmful. Client behavior that involves the user in decisions about handling inconsistencies or errors is beneficial.
  2. Since server headers are authoritative, a client SHOULD NOT ignore or override them without involving the user.
  3. Specifications MAY include hints about server headers but SHOULD NOT include requirements that a client override these headers without involving the user.

7 References

T. Berners-Lee, D. Connolly Hypertext Markup Language - 2.0, RFC1866, November 1995. (See http://www.ietf.org/rfc/rfc1866.)
Internet Assigned Numbers Authority (IANA) (See http://www.iana.org/.)
R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee Hypertext Transfer Protocol -- HTTP/1.1, RFC2616, June 1999. (See http://www.ietf.org/rfc/rfc2616.txt.)
N. Freed, N. Borenstein Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, RFC2046, November 1996. (See http://www.ietf.org/rfc/rfc2046.txt.)
The Text/Plain Format Parameter, RFC2646, August 1999. (See http://www.ietf.org/rfc/rfc2646.txt.)
D. Connolly, L. Masinter The 'text/html' Media Type, RFC2854, June 2000. (See http://www.ietf.org/rfc/rfc2854.txt.)

8 Acknowledgments

Dan Connolly generously provided significant input to this finding.