Authoritative Metadata

Draft TAG Finding 05 December 2005

This version:
Latest version:
Previous versions:
XML source, 25 February 2004
Roy T. Fielding, Day Software
Ian Jacobs, W3C


In Web architecture, communication between agents consists of exchanging messages with predefined syntax and semantics: a shared expectation of how each message's control data and payload (representation data and metadata) will be interpreted by the recipient. When supported by the communication protocol, the Web architecture uses representation metadata to indicate the sender's intentions regarding how the recipient should interpret the representation data. For example, HTTP and MIME use the value of the "Content-Type" header field to indicate the Internet media type of the representation, which influences the dispatching of handlers and security-related decisions made by recipients of the message. In this finding, we review the architectural design choice that metadata provided in a received message be considered authoritative. We examine why recipient behavior that fails to respect authoritative metadata can be harmful and under what conditions (user consent) such behavior is allowed. Finally, we consider how specification authors should incorporate these design constraints into their work.

Status of this Document

This DRAFT document has been developed for discussion by the W3C Technical Architecture Group as a finding to address the TAG issues contentTypeOverride-24, putMediaType-38, and portions of errorHandling-20. It is an update to the previously approved finding of 25 February 2004. Please send comments on this finding to the publicly archived TAG mailing list www-tag@w3.org (archive).

Publication of this finding does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. Additional TAG findings, both approved and in draft state, are also available. The TAG expects to incorporate this and other findings into a Web Architecture Document that will be published according to the process of the W3C Recommendation Track.

The terms MUST, SHOULD, and SHOULD NOT are used in this document in accordance with [RFC2119].

Table of Contents

1 Summary of key points
2 Defining authoritative metadata
3 Why metadata from an encapsulating container is authoritative
    3.1 Role of Internet Media Types
    3.2 Why embedded metadata is less authoritative
    3.3 Why external reference metadata is least authoritative
    3.4 What to do when there is no authoritative metadata
4 Overriding authoritative metadata
    4.1 Inconsistency between representation data and metadata
    4.2 Reducing inconsistency
    4.3 Avoiding silent recovery
    4.4 Obtaining user consent
5 Metadata hints in specifications
6 Scenarios
    6.1 Bad server configuration
    6.2 Good server configuration
    6.3 Misconfiguration and metadata hints
    6.4 Conflicting metadata during distributed authoring
7 Future Work
8 References
9 Acknowledgments

1 Summary of key points

The following are the key architectural points of this finding:

  1. Representation metadata received in an encapsulating container, such as within the header fields of a message, is authoritative in defining the nature of the representation received.

  2. Inconsistency between representation data and metadata is an error that should be discovered and corrected rather than silently ignored.

  3. It is an error for an agent to ignore or override authoritative metadata without the consent of the party the agent represents.

  4. Specifications MUST NOT work against the Web architecture by requiring or suggesting that a recipient override authoritatve metadata without user consent.

2 Defining authoritative metadata

The sequence of numbers "324033" might be a license plate number in the state of Arkansas or an old-style telephone number in Italy. Although there do exist some self-descriptive data formats, we generally rely on context to define the purpose, format, and meaning of data. One way to provide a context for interpretation is metadata.

Metadata is simply defined as data about other data. Metadata can be expressed while referencing data externally, while encapsuling data in a container, and by embedding metadata within the data being described. The following table provides examples of how various forms of metadata can be expressed during Web interactions:

resourceexternal referencemessage fieldsHTTP's "Allow" header field in a response message describes the request methods allowed by the resource for which the response was generated.
data formatLink relationship values (rel/rev attributes) are often used to describe metadata relationships between resources.
other sourcesRDF can associate metadata with a resource by reference to its URI.
messageencapsulatinglayersProtocols are often implemented as a stack of layered protocols, with each lower-layer protocol providing context for higher layers.
embeddedmessage syntaxHTTP's response messages begin with "HTTP/" and a version number.
message fieldsHTTP's "Date" header field describes the clock time at the origin when the message was generated.
representationexternal referenceidentifiersSchemes based on old (non-metadata) protocols, such as gopher and ftp, include or imply metadata information about the representation as part of the identifier.
data formatType attributes are sometimes used to express expectations about representation types for pre-access content selection.
encapsulatingmessage fieldsHTTP and MIME use the value of the "Content-Type" header field to indicate the representation's media type.
archival formatsArchives often include catalog data that associates metadata with parts of the archive.
embeddeddata formatMagic numbers, DOCTYPEs, and XML namespaces are all means for making data formats self-descriptive. HTML's "META" elements and RDF/XML assertions can describe metadata about the enclosing representation.

The table above demonstrates that the same metadata may be expressed in various forms. The representation media type [RFC2046], in particular, plays such an important role in the Web architecture that its value can be described in many different locations. Given multiple sources of metadata and the possibility that those sources may be inconsistent, the architect must decide what source of metadata has the highest priority and thus shall be considered authoritive in determining the desired behavior of the recipient. Furthermore, given the presence of self-descriptive data formats, a decision must be made on whether to respect the declared metadata over whatever might be learned by inspecting the data itself.

For Web architecture, a design choice has been made that metadata received in an encapsulating container MUST be considered authoritative and used in preference to metadata found by inspection of the data, declared by embedded metadata, or provided by external reference. Although this design choice is generally applicable to any container format, including archival formats that encapsulate other data, the most significant interpretation for Web architecture is that representation metadata found within the header fields of a received message shall be considered authoritative for the representation encapsulated within that message.

Representation metadata does not constrain the receiving agent to process the representation data in one particular way. What it does is allow the sender of a representation to express its intentions regarding how the data should be interpreted by a recipient. A recipient can then choose, based on its own purpose, design, and configuration, how it will react to those intentions on behalf of the party employing the agent. For example, a browser traversing a link may behave differently depending on how the link was selected, a maintenance spider may ignore a data format's rendering instructions, and an editor may treat every representation as a source for editing rather than display.

This treatment of authoritative metadata applies equally to clients, servers, and intermediaries. A server receiving a representation MUST respect the client's expressed intentions regarding the metadata for that representation and either act in accord with those intentions or respond with an appropriate redirection or error message.

3 Why metadata from an encapsulating container is authoritative

The rationale for our choice of authoritative metadata is difficult to describe using abstractions. Let's consider a specific example of the media type of a received representation and explain why each of the other sources of metadata are not considered authoritative.

3.1 Role of Internet Media Types

An Internet media type [RFC2046] is a short name, such as "text/html", that is associated with a data format specification and processing model through registration in the IANA media type registry. For example, "text/html" in the IANA registry is associated with [RFC2854], which in turn states that:

The text/html media type is now defined by W3C Recommendations; the latest published version is [HTML401].

The media type indicates the intended processing model for a representation, including such issues as whether the data should be rendered, stored, or executed. In practice, media types are thus usable for selecting handlers to implement those functions. A media type, therefore, is not simply an indication of data format; it also refers to a standardized interpretation of that data format. In fact, many different media types share a single data format, while others represent a superset of formats.

3.3 Why external reference metadata is least authoritative

If the authoritative media type of a representation were to be determined by external reference, then resources could be prevented from evolving independently from their references. For example, standards for hypermedia data formats evolve over time, whereas it is preferred that URIs remain persistent over time. If metadata guessed by inspecting the identifier were to be considered authoritative, then references would break when the representation media type changes. Similarly, a type attribute provided with a reference would suffer the same problem it were considered authoritative.

Intermediaries (i.e., proxies and gateways) perform significant functions in Web architecture, such as encapsulating legacy services, enhancing client functionality, and moderating the risk of interactions across firewalls. Those functions can only be performed correctly if the semantics of a given message are expressed within that message. In contrast, metadata associated by an external reference is only visible to the user agent that selects the reference: intermediaries are not aware of that context. If a message recipient treats external metadata as authoritative over that found in the message, then the intermediaries are effectively bypassed and their functionality is lost.

Finally, external references are usually made by third-parties: people who are neither the resource owner nor the user. Allowing a third-party to override the intent of the sender of a message means that the client must trust both the resource owner and the supplier of the reference, introducing yet another attack vector and its associated complications to secure configuration and monitoring.

3.4 What to do when there is no authoritative metadata

There are, of course, times when a representation is provided without any containing metadata, such as when the sender is not certain of the intended metadata or when the protocol being used does not support metadata. That is why the HTTP/1.1 specification [RFC2616] states:

"If and only if the media type is not given by a "Content-Type"
field, the recipient MAY attempt to guess the media type via inspection of
its content and/or the name extension(s) of the URI used to identify the

In other words, when there is no authoritative metadata, the receiving agent MAY attempt to guess the appropriate metadata based on inspection of the data and/or the reference, though such guessing should be limited to media types that are safe to use in that context.

4 Overriding authoritative metadata

Recognition of authoritative metadata is important because it influences the default processing behavior for Web interactions. However, representation metadata is also susceptible to misconfiguration, and user agents frequently try to "simplify" the Web by automatically "correcting" perceived "errors" in those configurations.

Choosing to ignore or override authoritative metadata is only allowed within the Web architecture when the user has given consent. Recipients SHOULD detect inconsistencies between representation data and metadata but MUST NOT resolve them without the consent of the user.

4.4 Obtaining user consent

A user agent represents the user for protocol-level interactions with resource providers. A user agent that does not respect the Web protocol specifications can violate user privacy, introduce security holes, and otherwise create confusion. For example, a broken user agent could trigger a security failure by ignoring a received "Content-Type" header with value "text/plain", guessing that representation data is a shell script, and then executing the script on the user's machine without the user's awareness. The other agents in the system (origin server and intermediaries) have sent or forwarded the message with the expectation that the user agent will not attempt to execute the script, at least not without some additional action deliberately chosen by the user. If the user agent violates those expectations, it violates the protections that may have been put in place for the user's self-protection.

Because of those risks, it is an error for an agent to ignore or override authoritative metadata without the consent of the party employing the agent.

Consent does not imply that the receiving agent must interrupt the user and require selection of one option or another. User consent may be achieved in the form of pre-selected configuration options, modes, or selectable user interface toggles, with appropriate reporting to the user when the agent detects an error. Naturally, the appropriate consent mechanism will be unique to each type of receiving agent and application context. It is therefore beyond the scope of this finding to anticipate the range of possible errors and ways in which interface designers might obtain user feedback to address them.

5 Metadata hints in specifications

Some format specifications allow content authors to provide metadata hints for servers and clients. For instance, the http-equiv attribute of the HTML meta element was intended for servers (not clients). In HTML 2.0 [RFC1866], section 5.2.5, the attribute is specified as follows:

HTTP servers may read the content of the document <head> to generate header fields corresponding to any elements defining a value for the attribute HTTP-EQUIV.

The HTML 4.01 link element has an attribute type that gives clients a hint about the likely media type if one were to retrieve a representation of the identified resource.

A format specification that includes metadata hints for clients must make clear that, when these hints interact with server metadata, they are advisory only. Format specifications MUST NOT include requirements for clients to override server metadata without user consent.

An architecturally sound description of an advisory attribute might read:

The author may provide a hint to the client about the likely Internet media type of representations of the designated resource. Although the client MUST treat server metadata (including that provided by the file system) as authoritative, the client MAY use the hint in a number of ways, including as a preference when negotiating with the server, as input to a decision to retrieve a representation, or to recover from a misconfigured server. However, the client MUST NOT override the server's authoritative metadata without the consent of the user.

A good example of such a description can be found in the W3C Recommendation Speech Recognition Grammar Specification Version 1.0 [SRGS10], which describes agent behavior that is consistent with this finding in section 2.2.2.

In contrast, the W3C Recommendation Synchronized Multimedia Integration Language (SMIL 2.0) [SMIL20] is inconsistent with this finding. The definition of the type attribute in section 7.3.1 specifies that the value of type takes precedence over authoritative metadata for some protocols. The specification is in error. Under no circumstances can a format specification change the meaning of protocol interaction on the Web. Implementers MUST disregard that statement in SMIL 2.0 and treat the type attribute as merely a means for content selection or for when authoritative metadata is unavailable.

6 Scenarios

The scenarios in this section illustrate some issues that arise when the architectural points described in this finding are ignored.

6.1 Bad server configuration

Stuart runs his own Web server at "http://www.example.org/". He creates an HTML page and means to serve it as "text/html", but misconfigures the Web server so that the content is served via HTTP/1.1 [RFC2616] as "text/plain". Janet's browser retrieves the page and displays the content as plain text. Tim's browser retrieves the page, detects some markup that suggests it is an HTML document (e.g., a <!DOCTYPE declaration or <html> element) and, without informing Tim, proceeds as though the content was declared to be "text/html", rendering it according to the HTML and CSS specifications.

Which party has neglected a principle of Web architecture: Stuart for the server misconfiguration, Tim's browser for silently overriding the HTTP headers from the server, or Janet's browser for not detecting that the content looked like HTML?

Answer: By silently overriding metadata from the representation provider in the HTTP headers, Tim's browser did not respect Web architecture principles that promote shared understanding and security.

Misconfiguration of the server is a fixable error. If Stuart was using Janet's browser, he would see that error immediately and fix it. However, if Stuart uses the same browser as Tim for his testing, Stuart would not be informed of the error. Tim's browser is the culprit here because it misrepresents the resource owner by ignoring the authoritative metadata without Tim's consent. Janet's browser respected the "Content-Type" header field and, in doing so, helps Janet detect a server misconfiguration.

6.3 Misconfiguration and metadata hints

Norm publishes an XHTML document that includes this link:

Although the link refers to an XSLT style sheet, Norm has set the type attribute to "text/css". Stuart has configured the Web server so that the style sheet is served via HTTP/1.1 as "application/xslt+xml". With a user agent that understands XSLT but not CSS, Janet requests the content that includes this link. As it interprets the representation data, Janet's user agent reads the type hint and does not fetch the style sheet."

Which party is responsible for the fact that Janet did not receive content she should have: Stuart for the server configuration, Norm for stating that the style sheet is served as "text/css" when in fact it's served with a different media type, or Janet's user agent for not double-checking the media type with the server?

Answer: Norm's mislabeling of content deprived Janet of content she should have received.

Norm is responsible for Janet not having access to representation data she was meant to receive. The HTML 4.01 Recommendation states that "Authors who use [the type] attribute take responsibility to manage the risk that it may become inconsistent with the content available at the link target address." Janet's client could have done more than merely read the type hint and decide to skip the style sheet. Users benefit from clients that allow different configurations for handling hints, including:

6.4 Conflicting metadata during distributed authoring


The meaning of any HTTP message is defined by the contents of that message as interpreted according to the HTTP standard. If a client requests that a server store a representation at a given URI and the server's configuration states that the given URI implies metadata inconsistent from what has been provided by the client, then the server should reject the request using an appropriate HTTP status code.

In other words, if a webdav client performs a

and example.org knows that it has been configured such that all resources with identifiers ending in in ".html" are represented in the "text/html" format, then the server has four choices:

  1. ignore the "application/pdf" metadata provided by the client, store the representation as-is, and serve it later as "text/html".

  2. change the configuration such that future 200 responses to GET /something.html will be served as "application/pdf", thus preserving the client's stated intent.

  3. accept the request only in the sense of it being a requested change of resource state, resulting in the PDF representation being "converted" to HTML for later responses.

  4. respond with "415 Unsupported Media Type" and a message stating why the request is inconsistent with the resource.

(1) is clearly a bad idea because the inconsistency is an error and failing to report an error is bad design.

(2) may be feasible on some HTTP servers that combine configuration for both authoring and read-only services, but most production HTTP servers do not work that way, and automatically overriding a server configuration is more likely to hide pilot-error rather than do what the user actually wants.

(3) is a complicated option that preserves REST semantics but not those of a dumb filesystem. It is one of those server-side magic tricks that tends to annoy people who think HTTP is a file protocol, which suits me just fine provided that it isn't mandatory.

(4) properly informs the user of the inconsistency (enabling them to choose the right workaround), works in all cases, but wastes some bandwidth.

Answer: (1) is a bug, (2) is bad implementation, (3) is a nifty feature when the user is making an informed request, and (4) is the right answer in all other cases.

7 Future Work

The TAG is working with the authors of [RFC3023] to revise section 7.1 of that RFC, which suggests behavior regarding character encoding metadata that is inconsistent with this finding.

8 References

Internet Assigned Numbers Authority (IANA) (See http://www.iana.org/.)
T. Berners-Lee, D. Connolly. Hypertext Markup Language - 2.0, RFC1866, November 1995. (See http://www.ietf.org/rfc/rfc1866.)
N. Freed, N. Borenstein. Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, RFC2046, November 1996. (See http://www.ietf.org/rfc/rfc2046.txt.)
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels, RFC2119, March 1997. (See http://www.ietf.org/rfc/rfc2119.txt.)
R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee. Hypertext Transfer Protocol -- HTTP/1.1, RFC2616, June 1999. (See http://www.ietf.org/rfc/rfc2616.txt.)
D. Connolly, L. Masinter. The 'text/html' Media Type, RFC2854, June 2000. (See http://www.ietf.org/rfc/rfc2854.txt.)
M. Murata, S. St. Laurent, D. Kohn. XML Media Types, RFC3023, January 2001. (See http://www.ietf.org/rfc/rfc3023.txt.)
J. Ayars et al. Synchronized Multimedia Integration Language (SMIL 2.0), Second Edition, W3C Recommendation, 7 January 2005. (See http://www.w3.org/TR/2005/REC-SMIL2-20050107/.)
A. Hunt, S. McGlashan eds. Speech Recognition Grammar Specification Version 1.0, W3C Recommendation, 16 March 2004. (See http://www.w3.org/TR/2004/REC-speech-grammar-20040316/.)

9 Acknowledgments

The first edition of this finding was edited by Ian Jacobs and included substantial input from Roy T. Fielding, Stuart Williams, and Dan Connolly. Martin Dürst, Philipp Hoschka, Rob Lanphier, and Norman Walsh provided reviews of prior drafts that improved this finding. This second edition has additionally benefited from the comments of Noah Mendelsohn.