Copyright © 2008 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This document provides guidance to content transformation proxies and content providers as to how inter-work when delivering Web content.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This is the First Public Working Draft of the Content Transformation Guidelines; it has been produced by the Content Transformation Task Force of the Mobile Web Best Practices Working Group as part of the Mobile Web Initiative.
This document is the Guidelines document referred to in the Deliverables section of the Charter of the W3C Mobile Web Initiative Best Practices Working Group Content Transformation Task Force. It is based on the issues identified by the Content Transformation Landscape document.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
Please send comments on this document to the Working Group's public email list public-bpwg-comments@w3.org, a publicly archived mailing list. The Working Group is especially looking for feedback on the guidelines that control the Behavior of Components of the delivery chain (see part 4 below). The editorial notes underline the issues that still need to be addressed. Suggestions in these areas would be most appreciated as well.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. This document is informative only. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
1 Introduction
1.1 Purpose
1.2 Audience
1.3 Scope
2 Terminology
2.1 Types of Proxy
2.2 Types of Transformation
2.3 Normative Language for Conformance Requirements
2.4 Normative and Informative Parts of this Document
3 Requirements
3.1 Summary of Requirements
3.2 Control of the Behavior of the Proxy
3.2.1 Control by the User
3.2.2 Control by Server
3.2.3 Control by Administrative or Other Arrangements.
3.2.4 Resolving Conflicting Instructions
4 Behavior of Components
4.1 Proxy Treatment of Request
4.1.1 no-transform directive in Request
4.1.2 Proxy Decision to Transform
4.1.3 Proxy Indication of its Presence to Server
4.1.4 Altering Header Values
4.2 Server Response to Proxy
4.3 Proxy Receipt and Forwarding of Response from Server
4.4 Proxy Response to User Agent
5 Testing
6 Conformance
A References
B Scope for Future Work (Non-Normative)
C Acknowledgments (Non-Normative)
This section is informative.
From the point of view of this document, Content Transformation is the manipulation in various ways, by proxies, of requests made to and content delivered by an origin server with a view to making it more suitable for mobile presentation.
The W3C MWI BPWG neither approves nor disapproves of Content Transformation, but recognizes that is being deployed widely across mobile data access networks. The deployments are widely divergent to each other, with many non-standard HTTP implications, and no well-understood means either of identifying the presence of such transforming proxies, nor of controlling their actions. This document establishes a framework to allow that to happen.
The overall objective of this document is to provide a means, as far as is practical, for users to be provided with at least a "functional user experience" [Device Independence Glossary] of the Web, when mobile, taking into account the fact that an increasing number of content providers create experiences specially tailored to the mobile context which they do not wish to be altered by third parties. Equally it takes into account the fact that there remain a very large number of Web sites that do not provide a functional user experience when perceived on many mobile devices.
The document describes how the origin server may request conforming transforming proxies not to alter HTTP requests and responses, as well as describing control options that a transforming proxy offers users.
A more extensive discussion of the requirements for these guidelines can be found in "Content Transformation Landscape" [CT Landscape].
The audience for this document is creators of Content Transformation proxies, purchasers and operators of such proxies and content providers whose services may be accessed by means of such proxies.
The recommendations in this document refer only to "Web browsing" - i.e. access by user agents that are intended primarily for interaction by users with HTML Web pages (Web browsers).
Note:
The document is not intended as guidelines for delivery of WAP/WML. Some of its recommendations may, in some circumstances, disrupt the delivery of WML.
The BPWG is not chartered to create new technology, its role is to advise on best practice for use of existing technology. In satisfying Content Transformation requirements, existing HTTP headers, directives and behaviors must be respected, and as far as is practical, no extensions to [RFC 2616 HTTP] are to be used.
This section is normative.
Alteration of HTTP requests and responses is not prohibited by HTTP other than in the circumstances referred to in [RFC 2616 HTTP] Section 13.5.2.
HTTP defines two types of proxy: transparent proxies and non-transparent proxies. As discussed in [RFC 2616 HTTP] Section 1.3, Terminology:
[Definition: "A transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification."]
[Definition: "A non transparent proxy is a proxy that modifies the request or response in order to provide some added service to the user agent, such as group annotation services, media type transformation, protocol reduction, or anonymity filtering.] Except where either transparent or non-transparent behavior is explicitly stated, the HTTP proxy requirements apply to both types of proxies."
This document elaborates the behavior of non transparent proxies, when used for Content Transformation in the context discussed in [CT Landscape].
Editorial Note: The BPWG requests feedback on the degree to which it is necessary to distinguish between Content Transformation proxies that interact with user agents using HTTP, and other types of arrangements where a (proprietary) client application interacts with an in-network component using other techniques.
Transforming proxies can carry out a wide variety of operations. In this document we categorize these operations as follows:
Alteration of Requests
Transforming proxies process requests in a number of ways, especially replacement of various request headers to avoid HTTP 406 Status responses (the server can not provide content in the format requested) and at user request.
Alteration of Responses
There are three classes of operation on responses:
Restructuring content
[Definition: Restructuring content is a process whereby the original layout is altered so that content is added or removed or where the spatial or navigational relationship of parts of content is altered, e.g. by linearization or pagination. It includes also rewriting of URIs so that subsequent requests route via the proxy.]
Recoding content
[Definition: Recoding content is a process whereby the layout of the content remains the same, but details of its encoding may be altered. Examples include re-encoding HTML as XHTML, correcting invalid markup in HTML, conversion of images between formats (but not, for example, reducing animations to static images). ]
Optimizing content
[Definition: Optimizing content includes removing redundant white space, re-compressing images (without loss of fidelity) and compressing for transfer.]
Individual conformance requirements or testable statements are identified in this document by the use of specific key words. In particular, the key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional in this Recommendation are to be interpreted as described in [RFC 2119] .
This section is informative.
The purpose of this section is to summarize the communication requirements of actors (transforming proxies, origin servers, and to some extent users) to communicate with each other. The relevant scenario involving a content transformation proxy is as follows:
Note:
The interactions of several transformation proxies are encompassed by this document, but only in a rudimentary form.
The needs of these actors are as follows:
The user agent needs to be able to tell the Content Transformation proxy and the origin server:
what type of mobile device and what user agent is being used;
that all Content Transformation should be avoided.
The Content Transformation proxy needs to be able to tell the origin server:
that some degree of Content Transformation (restructuring and recoding) can be performed;
that Content Transformation will be carried out unless requested not to;
that content is being requested on behalf of something else and what that something else is;
that the request headers have been altered (e.g. additional content types inserted).
The origin server needs to be able to tell the Content Transformation proxy:
that it varies its presentation according to device type and other factors;
that it is permissible (or not) to perform Content Transformation of various kinds;
that it has media-specific representations;
that is unable or unwilling to deal with the request in its present form.
The Content Transformation proxy needs to be able to tell the user agent:
that it has applied transformations of various kinds to the content.
The Content Transformation proxy needs to be able to interact with the user:
to allow the user to disable its features;
to alert the user to the fact that it has transformed content and to allow access to an untransformed representation of the content.
A transforming proxy as described in this document must offer a level of control to users and to origin servers with which it communicates.
Transforming proxies should provide to their users:
an indication that the content being viewed has been transformed for mobile presentation;
an option to view the original, unmodified content.
They may also provide the ability for their users to make a persistent or semi-persistent expression of preferences. Examples of such settings are "Never transform", "Transform only if necessary to avoid mis-operation" and "Compress where possible".
Editorial Note: The BPWG is studying how to clarify the scope of "persistent" and "semi-persistent".
Transforming proxies must allow origin servers to control the Content Transformation process. The control mechanisms include use of HTTP conventions as discussed in the following section (4 Behavior of Components).
Editorial Note: The BPWG is also considering the possible use of other techniques, such as [POWDER] to allow origin servers to express preferences regarding transformation of their content and also to provide information about the availability of different representations of their content.
The preferences of users and of servers may be ascertained by means outside the scope of this document, for example:
the use by transforming proxies of a disallow list of Web sites for which Content Transformation is known to diminish the user experience of content or be ineffective;
the use by transforming proxies of an allow list of Web sites for which Content Transformation is known to be necessary;
terms and conditions of service, as agreed between the user and the Content Transformation service provider.
Note:
Allow and disallow lists generally cause intractable problems for content providers since there is no mechanism for them to establish which lists they should be on, nor any generic mechanism though which they can check or change their status.
There is the possibility for conflict between the desires of the content provider and the desires of users of that content. This document sets out to provide a framework within which, for matters of presentation, the desires of the content provider are usually accommodated but, where necessary, the user may expressly override those desires with their preferences.
Editorial Note: The principle behind this is that it is consistent with the operation, for example, of CSS, where the user may override a content provider's presentation choice.
This section is normative.
no-transform
directive in RequestIf the request contains a Cache-Control: no-transform
directive
the proxy must forward the request unaltered to the
server, other than to comply with transparent HTTP behavior and in particular to add a
Via
HTTP header.
Irrespective of the presence of the no-transform directive, the proxy
must behave transparently unless it is able
positively to determine that the user agent is a Web browser. The mechanism
by which the proxy recognizes the user agent as a Web browser should use
evidence from the HTTP request, in particular the User-Agent
and Accept
headers.
If there is no no-transform
directive present in the request the
proxy should analyze whether it intends to offer
transformation services by referring to:
any administrative arrangements that are in place with the user, or the server;
any knowledge it has of user agent capabilities;
any knowledge it has of server preferences, derived either from a repository of such preferences, or from previous interaction with the server;
the HTTP method of the request.
Editorial Note: POWDER would provide an additional and potentially more definitive means ascertaining server preferences.
Proxies should not alter HTTP requests unless:
unaltered headers would result in the user's request being rejected by the origin server;
the user has specifically requested a restructured version of a desktop presentation.
Note:
Rejection of a request by a server is taken to mean both a HTTP 406 Status as well as HTTP 200 Status, with content indicating that the request cannot be handled - e.g. "Your browser is not supported"
Editorial Note: The BPWG is studying heuristics for determining when a response with a 200 Status should be treated as a response with a 406 Status.
Proxies should not intervene in methods other than GET, POST, HEAD and PUT.
User agents sometimes issue HTTP HEAD requests in order to determine if a resource is of a type and/or size that they are capable of handling. A transforming proxy may convert a HEAD request into a GET request if it requires the response body to determine the characteristics of the transformed response that it would return were the user agent subsequently to issue a GET request for that content.
In this case, the proxy should (providing such action is in accordance with normal HTTP caching rules) cache the response so that it is not required to send a second GET request to the server.
Editorial Note: The BPWG is aware of a number of circumstances in which request bodies are altered by transforming proxies (for example: transformation of character set from one supported by the device to one required by the server, carrying out a POST of a form that has been delivered in a paginated form to the device to avoid memory limitations, changing multipart/form-data to application/x-www-form-urlencoded where the device does not support multipart/form-data). Further study is required as to what prescriptions, if any, should be made about transformation of request bodies.
Knowing that the browser has available a linearization or zoom capability and/or supports a broad range of content formats the proxy should not restructure or recode content.
Editorial Note: Further discussion is needed on the subject of "session", in which consistency of user experience needs to be maintained across a sequence of related requests. This affects the proxy decision to transform and whether it should alter request headers.
If, as a result of carrying out this analysis the proxy remains unaware of the server's preferences and capabilities it should:
Issue a request with unaltered headers and examine the response (see 4.4 Proxy Response to User Agent);
If it is still in doubt, issue a request with altered headers
Editorial Note: Further discussion is needed on the fact that this technique can cause two requests for the same resource. It is noted that GET is supposed to be "idempotent" and not change the state of the server, however, it is also noted that there are circumstances in which GET is often used for such purposes, e.g. following a link from email.
Editorial Note: The BPWG notes that some transforming proxies issue duplicate requests as a matter of course in order to compare responses. This practice may specifically be deprecated in a future revision of this document.
The proxy must (in accordance with compliance to RFC 2616)
include a Via
HTTP header indicating its presence.
The proxy should indicate its presence and capabilities by
including a comment in the Via
HTTP header.
Editorial Note: The BPWG intends to recommend a format for such comments.
When forwarding Via
headers proxies should not alter them
in any way.
Editorial Note: The BPWG is currently studying the feature of HTTP where it suggests that comments in Via headers may be removed by other proxies.
If the proxy has altered header values or intends to transform it must indicate this.
Editorial Note: The BPWG is studying means by which this may be indicated.
If it has altered HTTP headers the proxy must include
copies of the unaltered headers in the request in the form
"X-Device-"<original header name>
. For example, if
it has altered the User-Agent
header, an
X-Device-User-Agent
header must be added
with the value of the received User-Agent
header.
Editorial Note: The prefix X-Device- has been determined provisionally and the BPWG continues to study what value to allocate to the prefix, and how to handle any received headers that conflict with this usage.
Editorial Note: The BPWG is considering recommending the addition of a Warning Transformation Applied header to the request.
Servers should respond with a 406 HTTP Status if a request can not be satisfied with content that meets the criteria specified by values of request HTTP headers (and not a 200 Status).
If it is capable of varying its presentation it should take account of user agent capabilities and formulate an appropriate experience according to those criteria.
If the server varies its presentation according to examination of received HTTP
headers then it must include a Vary
HTTP header
indicating this to be the case. If, in addition to, or instead of HTTP headers,
the server varies its presentation on other factors (source IP Address ...) then
it must, in accordance with [RFC 2616 HTTP],
include a Vary
header containing the value '*'.
If the server has distinct presentations that vary according to presentation media, then the medium for which the presentation is intended should be indicated.
Editorial Note: The BPWG is studying the use of the link
element of HTML
which is used for this purpose. It is noted that the link
element is not available in formats other than HTML, and it is noted that
there is currently active discussion about the use of the
Link
HTTP header, which would serve this purpose well.
If the server creates a specific user experience according to device
characteristics or presentation media types it should inhibit
transformation of the response by including a no-transform
directive.
The server must include a no-transform
directive
if one is received from the user agent.
Note:
Including a no-transform
directive
can disrupt the behavior of WAP/WML proxies, because it can inhibit such
proxies from converting WML to WMLC.
Servers may base their actions on knowledge of behavior of
specific transforming proxies, as identified in a Via
header, but
should not choose a Content-Type
for their
response based on their assumptions about the heuristic behavior of any
intermediaries. (e.g. a server should not choose content-type:
application/vnd.wap.xhtml+xml
solely on the basis that it
suspects that proxies will not transform content of this type).
If HTTP header fields were altered in the request then the proxy
must be prepared to re-issue the request in an unaltered form
on receipt of a Vary
header in the response indicating that the server offers
variants of its presentation according to any of the HTTP header fields that
have been modified.
Editorial Note: The BPWG is aware that more precision may be needed in the above statement. If a transforming proxy has followed the guidelines in this document, then it should not receive a response with a Vary header if it has not already received such a response to a request with unaltered headers.
If the response includes a Warning: 214 Transformation Applied the proxy must not apply further transformation.
In the absence of a Vary
or no-transform
directive the
proxy should apply heuristics to the content to determine
whether it is appropriate to restructure or recode it (in the presence of such
directives, heuristics should not be used.)
Examples of heuristics:
The server has previously shown that it is contextually aware, even if the present response does not indicate this;
the Content-Type
or other aspects of the response are known to be specific to the device or class of device e.g. application/vnd.wap.xhtml+xml
or the Document Type is DOCTYPE XHTML-MP;
the response is an HTML response and it includes <link> elements specifying alternatives according to presentation media type.
Editorial Note: The BPWG May include further example heuristics.
A proxy should strive for the best possible user experience that the user agent supports. It should only alter the format, layout, dimensions etc. to match the specific capabilities of the user agent. For example, when resizing images, they should only be reduced so that they are suitable for the specific user agent, and this should not be done on a generic basis.
If the proxy alters the content then it must add a Warning 214 Transformation Applied HTTP Header.
If the response contains links whose URIs have the scheme https
the
proxy may only rewrite them so that it can transform the content,
if it meets the following provision. If the proxy does rewrite such links, it
must advise the user of the security implications of
doing so and must provide the option to avoid decryption and
transformation of the resources the links refer to.
If the response includes a Cache-Control: no-transform
directive
then the response must remain unaltered other than to comply with transparent
HTTP behavior and other than as noted below.
If the proxy determines that the resource as currently represented is likely to cause serious mis-operation of the user agent then it may, with the users explicit prior consent, warn the user and provide links to both transformed and unaltered versions of the resource.
The editors acknowledge contributions of various kinds from members of the MWI BPWG Content Transformation Task Force.
The editor acknowledges significant written contributions from: