I posted the following message in the WMLProgramming mailing list.
People have suggested that I publish it as a formal comment to the CTG
draft, so here it is, under the heading "Allowing modifications of the
HTTP header field user-agent: rationale missing".
Eduardo Casais
areppim AG
Bern Switzerland
------------------
I would like to review (a last time) an issue that reoccurs in all
discussions about transcoders.
> Changing User Agent or other headers is not prohibited by HTTP
The first thing to stress is that the user-agent is essential to drive
content selection and generation processes, both in the mobile Web and
in the desktop Web.
a) In the mobile Web, the user-agent is directly associated to the
actual device, and hence serves as a key to characteristics such as
screen dimensions, preferred content types, etc. The advent of
uaprof/ccpp was supposed to make this mapping unncessary, but it is
not the case: uaprof descriptions are often missing, point to invalid
URL, omit important information, or are just plain unreliable. Device
databases like WURFL, based on user-agent mappings, thus remain
indispensible.
b) In the desktop (non-mobile) Web, developers have long relied upon
the user-agent to identify the browsers issuing requests in order to
tailor content to their "quirks". This has been going on at least
since the times of the Netscape / IE wars.
Let us now examine the use cases of a mobile Web browser accessing the
Internet, and evaluate the relevance of the user-agent -- assuming
that transcoders systematically substitue the original value with a
new one.
1.a User-agent-switcher.
The Web server is able, based on the user-agent, to
provide a mobile-optimized or a full-web service.
It therefore needs the original user-agent; modifying
it is unhelpful.
2.a Mobile Web only.
2.a.1 Generic content.
The server returns generic mobile content, without
customizing it for any specific user-agent.
This kind of applications is rare, and often
corresponds to surviving examples of text-only
services developed for older PDA and WAP 1 devices.
Since the server does not use the original user
agent, modifying it is useless.
2.a.2 Mobile with default.
The server returns mobile-optimized content. When
not recognizing the user-agent, it returns a default,
best-effort representation, perhaps with a message
suggesting that the content is tailored for mobile
devices.
Since the server relies upon the user-agent, and is
able to return a default representation, modifying it
is unhelpful.
2.a.3 No default.
The server returns mobile-optimized content, but will
return an error (page with "unsupported browser" warning
or return code "request not acceptable") whenever it
does not recognize the user-agent. In this case, modifying
the original user-agent is most unhelpful, as it guarantees
that the server will not recognize it as a valid
mobile one. If the server does not, for whatever reason
(e.g. incomplete device database), recognize a mobile
user-agent, then there might be a case for modifying it
towards an acceptable mobile one -- but transcoders
precisely do the reverse: they change a mobile user-agent
to an exotic full-Web one. Hence, a modification of the
original user-agent is unhelpful all situations.
3.a Full Web only.
3.a.1 Generic content.
The Web server serves generic full-Web content, without
looking at the user-agent. In this case, modification of
the original user-agent is useless.
3.a.2 Tailored full-web with default.
The server returns full-web content customized for specific
full-web user-agents (e.g. IE 6.0, IE 7.0 and Firefox 2.0),
and serves a default representation, perhaps with a warning
("This site is best viewed with the following browsers:...")
for other user-agents. In this case, modification of the
original user-agent is either useless (in any case a default
representation will be returned), or unhelpful (the default
representation is probably better downgradable than one
specifically customized for a very specific full-Web browser).
3.a.3 Tailored with no default.
The server returns content tailored for specific full-Web
browsers, and an error for other unrecognized or unsupported
user-agents.
Here there is a case to substitute the original user-agent to
force retrieval of content. However, this works only if the
"fake" user-agent precisely corresponds to one of those
accepted by the server -- but transcoders do not tailor their
substitute user-agents with respect to the application server:
the include only general hints (like Mozilla/x.y) in the hope
this is enough to determine content generation.
Hence, the generic substitution of user-agents performed by
transcoders is not appropriate here.
Conclusion: in two cases, modification of the user-agent is useless,
in three it is detrimental, in one it is either useless or
detrimental, and in one it could be helpful, but it is currently done
inappropriately.
Let us consider the interesting symetric use-cases: a full-Web mobile
device accessing the Internet.
1.b User-Agent switcher
Following the same reasoning as in 1.a, we find that
modification of the original user-agent is unhelpful.
2.b Mobile Web only.
2.b.1 Generic content.
Whatever user-agent, the server returns generic mobile-optimized
content. A modification of the original user-agent is useless.
2.b.2 Tailored with default.
The server returns mobile-optimized content, and a default
representation for unrecognized user-agent. Modifying it is
therefore useless -- the default representation will be returned
whether the original (full-Web) or the substitute (pseudo-full
Web) agent appears in the request.
2.b.3 Tailored, no default.
Following the same reasoning as in 2.a.3, the substitution of
original (full-Web) user-agent by a fake (full-Web) one is
useless, as it will anyway return an error.
3.b Full Web only
3.b.1 Generic content.
If the server returns generic full-Web content whatever the user
agent, then modifications of the user-agent are useless.
3.b.2 Tailored with default.
Following the reasoning in 3.a.2, modifying the original
full-Web user agent is either unhelpful (because the server
could have recognized the mobile device's agent), or useless
(the same default representation would be returned).
3.b.3 Tailored, no default.
Here the server may recognize the full-Web user-agent of the
mobile device; it is therefore unhelpful to modify it. Or it
might not support that specific user-agent, in which case it
would be sensible to substitute one that is effectively
supported by the server; however, this is not what transcoders
do: they provide a generic, not a real user-agent instead --
this is inappropriate.
So one situation where it is detrimental, four where it is useless,
one where it is either useless or detrimental, and one where it is
either useless or inappropriately done. It is also an acid test: do
transcoders modify requests from full-Web capable mobile browsers? If
so, something seriously weird is going on, as the excuse has generally
been to make full-Web content available to non-full-Web capable devices.
>From this examination, one can only conclude that proponents of the
preservation of the original user-agent do not have to justify their
position and established practice. Rather, the onus is on the
proponents of the substitution of the user-agent to argue in favour of
their approach, which disrupts established practice. There is
basically only one use case where changing the mobile user agent to a
desktop user agent might help, but it remains to demonstrate:
a) The relevance of the scenario. Perhaps people at Google could let
one of their crawlers roam over a few tens of thousands of WWW sites
to gather statistics on the relative frequency of each aforementioned
scenario.
b) The benefits resulting from handling that specific scenario.
c) That (a) and (b), taken together, are so overwhelming that they
more than compensate the disruptions caused in all other use-cases.
If another use case outside the framework I have presented here pops
up, this does not reduce the need for an assessment based on (a), (b),
(c).
As a final remark, I would like to note that transcoders have been
operating in the mobile Web for a long time. It started with
adaptation of HTML for PDA (Web clipping) and HTML to HDML conversion,
continued with HTML to WML, before arriving at the current crop of
content adaptation. In the old times, developers of content adaptation
software were wary of modifying the user-agent: turning generic WWW
content into a form suitable for mobile devices is so fraught with
difficulties that one would take every chance to let a server return
mobile optimized content (based on the user-agent) if it could. It is
only fairly recently that, without much justification, transcoders
have started in a
systematic way to overwrite the user-agent field.
I think I have said everything I wanted regarding the CTG. The
document requires quite some rework -- nothing exceptional, since it
is a draft. I will lean back and wait for the results of this round of
revisions. Till then, readers of the WMLprogramming and W3C BPWG lists
can rejoice in the knowledge that my long-winded posts are abating at
last.
E. Casais