I think this sounds like a good idea in general. Would you suggest that a single canonical User-Agent string be specified, or should there still be some implementation-specific information in it?

(In reply to comment #0)
> REFERENCES:
> * The UA string format is defined in HTTP:
>   http://tools.ietf.org/html/rfc1945#section-10.15

Hm, no.

Currently:

http://tools.ietf.org/html/rfc2616#section-14.43

In the future:

http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-latest.html#header.user-agent

(In reply to comment #1)
> I think this sounds like a good idea in general. Would you suggest that a
> single canonical User-Agent string be specified, or should there still be some
> implementation-specific information in it?

A single canonical User-Agent string does not sound realistic. Such a string would not be a User-Agent string. Or, at least its meaning would then only become "This is a HTML5-compatible user agent", or something like that. (In a way: no different from the accept header that tells that it accepts 'text/html' ...) Nevertheless, this would not be of much interest for those who read user agent statistics - such as the vendors themselves.

So yes, there must be implementation-specific info there. Basically, the de-facto requirements for UA strings should be collected and documented, so that old and new vendors can know what possible pitfalls there could be. The key things would be expected tokens and expected (sub)formats. For example "Mozilla/0" seems to be synonymous with "a UA that pretends to be a GUI UA".

One additional idea: Keep a registry of HTML UA strings? The list could document the support level of each UA together with its official UA string. There seems to be two kinds of Web page makers that vendors/UA develoeprs need to be aware of:
 a) Active developers, like Facebook. Such Web developers
    could probably make use of such list.
 b) Passive, those who use some software which, perhaps without
    their knowledge discriminate users of less common browsers

So, one could have two lists: A list of limitations when it comes to picking a UA string. And a list of the actual UA strings/string formats in use.

The reason I ask is that if the string still contains something UA-specific, is there any reason to think that the de-facto requirements for what to include in will now keep expanding with the ebb and flow of browser market share?

(In reply to comment #4)
> The reason I ask is that if the string still contains something UA-specific, is
> there any reason to think that the de-facto requirements for what to include in
> will now keep expanding with the ebb and flow of browser market share?

(I suppose you meant "not" and not "now")

You ask a rhetorical question. Hence it is possible that you only want to hear the "no" that you expect.

But what can I say? I guess there is a cost to changing the UA string. May be that is why Opera chose to "fall back" from XML to HTML, rather than adapting to UA string format that "works", which thus would have allowed you to not take this questionable step? Or was it because you did not look enough at that option? At least, none of the Opera employee's postings on this topic seem to have discussed the UA string change option (though I have no doubt that they knew that it they could use IE's string, directly - but that was of course no option). I don't know how long the ASP problem has lasted - probably several years.

Now, let us assume that Opera *had* changed the UA string to something that worked: Would you have done do so silently? Or would you have told the world? If the UA string format was kept track of somewhere, then you could have reported your change and your incompatibility findings to that UA string tracker, and thereby benefited "the Web"?

There is every reason to think that such a UA string registry would not put an end to the "development" of new UA strings. But then, that has never quite been the purpose of such registries either. (Also, it could be that XML5 will allow fatal errors to be handled differently.) But the registry would let us - vendors and site/server developers - to have this important detail under systematic scrutiny. E.g. perhaps Opera would have discovered this problem much earlier. ANd perhaps ASP would have been fixed also ...

Btw: As is, if a web developer want to discern between GUI browsers and other user agents, then the token 'Mozilla/\d' is the the safest, single-token. The *important* *exception* is Opera.

(In reply to comment #4)
> The reason I ask is that if the string still contains something UA-specific, is
> there any reason to think that the de-facto requirements for what to include in
> will now keep expanding with the ebb and flow of browser market share?

The purpose would this defined format would be to allow UAs to become popular, without such unexpected problems as the ASP issue. Hence, in a way, I feel your question is on the head.

(In reply to comment #5)
> (In reply to comment #4)
> > The reason I ask is that if the string still contains something UA-specific, is
> > there any reason to think that the de-facto requirements for what to include in
> > will now keep expanding with the ebb and flow of browser market share?
> 
> (I suppose you meant "not" and not "now")

Right.

> You ask a rhetorical question. Hence it is possible that you only want to hear
> the "no" that you expect.

Actually, I was interested to hear everyone's opinions on this. Specifying part of the UA string is obviously better than specifying none of it, but is it too far-fetched to suggest that all of it be specified? There are downsides of course, but imagine to remove this source of incompatibilities forever!

> But what can I say? I guess there is a cost to changing the UA string. May be
> that is why Opera chose to "fall back" from XML to HTML, rather than adapting
> to UA string format that "works", which thus would have allowed you to not take
> this questionable step? Or was it because you did not look enough at that
> option? At least, none of the Opera employee's postings on this topic seem to
> have discussed the UA string change option (though I have no doubt that they
> knew that it they could use IE's string, directly - but that was of course no
> option). I don't know how long the ASP problem has lasted - probably several
> years.

I wasn't at all involved, but my guess is that everyone knows that changing the UA string *at all* is guaranteed to cause compat issues somewhere, so it's not exactly a safe way to fix compat issues elsewhere.

> Now, let us assume that Opera *had* changed the UA string to something that
> worked: Would you have done do so silently? Or would you have told the world?
> If the UA string format was kept track of somewhere, then you could have
> reported your change and your incompatibility findings to that UA string
> tracker, and thereby benefited "the Web"?
>
> There is every reason to think that such a UA string registry would not put an
> end to the "development" of new UA strings. But then, that has never quite been
> the purpose of such registries either. (Also, it could be that XML5 will allow
> fatal errors to be handled differently.) But the registry would let us -
> vendors and site/server developers - to have this important detail under
> systematic scrutiny. E.g. perhaps Opera would have discovered this problem much
> earlier. ANd perhaps ASP would have been fixed also ...

I agree 100%, the more of the UA string that is specified the better for the Web in general and small players in particular.
 
> Btw: As is, if a web developer want to discern between GUI browsers and other
> user agents, then the token 'Mozilla/\d' is the the safest, single-token. The
> *important* *exception* is Opera.

Huh, I did not know that, but I'm guessing that there's some historical reason for why Mozilla/0 is not part of our UA. It seems likely that adding it now would cause all kinds of things to break, but perhaps it would fix some as well..

(In reply to comment #7)

> > > The reason I ask is that if the string still contains something UA-specific, is
> > > there any reason to think that the de-facto requirements for what to include in
> > > will now keep expanding with the ebb and flow of browser market share?
  [... snip ...]
> Actually, I was interested to hear everyone's opinions on this. Specifying part
> of the UA string is obviously better than specifying none of it, but is it too
> far-fetched to suggest that all of it be specified? There are downsides of
> course, but imagine to remove this source of incompatibilities forever!

What did you mean by "still contains something UA-specific" above? I think it could be quite ideal if there were rules for "all of it", that would not be far fetched, at all. However, previoiusly I read you to suggest that all UA strings should be one and the same - identical. I probably misread you then, sorry.

    [... snip ...]
> I wasn't at all involved, but my guess is that everyone knows that changing the
> UA string *at all* is guaranteed to cause compat issues somewhere, so it's not
> exactly a safe way to fix compat issues elsewhere.

That sounds true. Nevertheless: Trident, Gecko and Webkit all updated their UA strings this year. Are there important incompatibilities from that? Also, each time the browser is released on a new hardware platform - or a new version of the browser is released, the string updates too ...  (Opera has, because of this, stopped updating the first token of its string - which now just remains Opera/9.80 ... guess it can't do that forever? Opera back then, at version 10, asked what the other vendors would do - and perhaps they solved it by always starting with the token Mozilla/\d ???)

A most interesting detail in regard to what Gecko and Webkit did is that they standardized on strings of the following format:

"[mainproduct string]" (e.g. Firefox)
"[mainproduct string]+[sideprodstring]" (e.g. SeaMonkey)

Example:

   Firefox:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
   SeaMonkey:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 SeaMonkey/2.1.1

This shows that 
a) a realization of the need streamline the strings.
b) streamlined strings can without much risk be made longer

> > Now, let us assume that Opera *had* changed the UA string to something that
> > worked: Would you have done do so silently? Or would you have told the world?
> > If the UA string format was kept track of somewhere, then you could have
> > reported your change and your incompatibility findings to that UA string
> > tracker, and thereby benefited "the Web"?
> >
> > There is every reason to think that such a UA string registry would not put an
> > end to the "development" of new UA strings. But then, that has never quite been
> > the purpose of such registries either. (Also, it could be that XML5 will allow
> > fatal errors to be handled differently.) But the registry would let us -
> > vendors and site/server developers - to have this important detail under
> > systematic scrutiny. E.g. perhaps Opera would have discovered this problem much
> > earlier. ANd perhaps ASP would have been fixed also ...
> 
> I agree 100%, the more of the UA string that is specified the better for the
> Web in general and small players in particular.

Great to hear!

> > Btw: As is, if a web developer want to discern between GUI browsers and other
> > user agents, then the token 'Mozilla/\d' is the the safest, single-token. The
> > *important* *exception* is Opera.
> 
> Huh, I did not know that,

Mosaic killers, that was what they all wanted to back then ...  As for "know": I just counted the strings at http://www.useragentstring.com/pages/All/. 81% of them begin with the token "Mozilla/\d.\d". (But only 1000 - 10% - contains "Mozilla/\d.\." *and* a product token (or a *comment*) which says "MSIE \d.\d" OR "Firefox/\d" OR "Safari/\d". Note that Microsoft always places the "MSIE \d.\d" inside a comment - it is not a product token.

> but I'm guessing that there's some historical reason
> for why Mozilla/0 is not part of our UA. It seems likely that adding it now
> would cause all kinds of things to break, but perhaps it would fix some as
> well..

You could add it at the *end* of your curren string - see the SeaMonkey example above.

   This is the string I "made" in my www-tag@ message,
   where I added it inside a comment (the parenthesis):

Opera/11 Version/11.60 Presto/2.2.15 (boilerplate:Mozilla/5.0/msie 10.0)

   This is a current, "real" string (from
   http://www.useragentstring.com/pages/Opera/):

Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 Version/11.52'

   As I said above, Opera would probably need to fix
   the "9.80" part at some point. So a new string could
   look like this, where I replaced 'boilerplate' with 
   'compat'):

Opera/11 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 (compat: Mozilla/5.0; msie 10.0;) Version/11.52

   Compared to the current, real string, the changes are
   quite minimal, and it makes it work with the ASP 
   affected pages, like this this one: 
   http://home.mcafee.com/Default.aspx
   Note, as well, that the above string follows the advice
   in HTTP 1.1 about having the most specific product token
   first and then becomes more and more unspecific.

   The more "traditional" option would be to start with the 
   Mozilla/0 product token, where I guess you would 
   coordinate the number with your competitors:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 Opera/11 Version/11.52 (compatible: msie 10.00)

   This too works with that ASP page.
   Note that this option starts with the more general info
   and ends with the most specific info. (Opera's 
   Version/\d does perhaps not fit in, though.)

Perhaps it makes sense, from a compatible-with-the-Web point of view, to start with the least specific and end with the most specific. That approach seem more compatible with the thought of a - ideally - versionless Web than the opposite - to let the string start with a product token with a speedily increasing number.

(In reply to comment #8)
> (In reply to comment #7)
> 
> > > > The reason I ask is that if the string still contains something UA-specific, is
> > > > there any reason to think that the de-facto requirements for what to include in
> > > > will now keep expanding with the ebb and flow of browser market share?
>   [... snip ...]
> > Actually, I was interested to hear everyone's opinions on this. Specifying part
> > of the UA string is obviously better than specifying none of it, but is it too
> > far-fetched to suggest that all of it be specified? There are downsides of
> > course, but imagine to remove this source of incompatibilities forever!
> 
> What did you mean by "still contains something UA-specific" above? I think it
> could be quite ideal if there were rules for "all of it", that would not be far
> fetched, at all. However, previoiusly I read you to suggest that all UA strings
> should be one and the same - identical. I probably misread you then, sorry.

I was suggesting, half seriously, that all browsers have the same User-Agent string. It seems the logical the way to put an end to server-side sniffing (and browser stats, oops).

>     [... snip ...]
> > I wasn't at all involved, but my guess is that everyone knows that changing the
> > UA string *at all* is guaranteed to cause compat issues somewhere, so it's not
> > exactly a safe way to fix compat issues elsewhere.
> 
> That sounds true. Nevertheless: Trident, Gecko and Webkit all updated their UA
> strings this year. Are there important incompatibilities from that? Also, each
> time the browser is released on a new hardware platform - or a new version of
> the browser is released, the string updates too ...  (Opera has, because of
> this, stopped updating the first token of its string - which now just remains
> Opera/9.80 ... guess it can't do that forever? Opera back then, at version 10,
> asked what the other vendors would do - and perhaps they solved it by always
> starting with the token Mozilla/\d ???)

AFAIU, it's locked at 9.80 because of crummy script that took the first digit as the version number and assumed things based on that. The actual version is at the end of the string, e.g. Version/12.00. I guess when Opera reaches somewhere between 40 and 80 it could be changed back, but by then it's safe to assume that scripts will depend on Version/ instead :/

(Note that I'm not representing Opera here, I won't have anything to do with testing or implementing any changes to the UA string, I'm just speaking as someone with an interest in the topic.)

(In reply to comment #9)

> I was suggesting, half seriously, that all browsers have the same User-Agent
> string. It seems the logical the way to put an end to server-side sniffing (and
> browser stats, oops).

Opera would be closer to that goal, if it aligned more with the rest by placing "Mozilla/5.0" as the first token.
 
> AFAIU, it's locked at 9.80 because of crummy script that took the first digit
> as the version number and assumed things based on that. The actual version is
> at the end of the string, e.g. Version/12.00. I guess when Opera reaches
> somewhere between 40 and 80 it could be changed back, but by then it's safe to
> assume that scripts will depend on Version/ instead :/

Webkit too uses Version/*. So may be that is fair enough - perhaps they had a similar problem.

I don't know about that crummy script and how important that issue was or is. But perhaps it is more serious to be served XML? At any rate: Even Opera/9.80 is not without problems, see the PS below.
 
> (Note that I'm not representing Opera here, I won't have anything to do with
> testing or implementing any changes to the UA string, I'm just speaking as
> someone with an interest in the topic.)

OK.

(In reply to comment #8)
> Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 Opera/11
> Version/11.52 (compatible: msie 10.00)
> 
>    This too works with that ASP page.
>    Note that this option starts with the more general info
>    and ends with the most specific info.

This perhaps makes more sense, w.r.t. the order of specificity, and adds another twist on the compat comment:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6.8; U; de) Presto/2.9.168 Opera/1152 (Moz-compat: Firefox/9.0) Version/11.52

PS: Having "Opera/11.52" (or "Opera/9.80") anywhere, does not work, weird enough, for that ASP solution. Only "Opera/11" or "Opera/1152" works. Thus it would probably, w.r.t. that ASP solution, be impossible for a competitor to include a string such as "(Opera-compatible: Opera/11.52)" - one would have to say  "(Opera-compatible: Opera/1152)". (That appears to, somehow, be directed at Opera, specifically, even if it makes no real sense since, as told, e.g. Opera/11 does work.)

FWIW, https://bugzilla.mozilla.org/show_bug.cgi?id=690287 is Mozilla's tracking bug for problems due two-digit version numbers.

This is out of scope for the HTML spec. Should be defined by the HTTP group.

I really don't see how this falls into HTTP's area. HTTP defines the base syntax.

If format subset specifically for browsers would need to be defined in a spec specific to browsers.

It has nothing to do with HTML or browsers. There are more clients on the Web than browsers, you know. For example, wget has a User-Agent string, msnbot has a User-Agent string, etc.

(In reply to comment #14)
> It has nothing to do with HTML or browsers. There are more clients on the Web
> than browsers, you know. For example, wget has a User-Agent string, msnbot has
> a User-Agent string, etc.

Yes, I happen to know that.

My understanding was that the intent was to harmonize UA strings for browsers.

Who cares what the UA string for "wget" is?

(In reply to comment #15)
> (In reply to comment #14)
> > It has nothing to do with HTML or browsers. There are more clients on the Web
> > than browsers, you know. For example, wget has a User-Agent string, msnbot has
> > a User-Agent string, etc.
> 
> Yes, I happen to know that.
> 
> My understanding was that the intent was to harmonize UA strings for browsers.
>
> Who cares what the UA string for "wget" is?

The copy of wget that I use (http://www.hexcat.com/deepvacuum/) defaults to identify itself as Safari … (Can be disabled in the preferences.)

Intent was that a subset format should be defined - a harmonization around something that does not create problems. Clearly, to the extent that they are affected, non-browsers could also benefit from these things. E.g. I consider that the googleboot wants to catch "the Web" as she is seen by users? And - btw - the google bot UA string *does* cause pages to be served with the wrong MIME type - same as happens with Opera.

If "browsers" means "the browsers based on the 4-5 big GUI vendors", then they are not the only ones that are affected: E.g. according to my tests, text browsers are affected by the same thing that affects Opera.

When a browser vendor - Opera - changes how their browsers handle application/xml+xhtml because of a UA string issue, then I think that the issue is related to (X)HTML5.  Clearly, the HTML spec would want to say something about that?

Hm ... I can see that what I said about text browsers and google bot, could justify that the issue should be worked out in HTTP ...  ?

Julian, is there no room for e.g. adding a "recommended subset" in the HTTP spec?

(In reply to comment #16)
> Julian, is there no room for e.g. adding a "recommended subset" in the HTTP
> spec?

At this point, given the charter (no new features) and the timing: I don't think so.

But if you believe this is useful work then there's no reason why you couldn't define it in a separate document. If you need help with the IETF process, let me know.

(In reply to comment #17)
> But if you believe this is useful work then there's no reason why you couldn't
> define it in a separate document. If you need help with the IETF process, let
> me know.

A UA string spec written without the developers of the products that send UA strings is going to fail. Don't bother.

UA strings are compromises between various considerations:
 * Wishing to look similar enough to what came before to satisfy existing sniffers.
 * Wishing to look different from what came before to be recognized as something different for promotional purposes.
 * Wishing to reveal as little information as possible to avoid fingerprinting and to avoid giving sites the opportunity to sniff for inessential things and ruin UX for some users whose browser sends something different for inessential parts of the UA string.
 * Wishing to reveal as much information as possible for support forums, software downloads, bug trackers or plain curiosity.
 * Etc., etc.

It's hard to get consensus on the balance of these things even within one vendor. And if the end state of what you are trying to do results in different UA strings between vendors, sites will continue to target browsers by vendor and you haven't really accomplished a level playing field by standardization.

Ask yourself:
 * What problem are you trying to solve?
 * What's your solution?
 * Why would the stakeholders accept your solution?
 * Would your solution actually end up solving the problem?
 * How much damage would implementing your solution inflict?
 * Is your solution implementable on a vendor-by-vendor basis or does it require everyone to ship at the same time?

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Rejected
Change Description: no spec change
Rationale: Not HTML's problem, and not a space that we're likely to get interop on anyway