Bug 20214 - srcset: srcset="" lacks way to support different image types
srcset: srcset="" lacks way to support different image types
Status: NEW
Product: WHATWG
Classification: Unclassified
Component: HTML
unspecified
PC All
: P2 enhancement
: Needs Impl Interest
Assigned To: Ian 'Hixie' Hickson
contributor
:
Depends on: 20174
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-03 15:47 UTC by Marcos Caceres
Modified: 2013-11-13 21:27 UTC (History)
14 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marcos Caceres 2012-12-03 15:47:30 UTC
+++ This bug was initially created as a clone of Bug #20174 +++

The img@srcset spec currently lacks a way for alternative image formats to be supported. This means that content negotiation must be done on the server, which is known to be problematic ( see: http://lists.w3.org/Archives/Public/public-html-bugzilla/2011Nov/0309.html ). Or it must be done on the client using a whole bunch of nasty hacks (e.g., downloading a 1x1 pixel image, seeing if it's supported, etc.).
Comment 1 Ian 'Hixie' Hickson 2012-12-07 23:14:03 UTC
Can you elaborate on the use case?
Comment 2 Marcos Caceres 2012-12-08 18:46:19 UTC
Short version: we want to use WebP or JPEG-XR without jumping through client or server hoops. 

Long version: Some images are best suited to a specific file type for reasons such as file size optimization, alpha transparency, scalability, animation, etc. For example, a photo usually requires good color depth, but does not require alpha transparency or animation; JPEG or WebP are well-suited to these needs and offer good optimization between image quality and file size. Icons are often simpler in terms of color depth, but may require alpha transparency; the PNG format is better-suited to these needs.

Sharing URLs for formats that are not interoperable across browsers is a problem in the wild. 

In a responsive design, images need to be displayed at different sizes. When possible, a vector format such as SVG might be most appropriate. 

Although a web developer may want to use a specific image format, new or otherwise, the browser may not always support it. Currently, developers are forced to abandon the most suitable image format in favor of one that has good user agent support.
Comment 3 Ian 'Hixie' Hickson 2012-12-08 23:14:37 UTC
Why doesn't content negotiation work on the short term? On the long term, presumably it won't matter since all browsers will support the formats.
Comment 4 Yoav Weiss 2012-12-10 15:48:36 UTC
Server-side content negotiation for the "image type" use case has several drawbacks:
* It renders the content practically uncacheable by intermediary proxies - Current HTTP mechanisms do not have fine-graineed mechanisms that can enable caching for content-negotiated payload. The "key" HTTP header may add these mechanism, but since many cache servers still do not support "Vary", it will take many years before it is relevant.
* It relies on the "User-Agent" header, which is inaccurate in some cases (UA spoofing) and not future-friendly.
* The server side environment is fragmented, and there are many frameworks. Server-side content negotiation must therefore be implemented over and over for each of these frameworks in order for it to be accessible to content authors.

Client-side content negotiation have a different set of drawbacks:
* Images are fetched only after JavaScript runs, causing them to load later on in the page's loading process (since that cannot be downloaded by the PreloadScanner) and delay the overall page's load time.
* It adds a javaScript dependency that could be avoided if `type` was baked in.

If we take the current situation with WebP as an example, most authors don't use this format (which is superior to current formats in several use cases, and slightly better in many others) because it requires a complex server-side solution. Many authors do not have access to their server's configuration (shared hosting, etc).
Some Web site optimizers (e.g. Torbit) offer such content-negotiation services, but that's hardly ideal for wide adoption on the Web.

Mozilla have stated that the main reason it does not implement WebP support is because MS & Apple are not likely to support it either. 

This is a "chicken and egg" situation that I believe a markup based fallback for image types can get us out of. It would enable authors to include multiple versions of the same content image, and let the browser pick the most suitable format it supports.
Comment 5 Marcos Caceres 2012-12-10 17:17:49 UTC
I think a server-side solution is problematic for the same reasons Appcache had issues with the content type. If devs had trouble with setting a simple "content-type" header for .appcache, then imagine how much worse it's going to be with having to do content negotiation on the server. If possible, we want to avoid another situation like we had with appcache, where we would be forced to remove the server-side requirement post-implementation due to developer and implementor feedback. [1]

Additionally, Adobe (from their experience running TypeKit) has told us that the server-side requirement adds additional complication at scale [2]:
[[[

Just to offer some insight on this, given that I work on a product (Typekit) that is all about content negotiation, one reason I'm a strong proponent of doing this in the client is that server-side techniques can become very difficult — and very costly — at scale. I wrote a post for our blog last year about why we use JavaScript-based client-side content negotiation for web fonts instead of doing it on the server: http://blog.typekit.com/2011/08/11/better-web-font-loading-with-javascript/

The short version is: commercial CDNs either don't offer a way to do this on the server at scale, or will do it but only as a custom deployment that can be very expensive at virtually any scale. Google can do it server-side for their Web Fonts API because they're Google, and have their own, very robust CDN infrastructure. We initially did it client side because we were on a startup's budget, however using JavaScript we're also able to target browser edge cases (such as the IE rendering mode) that aren't reliably detectable with just a UA string.

I practically guarantee you that if responsive images don't have native content negotiation in the client, adding a JS-based solution is one of the first things I'd need in order to recommend using responsive images at scale.

]]]

[1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=14701
[2] http://lists.w3.org/Archives/Public/public-respimg/2012Oct/0069.html
Comment 6 Marcos Caceres 2012-12-10 17:35:15 UTC
Data point: how people are currently detecting WebP. https://groups.google.com/a/webmproject.org/forum/?fromgroups=#!topic/webp-discuss/7PE2lZX9yOs
Comment 7 James Zern 2012-12-11 01:40:10 UTC
> Long version: Some images are best suited to a specific file type for reasons
> such as file size optimization, alpha transparency, scalability, animation,
> etc. For example, a photo usually requires good color depth, but does not
> require alpha transparency or animation; JPEG or WebP are well-suited to these
> needs and offer good optimization between image quality and file size. Icons
> are often simpler in terms of color depth, but may require alpha transparency;
> the PNG format is better-suited to these needs.
> 

Note that WebP (0.2.0+) has lossless and transparency support [1] giving further overlap with PNG.

[1] https://developers.google.com/speed/webp/
Comment 8 Ilya Grigorik 2012-12-11 07:45:22 UTC
Client-driven negotiation, such as adding multiple image formats into srcset doesn't solve the problem. 

As a practical example, let's imagine I invent and introduce a new magic image format (.mif). Next, I embed it alongside the other image formats (.png, and .jpeg) in my markup, and rely on the user agent to make the choice - so far, so good. Next, the user agent comes to the page and decides to show .mif to the user. The user loves the image, right clicks, copies the URL, or drags into their favorite bookmarklet to share on Tumblr / Facebook / etc, and *boom*  a MIF file has been shared with others, who may not actually support it. Other UA's viewing the image on Facebook, or elsewhere, get a broken resource. This is a non-starter. Client-driven negotiation breaks down the moment the resource leaves the page where all of the representations are available.

The good news is, this is exactly what content negotiation is for, and it works.. mostly*. Instead of polluting our markup with multiple image types, the UA declares what it supports (Accept: image/mif, image/png), and the server responds with the best format for the agent. If the user then shares the mif URL, the same negotiation is performed for all other UA's and if mif is not supported, an alternate format can be served. (The extension on the actual filename is not important, all UA's look at content-type over extensions + actual data in the resource -- this works today). 

The *mostly part is: as a practical example with WebP, neither Chrome nor Opera specify 'image/webp' in their Accept header, but for all intents and purposes, this is a bug, and it should be fixed. With that in place, the server can serve webp safely with an additional 'Vary: Accept'. 

Long story short, server-side negotiation is the only way to resolve this problem properly.
Comment 9 Odin Hørthe Omdal 2012-12-11 09:39:48 UTC
> Long story short, server-side negotiation is the only way to resolve
> this problem properly.

Ilya: Perfect in theory, not so much in practice.


Currently we (as in Opera) do send image/webp etc in our Accept-header (Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1). Although I myself have been running with Accept: */* as a test for a long time to see if we can simplify the header to that. At least aligning with Chromium/Firefox seems reasonable.


Content negotiation has sadly failed in practice. Browsers don't actually give any useful information any more, e.g. Chromium sends Accept: */* for image requests and Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 for page requests.

Firefox sends Accept:image/png,image/*;q=0.8,*/*;q=0.5 for images, and Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 for page requests.
Comment 10 Ilya Grigorik 2012-12-11 19:51:04 UTC
Yup, good points. That's the *mostly part I was referring to. :-) 

If the mechanism is not working as it should, we should fix it, as this case is precisely what it is designed for, and the client-solution doesn't actually solve the problem (for the reasons documented earlier). The fact that Opera is indicating webp support is awesome - and that should remain there! As you suggested we should align the headers between browsers. I'm happy to help motivate this from and to the Chrome team.

Also, while note directly relevant, I think it's important to note that in combination with the "Key" header field [1], using the Accept header gives the server fine-grained control over what should be the cache key for upstream proxies.

[1] http://tools.ietf.org/html/draft-fielding-http-key-01
Comment 11 Mark Nottingham 2012-12-14 00:27:37 UTC
Odin, 

Accept: */* should be functionally equivalent to sending no accept header at all; you might want to try your experiment with that and save even more bytes.

FWIW, some background:
  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/81
  https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#header.accept
  https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p2-semantics.html#proactive.negotiation
Comment 12 Odin Hørthe Omdal 2012-12-14 10:00:38 UTC
Mark: True, true. The key here is "should be". It is not, it breaks servers so that is not web compatible. It would surely be coolest, but too many sites break when you omit the Accept header.
Comment 13 Mark Nottingham 2012-12-14 11:56:27 UTC
Odin,

Do you have data around that? Examples? Would be very interested.
Comment 14 Ian 'Hixie' Hickson 2012-12-14 22:36:24 UTC
I'm really unconvinced that introducing new image file formats is a common enough problem that supporting it in srcset="" makes sense. After all, we've gone 22 years without supporting it in <img src> and we seem to have done fine. This kind of thing happens once a decade, and the sites who try to use the formats before they are widely supported are always going to be run by the most advanced authors, who are the most able to work around limitations here.
Comment 15 Yoav Weiss 2012-12-17 15:59:46 UTC
>After all, we've gone 22 years without supporting it in <img src> and we seem to have done fine.

Let's look at the "We have done fine" part:

One example of introducing a new image format (or format extension) is PNG with an alpha channel, which was poorly supported by IE6 and caused unnecessary developer pain for nearly a decade. Client side markup based on content type (Along with a seperate MIME type for alpha channel PNG) would have given authors a choice between using GIF or PNG during this transition period, benifiting both users and developers. Given the lack of an easy to understand fallback pattern, we saw memory-draining hacks using non-standard properties gain wide acceptance [1] in an effort to work around these limitations, many of which are still in use today. 

Another example is SVG, which has been a W3C recommendation since 2001 and been supported by browsers since 2005. It is still not supported in IE8 and Android 2.X devices (about 80% of Android's install base). The RICG believes this is one of the main reasons SVG sees poor adoption among developers, which must support these older browsers. 

In both cases, the lack of an image fallback mechanism significantly slowed down the wide adoption of a useful format.

Furthermore, the technological landscape has changed significantly over the last 6 years (particularly with the rise of web-capable mobile devices) which has put increasing pressure on standards organizations and related entities to come up with new image formats that better address the changes in the landscape. 

New image formats that have appeared in the last decade include: JPEG2K, JPEGXR, WebP. All of them supported by some browsers but with limited means for authors to make use of them. In this respect, the img tag is deliberately hostile towards new formats by not providing any means to perform client-side negotiation based on format support. We believe that a client based standard mechanism to help authors adopt new image formats could have helped the adoption of new formats. At the very least, it gives developers a choice to use the new formats without having to jump through hoops on the server side or use nasty hacks on the client side (e.g., including a 1x1px image of a particular format and then testing for an error event when loaded). 

Extensions to JPEG, including arithmetic coding (which gives a lossless ~12% gain) and hierarchical coding which could have been used as a responsive image format, were never added to the JPEG implmentation in browsers because of patent concerns that are no longer relevant. Again, there is no mechanism today that would ease their introduction.


> ... and the sites who try to use the formats before they are widely supported are always going to be run by the most advanced authors, who are the most able to work around limitations here.

If anything, this makes a case *for* introducing a simple fallback mechanism. The fact that more advanced authors are quicker to adopt formats with greater potential benefit to users speaks to a complexity barrier. As newer formats cannot be used in an unqualified manner but qualifying them requires additional content negotiation on the server or custom client-side testing—both of which may prove unreliable from platform to platform — it stands to reason that the lack of a simple, reliable fallback pattern is a huge barrier to the adoption of these formats. For example, we’ve seen the rapid adoption of webfonts by novice authors despite the lack of a standardized file format, because a reliable fallback was available. Later, when the WOFF format was introduced, it was adopted very quickly even by less advanced authors.

Today server-side content negotiation cannot use the `Accept` header sent by browsers, which means it would have to rely on User-Agent sniffing. UA Sniffing is costly, future hostile, and inherently unreliable. 

Further adding to the difficulty of server-side content negotiation, the RICG have conducted a survey [2] currently including 137 web developers, which showed that ~26% of Web developers have no access to their server configuration, which is to say nothing of those that would have trouble implementing a complex UA-sniffing solution. Although the survey is not completely representative, we believe it to be highly indicative of the landscape. We will continue to run this survey and gather more data. However, the yes/no ratio seems to be stable.

The above complications are likely the reason that newer formats see relatively slow adoption by more advanced authors, despite their potential benefits.
Client-side fallback mechanisms exist for `<video>`, web fonts, and CSS background images. We do not understand why content images should be deprived from such a mechanism when there is already precedence on the Web Platform for this? (particularly when the current mechanism exclusively works only with legacy formats on the client side).

One of the strongest arguments against introducing such a mechanism to `<img src>` is that it would cause compatibility issues with current content. As there is no content yet on the Web using srcset, that burden does not exist for `srcset`. It is the RIGC's opinion that we should embrace this opportunity to introduce easier mechanisms for new Web image formats (which already exist, are supported in browsers, and are not mythical).

The sharing of an image resource of a newly introduced format does introduce a user interface problem, though a solvable one. It is merely convention that raises the expectation of right-click leading to a method to save a resource or copy a link to it, and it's a convention that does not hold up for some assets without developer tools installed - which most people don't have (CSS background images, a number of video sites, images inside of Canvas, etc). An example of a solution to this problem can be a context menu option that will save/share a link to the widely supported version of the resource, rather than the displayed one.


[1]: http://24ways.org/2007/supersleight-transparent-png-in-ie6/
[2]: http://freeonlinesurveys.com/app/showpoll.asp?qid=169055&sid=9re1nr8eed91gge169055&new=True
Comment 16 Ian 'Hixie' Hickson 2013-04-22 23:53:32 UTC
How much do authors use <object> for fallback from SVG to something else?
Comment 17 Ian 'Hixie' Hickson 2013-06-04 21:33:03 UTC
Design note: If we did do a solution to this, it would have to declaratively give the format type (since otherwise both images would get downloaded by downlevel clients, which is bad since they're the ones most likely bandwidth-constrained, and the whole point of a new format is probably to alleviate bandwidth concerns).

(In reply to comment #15)
> 
> Let's look at the "We have done fine" part:
> 
> One example of introducing a new image format (or format extension) is PNG
> with an alpha channel, which was poorly supported by IE6 and caused
> unnecessary developer pain for nearly a decade.

Sure, but there's no guarantee that if we add a way to do it for, say, WebP, that IE won't poorly implement that too. No solution we provide can defend against a future browser with half-baked support. So this isn't an argument in favour of adding such a feature.


> Another example is SVG, which has been a W3C recommendation since 2001 and
> been supported by browsers since 2005. It is still not supported in IE8 and
> Android 2.X devices (about 80% of Android's install base). The RICG believes
> this is one of the main reasons SVG sees poor adoption among developers,
> which must support these older browsers. 

Why doesn't <object> work for these browsers?


> In both cases, the lack of an image fallback mechanism significantly slowed
> down the wide adoption of a useful format.

For PNG, what slowed adoption was a bad browser, not the lack of a fallback mechanism (indeed there _was_ a fallback mechanism at the time, <object>). For SVG, the issues are a bit more complex, and I'm not sure what conclusion can be drawn really.


This request would be more convincing, IMHO, if authors used <object> for format fallback (the current feature in HTML for this). The sad truth is, it's hardly ever used for that at all.

Note that advanced users can use scripts to hack around this relatively easily. Also, future features such as those for intercepting network requests could be leveraged to do this at a much lower level (catching all requests to .../png/* and changing them to .../webp/*, for instance).
Comment 18 Yoav Weiss 2013-06-05 07:12:46 UTC
> Design note: If we did do a solution to this, it would have to 
> declaratively give the format type (since otherwise both images would get 
> downloaded by downlevel clients, which is bad since they're the ones most 
> likely bandwidth-constrained, and the whole point of a new format is probably
> to alleviate bandwidth concerns).

I agree completely. The solution within srcset could be similar to the proposed CSS4-images solution [1] to the same problem.

>    (In reply to comment #15)
>    >
>    > Let's look at the "We have done fine" part:
>    >
>    > One example of introducing a new image format (or format extension) is PNG
>    > with an alpha channel, which was poorly supported by IE6 and caused
>    > unnecessary developer pain for nearly a decade.
>
>    Sure, but there's no guarantee that if we add a way to do it for, say, WebP,
>    that IE won't poorly implement that too. No solution we provide can defend
>    against a future browser with half-baked support. So this isn't an argument in
>    favour of adding such a feature.

 
I agree that there are no guaranties against bad implementations, even though the solution in [1] can decouple browser support from the image's MIME-type, which can enable finer granularity (e.g. Regular PNG would have been declared as format("image/png"), while a PNG with an alpha channel would have been declared as format("image/png", "alpha"). IE6 would download the former, but not the latter)



>    > Another example is SVG, which has been a W3C recommendation since 2001 and
>    > been supported by browsers since 2005. It is still not supported in IE8 and
>    > Android 2.X devices (about 80% of Android's install base). The RICG believes
>    > this is one of the main reasons SVG sees poor adoption among developers,
>    > which must support these older browsers.
>
>    Why doesn't <object> work for these browsers?

 
Using <object> as a fallback results in a double download in browsers that support both types, so it is not something you'd use when you introduce new formats for optimization purposes.


>    > In both cases, the lack of an image fallback mechanism significantly slowed
>    > down the wide adoption of a useful format.
>
>    For PNG, what slowed adoption was a bad browser, not the lack of a fallback
>    mechanism (indeed there _was_ a fallback mechanism at the time, <object>). For
>    SVG, the issues are a bit more complex, and I'm not sure what conclusion can be
>    drawn really.
>
>
>    This request would be more convincing, IMHO, if authors used <object> for
>    format fallback (the current feature in HTML for this). The sad truth is, it's
>    hardly ever used for that at all.

 
Again, <object> as a fallback results in double download, which means it's of no use for optimization purposes.


>    Note that advanced users can use scripts to hack around this relatively easily.

 
Yes, but this results in significant delay in the images download (the preloader can't detect them), and requires adding a <noscript> tag for each image, in order to get the site also working without scripts.

>    Also, future features such as those for intercepting network requests could be
>    leveraged to do this at a much lower level (catching all requests to .../png/*
>    and changing them to .../webp/*, for instance).

If you are referring to the NavigationController, it may be able to somewhat help here, but since it is only planned to work starting from the second load of the page, it would not be a full solution.

[1] http://comments.gmane.org/gmane.comp.web.css.general/42726
Comment 19 Marcos Caceres 2013-06-05 11:07:37 UTC
Object causing double download test page:
http://codepen.io/seraphzz/full/eduzn
Comment 20 Yoav Weiss 2013-06-05 21:12:19 UTC
A less noisy example of the object fallback double download in supporting browsers can be found at [1]
In Chrome the SVG is not displayed properly (does not fit itself to the object's dimensions). While probably a bug, that can easily explain (along with the double download) why this pattern is not used more often.

[1] http://responsiveimagescg.github.io/tests/ObjectFallbackForImageType/index.html
Comment 21 Marcos Caceres 2013-06-07 07:19:30 UTC
The (In reply to comment #20)
> A less noisy example of the object fallback double download in supporting
> browsers can be found at [1]
> In Chrome the SVG is not displayed properly (does not fit itself to the
> object's dimensions). While probably a bug, that can easily explain (along
> with the double download) why this pattern is not used more often.

The test case has moved here:
http://responsiveimagescg.github.io/experiments/ObjectFallbackForImageType/index.html
Comment 22 Ian 'Hixie' Hickson 2013-06-13 22:39:29 UTC
Those cases don't use <object> for fallback, they use <img> for fallback.

I mean something like:

   <object data="image.webp">
      <object data="image.png">
         text equivalent
      </object>
   </object>
Comment 23 Marcos Caceres 2013-06-15 17:44:19 UTC
Rep(In reply to comment #22)
> Those cases don't use <object> for fallback, they use <img> for fallback.
> 
> I mean something like:
> 
>    <object data="image.webp">
>       <object data="image.png">
>          text equivalent
>       </object>
>    </object>

Moved something similar to the above here:

http://responsiveimagescg.github.io/experiments/nestedObject/object.html

Can be modified, updated here:
https://github.com/ResponsiveImagesCG/experiments/tree/gh-pages/nestedObject
Comment 24 David Newton 2013-06-15 20:00:50 UTC
> Moved something similar to the above here:
> 
> http://responsiveimagescg.github.io/experiments/nestedObject/object.html

This structure fails rather spectacularly in iOS:

http://responsiveimagescg.github.io/experiments/nestedObject/results/MobileSafari_iPad_iOS6.1.3.png
Comment 25 Yoav Weiss 2013-06-16 08:52:19 UTC
One more thing:
In Firefox this pattern downloads all the resources, even if "type" is specified [1]
So, the fact that authors don't use this pattern, which is not commonly known, buggy, and not detected by the preloader (so, has performance implications), is not indicative to the lack of need for a image type fallback solution (which exists for fonts & video and is used extensively there)

[1] http://responsiveimagescg.github.io/experiments/nestedObject/object_with_type.html
Comment 26 Ian 'Hixie' Hickson 2013-06-21 16:24:27 UTC
Well before we add yet more ways for browsers to be buggy, we should get the browsers to implement the existing features correctly. There's not much point adding features to theoretically do something that's already theoretically possible, if the browsers aren't supporting what's already possible.
Comment 27 David Newton 2013-06-21 17:37:50 UTC
This pattern is unused by devs, and broken in browsers. I also don't think it's realistic to expect browser vendors to change their implementations of <object> here, in no small part because there's no real demand for it.

IMO the pattern is not wanted by devs, simply because it's so convoluted. If a dev wanted to offer WebP, JPEG-XR, and JPEG versions with appropriate fallbacks, they'd need three nested <object> elements, with a final fallback to text. Personally, I find that ugly. I suspect if devs actually liked this pattern and wanted to use it, there would be a lot more noise to browser makers to get it working.

In addition to just being syntactically ugly, it's semantically ugly too. An image should not be represented by an <object>. Maybe it's not as bad as having an image represented by a <div>, but it's not as nice as using an element *designed for images*. If this pattern were fixed and promoted (assuming the implementations actually worked), it would require asking devs to choose between semantics and optimization, which is unappealing at best.

Finally, this pattern of using objects for images can't coexist with srcset, which is what this bug is about. Once we consider srcset, a dev is now faced with choosing between semantics + resolution optimization (<img srcset />) and format optimization (<object><object><object></object></object></object>). Expecting devs to use the latter is a non-starter.
Comment 28 Ian 'Hixie' Hickson 2013-07-02 22:43:45 UTC
> This pattern is unused by devs, and broken in browsers. I also don't think
> it's realistic to expect browser vendors to change their implementations of
> <object> here, in no small part because there's no real demand for it.

If there's no real demand, then we shouldn't be adding another mechanism for it either. I don't think it makes sense to argue that there's demand for a feature on the one hand, but that there's not demand for the same feature on the other.

Authors will go to extreme lengths to use features we provide even when they're broken and extremely obtuse, if they actually want to use the Web. (Just look at the many libraries that have been made to work around lots of browser bugs in things like the DOM.)


> IMO the pattern is not wanted by devs, simply because it's so convoluted.

It's really not that convoluted.

   <object data="tree.webp">
     <object data="tree.wpn">
       <object data="tree.gif">
          I have a tree in my back yard.
       </object>
     </object>
   </object>

It's amongst one of the simpler things in HTML.


> Personally, I find that ugly.

I agree it's ugly, but then so's the rest of HTML. Since when has that stopped anyone from using something?


> I suspect if devs actually liked this
> pattern and wanted to use it, there would be a lot more noise to browser
> makers to get it working.

I agree. My point is that this means we shouldn't add a way to do it with srcset either — if there was demand for it, devs would have been asking for <object> to work to solve this before srcset="" was invented.


> In addition to just being syntactically ugly, it's semantically ugly too.

That has _definitely_ never stopped anyone before. :-)


> An image should not be represented by an <object>.

I don't understand why. Can you elaborate? <object> was specifically designed for the use case of embedding objects, such as images.


> If this pattern were fixed and promoted
> (assuming the implementations actually worked), it would require asking devs
> to choose between semantics and optimization, which is unappealing at best.

Granted that it's unappealing, but it's also a decision that devs have made without the slightest difficulty for years, always choosing optimisation.


> Finally, this pattern of using objects for images can't coexist with srcset,
> which is what this bug is about.

I agree that <object> doesn't solve the problems srcset="" solves.


My thesis is that we don't need to support this in srcset="" because most people never bother trying to support multiple formats. My supporting evidence for this is that we've had a mechanism to do exactly this for a decade and a half now, and there's basically zero demand for it. I don't understand why we would conclude that authors would want this for srcset="" when they've shown zero interest in using it with <object> over the past 15 years, to the point that browsers apparently haven't felt any pressure to implement <object> correctly.
Comment 29 David Newton 2013-07-18 15:59:22 UTC
(In reply to comment #28)
> > This pattern is unused by devs, and broken in browsers. I also don't think
> > it's realistic to expect browser vendors to change their implementations of
> > <object> here, in no small part because there's no real demand for it.
> 
> If there's no real demand, then we shouldn't be adding another mechanism for
> it either. I don't think it makes sense to argue that there's demand for a
> feature on the one hand, but that there's not demand for the same feature on
> the other.

Stating that there is no demand for this syntactic construction is not equivalent to stating that there is no demand for this use case. There are many examples of devs finding other ways to tackle this problem, which don't involve using <object>, including JS- and server-based solutions.

Some examples:
CSS4 image fallback [2] (format agnostic, developing standard)
weppy.js [3] (WebP, client-side)
Modernizr [4] (APNG/WebP/SVG/, client-side) note: JPEG-XR support pending [19]
WebPJS [5] (WebP, client-side)
webp-detect [6] (WebP, server-side)
svgeezy [7] (SVG, client-side)
svgweb [8] (SVG, client- & server-side)
canvg [9] (SVG, client-side)
SIE [10] (SVG, client-side)
fabric.js [11] (SVG, client-side)
apng-canvas [12] (APNG, client-side)
png.js [13] (APNG, client- & server-side)

Demand for *a* solution is there as shown above; demand for <object> as a solution is not because it evidently does not work.


> Authors will go to extreme lengths to use features we provide even when
> they're broken and extremely obtuse, if they actually want to use the Web.
> (Just look at the many libraries that have been made to work around lots of
> browser bugs in things like the DOM.)

Agree. Authors *have* gone to extreme lengths, they just haven't done so in the way you're prescribing: it's much more sane to use a solution like picturefill, or a server side solution for them. 

See the ever growing number of polyfills to do responsive images:
https://github.com/ResponsiveImagesCG/picture-element/wiki/Current-users,-polyfills,-prototypes,-and-implementations

Our argument here is that there is no demand for using the <object> syntax, which I think we can agree on, and without that demand it is unrealistic to expect browser vendors to change how <object> behaves. Again, this is not equivalent to saying that there is no demand for the feature.

Some of the reasons there is no demand for this syntax are outlined above. Chief among them is that it doesn't work on where it's actually needed (on mobile devices, especially on the most popular platform).


> > > IMO the pattern is not wanted by devs, simply because it's so convoluted.
> 
> It's really not that convoluted.
> 
>    <object data="tree.webp">
>      <object data="tree.wpn">
>        <object data="tree.gif">
>           I have a tree in my back yard.
>        </object>
>      </object>
>    </object>
> 
> It's amongst one of the simpler things in HTML.

You can argue that it's simple (we disagree), but you can't argue that it is working today. In this pattern, a browser not supporting the latest format will download WebP, realize that it cant use it, then download the "wpn" image, realize it can't use it, then download the gif image and will use it.

So we've downloaded 2 extra images for no good reason, and added at least 2 RTTs + image download time to this specific image's load time (and therefore to the page's onload event)

Adding a 'type' attribute to this pattern helps significantly, but it is not widely supported (Firefox downloads all the resources anyway, IE simply chokes on it and displays nothing)

So, using the fact that this broken pattern is not widely used to prove there's no need for img format fallback is an equivalent to a non-democratic regime, where the lack of need for democratization is demonstrated by the fact that no one shows up to vote anyway.


> > Personally, I find that ugly.
> 
> I agree it's ugly, but then so's the rest of HTML. Since when has that
> stopped anyone from using something?

That's not what's stopping people from using it. The fact that it doesn't work is what's stopping people from using it. That it's ugly is an argument against attempting to "fix" it and make it standard practice.


> > I suspect if devs actually liked this
> > pattern and wanted to use it, there would be a lot more noise to browser
> > makers to get it working.
> 
> I agree. My point is that this means we shouldn't add a way to do it with
> srcset either — if there was demand for it, devs would have been asking for
> <object> to work to solve this before srcset="" was invented.

Since the introduction of WebP and JPEG-XR we have seen numerous attempts from devs to solve this problem [3, 4, 5, 6]. That they didn't try to use <object> is not an indication that they don't want the problem solved, only that they don't want it solved using object.


> > In addition to just being syntactically ugly, it's semantically ugly too.
> 
> That has _definitely_ never stopped anyone before. :-)

Agree. Again, not arguing that this is what stopped people from using it. Being broken is what stopped people from using it. Being semantically ugly is a reason to not "fix" and adopt going forward.


> > An image should not be represented by an <object>.
> 
> I don't understand why. Can you elaborate? <object> was specifically
> designed for the use case of embedding objects, such as images.

Because, all things being equal, <img> is *more* correct than <object>, because <img> is more semantically specific. If this was not a relevant concern, then there would be no reason to use <video> or <audio> instead of <object>, or <section> or <article> instead of <div>. There's nothing incorrect about <object>, it just lacks structural semantics, and will likely have negative side effects when one tries to use an object where an image is expected. 


> > If this pattern were fixed and promoted
> > (assuming the implementations actually worked), it would require asking devs
> > to choose between semantics and optimization, which is unappealing at best.
> 
> Granted that it's unappealing, but it's also a decision that devs have made
> without the slightest difficulty for years, always choosing optimisation.

Agree, which is why devs have chosen JS- and server-based solutions to this problem that may not be great semantically. What we're discussing is how best to address this going forward. If something needs to be changed anyway, why not make a change that allows for both optimization and semantics?


> > Finally, this pattern of using objects for images can't coexist with srcset,
> > which is what this bug is about.
> 
> I agree that <object> doesn't solve the problems srcset="" solves.
> 
> 
> My thesis is that we don't need to support this in srcset="" because most
> people never bother trying to support multiple formats. 

Facebook [14] and the Google Store [15] have both recently made moves towards supporting WebP, with fallbacks. Facebook has a couple of users, no? 


> My supporting
> evidence for this is that we've had a mechanism to do exactly this for a
> decade and a half now, and there's basically zero demand for it. 

Attempted implementation from two huge content providers with many millions of users [14, 15]
+ at least 10 JS solutions [1, 3, 4, 5, 6, 7, 9, 10, 11, 12]
+ at least one server-side solution [6]
+ at least two proposed/developing standards [2, 16] 
+ bugs filed with browser vendors [17, 18]
+ countless blog, email, and IRC discussions

The RICG is happy to do the due diligence for you. We are happy to provide you with more evidence if you need it - we hope that convinces you that there is actually significant demand for it.  


> I don't
> understand why we would conclude that authors would want this for srcset=""
> when they've shown zero interest in using it with <object> over the past 15
> years, to the point that browsers apparently haven't felt any pressure to
> implement <object> correctly.

*Most* don't, because all the options right now are bad. Many do, by going to the extreme lengths outlined above.

The "mechanism for doing this" may have existed for 15 years, but that's not relevant - <object> was primarily synonymous with Flash. This responsive images problem is new on the Web: In the past, the biggest new image format adoption was PNG. In late 1997, both IE and Netscape -- which, combined, covered the vast majority of browser market share -- introduced support near simultaneously. Before this support was there, PNG didn't get used. After support was there, PNG could be used. For many devs, the advantages of PNG were negligible, so there was no pressing reason to change formats. There were also delays in adoption due to use of old browser versions, but this version fragmentation was nothing like the problem we're seeing now.

The current problem, in the last couple of years, is that device, browser, and version fragmentation means that a dev cannot be reasonably sure whether a new format (e.g. WebP or JPEG-XR) can be viewed by a user, without resorting to server-side or JS hacks. Furthermore, there is more of a demand for these new formats because they offer significant advantages (e.g. file size), especially for large and high-resolution images - today, there is more demand for these higher resolution images because of the trend towards devices having high-density displays. Since these formats have been introduced, there has been demand to use them, and to create a fallback strategy, as shown above.

The RICG's thesis is that a new HTML-based fallback strategy is needed. Our supporting evidence for this is that:
    * the market continues to pack more pixels into displays (up to 3x or 4x) - there is no reason to believe it will not continue to increase given 4k display and 8k displays arriving over the next decade (see our wiki page [20] for a list of upcoming devices or devices currently on the market).    
    * there is developer interest in fallbacks, as shown above
    * the existing fallback strategies in HTML (i.e. <object>) do not work
    * Javascript-based solutions fail for users that don't have JS enabled, potentially create accessibility issues, are not available to the pre-parser, and cause performance issues
    * Server-side solutions are unavailable to authors that don't have access to the server. They also require inaccurate UA sniffing (even though that's being worked on) and cause caching issues, since the "Vary" header is not accurate enough.


[1] https://github.com/scottjehl/picturefill/
[2] http://www.w3.org/TR/css4-images/#image-fallbacks
[3] http://nnucomputerwhiz.com/use-webp-images-with-jpeg-fallback.html
[4] http://www.stucox.com/blog/using-webp-with-modernizr/
[5] http://webpjs.appspot.com/
[6] https://github.com/igrigorik/webp-detect/
[7] https://github.com/benhowdle89/svgeezy
[8] https://code.google.com/p/svgweb/
[9] https://code.google.com/p/canvg/
[10] http://sie.sourceforge.jp/
[11] https://github.com/kangax/fabric.js
[12] https://github.com/davidmz/apng-canvas
[13] https://github.com/devongovett/png.js
[14] http://news.cnet.com/8301-1023_3-57580664-93/facebook-tries-googles-webp-image-format-users-squawk/
[15] http://techcrunch.com/2013/02/07/google-now-uses-its-own-webp-format-instead-of-pngs-in-the-chrome-web-store/
[16] http://picture.responsiveimages.org/
[17] https://bugzilla.mozilla.org/show_bug.cgi?id=webp
[18] https://bugzilla.mozilla.org/show_bug.cgi?id=856375
[19] https://github.com/Modernizr/Modernizr/pull/895
[20] https://github.com/ResponsiveImagesCG/picture-element/wiki/Display-technologies:-beyond-Retina
Comment 30 Ian 'Hixie' Hickson 2013-07-18 18:01:33 UTC
Ok, there's demand. Thanks for the data on that.

Why are browsers going to implement a new feature to do this better than the 15 year old feature?
Comment 31 Mat Marquis 2013-10-01 17:09:04 UTC
(In reply to Ian 'Hixie' Hickson from comment #30)
> Ok, there's demand. Thanks for the data on that.
> 
> Why are browsers going to implement a new feature to do this better than the
> 15 year old feature?

I’m happy to speak to the authors’ side of this equation—current usage and prevailing developer sentiment, detailed extensively above—but I’m sure you don’t expect us to speak on behalf of every browser.

My hope is that the feature would be “done better” based on thoughtful consideration of the feedback we’ve provided being reflected in the specification. Perhaps I’m a bit confused about what form of response you’re looking for, here. Could you clarify?
Comment 32 Ian 'Hixie' Hickson 2013-10-01 20:30:31 UTC
I just don't understand why, if browsers haven't felt the pressure to implement this right in the past with <object>, they would feel the pressure to implement it right in a new feature.