Where Things Stand

Mathew Marquis | Posted on: March 7, 2012

Our goal is a markup-based means of delivering alternate image sources based on device capabilities, to prevent wasted bandwidth and optimize display for both screen and print. From an accessibility standpoint, it should at the very least have something equaling img’s alt tag. For non-supporting browsers, fallback content should be displayed.

This Community Group is my effort to bring a growing number of developers together with some like-minded browser and standards folk, so we can iron out any issues we may not have considered and keep things moving forward.

The idea is to use the video tag’s markup pattern as the inspiration, as it’s specced to allow the use of media queries within attributes on its source elements and reliably displays the markup inside the tag in any browser that doesn’t recognize it. As with the video and audio tags, this solution shouldn’t require any custom scripting, HTTP/SPDY headers, or server-side technologies to reliably deliver content tailored for the end user’s context. Through use of media attributes we would not only be able to reduce wasteful image requests for the sake of users with smaller displays, but we could tailor our images’ resolutions for users with high-res displays, or for printing.

You can see part of our brainstorming process out in the open here:
https://etherpad.mozilla.org/responsive-assets

I’ve also published the most viable-seeming markup pattern at A List Apart:
http://www.alistapart.com/articles/responsive-images-how-they-almost-worked-and-what-we-need/

That markup pattern is something to the tune of:

<picture alt="Alt tag should accurately describe the image represented by all sources, though cropping and zooming may differ.">
  <source src="mobile.jpg" /> <!-- Matches by default. -->
  <source src="high-res.jpg" media="min-width: 800px" /> <!-- Overrides the previous source over 800px before any assets are fetched, resulting in a single request. -->
  <img src="mobile.jpg" /> <!-- Fallback content, in the event the <picture> tag is completely unsupported by the user’s browser. -->
</picture>

It turns out we reached much the same conclusion as many others have previously—in some cases, right down to the semantics:

We posted all this information to the WHATWG mailing list recently, but ended up covering a lot of the same “what if we”/“how about” ground. While a little disjointed, you can mine through said thread at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2012-February/ by doing a search for “responsive images”.

I’m looking forward to some focused, productive discussion here. Let’s get this thing done.

39 Responses to Where Things Stand

Jason Grigsby ✪ February 21, 2012 at 4:03 am

Rereading the thread has me more encouraged than I felt before. There were two conversations going on about images on the WhatWG list at the same time.

One was about having the client send back headers. There was a lot of pushback on that discussion.

The discussion about a new element seemed to primarily revolve around different ideas on implementation. I didn’t take the time tonight to reread every single comment in the thread, but I looked at quite a few of them and didn’t bump into anyone pushing back on the idea in general.

Mat, is that consistent with your sense of how the conversation about a new element went?

Scott Jehl ✪ February 21, 2012 at 8:10 am

Great intro, thanks Mat.

Regarding the markup pattern: In order to support full media query syntax, I think the media attributes above need to be more like media="(min-width: 500px)", basically, just adding some parentheses. They could just as well be more specific, like media="screen and (min-width: 500px) and (orientation: landscape)".

Seem accurate?

Scott

Matt Wilcox ✪ February 21, 2012 at 12:29 pm

Can I ask why we are ruling out servers, and why we are assuming CSS media queries are the right way to go from the offset?

If it’s simply because this group is only interested in client-side solutions thats fine (I’m well up for that!), but if the group is about finding strategies to adapt content then I think it is entirely valid to talk about server adaption too. There is a place for both – where is the focus for the group?

Matt Wilcox ✪ February 21, 2012 at 12:31 pm

For me Responsive Design is about the following:

* Respond to the needs of the user.
* Present information as clearly as possible.
* Be as efficient as possible.

I believe that any image solution needs to respect these principals as responsive images are merely an extension of this mindset.

There are problems with our technologies; what are we responding *to*? People often say screen size, but this very often simply a proxy for other things we would rather be responding to but can’t: bandwidth, usage metering, memory size, device speed, etc.

I propose that any solution should define what we’re responding to as well as a mechanism for responding.

I also propose that any solution must:

* Be easy to achieve, or it won’t get used.
* Obey progressive enhancement methodologies.
* Work with the eco system of the web.

I’d also like to point out that alternate image resources need not be simple re-sized versions of the same resource. It is appropriate to substitute different images with the same semantic meaning at certain times. E.g., a profile page may have a photo of a person in it. At large display sizes it’s OK to have a full body shot with the person in a setting. At small sizes the important details may be lost, and it makes sense to switch to a head-shot.

(PS, sorry for the slightly stilted comment – this is all knicked from a presentation I’m writing for StandardsNext in Manchester next month)

John Albin Wilkins ✪ February 21, 2012 at 1:58 pm

Jason,

That was my feeling as well with the WhatWG list. The discussion about HTTP headers really took off! And the arguments made in the push-back confirmed my belief that a client-side solution is the way forward.

Just after the holidays, I was trying to ping Mat over twitter about the necessity of explaining the “why we got here” for <picture> but Twitter is crap for starting lengthy discussions. Here is a much better forum!

The Drupal community has recently created “Initiative Leaders” that organize discussions around large architectural changes to the codebase. The Leads have usually had public discussions and then come to a consensus-based solution. The problem we’ve repeatedly faced is when presenting “the solution”. If we present the solution without also presenting all the steps for how that decision was arrived, we always end up repeating a lot of the same arguments (but with a much larger audience and in a less productive way).

The responsive image problem faces the same issue.

The giant tangential threads we saw in the WhatWG email list were because, while we did link to that etherpad, no one read the background. We need to present the background right along with the solution or we’ll end up repeating the same arguments ad nauseum.

🙂

edit: Looks like Matt posted before I had a chance to submit and proved my point. People just don’t follow the background material links and read the lengthy material. It’s human nature.

We’ll need a wiki-like page for a document describing various proposed solutions and their pro/cons. Again. 🙂

– John

John Albin Wilkins ✪ February 21, 2012 at 2:27 pm

But to respond to Matt, I agree that we should consider any solution. I’m personally convinced client-side is the way to go, but we need to convince everyone here and then everyone everywhere that a specific client-side way is _the_ way. And the only way we’ll get there is by documenting all the wrong turns.

I’d like to partially frame the problem this way…

Designers want to have the editorial ability to have different “displays” of a single image resource, depending on the context that resource is presented in. As such, they’ll need to create rules for when each display is used for a resource. Those rules can either be put into a server-side or client-side mechanism. (There’s a further background discussion to prove this is true, but I’ll save it for later. i.e. explaining why “waiting for everyone to get faster connections” isn’t a solution.)

How the rules are evaluated is based solely on the context. And all of that context is firmly on the client side of things. Its the browser that knows what the hell is going on.

Can we have the browser send context to the server? Sure. But we can’t send all of the possible context. If we limit the context to a specific set of pre-defined criteria, we limit our future ability to have different kinds of rulesets. There was also a lengthy discussion on the WhatWG list about how adding weight to the HTTP header is counter-productive since it adds a performance penalty for every single request. Its only a small set of HTTP resources that would need the context. GET /favicon.ico, GET /scripts/jquery.js, GET /README.txt, etc., etc. ALL of those would be penalized. Its simply not a good trade-off.

Could we negotiate the context sent? The negotiation would violate our “first request” problem (we’d waste time with back and forth HTTP requests).

So do we send all the context from the browser to the server? Or, instead, do we send all the rules from the server to the browser?

I think the latter wins.

Matt Wilcox ✪ February 21, 2012 at 2:45 pm

@ John – I hope I didn’t prove that point, considering I’ve read a lot of that stuff. My post above was to clarify what the intention of this group is:

a) to provide a mark-up solution
b) to look into *all* solutions, of which mark-up may be a part.

That’s why I asked that explicitly.

I think client side is generally the preferable approach and would strongly back a solution like the one suggested. But I also feel there is a need for server side responsiveness too. I’d rather see *both* than just one, but if I was to pick just one then it would be the client side solution.

I’ve already discussed mechanisms to make client/server header communication efficient in other places and argued that the “every request” argument is poor, especially given SPDY rather than HTTP. For clarity my thoughts were these:

1) Client requests as normal
2) Server responds with normal content but accompanied by a feature request header [request-bandwidth|request-screen-dimensions|request-on-jpg]
3) Non-conformant browser does nothing with that. No impact has been had.
4) Conformant browser sends future requests on the domain with matching file-types along with requested headers.

SPDY can also push content that hasn’t been requested, so in the event the first request got a sub-optimal response the server could re-send with a better one. There would be no more than one potential additional outgoing request per *website*. SPDY also Gzips headers, so there’s not much overhead.

- John Albin Wilkins ✪ February 21, 2012 at 5:30 pm
  
  My apologies! I felt like the WhatWG discussion kind of put an end to the idea of header-based negotiations. But your proposal sounds interesting. I think I may have missed it when you posted that in the WhatWG list. But I didn’t read all the posts in that discussion, so maybe I proved my own point! 😀
  
Mathew Marquis ✪ February 21, 2012 at 3:40 pm

I’m gonna write this up in more detailed post, but I really want to keep this group laser-focused on the markup based solution. Don’t get me wrong—I’d love to have detailed gadget-describing headers available. I think we’d all rather have more device information readily available than not.

I’m just really uneasy about treating headers as a solution to this problem, since there’s a big gap between “having the user’s screen size” and an implementation—and that’s assuming the information contained in the headers is always correct. My concern is that it would basically put the onus on developers to implement their own versions of the element we’re proposing—which not only makes implementation more difficult for the developer, but varying quality implementations of “responsive images” are gonna impact the user in terms of wasted bandwidth or—at worst—completely broken images.

I do think the header conversation is worth having, and I’d love to be involved in it. For now, though, I want to keep our focus on the front-end solution.

- Matt Wilcox ✪ February 21, 2012 at 3:45 pm
  
  Totally agree, I would not want to push server-side stuff as “the” solution, and I’m happy with this group being focussed on the mark-up side of things. I just wanted to make sure that’s what the idea is, rather than anything more holistic, before I went off on any tangents! 🙂
  
bruce ✪ February 21, 2012 at 4:28 pm

I’m convinced that for simple situations (I want to send a low-res image for very small device width, and a hi-res, wide picture for widescreen TVs) that we need a declarative, client side solution.

Given that we have that, combining video element with CSS MQs, we need something equally simple for images.

I’m delighted if we have another mechanism based on content negotiation, more headers or robust server-side detects for those developers who are comfortable with server-side work, and who even have access to their servers. But that’s a discussion for another day.

The urgent need – and low-hanging fruit – is a declarative, mark-up based solution,

- Matt Wilcox ✪ February 21, 2012 at 5:34 pm
  
  I’d agree with all of that, including the lifting of CSS-MQ syntax.
  
  That said, a concern I have at the moment is to avoid repetitive declaration of identical trigger points in multiple instances of an element. That’s potentially a lot of identical CSS-MQ statements cropping up which is a waste of time, bandwidth, and bad for maintenance (re-designs may call for different sized *content* images) 🙂
  
Denys Mishunov ✪ February 21, 2012 at 5:54 pm

I think there is an agreement on focusing this group on a client-side solution. More so because there is no server-side solution yet ready and no matter what we decide, this has yet to be done (as in “built” or “created”). To me personally, client-side solution feels like an easier approach to the issue.

What I am not convinced with is the suggested element. First of all I really like it being similar to that gives us predictable and re-usable mechanisms for solving potentially similar issues in the future. I like that there is nothing limiting authors from declaring 2, 3, 5 and so on image sizes if there is a real need for this. These are all strong points and the proposed solution sounds very flexible and quite convinient.

But there is one thing that bothers me. We do have tag for representing images. It’s been here for years. And there will always be cases when new responsive images and good old conventional images are used together on the same page. Now, we want people to use not one, but two different tags that have the same semantical meaning — they are images no matter whether responsive or not. I see (in the etherpad) that there are quite a few ideas of re-using tag. But all of the solutions have some different issues. Wouldn’t it be the right thing to do — to review those potential solutions with tag and make a decision whether we can solve those OR live with those being present OR they are a real blockers for solution. If we come to an agreement that can not be used for the responsive images, I am fine with having two different tags of course. But then, even if it still bugs me, we will have good arguments to explain our decision.

- Denys Mishunov ✪ February 21, 2012 at 5:59 pm
  
  Hmm… all my tag references in the comment got stripped out. Should I re-post I wonder? Or the idea is clear?
  
  - Denys Mishunov ✪ February 21, 2012 at 6:00 pm
    
    D’oh, <img>… this form needs a preview
    
- Denys Mishunov ✪ February 21, 2012 at 6:09 pm
  
  I’ll better re-post my comment. Sorry for the inconvinience…
  ——————–
  I think there is an agreement on focusing this group on a client-side solution. More so because there is no server-side solution yet ready and no matter what we decide, this has yet to be done (as in “built” or “created”). To me personally, client-side solution feels like an easier approach to the issue.
  
  What I am not convinced with is the suggested <picture> element. First of all I really like it being similar to &ltvideo&gt that gives us predictable and re-usable mechanisms for solving potentially similar issues in the future. I like that there is nothing limiting authors from declaring 2, 3, 5 and so on image sizes if there is a real need for this. These are all strong points and the proposed solution sounds very flexible and quite convinient.
  
  But there is one thing that bothers me. We do have <img> tag for representing images. It’s been here for years. And there will always be cases when new responsive images and good old conventional images are used together on the same page. Now, we want people to use not one, but two different tags that have the same semantical meaning — they are images no matter whether responsive or not. I see (in the etherpad) that there are quite a few ideas of
  re-using <img> tag. But all of the solutions have some different issues. I would suggest that the first thing we do is review those potential solutions with <img> tag and make a decision whether we can solve those, live with those being present or they are a real blockers for <img> solution. If we come to an agreement that <img> can not be used for the responsive images, I am fine with having two different tags of course. But then, even if it still bugs me, we will have good arguments to explain our decision.
  
  - Matt Wilcox ✪ February 21, 2012 at 6:17 pm
    
    What about re-purposing <picture> and calling it <media> ? I’d quite like to have seen a blanket element like this that could handle images, video, audio, and anything else media related. All of which may end up wanting to be responsive and all of which may want different source.
    
    Is it too late to does something like that with a flag of media-type to indicate whether it’s an audio, video, etc?
    
    IMG is a necessary fallback no matter what we end up with, be that a new tag or attributes on an img.
    
    - Mathew Marquis ✪ February 21, 2012 at 6:28 pm
      
      Hah, scrapping video and audio in favor of a combined element is a big ask—I think we’re better off sticking with solving the single problem, for now.
      
      I figure: the tighter our focus, the better our chances of this thing seeing the light of day, yeah?
      
      - Matt Wilcox ✪ February 22, 2012 at 8:40 am
        
        Yeah, that idea was never likely, but I figure we may as well air everything. I think if we could re-start on HTML there would be quite a bit different 🙂
        
        FWIW I think the current syntax is pretty much the best we’re going to be able to get in a realistic world. That doesn’t make it ideal conceptually though.
    - Denys Mishunov ✪ February 21, 2012 at 6:31 pm
      
      Hi Matt,
      
      I don’t really think that an “umbrella” tag like <media> is solving my concern of having two different tags for the same thing — image. But I do agree that it’s better than multiple tags for different media indeed — this way we can have unified and predictable behavior for all of them that is an advantage.
      
      I realize that IMG is a necessary fallback of course. But ideally, I would like to have one IMG tag that can be either responsive or not. It’s ONE tag because no matter whether an image is responsive or not, it is still an image and in most cases, it’s the same image but in different resolutions.
      
      - Mathew Marquis ✪ February 21, 2012 at 6:39 pm
        
        We spent a great deal of time looking for ways to work with/around the img tag (and its alias in many browsers, “image”)—unfortunately, it was established early on that it was nigh impossible to modify the behavior of img to “look ahead” for multiple sources, and that any modifications we made stood to break things in terms of backwards-compatibility. A lot of that history is documented at https://etherpad.mozilla.org/responsive-assets
      - Denys Mishunov ✪ February 21, 2012 at 6:48 pm
        
        Thanks Mathew. This is enough for me to ditch the hope of re-using IMG for the responsive images. Now, if possible, I would like us to formulate the grounded answer why a new PICTURE (or any other element we can come up with) element is better than the existing solutions. Please read my response to Nicolas.
  - Nicolas Gallagher ✪ February 21, 2012 at 6:18 pm
    
    Hi Denys, yeah someone proposing a modification of `img` is what triggered this etherpad https://etherpad.mozilla.org/responsive-assets.
    
    But changes to `img` parsing is not going to happen and wouldn’t be backwards compatible. I asked Hixie about it directly and he said as much.
    
    The `picture` idea would only require 1 new element to be forged, because `source` already exists for `video` and `audio`.
    
    - Denys Mishunov ✪ February 21, 2012 at 6:43 pm
      
      Hi Nicolas,
      
      Yeah, that’s the etherpad I was telling about. That’s good to know that changes to the IMG parsing have been discussed in the WG and there is a response. So to summarize this — there have been discussions in a wider circle about this. And re-purposing IMG is not an option due to compatibility issues and reasons mentioned in the etherpad. Fair enough. Now to the new potential solution being PICTURE.
      
      I wonder what makes PICTURE better comparing to, for example NOSCRIPT solutions like Head’s one or the one I proposed as an extension of it? Please don’t get me wrong, I am not opposed to PICTURE at all. But if we are about to introduce a new element I would like to know what are the reasons for that and what makes it better than the existing solutions. Is it just the similarity with VIDEO/AUDIO and contemporary markup (due to use of SOURCE)? I would appreciate if anybody could explain the benefits or just point me to any reference with such explanation.
      
      - Denys Mishunov ✪ February 21, 2012 at 8:05 pm
        
        Ok, gave it another thought (need to take brakes from computer from time to time :)).The reason why we want a new element is obvious, I suppose, — it would let us have adaptive images without javascript (in capable browsers that is). Scratch that silly question. Had to figure that out myself 🙂
Matt Wilcox ✪ February 24, 2012 at 5:12 pm

What are the implications of http://dvcs.w3.org/hg/dap/raw-file/tip/network-api/index.html with regard to our <picture> element?

Can we leverage the Network API through HTML, or CSS? We’re already stealing media-queries from CSS to get <picture> functional.

Satoru Takagi ✪ February 27, 2012 at 7:17 am

It was based on considerably different use-cases, but, as a result, I suggested a function considerably similar to Responsive Images.
I feel the need to examine relations of CSS media query and SVG.

Pingback: The slow elephant in the responsive images room | Bram.us
Kevin Suttle ✪ March 3, 2012 at 9:37 pm

Hey everyone,

I’m going to attempt to share some holistic, self-contradicting ideas I’ve been kicking about this problem that I haven’t seen suggested or mentioned yet (though there may be a reason for that).
Warning: entering the wavy-hand zone.

1. img is content, so having it in markup makes sense. However, the size of an img is presentational. By design, this is most likely a problem that CSS needs to solve, though I’m going to use this post to propose markup-based solutions. CSS solutions I’ll save for another post. Also, as much as I love JS, I have a very hard time thinking it needs to solve this problem. We’re better off implementing a solution natively in the long run.

2. I also don’t think adding picture to the spec is going to add much other than confusion. Having img and picture? The differences could only be minimal at best, and I sincerely doubt the tradeoff would be worth the mud it adds to the water. Unless there’s a very compelling and clear use case, I think we can do better.

3. If the img tag is locked down according to the powers that be, perhaps we can invent more semantic tags to handle this problem, while still keeping the img namespace.

Examples (100% off of the top of my head):

mobileimg
tabletimg
desktopimg
tvimg
hdimage

In these cases, the tags themselves have media query like range defaults in CSS. For example, defaults to 240px min up to 480px.
defaults to 481px up to 768px, etc. This still needs to take landscape orientations into account and other such niceties, but I’m just spit-balling here.

These can be overridden in CSS, just like any other set of properties, obviously.

4. I also think the foundation of whatever solution we end up proposing should establish the mantra of approaching images mobile-first.

5. Another possible approach could be to standardize a file naming convention like in iOS. i.e. ‘icon@2X.png’, but instead of focusing on pixel density, focusing on dimension. So, perhaps something like banner@mobile.png, banner@tablet.png, etc.

6. The topic of bandwidth keeps coming up, but addressing it is tricky at best. When considering data like “the majority of mobile usage happens at home”, the Rubik’s cube starts to grow sides.

I realize that the lines of what exactly mobile, tablets, and HD can be quite blurry, but this post is really to propose some ideas outside of the conversations I’ve seen. It’s entirely possible that we not use any specific device tag like mobile, and decide on something a little more familiar, like ‘fleximg’, with multiple source attribute/value pairs. We already have flexbox, and this is similar enough to keep the mental model in check, and express semantics at the same time.

Yeah, I like this better, forget all my other ideas. 🙂

Kevin Suttle ✪ March 4, 2012 at 8:23 pm

I also wonder if augmenting the scheme property of the meta tag could be helpful somehow.
http://www.w3.org/TR/REC-html40/struct/global.html#adef-scheme

We already use the meta tag for view options on mobile.
meta content=”width=device-width, target-densitydpi=160dpi, initial-scale=1.0″ name=”viewport”‘

Anselm Hannemann ✪ March 4, 2012 at 9:52 pm

Indeed you hit a valid point with no. 1 for 90 percent of the images. But I sometimes would like to change an image on a media-query. And this will be more and more the more devices we have. And then we have a different content which should not be changed in CSS but in HTML itself.
This is the major point for the new syntax in HTML. For all other presentational things you are totally right.

3.) Nope that isn’t a good idea because if a new device comes up we need another new tag. It should be done with media-attribute on a specified universal tag.

5.) Same problem as in 3.)

fleximg.) This might be an interesting approach but you’re mixing up HTML and CSS now. But it’s a good idea although.

6.) All I can say about this is “we don’t know and will never know how our user is viewing the site and under what circumstances”. If we always have this in mind we’ll end up with a pretty good solution I think.

- Kevin Suttle ✪ March 4, 2012 at 10:30 pm
  
  Thanks Anselm,
  
  Definitely valid arguments there. So, scratch the device-specific tags. About ‘fleximg’, my purpose here is two-fold. 1) ‘picture’ is too generic and too close to ‘img’ and trying to define it will reinforce that. It also adds no real semantics in regards to the fact that the image will somehow be responsive or have multiple sources. 2) fleximg solves this and also creates a single, new, unified tag to work with and adheres to a standard that was set with ‘flexbox in CSS. This same namespace could also add ‘flexvideo’ and ‘flexaudio’ in the future. How does everyone feel about this? I can’t be the only one thinking ‘picture’ is not the solution.
  
  - Anselm Hannemann ✪ March 5, 2012 at 6:51 am
    
    No, you aren’t the only one. I agree with your last arguments. I certainly like the fleximg idea.
    
Kevin Suttle ✪ March 13, 2012 at 5:16 pm

So, Apple is updating their site to be compatible with the new iPad’s Retina Display, employing a progressive download technique in JS.

http://www.appleinsider.com/articles/12/03/13/applecom_upgrading_to_high_resolution_images_ahead_of_retina_ipad_launch.html

https://gist.github.com/2029936

https://ssl.apple.com/v/ipad/a/scripts/image_replacer.js

Also see http://timkadlec.com/mq/test7.php for resolution tests.

Thanks to Jason Grigsby, Scott Jehl, Brad Frost, and Jim Newbury for the links.

Pingback: WTFWG | TimKadlec.com
Pingback: Definición de imágenes responsive como atributo de img o nuevo elemento picture? | Desarrollo Web Avanzado en Guatemala
Pingback: HTML5 responsive images spat explodes » b.c.s.
Pingback: » The real conflict behind <picture> and @srcset Cloud Four Blog
Pingback: Picture 元素的故事 – 陈三

Community & Business Groups

Where Things Stand

39 Responses to Where Things Stand

Leave a Reply Cancel reply

Archives

Categories

Footer Navigation

Navigation

Contact W3C

W3C Updates