An update on responsive images

Hey everyone,

I guess a lot of you guys aren’t signed up to the whatwg mailing list so I thought it’d be good to give an update on what’s going on. I don’t claim to be an authority, in fact I’m sure my knowledge is flawed; Also things are moving so fast that any information here might be plain wrong by the time you read it! (please leave a comment if you see an error)

How srcset really works (-ish!)
When an image tag reads <img src=”mobile.jpg” alt=”” srcset=”medium.jpg 600w, desktop.jpg 1000w”> what this means is: default to mobile.jpg, unless the viewport width is greater* than 600px then the browser is free to choose mobile.jpg or medium.jpg as it sees fit. Once the viewport width is greater than 1000px the browser is free to choose any of the three images.

For a much better explanation read Bruce Lawson’s great html5doctors post

Tab’s suggestion
On the mailing list Tab Atkins (& others) suggested the following CSS like notation might be more helpful:
<img src=”mobile.jpg” alt=”” srcset=”medium.jpg min-width:600px, desktop.jpg min-width:1000px”>
And the browser would be free to choose any image where it’s corresponding media query is satisfied (this is different to <picture> where the browser gets no choice). Also only a subset of media queries would be allowed (min-width, max-width and maybe min-pixel-density). This is more friendly to developers that know CSS, it’s more readable and allows for units other that px to be used. On the downside it’s slightly more verbose.

Going forward
I’d suggest you all join up to the mailing list, but if you don’t want to receive dozens of emails a day, feel free to leave comments/thoughts below.

* there is discussion whether “600w” should reference the maximum viewport or minimum viewport, however for the sake of brevity I’m going to just to talk about minimum.

srcset added to WHATWG spec

Despite significant opposition from developers, the srcset attribute has been added to the WHATWG’s HTML spec by Ian Hickson. He has written a long email to the mailing list commenting on many of the issues and explaining his actions.

As the news broke and the uproar began on Twitter, the W3C replied, saying they are “looking at this” and to “stay tuned”.

Expect more developments soon.

More on the srcset proposal

After a lot of reading over the weekend and a fair bit of discussion on the WHATWG mailing list,  I’ve come to a couple of conclusions:

1. Initially, I totally misunderstood how <img srcset> works
2. I still don’t want it, and still do want <picture>

The appeal of <picture> is that you can select different image sources based on any media query, giving you fine control over which version is served under various conditions. Currently this includes viewport width/height, device orientation, pixel density, colour capability etc, and as more media queries arrive (surely there will be a connection speed query sooner or later, and I’d love to see containing element width/height as well) the element becomes even more useful and flexible. Initially, I thought <img srcset> was an attempt to avoid creating a new element by shoehorning this functionality into a new attribute.

In fact, <img srcset> comes from a radically different angle. For each version of the image, we add an entry to the srcset attribute with the filename, and optionally the width, height and resolution (expressed as “1x”, “2x” etc). The user agent then decides which version to serve, based on the metadata we’ve given it and the current conditions on the client (viewport, connection speed, pixel density, user settings, etc). Unlike <picture>, authors aren’t specifying which source to use and when; rather, we’re just telling the user agent what’s available and letting it make the call on its own. <img srcset> allows authors to provide lots of different versions of an image, but affords us no control over when they are used.

Update: I had this wrong, and so did quite a few other people. There’s still some ambiguity, with the spec contradicting itself in places. Jeremy Keith has done an excellent job of clearing up what the parameters in srcset actually do in his post today.

When we practise responsive design, we are trying to give the best possible experience to our users by adapting our websites to work with whatever they are using to access them. <picture>, by design, gives us the control to do this with content images. <img srcset>, by design, takes it away.

Of course, <picture> is imperfect (duplication of alt attributes is annoying, as is the potential duplication of media queries on some pages), but in my view it’s the best path to continue down.

An alternative proposition to picture and srcset, with wider scope

Managing responsive designs is hard, so let’s use our <head>

Here is what I believe is the best proposition yet for managing our responsive designs. This includes managing our CSS, JavaScript, and inline <img>’s.

<head>
  <meta name='case' data='breakpoint1' media='min-width:350px' />
  <meta name='case' data='breakpoint2' media='min-width:1000px' />
</head>

Deconstructing the case for breakpoint management in <head>

To re-cap our current problems with breakpoint management; we write our Media Query’s dozens of times in CSS files, we write the same tests in our JS files, and we will soon write them inside every <picture> element too. The same tests, in every case, but with a different “do something if true” instruction. That’s wasteful in terms of repetition, becomes very hard to manage, requires continuous re-processing of a test that never changes, and is future unfriendly. Read more detail here.

So why does that code up there solve all these issues?

At first glance, that code is going to cause many people a few frowns. We are not used to seeing this sort of thing – meta elements are supposed to describe a property of the URI aren’t they? Well, no. We already set meta elements that do not describe a property of the loaded URI – all of us do. Every time we set <meta name=”robots” content=’index,follow’ /> or any associated instruction. Or when we tell Google Translate to not run on the page. Or when we tell IE not to use its picture menus. These are all examples of using a meta tag to control the behaviour of some software that’s consuming the URI’s HTML, and not to describe properties of the page itself.

What the code above does is set ‘case’ to equal a particular value, when a media query expression is matched. And if that’s done in the HTML <head> we know that absolutely everything else can rely on it – the <head> is the very first thing parsed by a browser, even before anything inside <body>. Anyone familiar with working on the ‘responsive images’ problem should at this point be getting very excited. It means that the browser can be aware of the need to adapt before any pre-fetch behaviours have been triggered by finding an <img /> element in the mark-up. That is a major advantage and is how this solution solves the problem of image pre-fetch.

Some people will be uncomfortable with having the queries in the head, but I’d say putting the queries into the CSS is, most of the time, backward – it’s just that we’re used to doing it that way so it makes a certain kind of sense. Remember that the order we actually build a site goes like this:

1) Consider a device size
2) Create a design for that size on paper or in Photoshop
3) Write the code to implement it
4) Increase device size and goto 1

All of our code is actually hanging off the same media query breakpoints, because they’re the design breakpoints. So why re-define the same tests in each technology rather than do it once in HTML and have the CSS and JS just reference them? We could still do specific tasks in the “old” way, if desired.

How would we use these <meta> elements?

In HTML

Addressing first how we might use it to control <img/> in our mark-up:

<body>
  <img src='/content/images/{case}/photo.jpg' alt='' />
</body>

What has this bought us?

1. Well, we have a single image element with no custom properties or other code, which will adapt to any number of breakpoints you want to declair, without ever adding another character to the image code.
2. We have something that is backward compatible right now: either create a directory called “[case]” so that a file exists there, or if you want to be cleverer about it, set your server to alias [case]’s path to an already existing path.
3. The image is cachable by proxies and CDNs because the source for each version of the image has a unique URI.
4. UPDATE: By using {} instead of [] this solution is actually complying with the URI Template too (See the RFC) – thanks Brett!

Are there any drawbacks?

None that I can see. Even with the current proposal, we should never change the alt attribute because all resources are supposed to have the same semantic meaning.

In CSS

Rather than writing endless clauses like this, where a test must be performed each time:

@media only screen and (min-width: 700px) { ... }

We could write this instead:

@media (case: breakpoint1) { ... }

What has this bought us?

1. We are abstracting the test itself away from the block of code that defines a response. This means if we decide to change the breakpoint condition, we do it once in the <head> and everything else works automatically.
2. We reduce the number of tests required of the browser. It’s not re-running a test, it’s instead checking the value of case – which has already been computed.
3. We have smaller code. Yeah, i know, it’s a tiny point but it’s a benefit non the less.

Are there any drawbacks?

Not that I can see, because you can still carry on using existing syntax if you like, this is just an additional capability we may want to take advantage of. It can also be used in all the same ways you use any existing media-query.

In the case of JavaScript

It’s largely the same as for CSS. Instead of doing its own tests, it can just use whatever value is stored in the ‘case’ node. There is prior art for this: the behaviour here would be exactly the same as the wonderful shiv that a lot of people are now using Conditional CSS, but it’d check against the meta element and not the body:before pseudo element.

So why might this not work?

I’ve only analysed this from the perspective of an author. I do not know how much effort it might be for a browser vendor to actually implement, and whether they would be willing to do so. For me, this is the only potential downside to the whole idea – it needs the vendors to implement it. In terms of functionality and benefits of the approach, i can see no downsides to the idea itself.

I think vendors may object initially to the idea of browsers having to do this:

1) detect the existence of a specific meta tag

2) if it exists scan all src=”” properties for the variable string to replace with the variable value

If no meta tag was there, it wouldn’t have to do anything with the src strings.

How likely are variables?

There is a history of our core technologies not supporting Variables, but there is a trend of adding them recently:

1) We now have variables in CSS ( http://dev.w3.org/csswg/css-variables/ )
2) We now have variables in URIs ( http://code.google.com/p/uri-templates/ )
3) We’ve always had variables in JS

So, HTML’s the last one standing without support for variables.

———–

N.b., This post is a re-post from a blog entry I wrote, it was suggested on Twitter that the Community Group should see it. Source: http://mattwilcox.net/archive/entry/id/1091/

The suggestions here are a mix from an older suggestion of mine which worked with <picture>, and the ideas of Denis LeBlanc

———–

UPDATE: Philip Ingrey has written a polyfil for this soultion!

Responsive image element proposed by the WHATWG

After months of discussing possible solutions for responsive images on the WHATWG mailing list, I discovered yesterday that another, different syntax was proposed on the WHATWG IRC channel. Additionally, when I joined the discussion, it became apparent that the majority of the WHATWG leads present in IRC weren’t aware of the proposed picture element—or, in fact, this Community Group. Thankfully, after being informed about our efforts, I was assured they would attempt to incorporate our input.

The majority of the people involved in the IRC discussion had not heard of our Community Group, and —though none of us were contacted for input when the time came to propose solutions.

So we’re all in the loop here, the currently proposed solution is a new set attribute on the img element. If that link’s not working for you, the proposed syntax is:

<img src="face-600-200@1.jpg" alt="" set="face-600-200@1.jpg 600w 200h 1x, face-600-200@2.jpg 600w 200h 2x, face-icon.png 200w 200h">

To recap, the markup pattern proposed by this community group is as follows:

<picture alt=""> <source src="mobile.jpg" /> <source src="large.jpg" media="min-width: 600px" /> <source src="large_1.5x-res.jpg" media="min-width: 600px, min-device-pixel-ratio: 1.5" /> <img src="mobile.jpg" /> </picture>

I noted a number of concerns with the new, WHATWG-proposed img syntax in IRC, namely that it seems far less developer-friendly than the proposed picture markup structure.

But now? Now we need to hear from you.

The WHATWG, having now realized there’s been significant work being done in this group, wants to hear what you think. I’ve been asked to provide citations of developer opinion on the topic, so I’d like to ask you to sound off in the comments:

Does picture seem like something you’d like to write, or does the img set attribute seem more accessible to the average developer?

Additionally, if you have more thoughts on the topic you’d like to share with the WHATWG, here’s how to continue the discussion:

• Join the WHATWG mailing list, and help keep picture at the forefront of the discussion.
• Join the #WHATWG channel on IRC, since apparently this is where the WHATWG leads’ attention is focused.
• Talk to the WHATWG on Twitter.

If you have a clear argument for or against the proposed syntax here, for or against the picture element, or a decided preference between the two: it seems like now is our opportunity to have our voices heard.

If you care about responsive images, I’d love for you to join in.

Syntax issue with displaying high-resolution images in the picture element

EDIT: We can all agree that the picture element can become very verbose. Especially if you are thinking about supporting a wide range of devices with multiple screen sizes/resoltions.

I’ve put together an example of some markup that would help allievate this issue based on image-set() CSS and URI Temapltes.

View Example Mark-Up – https://gist.github.com/2572749

Big thanks to Adam Bradley for his feedback on this and helping me understand why this makes so much sense as an alternative way to author picture elements which makes them easier to maintain and keeps our markup DRY.

Also, Philip Ingrey reminded me that these can all be in one media query and that each vendor prefixed device pixel ratio doesn’t need to be in an individual media query. I must have had a mind blank when I originally wrote this. I’ve updated the markup to show his example which I think is a perfect solution to the “issue” I was describing. I’ve also moved some of the code examples to github.

 <picture alt="Alt tag describing the image represented">
     <source src="photo.jpg"/>
     <source src="photo@2x.jpg" media="-webkit-min-device-pixel-ratio:2,-moz-min-device-pixel-ratio:2,-o-min-device-pixel-ratio: 2/1,min-device-pixel-ratio:2"/>
     <img src="photo.jpg" />
 </picture>

As more and more high-resolution devices come out, from smart phones, to tablets, to laptops and desktops, I think we will start to need 2 images for each breakpoint.  Where originally the picture element might have been used like so: https://gist.github.com/2509519

I feel like a more realistic example of the picture element being used in the future will be: https://gist.github.com/2509534

I’m glad Philip reminded me of my mistake, because the markup I had imagined to handle all the vendor prefixes to do this was crazy.

On a side note, I’d love to see the user be able to choose if they want high-resolution graphics in their browser, just like they can choose their default font-size. Maybe an option where you can pick SD graphics and HD graphics. If they choose SD, it would say their device had a pixel ratio of 1 rather than 2 even if they were on a retina device.

Summary of use cases and requirements

This post is from the public mailing list where Kornel Lesiński has summarized this.

DPI/optimization problems:

1. Regular DPI images look bad on Retina iPad.
2. Regular images look bad on Retina iPhone if zoomed in, and on desktop screens too when large zoom factor is used.
3. Regular images sometimes look bad when printed.
4. High-DPI icons/line-art images scaled down are blurry.
5. High-quality images take long time to load, especially on slow connections.
6. High-quality images may be expensive to load on mobile connections.
7. Image galleries with lots of large images (e.g. worth1000 threads, Tumblrs) strain phones’ memory (cause other tabs to be “flushed” or don’t load completely).

Layout problems:

1. Images designed/cropped for wide screens are poor fit on portrait screens and vice versa.
2. Images designed for large screen lose too much detail when scaled down to physically smaller screen (a differently cropped image could be more readable, e.g. portrait instead of a group shot).

Requirements:

1. No wasted bandwidth.
2. No extra HTTP requests.
3. No complex or overly verbose markup.
4. Hard to get wrong (should survive mindless copy&pasting from w3schools’ misinterpretation of the feature).
5. Should not deteriorate badly over time, even when misused (e.g. like User-Agent strings, which poorly sniffed caused browsers to lie, which caused more sniffing and lying).
6. Needs to work with new/unusual types of networks and screens, and should not get in the way of browsers innovating in this area.
7. Easy to deploy on existing infrastructure (shared hosting, CDNs).

Nice to haves:

• Easy to get right (non-geeks who “just want their blog to look nice” should be able to figure it out).
• Allow users (user-agents on their behalf) to influence choice of quality/bandwidth trade-off (e.g. an impatient person with poor vision may not give a damn about Retina images. An RSS reader preloading feed in background or HTML5 offline cache may want high-res even on slow network).
• Allow authors to say when quality is most important.
• Be familiar to users of current solutions, follow existing practices where possible.
• Achieved quality isn’t worse than the best UA/device can provide.

Jad Sarout answered:

The only way to cater for all these (sorry, I am just chiming in, never participated before so if I derogate to rules or etiquette, please tell me), is to have the browser handle most of it. Also, please bear with the awkward English, I’ll try to be as clear as possible.

I would imagine something like the automated call for favicon. So the browser decides what to do (try to load favicon from the root) but we can influence the behavior through meta-tags and css.

Let’s say that for a file (lowest possible resolution) “image.jpeg”, it would be expected that the browser could ask for “image.jpeg.medium”, “image.jpeg.large”, “image.jpeg.landscape” and so on (mind you, these names do not match to the user’s resolution, but standards, so an iphone browser could very well request the “image.large.jpeg” if its browser was set on “high bandwidth”. This said, they should probably be numbers so we can add to them later).

In order to avoid unnecessary requests, the browser would attempt the loading only if a specific meta-tag is present.

We would have to invent a new sort of CSS rule, that would allow:

1. To disable these calls on certain images (let them be disallowed by default, and allow them for certain classes only)
   img.oneResolution{
        responsive:none;
   }
2. To specify how the rules work (it would serve to rename the defaults; or maybe you want to serve the same image “$imageName.big.jpg” for a certain number of requests)
   .img{
      responsive: 320(attr(src)’.small’), 420(attr(src)’.small’),640(attr(src)’.big’), 960(attr(src)’.big’), none// next resolutions get the last highest one.
   }
3. To refine behavior, for example:
   @media only screen and (max-device-width: 1200px) and low-bandwidth{
       img{responsive: attr(src).medium}
   }
   @media only screen and (max-device-width: 1200px) and high-bandwidth{
       img{responsive: attr(src).large}
   }

Benefits:

• On the server side, this could be handled in a variety of ways: automatic image generation and caching, user uploading several versions, etc…It leaves full flexibility on that point; If the feature is partially implemented, one could do a regex on server level to redirect calls to “image.jpeg.big” to “image.jpeg.medium”.
• On the browser, it is also very flexible and it is up to each vendor to decide how to deal with this, open options to the user, decide smartly what to do, or use plugins (for example, firefox might try to do this intelligently by default, but plugins would expose a slider that lets the user decide how to handle this). Also, full backward compatibility with no added markup. The img tag stays untouched.
• Full separation of concerns: The content is specified in the markup, refinement if presentation CSS.
• Very easy to implement and noob-ready: if you are not aware of the feature, just upload your biggest image and let your CMS handle the requests. Sort of complex for the CMSes, but a lot of similar solutions are already in use since years and there is almost nothing to change, just modify the regex to stick with the spec. All important (and less important) CMSes have that sort of feature.
• It is even easy to implement this on server-level. No additional http requests:
• browser-responsive-aware and server-not-responsive-aware:
• meta-tag not present, browser behaves as default
• browser-non-aware and server-aware: no requests, server serves
• the default image specified in the img tag
• browser-aware and server-aware: the browser only requests the
• correct image, and not the one in the markup.
• The only case in which there are wasted requests is if the server specifies it is responsive-aware, but the website is not. In which case the browser will request images that don’t exist. This could be a problem, but it can be solved in a variety of ways: specify “responsive:none” in CSS, include a meta-tag “not-responsive”…
• Easy to pollyfill

Drawbacks:

• Much larger space needed per site (but that’s the case for all solutions)
• Difficult to imagine the CSS “responsive” rule in a way that it reads easily and allows for all use-cases (and future ones), as well as deciding on the default naming of the images.
• Solely Dependant on browser adoption, and it is not an easy task for them (the quality of the algorithm that decides which image to load is a cornerstone, if badly implemented, nothing will work).

Am I way out of my league or is what I am saying making any sense?

Mimicking the noscript functionality: A new proposal

I wanted to float another idea which I’ve been toying with to hear what other people think and see if its a viable tool.

When we created our noscript solution we discovered that the image inside wasn’t being added to the DOM and therefore the src attribute wasn’t making a request to the server. So we were pleased we’d found a work around but it’s a pretty dirty hack and I’m not that keen on it even though it does the job.

What I’d really like is another way of adding content to the HTML which isn’t loaded into the DOM until either my CSS or JS has decided what it wants to do with it. I’m also wondering if this is a way I can get more info into the HTML page, without having to add weight to the HTTP header and if this idea could be extended beyond just responsive images, into other parts of responsive asset management and build (i.e. fonts, media, video etc. )

I was thinking of submitting a proposed spec for something like a defaults tag, or possibly  a line in the meta data to tell modern browsers to omit certain DOM requests, like img src until the client has decided which src request it wants to make. That way old browsers can still load the image src as normal, we don’t have to do dirty things like hiding stuff in the noscript and modern browsers can hijack the img tag to prevent the page making the img src request until its ready.

Would love to hear other peoples opinions on this technique and if anyone has any suggestions on ways to improve on it.

Move on!

As of today we now seem do have discussed nearly every different angle of responsive images with its problems.

We recently figured out that

• We don’t want to provide a general responsive element.
  We only want to provide a responsive image-element (no other media)
  Read here why
• We already have a working polyfill for our proposed solution by Scott Jehl.
  Read here
• There is a problem about bandwidth. We need to figure out how we can implement bandwidth queries into our picture-element. This needs to be solved now. Try to solve it!
• We saw several attempts of responsive image solutions. All are more a polyfill than a real solution. This is not what we want to have for production later.
• Yet another responsive image system
• Responsive Content Images using a spacer PNG and background-image
• The new elephant: Image Quality and Retina Displays
• We now have a CSS variant of responsive images with image-set(). This is great but not our goal. So please use this later for styling purposes but not for the real HTML-content. Read more
• We need a way to centralize media-queries. Read more

Then we have the second important point we recently figured out: A responsive file-format. This is what people think would be a great approach to work with responsive images. Unfortunately this is not easy and it is not our primary goal that we care about this. Nevertheless I would like to see some people pinging the right people (e.g. WebP-team) so they might be aware of the problem.

So we have all this solutions and thoughts written and discussed. When we have discussed and solved the outstanding issues, we need to start writing a specification for our picture-element I think.
If I have not missed a very important point here, let’s do this.

And don’t hesitate to maintain our Wiki please. This is the place where all our summarizing thoughts should be – not here in this blog-system.