Finally we’re able to provide an answer to the many rumors which have been going around in the past few months: Google Chrome is now available for Android devices running Ice Cream Sandwich. On my Galaxy Nexus, the browser scores 343 points on html5test.com, runs the SunSpider test in 1880ms and scores 1308 points on the v8 test.

Support for the Web Platform
The first beta uses the same basis as Google Chrome 16, meaning it has most Chromium and WebKit changes up to mid October 2011. I’ve already iterated through many of the changes which were included in that release, most of which will apply for the Android version as well.

To name some highlights, Chrome for Android supports pretty much all of the Web Platform’s exciting features, including CSS 3D Transforms, GPU accelerated canvas, CSS Animations, SVG, WebSockets (including binary messages!) and Dedicated Workers. It supports IndexedDB, Application Cache and the File APIs, date and time pickers, parts of the Media Capture API and mobile oriented features such as Device Orientation and Geolocation. Fixed positioned elements are also working and even look smooth while scrolling!

What the Android-based Chrome does not support is Flash, following Adobe’s announcement in November. Other unsupported features in the beta release are Extensions/Apps, WebGL, Shared Web Workers and the Web Audio API.

An issue that often pops up for mobile browsers is that text on the website may be too small to read properly. Where the Android Browser employs a text reflow algorithm to clarify the situation, Chrome for Android features a technique which we’ve called Font Boosting. It uses an algorithm to increase font sizes when necessary, aiming to make the text readable regardless of the zoom level.

And for the Web Developers among you, it’s also possible to utilize Web Inspector’s Remote Debugging feature to inspect and modify pages while viewing it on the device itself.

Google Chrome 16 and mobile limitations
Of course, bringing a browser to a different -much more limited- platform goes further than simply re-using code. Mobile devices have a lot of limitations compared to desktop and laptop machines. Besides the lower amount of available memory and CPU power, other constraints lie in less memory bandwidth and VRAM on the device’s GPU. Google Chrome has a complicated architecture which imposed some interesting challenges here: separating the browser from the renderers through its multiple process architecture, to name an example. Decreased rendering and scrolling performance were also an issue, to which Chrome’s GPU team provided an excellent answer in the form of a threaded compositor.

Another characteristic of the Android platform is that the APIs for most significant features are exposed through the Android SDK, making them available to Java code. To this end, part of the browser layer has actually been implemented in Java, communicating with the rest of the Chromium and WebKit code through Java Native Bindings.

Becoming part of the Chromium project: upstreaming our code
Chrome for Android has been developed in a separate repository as a fork, which means that most of the code will have to be upstreamed. To date, about 150 commits have landed in Chromium making many preparations, including the build environment and bots for Chromium and WebKit, not to mention WebKit’s umbrella bug and the two announcements.

As having a fork is far from ideal, one of the current top priorities will be to upstream most new and modified code to Chromium and WebKit, while the build and testing infrastructure on Chromium’s waterfalls is expected to evolve significantly. A snapshot of the current source-code of Chrome for Android can be downloaded as a tarball.

Of course, keep in mind that today’s release is just a beta; much more work, features and stability will be needed to actually be able to release the first stable version. Personally, I’m very glad that the project has been announced, and definitely am looking forward to continuing development in the open!

True story: I was tasked by the lovely Mrs Lawson to buy some curtains that match our carpet during the January sales. I dutifully did so — and had to return to the shop straight away because they didn’t match at all. Mrs Lawson accompanied me, and with a withering glance at her incompetent mate, immediately found some correctly hued fabric, and all was well.

But what’s a middle-aged colour-blind bloke to do? I had early in the curtain procurement process decided against cutting a hole in the carpet in order that I may take a sample with me. (All other mistakes aside, this was a correct decision.)

So, in order to ensure that I would never again repeat the mistake, I set out to make an app that would allow me to capture the colour of an image straight from my camera. Of course, it had to be a web app rather than a native app, because we’re web angels, not proprietary devils.

getUserMedia #

getUserMedia is an API that gives a web page access to a user’s camera and microphone via JavaScript. It’s supported in Opera Labs (latest Android build, latest desktop builds) and WebKit in Chrome Canary builds (instructions).

Like many other APIs, it’s not part of the “real” HTML5 spec. It started life as the HTML5 <device> element, then got moved into the W3C as part of the webRTC specifications. But let’s not taxonomise when we could be having fun.

The first thing we need to do when using super-cool new UIs is detect whether it’s supported:

 if (navigator.getUserMedia) {
  // do something cool
} else {
  // fallback code
}

For this article, our “do something cool” will be to grab the dominant colour of the current input stream from the camera. As a fallback, we have several options.

One option, if we’re using the default Android browser, is to show a form that automatically starts the camera when focussed:

<form enctype="multipart/form-data" method="post">
  <input type="file" accept="video/*;capture=camera" />
</form>

There’s also the capture attribute, a proposed extension from the W3C Device API Media Capture API that tells a user agent where to get the input data. Android extends the accept attribute. See more in David Calhoun’s Device API article. Alternative capture arguments are camcorder (for video), microphone, and filesystem.

Browsers that don’t understand the still-draft Media Capture API will simply ignore the capture attrbute and prompt the user to browse to a file on their file system for uploading.

Users on iOS are out of luck, as their devices don’t allow access to the file system.

Differences Between the Media Capture API and getUserMedia #

Robin Berjon, co-chair of the Device API (DAP) Working Group explains: The markup-only Media Capture API is a simpler, low-hanging-fruit version that can work easily for a lot of use cases. It also has a very simple security model (just a file upload).

In fact, it works exactly like <input type=file>, except that it knows to allow the user to grab a pic from the camera by default instead of hitting the filesystem. But just like with <input type=file>, once a picture has been taken (i.e., a file selected), you can access it and read from it using the File API, which means you can stuff it in an <img> or a <canvas> and do nasty things to it without annoying any faraway server.

But of course, if it’s part of a form and you submit it, it gets submitted just like a regular file upload.

The thing is, though, it’s always a snapshot. Even if you use it to take a video (which you can), the page only gets it after you’ve finished shooting. With getUserMedia (gUM), you get the live stream, which you can modify, record, or stream elsewhere. The gUM approach is more powerful but more complex, and it has more severe security issues.

getUserMedia API #

Let’s assume (because we’re optimists) that your user has a getUserMedia-enabled device. The API is nice and simple, as all Web APIs should be.

navigator.getUserMedia accepts two required arguments and an optional third.

The first argument tells the device which media you require access to, and it’s passed as a JavaScript object. So, if you only require access to the microphone, the first argument would be {audio: true}; for video-chatting, you would use {audio: true, video: true}.

The device decides which camera to use: User agents are encouraged to default to using the user’s primary or system default camera and/or microphone (as appropriate) to generate the media stream.

A previous version of the specification allowed hints to user agents about which camera to use. The API could specify “user” (the front camera on a phone) or “environment” (the rear camera on a phone). Debate continues about whether to reinstate this (read the ongoing conversation). One argument against is that letting sites know what cameras a device has could help Dr Malice of Evil Corp. fingerprint users after luring them to his site (which I find to be overly-cautious). Harald Alvestrand poses a conundrum: conferencing units may have a room camera, a document camera and a lectern camera, neither of which is attached to the physical box; which one of these is front?

But let’s not worry at the moment. Devices with more than one camera generally have a mechanism that allow the user to choose which one is used, so this gives the user control. Note also that the spec says User agents may allow users to use any media source, including pre-recorded media files.

The second argument is the success callback, the function to be executed on success, assuming that the user allows access and the device supports your request.

There is an optional third argument. This is the failure callback, the function to be executed if something went wrong. It’s optional, but only optional in the sense that washing your hands before you eat is optional: if you don’t do it, you leave yourself open to all sorts of bugs and your mum will shout at you.

Of course, it’s important to do feature detection before making an API call to ensure that the browser and device are capable, but even if feature detection succeeds, there is still much that can cause failure. For example, the user could deny permission or turn off the camera via a hardware switch, or the device itself could disable the camera.

So here’s how we recommend you use the API:

navigator.getUserMedia({audio: true, video: true}, success, error);

function success(stream) { 
  // ... use 'stream' ... 
} 

function error(){ 
  // display fallback content 
}

To check whether it’s working correctly, we can attach the output of the camera’s stream to the input of an HTML5 video on the page, like this:

<video autoplay></video>
<script>
if (navigator.getUserMedia) {
  navigator.getUserMedia({audio: true, video: true}, success, error);
  function success(stream)
  { 
    // ... use 'stream' ...
    var video = document.getElementsByTagName('video')[0];
    video.src = stream; 
  }
  // ...
</script>

Differences Between Webkit and Opera #

The two rendering engines that implement getUserMedia do so in two different ways. Hopefully, they’ll harmonise when the features reach the released browsers.

The differences:

• WebKit uses a prefix — navigator.webkitGetUserMedia — while Opera doesn’t.
• WebKit implements the options according to an old version of the spec, in which audio and video were passed as strings:

  navigator.webkitGetUserMedia("video, audio", /* ... */)

  whereas Opera implements the most recent version of the specification which uses JavaScript objects:

  navigator.getUserMedia({video: true, audio: true}, /* ... */)

• WebKit attaches the resulting stream like this:

  video.src = window.webkitURL.createObjectURL(stream);

  which is according to the current version of the spec. Opera uses:

  video.src = stream;

  and has proposed simplifying the spec (but discussion continues).

But this fragmentation, even though it’s only in experimental browser releases is, like, a total drag, man.

Introducing The gUM Shield #

Fortunately, we’ve got your back. Two of the finest minds I know — actually, two of the finest Mikes, Doctor Mike Robinson and Opera’s Mike Taylor — have created a snippet of script that you can copy to shield you from these annoying differences. We’ve named this getUserMedia syntax normalisation script snippet The gUM Shield.

It assumes that there’s a variable called video to which you’ve assigned a reference to the <video> element you’ll stream to. Do this with some mechanism like:

video = document.getElementById('video');

Simply hit the ‘Hub, download it, plug it into your script, and check it out!

Note that it’s experimental. Who knows what’ll happen when Mozilla, Microsoft, and Apple implement it, but please, have a play. Fork it and improve it!

Doing Exciting Things with the Video Stream #

So far, we’re only echoing the video stream. But, as you’ll know from Tab Atkins’ guest article video + canvas = magic, if we copy the video into a canvas, we can manipulate it:

var video = document.querySelector('video');
var canvas = document.querySelector('canvas');
var ctx = canvas.getContext('2d');

ctx.drawImage(video, 0, 0, video.width, video.height,
  0, 0, canvas.width, canvas.height);

A common mistake is assuming that this somehow establishes a “live connection” between the video stream and the canvas. It doesn’t. It only grabs the current frame. So you need to grab the current frame repeatedly in order to replicate a video. The magical number is 15 times a second (or every 67 milliseconds), but a slower frame rate would probably be okay for this demo:

setInterval(function () { 
  ctx.drawImage(video, 0, 0, video.width, video.height,
  0, 0, canvas.width, canvas.height);
}, 1000 / 67);

See the working demo.

What I want to do now is to grab the dominant colour from the canvas. I’m stealing a demo by Shwetank Dixit (who borrowed from Lokesh Dhakar‘s Color Thief demo). We won’t look at how it determines the dominant colour (because that’s nothing to do with getUserMedia).

Try this in Opera Labs and Chrome Canary!

Unfortunately, this works splendidly in Opera but it occasionally fails in Chrome, and we’re not sure why. (We’ve emailed the Chrome team to ask them.) Apart from the rather anti-climactic end to an article, you shouldn’t be dismayed by this. getUserMedia is only available in experimental builds that have plenty of work to be done before they hit production.

Back to our demo. Once you’ve got the dominant colour, you could prompt the user for a name for that colour (“carpet”, for example) and store that in local storage using its name as a key. Then, when you find a pair of curtains in the shop, you can find their dominant colour and easily compare it with that stored against “carpet” from earlier.

And you can return home, confident of delighting the love of your life with your chromatically appropriate choice of soft furnishings. That’s HTML5: enriching marriages since 2004.

Harshing Your Mellow #

Things need refining before web apps with getUserMedia are on a par with native applications. At the moment, there is no way of controlling which camera is to be used (and no agreement that a developer should be able to). The camera’s flash mechanism isn’t programmatically controllable, which could harm the user experience. Most importantly are the privacy implications of web pages being able to access your camera. Opera has an experimental privacy UI, but Chrome has no UI yet.

More Demos #

See Paul Neave’s HTML5 Webcam Toy and the links at the foot of my Opera Labs article.

Thanks #

Thank for help and code are due to Robin Berjon, Shwetank Dixit, Daniel Davis, Lokesh Dhakar, Mike Taylor, Mike Robinson, and Rich Tibbett.

It’s Curtains for Marital Strife Thanks to getUserMedia originally appeared on HTML5 Doctor on February 7, 2012.