IEBlogAnnouncing Dolby Audio for high performance audio in Microsoft Edge

Web browser media experiences have advanced significantly in recent years, from the first releases of HTML5 media, through the subsequent additions of Media Source Extensions for adaptive streaming and Encrypted Media Extensions for content protection.  And formats have been similarly advanced by the development of MPEG DASH and Common Encryption (CENC).  These important specifications offered the potential for interoperable, plug-in free web video.  We referred to the resulting experience as Professional Quality Video when we first shipped support for these specs in Internet Explorer 11 on Windows 8.1.

Advancement in media experiences continues in Windows 10 and Microsoft Edge. Today, we are announcing that Microsoft Edge includes HTML5, MSE and EME support for a new audio format:  Dolby Audio™.  It allows websites to match the compelling visuals of H.264 video with equally compelling multi-channel audio.  It works well with AVC/H.264 video and also with our previously announced HLS and MPEG DASH Type 1 streaming features, which both support integrated playback of an HLS or DASH manifest.

Dolby Audio logo

High Performance Media Experiences

We added Dolby Audio to Microsoft Edge so that web apps and sites can deliver the richest and most compelling media experiences with Windows 10.  In particular, we wanted Microsoft Edge to support the most demanding requirements from media websites.  Supporting Dolby Audio fits with this goal.

Dolby Audio

Dolby Audio supports the multi-channel Dolby Digital Plus™ audio format that provides crisp, clear audio in a broad range of channel configurations.  Microsoft Edge is the first browser to include support for Dolby Digital Plus.  Websites will now be able to selectively support Dolby Audio as a premium sound experience, and use HTML5, MSE and EME capability checks to fall back to other supported formats on browsers other than Microsoft Edge.

Windows 10 includes support for the Dolby Digital Plus Portable Mode, which is a dynamic range control setting that is optimized for playback over built-in speakers and headphones. Dolby content will play back louder and clearer with a dynamic range more appropriate for portable devices. This improves the experience from Windows 8 where content could occasionally render softly over built-in speakers and headphones, and the advantages will apply equally to websites running in Microsoft Edge.

Dolby Audio is compatible with current DASH (MP4) and HLS (M2TS) file formats, and so can be included with progressive download HTML5 content, adaptive streaming HLS or MSE/EME MP4 content.

Usage Guidance

There are two adaptive streaming approaches for websites to take advantage of these new formats:

  1. Use either DASH Type 1 streaming (where the DASH manifest (.mpd) or HLS manifest (.m3u8) is set as the src for an HTML 5 media element). Using our DASH Type 1 implementation, it’s important to use the DASH live profile with MPD Type=”static”.
  2. Build a custom adaptive streaming solution using MSE APIs directly. This is more work, but provides the most control over streaming details.

Both the DASH Type 1 and full custom MSE approaches may be combined with Encrypted Media Extensions apply DRM protection.  In Microsoft Edge, this would require supporting PlayReady DRM.

Capabilities

Dolby Audio offers superior performance, but isn’t currently supported by other browsers.  Websites that use it should feature detect on the format and be prepared to stream alternative audio formats on systems that don’t support Dolby Audio.  Javascript examples to check for format support are listed below.

For HTML5:

  • Dolby EC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = myvideo.canPlayType(‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = myvideo.canPlayType(‘video/mp4; codecs=”avc3”’);

For MSE:

  • Dolby EC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc3″‘);

For EME:

  • Dolby EC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc3″‘);

Dolby Audio Experience Demo

Dolby has created a Dolby Audio Experience demo using WebGL to provide an interactive environment where users can explore and play content on a variety of media devices (a media PC, desktop, laptop and tablet).   Each device demonstrates Dolby Audio with a different streaming video, either MPEG DASH H.264 or HLS.  The demo currently requires Microsoft Edge, since it is the first browser to include native support for Dolby Audio.

Conclusion

The addition of Dolby Audio to Microsoft Edge is intended to open new opportunities for website media experiences.  It is supported only in Microsoft Edge now, but fallback strategies are possible that allow sites to provide the highest quality media experience when supported.

We look forward to your feedback, and hope to see websites using Dolby Audio soon!

– Jerry Smith, Senior Program Manager, Microsoft Edge

Steve Faulkner et alCurrent Standards Work at W3C

People regularly ask me what I am working on at the W3C, here is a run down of standards/guidance documents I am editing/co-editing, contributing to (note: I am only 1 of the people at The Paciello Group involved directly in standards development at the W3C)

HTML 5.1

This specification defines the 5th major version, first minor revision of the core language of the World Wide Web: the Hypertext Markup Language (HTML). Editing updates to and maintenance of (mainly) accessibility related advice and requirements, with an emphasis on information for web developers.

ARIA in HTML

HTML5.1 specification module defining the web developer rules  for the use of ARIA attributes on HTML 5.1 elements. It also defines requirements for Conformance Checking tools. In HTML 5.1 this spec replaces the web developer (author) conformance requirements in section 3.2.7 WAI-ARIA of the HTML5 spec (titled 3.2.7 WAI-ARIA and HTML Accessibility API Mappings in HTML 5.1).

HTML Accessibility API Mappings 1.0

Defines how user agents map HTML 5.1 elements and attributes to platform accessibility application programming interfaces (APIs).  this spec replaces (and extends to all HTML elements/attributes) the user agent implementation requirements in section 3.2.7 WAI-ARIA of the HTML5 Recommendation (titled 3.2.7 WAI-ARIA and HTML Accessibility API Mappings in HTML 5.1).

ARIA in SVG1.1

This specification defines the web developer (author) rules (conformance requirements) for the use of ARIA attributes on SVG 1.1 elements. It also defines the requirements for Conformance Checking tools.

Custom Elements

This specification describes the method for enabling the author to define and use new types of DOM elements in a document.

Editing the Custom Element Semantics section of the specification.

Best Practice

HTML5: Techniques for providing useful text alternatives

This document contains best practice guidance for authors of HTML documents on providing text alternatives for images.  Edited until October 2104, the bulk of this document is  included in the HTML5 and HTML 5.1 specifications under section 4.8.5.1 Requirements for providing text to act as an alternative for images where I continue to update and maintain.

Notes on using ARIA in HTML

This document is a practical guide for developers on how to add accessibility information to HTML elements using the Accessible Rich Internet Applications (ARIA) specification.

 

 

Bruce LawsonReading list

W3C Team blogMake a Mark – Become Verified in HTML5 by W3C!

W3C’s ongoing mission is to make the Web better and one other way to enable this is to offer high quality training programs as a way to increase the skills of Web developers and empower them to become the next leaders and innovators on the Web.

W3Cx logoW3C has recently expanded its Training Program with the edX partnership that resulted in the creation of W3Cx. The first MOOC to be offered on that platform is Learn HTML from W3C (W3Cx HTML5.1a). This exciting 6-week course starts on 1 June 2015 and has been built by Michel Buffa who was the creator and instructor of the HTML5 Course on W3DevCampus. We think that the learning experience that you will have will result in a dramatic increase in your knowledge of HTML5. While you can sign up for the course on the Honor System, I would encourage you to sign up for a Verified Certificate as it provides a mark of distinction for you, the W3C Seal of Approval.

In fact, on W3DevCampus, we were asked by students of our courses to recognize the fact that they had successfully completed the course, so we did that via Open Badges in addition to certificates of completion. People who have earned these badges tell us they make a difference for them as they enter the job market, change opportunities or seek new clients. We believe employers will soon be asking for marks of distinction such as Verified Certificates.

One last comment for your consideration. We are making this a special offer course from a pricing perspective. W3C has set a price for Verified Certificates at $129 per course. For this inaugural course on W3Cx we are offering these at $99 which is an even better bargain!

We look forward to seeing you sign up for this exciting course and the many more that W3C will be offering in the coming years.

IEBlogBringing Web Audio to Microsoft Edge for interoperable gaming and enthusiast media

In March, we released initial preview support for Web Audio to Windows Insiders.  Today, we are excited to share details of our Web Audio implementation in Microsoft Edge.  Web Audio provides tight time synchronization not possible with HTML5 Audio Elements, and includes audio filters and effects useful for enthusiast media apps.  Additionally, Microsoft Edge includes Web Audio support for live audio streams using Media Capture and getUserMedia, and also works with an updated HTML5 Audio Element capable of gapless looping.

Web Audio is specified in the W3C Web Audio Specification, and continues to evolve under the guidance of the W3C Audio Working Group.  This release adds Microsoft Edge to the list of browsers that support Web Audio today, and establishes the specification as a broadly supported, and substantially interoperable web spec.

Web Audio Capabilities

Web Audio is based on concepts that are key to managing and a playing multiple sound sources together.  These start with the AudioContext and a variety of Web Audio nodes.  The AudioContext defines the audio workspace, and the nodes implement a range of audio processing capabilities.  The variety of nodes available, and the ability to connect them in a custom manner in the AudioContext makes Web Audio highly flexible.

Diagram of a conceptual Audio Context

Diagram of a conceptual Audio Context

SourceBufferNodes are typically used to hold small audio fragments.  These get connected to different processing nodes, and eventually to the AudioDestinationNode, which sends the output stream to the audio stack to play through speakers.

In concept, the AudioContext is very similar to the audio graph implemented in XAudio2 to support gaming audio in Classic Windows Applications.  Both can connect audio sources through gain and mixing nodes to play sounds with accurate time synchronization.

Web Audio goes further, and includes a range of audio effects that are supported through a variety of processing nodes.  These include effects that can pan audio across the sound stage (PannerNode), precisely delay individual audio streams (DelayNode), and recreate sound listening environments, like stadiums or concert halls (ConvolverNode).  There are oscillators to generate random signals (OscillatorNode) and many more.

Microsoft Edge all of the following Web Audio interfaces and node types:

Special Media Sources

There are two other node types that Microsoft Edge implements that apply to specific use cases.  These are:

Media Capture integration with Web Audio

The MediaStreamAudioSourceNode accepts stream input from the Media Capture and Streams API, also known as getUserMedia after one of the primary stream capture interfaces.  A recent Microsoft Edge Dev Blog post announced our support for media capture in Microsoft Edge.  As part of this implementation, streams are connected to Web Audio via the MediaStreamAudioSourceNode.  Web apps can create a stream source in the audio context and include stream captured audio from, for example, the system microphone.  Privacy precautions are considered by the specification and discussed in our previous blog.  The positive is that audio streams can be processed in web audio for gaming, music or RTC uses.

Gapless Looping in Audio Elements

The MediaElementAudioSourceNode similarly allows an Audio Element to be connected through Web Audio as well.  This is useful for playing background music or other long form audio that the app doesn’t want to keep entirely in memory.

We’ve both connected Audio & Video Elements to Web Audio, and also made performance changes that allow audio to loop in both with no audio gap.  This allows samples to be looped continuously.

Web Audio Demo

We’ve published a demo to illustrate some of Web Audio’s capabilities using stream capture with getUserMedia.  The Microphone Streaming & Web Audio Demo allows local audio to be recorded and played.  Audio is passed through Web Audio nodes that visualize the audio signals, and apply effects and filters.

The following gives a short walkthrough of the demo implementation.

Create the AudioContext

Setting up the audio context and the audio graph is done with some basic JavaScript.  Simply create nodes that you need (in this case, source, gain, filter, convolver and analyzer nodes), and connect them from one to the next.

Setting up the audioContext is simple:

var audioContext = new (window.AudioContext || window.webkitAudioContext)();

Connect the Audio Nodes

Additional nodes get created by calling node specific create methods on audioContext:

var micGain = audioContext.createGain();
var sourceMix = audioContext.createGain();
var visualizerInput = audioContext.createGain();
var outputGain = audioContext.createGain();
var dynComp = audioContext.createDynamicsCompressor();

Connect the Streaming Source

The streaming source is just as simple to create:

sourceMic = audioContext.createMediaStreamSource(stream);

Nodes are connected from source to processing nodes to the destination node with simple connect calls:

sourceMic.connect(notchFilter);
notchFilter.connect(micGain);
micGain.connect(sourceMix);
sourceAudio.connect(sourceMix);

Mute and Unmute

The mic and speakers have mute controls to manage situations where audio feedback happens.  They are implemented by toggling the gain on nodes at the stream source and just before the AudioDestinationNode:

var toggleGainState = function(elementId, elementClass, outputElement){
     var ele = document.getElementById(elementId);
     return function(){
          if (outputElement.gain.value === 0) {
               outputElement.gain.value = 1;
               ele.classList.remove(elementClass);
          } else {
               outputElement.gain.value = 0;
               ele.classList.add(elementClass);
          }
     };
};
 
var toggleSpeakerMute = toggleGainState('speakerMute', ‘button--selected', outputGain);
var toggleMicMute = toggleGainState('micMute', ‘button--selected', micGain);

Apply Room Effects

Room effects are applied by loading impulse response files into a convolverNode connected in the stream path:

var effects = {
     none: {
          file: 'sounds/impulse-response/trigroom.wav'
     },
     telephone: {
          file: 'sounds/impulse-response/telephone.wav'
     },
     garage: {
          file: 'sounds/impulse-response/parkinggarage.wav'
     },
     muffler: {
          file: 'sounds/impulse-response/muffler.wav'
     }
};
               
var applyEffect = function() {
     var effectName = document.getElementById('effectmic-controls').value;
     var selectedEffect = effects[effectName];
     var effectFile = selectedEffect.file;

Note that we took a small liberty in using the “trigroom” environment as a surrogate for no environmental effect being applied.  No offense is intended for fans of trigonometry!

Visualize the Audio Signal

Visualizations were implemented by configuring analyzerNodes for time and frequency domain data, and using the results to manipulate canvas based presentations.

var drawTime = function() {
     requestAnimationFrame(drawTime);
     timeAnalyser.getByteTimeDomainData(timeDataArray);
 
var drawFreq = function() {
     requestAnimationFrame(drawFreq);
     freqAnalyser.getByteFrequencyData(freqDataArray);

Record & Play

Recorder features use the recorder.js open source sample written by Matt Diamond, and used previously in other Web Audio based recorder demos.  Live audio in the demo uses the MediaStreamAudioSource, but recorded audio is played using the MediaElementAudioSource.  Gapless looping can be tested by activating the loop control during playback.

The complete code for the demo is available for your evaluation and use on GitHub.

Conclusion

There are many articles and other demos available on the web that illustrate Web Audio’s many capabilities, and provide other examples that can be run on Microsoft Edge.  You can try some out on our Test Drive page at Microsoft Edge Dev:

We’re eager for your feedback so we can further improve our Web Audio implementation, and meanwhile we are looking forward to seeing what you do with these new features!

– Jerry Smith, Senior Program Manager, Microsoft Edge

Jeremy Keith100 words 058

PPK writes of modern web development:

Tools don’t solve problems any more, they have become the problem.

I think he’s mostly correct, but I think there is some clarification required.

Web development tools fall into two broad categories:

  1. Local tools like preprocessors, task managers, and version control systems that help the developer output their own HTML, CSS, and JavaScript.
  2. Tools written in HTML, CSS, and JavaScript that the end user has to download for the developer to gain benefit.

It’s that second category that contain a tax on the end user. Stop solving problems you don’t yet have.

Planet MozillaFirst Panasonic Smart TVs powered by Firefox OS Debut Worldwide

The first Panasonic VIERA Smart TVs powered by Firefox OS are now available in Europe and will be available worldwide in the coming months.Firefox OS Smart TV“Through our partnership with Mozilla and the openness and flexibility of Firefox OS, we have been able to create a more user friendly and customizable TV UI. This allows us to provide a better user experience for our consumers providing a differentiator in the Smart TV market,” said Masahiro Shinada, Director of the TV Business Division at Panasonic Corporation.

The Panasonic 2015 Smart TV lineup includes these models powered by Firefox OS: CR850, CR730, CX800, CX750, CX700 and CX680 (models vary by country).

“We’re happy to partner with Panasonic to bring the first Smart TVs powered by Firefox OS to the world,” said Andreas Gal, Mozilla CTO. “With Firefox and Firefox OS powered devices, users can enjoy a custom and connected Web experience and take their favorite content (apps, videos, photos, websites) across devices without being locked into one proprietary ecosystem or brand.”

Panasonic Smart TVs powered by Firefox OS are optimized for HTML5 to provide strong performance of Web apps and come with a new intuitive and customizable user interface which allows quick access to favorite channels, apps, websites and content on other devices. Through Mozilla-pioneered WebAPIs, developers can leverage the flexibility of the Web to create customized and innovative apps and experiences across connected devices.

Firefox OS is the first truly open mobile platform built entirely on Web technologies, bringing more choice and control to users, developers, operators and hardware manufacturers.

Planet MozillaDropdown Component Using Custom Elements (vs. React)

Dropdowns have never been easier and native.

We have been building an increasing amount of Custom Elements, or Web Components, over at the Firefox Marketplace (using a polyfill). Custom Elements are a W3C specification that allow you to define your own HTML elements. Using Custom Elements, rather than arbitrary JS, encourages modularity and testability, with portability and reusability being the enticer.

Over the last several months, I worked on revamping the UI for the Firefox Marketplace. Part of it was building a custom dropdown element that would allow users to filter apps based on platform compatibility. I wanted it to behave exactly like a <select> element, complete with its interface, but with the full license to style it however I needed.

In this post, I'll go over Custom Elements, introduce an interesting "proxy" pattern to extend native elements, and then compare Custom Elements with the currently reigning Component king, React.

Custom Select source code.

Building a Custom Element

Custom Elements are still in working draft, but there is a nice cocument.registerElement polyfill. Here is an extremely simple Custom Element that simple wraps a div and defines some interface on the element's prototype.

document.registerElement('input-wrapper', {
    prototype: Object.create(HTMLElement.prototype, {
        createdCallback: {
            value: function () {
                // Called after the component is "mounted" onto the DOM.
                this.appendChild(document.createElement('input'));
            }
        },
        input: {
            get: function() {
                return this.querySelector('input');
            }
        },
        value: {
            get: function() {
                return this.input.value;
            },
            set: function(val) {
                this.input.value = val;
            }
        }
    })
});

var inputWrapper = document.createElement('input-wrapper');
document.body.appendChild(inputWrapper);

We define the interface using Javascript's Object.create, extending the basic HTMLElement. The element simply wraps an input, and provides a getter and setter on the input's value. We drop it into the DOM, and it will natively have whatever interface we defined for it. So we could do something like inputWrapper.value = 5 to directly set the inner input's value. Basic example, but being able to create these native Custom Elements can go far in modular development.

Proxy Pattern: Extending the Select Element by Rerouting Interface

Now we got a gist of what a Custom Element is, let's see how we can use it to create a custom dropdown by extending the native <select> element.

Here's an example of how our element will be used in the HTML:

<custom-select name="my-select">
  <custom-selected>
    The current selected option is <custom-selected-text></custom-selected-text>
  </custom-selected>
  <optgroup>
    <option value="1">First value</option>
    <option value="2">Second value</option>
    <option value="3">Third value</option>
    <option value="4">Fourth value</option>
  </optgroup>
</mkt-select>

What we'll do in the createdCallback is, if you check the source code, create an actual internal hidden select element, copying the attributes defined on <custom-select>. Then we'll create <custom-options>, copying the original options into the hidden select. We extend the custom select's interface to have an attribute pointing to the hidden select like so:

select: {
    // Actual <select> element to proxy to, steal its interface.
    // Value set in the createdCallback.
    get: function() {
        return this._select;
    },
    set: function(select) {
        copyAttrs(select, this);
        this._select = select;
    }
},

This will allow our custom element to absorb the functionality of the native select element. All we have to do is implement the entire interface of the select element by routing to the internal select element.

function proxyInterface(destObj, properties, methods, key) {
    // Proxies destObj.<properties> and destObj.<methods>() to
    // destObj.<key>.
    properties.forEach(function(prop) {
        if (Object.getOwnPropertyDescriptor(destObj, prop)) {
            // Already defined.
            return;
        }
        // Set a property.
        Object.defineProperty(destObj, prop, {
            get: function() {
                return this[key][prop];
            }
        });
    });

    methods.forEach(function(method) {
        // Set a method.
        Object.defineProperty(destObj, method, {
            value: function() {
                return this[key][method].call(arguments);
            }
        });
    });
}

proxyInterface(CustomSelectElement.prototype,
    ['autofocus', 'disabled', 'form', 'labels', 'length', 'multiple',
     'name', 'onchange', 'options', 'required', 'selectedIndex', 'size',
     'type', 'validationMessage', 'validity', 'willValidate'],

    ['add', 'blur', 'checkValidity', 'focus', 'item', 'namedItem',
     'remove', 'setCustomValidity'],

    'select');

proxyInterface will "route" the property lookups (the first array), and method calls (the second array) from the custom select element to the internal select element. Then all we need to do is make sure our select element's value is up-to-date while we interact with our custom select element, then we can do things like customSelectElement.selectedIndex or customSelectElement.checkValidity() without manually implementing the interface.

Note we could have simply looped over HTMLSelectElement.prototype rather than manually entering in each property and method name, but unfortunately that doesn't play well with some older browsers.

With all of this, we have a custom select element that is fully stylizable while having all the functionality of a native select element (because it extends it!).

Comparing Custom Elements to React

I love React and am using it for a couple of projects. How does Custom Elements compare to it?

  1. Custom Elements has no answer to React's JSX template/syntax. In most of our Custom Elements, we have to manually shuffle things around using the native DOM API. JSX is much, much easier.

  2. Custom Elements has no data-binding or automatic DOM updates whenever data updates. It's all imperative with Custom Elements, you have to listen for changes and manually update the DOM. React is, well, reactive. Whenever a component's state, so does its representation in the DOM.

  3. Custom Elements is a bit harder to nest components than React. In React, it's natural for a component to render a component that renders a component that renders other components. With Custom Elements, it's a bit difficult to connect components together in a nice heirarchy, and the only communication you get is through Events.

  4. Custom Elements, however, is smaller in KB. React is about 26KB after min+gzip whereas a Custom Elements polyfill is maybe a few KB. Though the 26KB might be worth it since you'll end up writing less code, and you get the performance of virtual DOM diffing.

  5. Custom Elements has no answer to React Native.

They're both just as portable, they both can be dropped into any framework. They both have similar interfaces as well in creating components. Although, React is more powerful. In React, I really enjoy keeping data compartmentalized in states and passing down data as props. Loose comparison, but it's like combining the data-binding and template power of Angular with the ideas of Custom Elements.

However, it doesn't have to be one or the other either. Why not both? React can wrap Custom Elements if you want it to. As always, choose the best tools for the job. I'm always chasing and adopting the newest things like an excited dog going after the mailman, but despite that I can say React is a winner.

Steve Faulkner et alShort note on aria-labelledby and aria-describedby

The ARIA attributes aria-labelledby and aria-describedby can be used to provide an accessible name or accessible description for a subset of HTML elements. It is important to note that by design, hiding the content (using CSS display:none or visibility:hidden or the HTML hidden attribute) of the element(s) referenced by these attributes does not stop the content from being used to provide the name/description.

Hidden or visible – makes no difference

By default, assistive technologies do not relay hidden information, but an author can explicitly override that and include hidden text as part of the accessible name or accessible description by using aria-labelledby or aria-describedby.
Accessible Name and Description: Computation and API Mappings 1.1

In the following example the description will be available to assistive technology users in both states:

Non error state: message not visible

<label>Name <input type="text"  aria-describedby="error-message"></label>
<span id="error-message" <mark>style="display:none"</mark>>
You have provided an incorrect name</span>

Note: addition of aria-hidden=true to the referenced element makes no difference:

<span id="error-message" style="display:none" <mark>aria-hidden="true"</mark>>
 You have provided an incorrect name</span>

Error state: message visible

<span id="error-message" <mark>style="display:inline"</mark>>
You have provided an incorrect name</span>

Methods to provide context sensitive name/description text

If you want to associate context sensitive text, such as an error message you can:

  • Add the referenced element to the DOM when the error state occurs.
  • Add the error text as child of the referenced element in the DOM when the error state occurs.
  • Add the id reference in the DOM to the aria-labelledby/aria-describedby attribute, when the error state occurs.

Further reading: Notes on Using ARIA in HTML

Planet MozillaUpdate on Digital Rights Management and Firefox

A year ago, we announced the start of efforts to implement support for a component in Firefox that would allow content wrapped in Digital Rights Management (DRM) to be played within the HTML5 video tag. This was a hard decision because of our Mission and the closed nature of DRM. As we explained then, we are enabling DRM in order to provide our users with the features they require in a browser and allow them to continue accessing premium video content. We don’t believe DRM is a desirable market solution, but it’s currently the only way to watch a sought-after segment of content.

Today, Firefox includes an integration with the Adobe Content Decryption Module (CDM) to playback DRM-wrapped content. The CDM will be downloaded from Adobe shortly after you upgrade or install Firefox and will be activated when you first interact with a site that uses Adobe CDM. Premium video services, including Netflix, have started testing this solution in Firefox.

Because DRM is a ‘black-box’ technology that isn’t open source, we have designed a security sandbox that sits around the CDM. We can’t be sure how other browsers have handled the “black-box” issue but a sandbox provides a necessary layer of security.  Additionally, we’ve also introduced the ability to remove the CDM from your copy of Firefox. We believe that these are important security and choice mechanisms that allow us to introduce this technology in a manner that lessens the negative impacts of integrating this type of black-box.

FX3

We also recognize that not everybody wants DRM, so we are also offering a separate Firefox download without the CDM enabled by default for those users who would rather not have the CDM downloaded to their browser on install.

As we’ve discussed, DRM is a complicated issue. We want our users to understand its implications and we have developed a teaching kit to introduce DRM, its challenges, and why some content publishers use it.

IEBlogMicrosoft Edge: Building a safer browser

With Microsoft Edge, we want to fundamentally improve security over existing browsers and enable users to confidently experience the web from Windows. We have designed Microsoft Edge to defend users from increasingly sophisticated and prevalent attacks.

This post covers some of the advanced technologies used to protect Microsoft Edge, including industry leading sandboxing, compiler, and memory management techniques developed in close partnership with Windows.

Web Security Threats

While the web is predominantly a safe environment, some sites are designed to steal money and personal information. Thieves by nature don’t care about rules, and will use any means to take advantage of victims, most often using trickery or hacking:

  • Trickery: in real life, a “con man” will use tricks to take advantage of a victim, e.g. “got two 10s for a 5?” On the web, attackers will try to fool victims using things like “phishing” attacks that convince a user to enter their banking password into a web site that looks like their bank, but isn’t.
  • Hacking: in real life, a mugger might assault you and take your money, or a burglar might break into your home and steal your valuables. On the web, attackers present a victim’s browser with malformed content intended to exploit subtle flaws in your browser, or in various extensions your browser uses, such as video decoders. This lets the attacker run their code on the victim’s computer, taking over first their browsing session, and perhaps ultimately the entire computer.

These are threats faced by every browser. Let’s explore how Microsoft Edge addresses these threats and is helping make the web a safer experience.

Defend Users Against Trickery

A web browser can help defend a user against trickery by identifying and blocking known tricks, and by using strong security protocols to ensure that you are talking to the web site you think you are talking to. Here are some of the ways that Microsoft Edge helps achieve this.

Stronger, More Convenient Credentials

One of the most common ways to attack users is phishing: enticing the user into entering their password into a fake version of a web site that they trust. Attempts to uniquely identify legitimate web sites, such as through the original HTTPS lock symbol, and more recently with the EV Cert green bar, have met with only limited success: attackers are just too good at faking a legitimate experience to expect most users to notice the difference.

To really defend against this requires removing the need for users to enter plain-text passwords into web sites. Instead, Windows 10 provides Microsoft Passport technology with asymmetric cryptography to authenticate you to your web sites. Windows 10 will also offer the most convenient way to unlock your device and access your Microsoft Passport, providing a truly seamless experience that is more secure than today’s world of complicated passwords.

Defending Against Malicious Web Sites and Downloads

Microsoft SmartScreen, originally introduced in IE8, is supported in Microsoft Edge and by the Windows 10 Shell. SmartScreen defends users against phishing sites by performing a reputation check on sites the browser visits, blocking sites that are thought to be phishing sites. Similarly, SmartScreen in both the browser and the Windows Shell defends users against socially-engineered downloads of malicious software to users being tricked into installing malicious software.

Defending Against Fake Sites with Certificate Reputation

Users have learned to trust sites that use encrypted web traffic. Unfortunately, that trust can be undermined when malicious sites use improperly obtained or fraudulent certificates to impersonate legitimate sites. Last year we announced Certificate Reputation – recently we have extended this system by allowing web developers to use the Bing Webmaster Tools report directly to alert Microsoft to fraudulent certificates.

Web Standards

As we announced recently, Microsoft Edge hosts a new rendering engine, Microsoft EdgeHTML. This engine is focused on modern web standards, allowing web developers to build and maintain one consistent site that supports all modern browsers. This greatly simplifies the hard work of building first class web sites, allowing more time and energy for web developers to focus on reliability and security rather than the complexities of interoperability.

Microsoft EdgeHTML helps in defending against “con man” attacks using new security features in the W3C and IETF standards:

Defending the Browser Against Hacking

Microsoft Edge does more than facilitate standards compatibility and users with defenses against trickery. It also includes a major overhaul of the DOM representation in the browser’s memory, making the browser’s code more resistant to “burglar” attacks that attempt to subvert the browser.

But Microsoft Edge has done more than just re-write the rendering engine. Here we describe the security mitigations baked into the Microsoft Edge browser.

Moving to a more secure extension model

Microsoft introduced ActiveX in 1996, part of a wave of support for web browser extensions, enabling 3rd parties to experiment with various forms of alternate content on the web. However, browser extensions come at a cost of security and reliability: binary extensions bring code and data into the browser’s process, with no protection at all, and so anything that goes wrong or is vulnerable in the extension can also take down or compromise the browser itself. It is rather similar to building a sun porch onto your house without locking the door to the sunporch: it is all too easy for a burglar to break into the sun porch, and from there loot the rest of your house.

So to make browsers safer against attacks, and just more reliable, it is important to create an extension model that is safer, by sharing less state between the browser itself and the extensions. Thus Microsoft Edge provides no support for VML, VB Script, Toolbars, BHOs, or ActiveX. The need for such extensions is significantly reduced by the rich capabilities of HTML5, and using HTML5 results in sites that are interoperable across browsers.

To enable extensibility beyond what is provided by HTML5, we are working on plans for a modern, HTML/JS-based extension model. We look forward to sharing more details on these plans, including guidance on migrating your scenarios from ActiveX controls to modern standards and extensions in the coming months.

Microsoft Edge is an App

The largest change in Microsoft Edge security is that the new browser is a Universal Windows app. This fundamentally changes the process model, so that both the outer manager process, and the assorted content processes, all live within app container sandboxes. This provides the user and the platform with the confidence provided by other Windows store apps.

App Container Sandbox by Default

IE7 on Windows Vista was the first web browser to provide a browsing sandbox, called protected mode, where the part of the browser that renders the web content runs with less privilege than the browser controls (e.g. the back button) and the user do. This provides the user with a level of protection should a malicious web site exploit a bug in the browser or one of its plugins, in that the attacker is now stuck inside the sandbox and does not have full access to all of the user’s resources.

IE7’s protected mode was built on Vista’s mandatory integrity control, which provides a degree of protection, but is limited. IE10 introduced EPM (Enhanced Protected Mode) based on Windows 8’s app container sandbox. EPM provided a much stronger sandbox than protected mode, including for example deny-by-default and no-read-up semantics. EPM was on by default in the immersive browser, but was only an option on the desktop in IE10 and IE11 because some browser extensions are not compatible with EPM.

Microsoft Edge is rebooting our browser extension model, allowing it to run its content processes in app containers, not just as a default, but all the time. Thus every Internet page that Microsoft Edge visits will be rendered inside an app container, the latest and most secure client-side app sandbox in Windows.

64-bit By Default

Microsoft Edge is also 64-bit, not just by default, but at all times when running on a 64-bit processor. 64-bit processes in general, and browser processes in particular, get significant security advantages by making Windows ASLR (Address Space Layout Randomization) stronger.

Attackers want to inject malicious code into your browser process via a coding bug, and then execute their malicious code. ASLR makes that harder by randomizing the memory layout of the process, making it hard for attackers to hit precise memory locations to achieve their ends. In turn, 64-bit processes make ASLR much more effective by making the address space exponentially larger, making it much more difficult for attackers to find the sensitive memory components they need.

Defending Against Memory Corruption

Memory corruption is a class of vulnerability so common that it deserves special attention. It is endemic to applications written in C/C++, because the programming language does not provide type safety or buffer overflow protection. Broadly speaking, memory corruption attacks involve the attacker providing malformed input to a program that the program does not handle properly, resulting first in corruption of the program’s memory state, and subsequently in the attacker gaining control of the program. A broad variety of memory corruption mitigations have been devised since the mid-1990s, and in the 2000s Microsoft has lead the way with advances including ASLR, DEP, and SeHOP.

MemGC

As these mitigations have rolled out, attackers have adapted, inventing new forms of attack. Microsoft in turn has responded with new memory safety defenses that mitigate the most common new forms of attack, including and especially use-after-free vulnerabilities.

MemGC (Memory Garbage Collector) is a memory garbage collection system that seeks to defend the browser from UAF (Use-after-free) vulnerabilities by taking responsibility for freeing memory away from the programmer and instead automating it, only freeing memory when the automation has detected that there are no more references left pointing to a given block of memory.

Control Flow Guard

The end-game in memory-corruption is for the attacker to gain control of the CPU program counter, and jump to a code location of the attacker’s choice. CFG (Control Flow Guard) is a Microsoft Visual Studio technology that compiles checks around code that does indirect jumps based on a pointer, restricting these jumps to only jump to function entry points that have had their address taken. This makes attacker take-over of a program much more difficult by severely constraining where a memory corruption attack can jump to.

These new memory safety protections have been enabled and shipped out to Windows and IE users over the last year, and are on all the time in Microsoft Edge.

Bug Bounty

Despite all efforts, there will be security vulnerabilities in Microsoft Edge that we do not yet know about. To minimize customer impact, we will be offering a Windows 10 Technical Preview Browser Bug Bounty program, intended to incent security researchers to report browser vulnerabilities to Microsoft during the Technical Preview period rather than after Microsoft Edge ships as a general use product. Security researchers interested in the bounty program should look here for more information.

Security Is A Process, Not A Destination

Microsoft Edge is a brand new browser, with new goals and requirements. This has allowed us to include these security enhancements, both brand new security features, and moving older opt-in features to be always-on. For this reason, we believe Microsoft Edge will be the most secure web browser that Microsoft has ever shipped. As security is a process, not a destination, we will continue to work on browser security improvements. Let us know what you think in the comments below or @MSEdgeDev.

– Crispin Cowan, Senior Program Manager, Microsoft Edge

[Corrected on 5/11 at 5:35 PM to reflect that Protected Mode was introduced in IE7 on Windows Vista, not Windows 7.]

IEBlogBringing Asm.js to Chakra and Microsoft Edge

A couple of months back, we announced that we had started development on Asm.js. Support for Asm.js has been one of the top 10 most-requested items at the Microsoft Edge Developer Suggestion Box on UserVoice since we launched it in December 2014. Since that time, we’ve made good progress, and in Windows 10 Insider Preview builds starting with 10074, we are now previewing the early work that we’ve been doing to enable Asm.js support in Chakra and Microsoft Edge.

What is Asm.js?

Asm.js is a strict subset of JavaScript that can be used as a low-level, efficient target language for compilers. As a sublanguage, asm.js effectively describes a sandboxed virtual machine for memory-unsafe languages like C or C++. A combination of static and dynamic validation allows JavaScript engines to employ techniques like type specialized compilation without bailouts and ahead-of-time (AOT) compilation for valid asm.js code. Such compilation techniques help JavaScript execute at “predictable” and “near-native” performance, both of which are non-trivial in the world of compiler optimizations for dynamic languages like JavaScript.

Given the complexity of writing asm.js code by hand, asm.js is currently produced principally by transpiling C/C++ code using toolchains like Emscripten to run on the Web platform, utilizing technologies such as WebGL and Web Audio. Game engines like those from Unity and Unreal are starting to land early/experimental support of plug-in free games on the web using a combination of asm.js and other related technologies.

How do I experiment with Asm.js in Microsoft Edge?

To enable and experiment with the current Asm.js support in Microsoft Edge, navigate to about:flags in Microsoft Edge and select the “Enable asm.js” flag as below:

Screenshot of about:flags interface for asm.js

Engineering Asm.js into Chakra’s Execution Pipeline

To enable Asm.js in Chakra, the components marked in green below have been added or modified vis-à-vis the typical code flow though Chakra’s pipeline outlined and explained in this blog.

Diagram of the Chakra pipeline

Key changes include:

  • Addition of an asm.js validator, which helps Chakra identify and validate that the asm.js code adheres to the asm.js specification.
  • Generation of optimized Type Specialized bytecode. Given that asm.js supports only native types (int, double, float or SIMD values), Chakra takes advantage of the type information to generate optimized Type Specialized bytecode for asm.js code.
  • Faster execution of Type Specialized bytecode by Chakra’s interpreter by utilizing the type information to avoid boxing and unboxing of numeric values, and removing the overhead of generating profile data while interpreting. For non-asm.js JavaScript code, Chakra’s interpreter generates and feeds profile data to the JIT compiler to enable generating highly optimized JIT code. The type specialized bytecode generated for valid asm.js code already has all the data needed by Chakra’s JIT compiler to produce highly optimized machine code. The type specialized interpreter is 4x-5x magnitude faster than the profiling interpreter.
  • Faster code compilation by Chakra’s JIT compiler. Asm.js code is typically generated by using the LLVM compiler and Emscripten toolchain. Chakra’s JIT compiler takes advantage of some of the optimizations that are already available in the asm.js source code generated by the LLVM compiler. For example, Chakra’s JIT compiler avoids doing an inlining pass and takes advantage of code inlining that has been performed on the asm.js code produced by the LLVM compiler. Avoiding analysis for such JIT optimizations not only helps save CPU cycles and battery usage to generate highly optimized code, but also improves the throughput of Chakra’s JIT compiler tremendously.
  • Predictable performance by removal of bailouts from compiled asm.js code. Given the dynamic nature of JavaScript, all modern JavaScript engines support some form of migration to non-optimized code execution, called bailouts, when they determine that the assumptions made by the JIT compilers are no longer valid for the compiled code. Bailouts not only impact performance but also make it unpredictable. Taking advantage of type-safety restrictions posed on valid asm.js code, the code generated by Chakra’s JIT compiler for valid asm.js code is guaranteed to not require bailouts.

JIT Compiler Optimizations for Asm.js

Apart from changes to Chakra’s execution pipeline described above, Chakra’s JIT compiler has also been enhanced to take advantages of the constraints laid out by the asm.js specification. These enhancements help Chakra’s JIT compiler to generate code that could execute at near native code performance – the golden target for all dynamic language JIT compilers. Let’s drill into two such optimization techniques.

Helper and bailout call elimination

Helper or bailout code acts like a power backup for compiled code in dynamic languages. If any of the assumptions that the JIT compiler makes while compiling the code is invalidated, there needs to be fail safe mechanism to continue execution with correctness. Helpers can be used to handle un-anticipated situations for an operation, and control can then return to the JIT’d code. Bailouts handle situations where the JIT’d code cannot recover from the un-anticipated conditions and control needs to be transferred back to the interpreter to execute the remainder of the current function. For example, if a variable is seen as integer and the compiled code is generated with that assumption, the occurrence of a double should not impact correctness. In such cases, helper or bailout code is used to continue execution, albeit with slower performance. For asm.js code, Chakra does not generate any helper or bailout code.

To understand why, let’s take an example of a function square in an asm.js module, which has been over simplified to help explain the concept.

function PhysicsEngine(global, foreign, heap) {
    "use asm";
    …
    // Function Declaration
    function square(x) {
        x = +x;
        return +(x*x);
    }
    …
}

This function square in the code sample above translates to two x64 machine instructions in asm.js code, excluding the prolog and the epilog that is generated for that function.

xmm1 = MOVAPS xmm0 //Move native double value to a temporary register
xmm0 = MULSD xmm0, xmm1 //Multiplication

In contrast, the compiled code generated for the square function when asm.js is not enabled generates over 10 machine instructions. This includes:

  • Checking if the parameter is a tagged double, if not jump to a helper which handles conversions from any type to double
  • Extracting the double value from tagged double
  • Multiplying the value
  • Converting the result back to tagged double

Chakra’s JIT compiler is able to generate efficient code for asm.js by taking advantage of the type information of the variable, which cannot change through the lifetime of the program. The Asm.js validator & linker ensure the same. Given that all internal variables in asm.js are native types (int, double, float or SIMD values), internal functions freely pass native types without boxing them to JavaScript variables between function boundaries. In compiler terminology these are often called direct calls and avoid boxing or conversions when operating on data in asm.js code. Likewise for external calls into or from JavaScript, all the incoming variables are converted to native types and boxed while all the outgoing native types are converted to variables.

Bounds check elimination for typed array accesses

In a previous blog, we explained how Chakra’s JIT compiler supports auto-typed array optimizations, as well as hoisting of bounds check outside of loops thus improving the performance on array operations inside loops by up to 40%. With asm.js type constraints, Chakra’s JIT compiler completely eliminates bounds checking for typed array accesses for all of asm.js compiled code, regardless of the location (inside or outside loops or functions) or the type of the typed array. It also eliminates bounds checks for constant and non-constant index accesses for typed arrays. Together, these optimizations help bring near native performance when operating on typed arrays in asm.js code.

Following is a simplified example of typed array store at a constant offset and the single x64 machine instruction generated by the Chakra’s compiler for that code in asm.js.

int8ArrayView[4] = 25; //JavaScript code
[Address] = MOV 25 //Single machine instruction to store value

When compiling asm.js code, Chakra’s JIT compiler is able to pre compute the final address corresponding to int8ArrayView[4] during link time of asm.js code (first instantiation of the asm.js module) and hence can generate a single instruction corresponding the typed array store. In contrast, the compiled code generated for the same operation in non-asm.js JavaScript code results in around 10-15 machine instructions with path length being considerable. The steps include:

  • Loading the typed array from closure variable (typed array is closure captured in asm.js)
  • Checking if the closure loaded variable is indeed a typed array, and if not, bailout
  • Checking the index is within the length of the typed array; if any of the conditions fail, jumping to helper or bailout
  • Storing the value in the array buffer

The above checks are removed from asm.js code due to the strict constrains on what can be changed inside an asm.js module. For example, a variable pointing to typed array in an asm.js module is immutable and cannot change. If it does, it’s not a valid asm.js code and the asm.js validator fails to validate it as asm.js code. In such cases the code is treated like any other JavaScript function.

Scenario light-ups and performance wins from Asm.js

From the early work that shipped in the latest Windows 10 preview, we have started to see some great scenarios light-up apart from the performance wins by bringing asm.js to Chakra and Microsoft Edge. If you want to take it for a spin, try out games like Angry Bots, Survival Shooter, Tappy Chicken, or try out some fun asm.js demos listed here.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="https://www.youtube.com/embed/Ge6NZT9mbA8?feature=oembed" width="640"></iframe>

Unity WebGL Player | Prime World Defenders

 In terms of performance wins, there are a crop of asm.js benchmarks that are emerging. With the preliminary asm.js support, Chakra and Microsoft Edge now perform more than 300% faster on Unity Benchmark and around 200% faster on individual tests like zlib, which features in Google’s Octane and Apple’s Jet Stream benchmarking suites.

Unity Benchmark scores for 64-bit browsers

Unity Benchmark scores for 64-bit browsers (Click to enlarge)
(System info: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz (4 cores), 12GB RAM running 64 bit Windows 10 Preview)
(Scores have been scaled so that IE11 = 1.0, to fit onto a single chart)

 

So, what’s next?

This is an exciting step towards bringing asm.js to the web platform underneath Microsoft Edge. However we are not fully done yet. We are working on fine tuning Chakra’s pipeline for ASM.js support – gathering data to validate if the current design approach performs well for real world asm.js payloads, understanding and addressing outstanding performance, functionality and tooling issues etc. before enabling this feature by default. Once enabled, asm.js would not only be available in Microsoft Edge, but would also be available in Windows store apps written using HTML/CSS/JS as well as Web Views that target the new EdgeHTML rendering engine that is shipping with Microsoft Edge in Windows 10.

We’d also like to thank Mozilla, with whom we’ve been working closely since we started implementing asm.js, and Unity for their support and partnership in helping bring asm.js to Chakra and Microsoft Edge. And a big thanks to all the developers and users of Windows 10 and Microsoft Edge, who have been providing us with valuable feedback in this journey. Keep it coming! You could reach us and let us know your feedback on Twitter at @MSEdgeDev or on Connect.

– Abhijith Chatra, Senior Software Engineering Manager, Chakra
Gaurav Seth, Principal PM Manager, Chakra

IEBlogA break from the past, part 2: Saying goodbye to ActiveX, VBScript, attachEvent…

We recently posted “A break from the past: the birth of Microsoft’s new web rendering engine”, an in-depth look at the background and motivation behind building a new rendering engine to power Microsoft Edge. A key factor described was the ability to make a break from legacy Internet Explorer-specific technologies that had been built up over the years.

In this post, we are sharing a list of legacy IE technologies and features that are not part of Microsoft Edge. Not supporting these legacy technologies in Microsoft Edge has a number of benefits: better interoperability with other modern browsers, improved performance, security & reliability, and reduced code complexity, just to name a few. And just like all good developers, ours love to delete code – over 220,000 lines of code in MSHTML have been removed from Microsoft EdgeHTML!

Graphic showing lines of code removed and interoperability fixes

We’re also sharing more about how we plan to introduce new web platform features to Microsoft Edge using concepts like feature flags and API trials, with the goal of preserving the ability to experiment with new capabilities which eventually lead to stable, interoperable implementations across browsers.

Technologies and APIs not supported in Microsoft Edge

Here are some key technologies which will not be supported in Microsoft Edge. A number of these had been deprecated in the past few releases, but moving to a new engine allows us to remove support completely. To ensure future compatibility with Microsoft Edge and other modern browsers, we recommend against any future development or deployment of these technologies. These technologies will continue to be supported in Internet Explorer on Windows 10 for users and enterprises who depend on them.

Technology Why it existed Why we don’t need it anymore
ActiveX ActiveX is a binary extension model introduced in 1996 which allowed developers to embed native Windows technologies (COM/OLE) in web pages. These controls can be downloaded and installed from a site and were subsequently loaded in-process and rendered in Internet Explorer. The need for ActiveX controls has been significantly reduced by HTML5-era capabilities, which also produces interoperable code across browsers. Microsoft Edge will support native PDF rendering and Adobe Flash as built-in features rather than external add-ons.We recently demonstrated our early work on a modern, HTML/JavaScript-based extension model to provide extensibility beyond what is available with HTML5. We will enable this new model after our initial release of Microsoft Edge this summer and we look forward to sharing more details soon.
Browser Helper Objects (BHO) BHOs are a binary extension model introduced in 1997 which enabled developers to write COM objects that were loaded in-process with the browser and could perform actions on available windows and modules. A common use was to build toolbars that installed into Internet Explorer.
Document modes Starting with IE8, Internet Explorer introduced a new “document mode” with every release. These document modes could be requested via the x-ua-compatible header to put the browser into a mode which emulates legacy versions. Similar to other modern browsers, Microsoft Edge will have a single “living” document mode. In order to minimize the compatibility burden, features will be tested behind switches in about:flags until they are stable and ready to be turned on by default.
Vector Markup Language (VML) VML is an XML-based format for 2D vector graphics, originally proposed in 1998 and originally supported in IE5. 2D vector graphics are supported via Scalable Vector Graphics (SVG) which provides interoperability across modern browsers.
VBScript VBScript is a scripting language modeled after Visual Basic. It was introduced in 1996 to provide web developers with a language option when scripting on the web. JavaScript has become the de-facto language of the web and is evolving quickly with ECMAScript 6 and beyond. Microsoft Edge delivers the leading implementation of ES6.
attachEvent / removeEvent IE8 and below supported a proprietary model for DOM events, which supported bubbling but not capture and used a global event object rather than passed as an argument to the event listener. IE9 and above support the DOM Level 3 Events standard with addEventListener/ removeEventListener.
currentStyle The currentStyle property retrieves the cascaded style of the element. IE9 and above support the getComputedStyle property (part of DOM L2 standard) to retrieve the final computed style of any element.
Conditional Comments Conditional comments provided a way to embed code that was only targeted at specific versions of Internet Explorer. It is more effective to use feature detection and other progressive enhancement rather than browser specific code in conditional comments.
IE8 layout quirks IE8 specific layout behaviors that were then emulated in later versions of Internet Explorer. Most websites are written for standards-based layout. Sites that require IE8 specific behavior on the intranet can be rendered using Enterprise Mode.
DirectX Filters and Transitions DX filters and transitions enabled web developers to apply various visual effects to elements on the web page. Similar effects can be achieved using standards-based features in CSS3 and SVG.

In addition to ActiveX and BHOs, there are a number of other legacy extensibility points that are not in Microsoft Edge and will be replaced by a unified, modern extensibility model. These include:

  • Binary behaviors
  • Pluggable protocols
  • Shell Helper API
  • Active Documents
  • Custom Download Managers
  • Custom Security Managers
  • MIME filters
  • Custom Print and Print Preview Handlers
  • Explorer Bars
  • Custom Context Menu entries (via registry)
  • Edit Designers
  • Timers
  • Accelerators
  • Webslices

There are also hundreds of other non-interoperable APIs which have been removed. In most cases, these APIs now have interoperable, standardized replacements—removed APIs do not necessarily mean removed capabilities. All too often, the presence of these non-interoperable APIs cause site compatibility issues. Removing them makes a substantial improvement to user experience in Microsoft Edge and doubles down on our commitment to the web’s best feature: interoperability.

Preserving interoperability and experimentation on the web

Some of the more notable removals are those where Microsoft Edge supports the latest standard API definition and removes support for MS prefixed versions of the APIs. Examples include CSS Transforms, Fullscreen API, and Pointer Events. This brings up an important topic: vendor prefixes. You’ll see a trend in the vendor prefixes in Microsoft Edge compared to IE:

Graphic showing vendor prefixes in IE11 versus Microsoft Edge

Our support (or lack thereof) of prefixed APIs is data-driven with compatibility as the sole purpose for any that remain. As examples, -webkit-border-radius is still in use on over 60% of page loads and the MS-prefixed Encrypted Media Extensions APIs are still in use on top video streaming services. Because of the impact on interoperability across the Web, these are supported in Microsoft Edge for the time being.

The consensus across the industry is that vendor prefixes cause more harm than good and the necessity of browsers supporting other vendor’s prefixes is a good example of why. Going forward, we’re using feature flags (accessible by navigating to about:flags) to introduce experimental APIs and minting new vendor prefixed APIs sparingly, if ever. Today, you’ll find examples like early implementations of CSS Filters, Touch Events, and HTML5 date inputs in about:flags. We’re also working with other browser vendors on exploring new approaches to experimental APIs that enable easier and broader access to experimental APIs while minimizing the compatibility burden that prefixes incur. Check out this talk (starts around 31 min mark) for an initial proposal we’ve made, tentatively named “API Trials”.

Ready for the future

We hope that this provides some insight into of how Microsoft Edge is leaving behind some legacy in order to provide a great foundation for the future. Please continue to test your sites in Microsoft Edge via the Windows Insider Program or RemoteIE, and give us your feedback via @MSEdgeDev or Connect.

If you have any questions about legacy technologies in Microsoft Edge, or about Microsoft Edge in general, join us live at 2:30 PM PDT on Channel 9 for an open Q&A with the Microsoft Edge engineering team! We’ll be taking audience questions from Microsoft Edge Web Summit attendees, as well as questions to @MSEdgeDev on Twitter with the hashtag #AskMSEdge.

Charles Morris, Principal Program Manager Lead, Microsoft Edge
Jacob Rossi, Senior Program Manager, Microsoft Edge

Planet MozillaServo Continues Pushing Forward

Servo is a new prototype web browser layout engine written in Rust that was launched by Mozilla in 2012 with a new architecture to achieve high parallelism on components like layout and painting.

It has been progressing at an amazing pace, with over 120 CSS properties currently supported, and work is ongoing to implement the remaining properties. For a full list of the current set of CSS properties with initial support in Servo, check out the Google Docs spreadsheet servo team is using to track development.

The current supported properties allow Servo to be mostly operational on static sites like Wikipedia and GitHub, with a surprisingly small code footprint. It has only about 126K lines of Rust code, and the Rust compiler and libraries are about 360K lines. For comparison, in 2014 Blink had about 700K lines of C++ code, and WebKit had around 1.3M lines, including platform specific code.

Another exciting development is servo-shell, which allows the implementation and customization of a WebBrowser using only Javascript and CSS. It’s essentially a browser chrome that uses mozbrowser APIs (i.e. iFrame extensions) running on top of Servo and provides the ability to separate the browser content from loaded pages, which has led to fairly good performance so far.

Finally, Rust (the programming language used to implement Servo) is approaching the 1.0 launch and a big group of people are ready to celebrate the occasion in San Francisco.

Improving Debugging and Testing

One of the most challenging parts of developing a browser engine from scratch is re-implementing all of the CSS features, because they often have complicated interactions. For a developer to solve any layout rendering bugs they run into, they must first inspect the graphical representation of the DOM tree to see if it is correct. In case of Servo, the DOM tree will generate a FlowTree and DisplayLists while performing layout and rendering, compared to WebKit and Blink, which uses a RenderTree as graphical representation (and features DumpRenderTree tool for accessing the RenderTree). Debugging support was improved remarkably with addition of the ability to dump Optimized display lists, Flow tree, and Display List, as well as the implementation of reflow events debugging, which can be used to inform developers when and why a layout was recalculated.

Integration of the Firefox timeline has recently been started on Servo. This is a tool that allows tracking of operations performed by the web engine and is useful for debugging and profiling a site. Additionally, W3C organization has created a test suite to help in verifying CSS features across browsers, which enhances interoperability. Servo now has support for running these W3C CSS tests.

Additional Servo Highlights

General Developments

  • Servo was ported to Gonk (the low level layer of Firefox OS) last February.
  • Servo has some state of the art components (e.g. HTML5 parser, CSS parser) implemented in Rust as independent libraries, which may be beneficial to integrate with Firefox. Work has started on this integration, but whether the image decoder or the URL parser will be integrated first is undefined at this time.
  • WebGL implementation has begun.
  • Another cool feature is the visualization of parallel painting, where Servo executes in a mode in which tiles rendered by each distinct thread will have an overlay rendered on top of it. This makes it possible to visualize Servo’s parallel painting.
  • Support for displaying a placeholder when an image link is broken.
  • Cookies are now supported as well as SSL certification verification. These allow users to login to most websites that have user accounts.
  • Providing the ability to embed Servo on applications in the future is important, and work on this subject is progressing. Instead of creating a new API for developers, the community decided to use the Chromium Embedded Framework (CEF): an API that is quite successful and stable. Servo has a CEF-like API that provides the ability to embed a Servo-powered webview on native apps, as demonstrated by Miniservo on Mac and Linux. Work on supporting the API has been progressing well.

Improved HTML/CSS Support

The Road Ahead

As you can see, Servo has advanced remarkably in the last few months by implementing many new features that benefit both Servo developers as well as future users. It is moving at a fast pace, implementing support for several of the web features needed by any modern browser engine while proving that Rust, as a systems-level programing language, is up to the task of writing a web engine from scratch.

Are you interested in contributing to Servo or just curious to give it a try? Visit the project site, or feel free to chat with the developers on #servo on the mozilla IRC server.

This post was originally published on the Samsung OSG Blog, which is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Planet MozillaTalk Script: Firefox OS Email Performance Strategies

Last week I gave a talk at the Philly Tech Week 2015 Dev Day organized by the delightful people at technical.ly on some of the tricks/strategies we use in the Firefox OS Gaia Email app.  Note that the credit for implementing most of these techniques goes to the owner of the Email app’s front-end, James Burke.  Also, a special shout-out to Vivien for the initial DOM Worker patches for the email app.

I tried to avoid having slides that both I would be reading aloud as the audience read silently, so instead of slides to share, I have the talk script.  Well, I also have the slides here, but there’s not much to them.  The headings below are the content of the slides, except for the one time I inline some code.  Note that the live presentation must have differed slightly, because I’m sure I’m much more witty and clever in person than this script would make it seem…

Cover Slide: Who!

Hi, my name is Andrew Sutherland.  I work at Mozilla on the Firefox OS Email Application.  I’m here to share some strategies we used to make our HTML5 app Seem faster and sometimes actually Be faster.

What’s A Firefox OS (Screenshot Slide)

But first: What is a Firefox OS?  It’s a multiprocess Firefox gecko engine on an android linux kernel where all the apps including the system UI are implemented using HTML5, CSS, and JavaScript.  All the apps use some combination of standard web APIs and APIs that we hope to standardize in some form.

Firefox OS homescreen screenshot Firefox OS clock app screenshot Firefox OS email app screenshot

Here are some screenshots.  We’ve got the default home screen app, the clock app, and of course, the email app.

It’s an entirely client-side offline email application, supporting IMAP4, POP3, and ActiveSync.  The goal, like all Firefox OS apps shipped with the phone, is to give native apps on other platforms a run for their money.

And that begins with starting up fast.

Fast Startup: The Problems

But that’s frequently easier said than done.  Slow-loading websites are still very much a thing.

The good news for the email application is that a slow network isn’t one of its problems.  It’s pre-loaded on the phone.  And even if it wasn’t, because of the security implications of the TCP Web API and the difficulty of explaining this risk to users in a way they won’t just click through, any TCP-using app needs to be a cryptographically signed zip file approved by a marketplace.  So we do load directly from flash.

However, it’s not like flash on cellphones is equivalent to an infinitely fast, zero-latency network connection.  And even if it was, in a naive app you’d still try and load all of your HTML, CSS, and JavaScript at the same time because the HTML file would reference them all.  And that adds up.

It adds up in the form of event loop activity and competition with other threads and processes.  With the exception of Promises which get their own micro-task queue fast-lane, the web execution model is the same as all other UI event loops; events get scheduled and then executed in the same order they are scheduled.  Loading data from an asynchronous API like IndexedDB means that your read result gets in line behind everything else that’s scheduled.  And in the case of the bulk of shipped Firefox OS devices, we only have a single processor core so the thread and process contention do come into play.

So we try not to be a naive.

Seeming Fast at Startup: The HTML Cache

If we’re going to optimize startup, it’s good to start with what the user sees.  Once an account exists for the email app, at startup we display the default account’s inbox folder.

What is the least amount of work that we can do to show that?  Cache a screenshot of the Inbox.  The problem with that, of course, is that a static screenshot is indistinguishable from an unresponsive application.

So we did the next best thing, (which is) we cache the actual HTML we display.  At startup we load a minimal HTML file, our concatenated CSS, and just enough Javascript to figure out if we should use the HTML cache and then actually use it if appropriate.  It’s not always appropriate, like if our application is being triggered to display a compose UI or from a new mail notification that wants to show a specific message or a different folder.  But this is a decision we can make synchronously so it doesn’t slow us down.

Local Storage: Okay in small doses

We implement this by storing the HTML in localStorage.

Important Disclaimer!  LocalStorage is a bad API.  It’s a bad API because it’s synchronous.  You can read any value stored in it at any time, without waiting for a callback.  Which means if the data is not in memory the browser needs to block its event loop or spin a nested event loop until the data has been read from disk.  Browsers avoid this now by trying to preload the Entire contents of local storage for your origin into memory as soon as they know your page is being loaded.  And then they keep that information, ALL of it, in memory until your page is gone.

So if you store a megabyte of data in local storage, that’s a megabyte of data that needs to be loaded in its entirety before you can use any of it, and that hangs around in scarce phone memory.

To really make the point: do not use local storage, at least not directly.  Use a library like localForage that will use IndexedDB when available, and then fails over to WebSQLDatabase and local storage in that order.

Now, having sufficiently warned you of the terrible evils of local storage, I can say with a sorta-clear conscience… there are upsides in this very specific case.

The synchronous nature of the API means that once we get our turn in the event loop we can act immediately.  There’s no waiting around for an IndexedDB read result to gets its turn on the event loop.

This matters because although the concept of loading is simple from a User Experience perspective, there’s no standard to back it up right now.  Firefox OS’s UX desires are very straightforward.  When you tap on an app, we zoom it in.  Until the app is loaded we display the app’s icon in the center of the screen.  Unfortunately the standards are still assuming that the content is right there in the HTML.  This works well for document-based web pages or server-powered web apps where the contents of the page are baked in.  They work less well for client-only web apps where the content lives in a database and has to be dynamically retrieved.

The two events that exist are:

DOMContentLoaded” fires when the document has been fully parsed and all scripts not tagged as “async” have run.  If there were stylesheets referenced prior to the script tags, the script tags will wait for the stylesheet loads.

load” fires when the document has been fully loaded; stylesheets, images, everything.

But none of these have anything to do with the content in the page saying it’s actually done.  This matters because these standards also say nothing about IndexedDB reads or the like.  We tried to create a standards consensus around this, but it’s not there yet.  So Firefox OS just uses the “load” event to decide an app or page has finished loading and it can stop showing your app icon.  This largely avoids the dreaded “flash of unstyled content” problem, but it also means that your webpage or app needs to deal with this period of time by displaying a loading UI or just accepting a potentially awkward transient UI state.

(Trivial HTML slide)

<link rel=”stylesheet” ...>
<script ...></script>
DOMContentLoaded!

This is the important summary of our index.html.

We reference our stylesheet first.  It includes all of our styles.  We never dynamically load stylesheets because that compels a style recalculation for all nodes and potentially a reflow.  We would have to have an awful lot of style declarations before considering that.

Then we have our single script file.  Because the stylesheet precedes the script, our script will not execute until the stylesheet has been loaded.  Then our script runs and we synchronously insert our HTML from local storage.  Then DOMContentLoaded can fire.  At this point the layout engine has enough information to perform a style recalculation and determine what CSS-referenced image resources need to be loaded for buttons and icons, then those load, and then we’re good to be displayed as the “load” event can fire.

After that, we’re displaying an interactive-ish HTML document.  You can scroll, you can press on buttons and the :active state will apply.  So things seem real.

Being Fast: Lazy Loading and Optimized Layers

But now we need to try and get some logic in place as quickly as possible that will actually cash the checks that real-looking HTML UI is writing.  And the key to that is only loading what you need when you need it, and trying to get it to load as quickly as possible.

There are many module loading and build optimizing tools out there, and most frameworks have a preferred or required way of handling this.  We used the RequireJS family of Asynchronous Module Definition loaders, specifically the alameda loader and the r-dot-js optimizer.

One of the niceties of the loader plugin model is that we are able to express resource dependencies as well as code dependencies.

RequireJS Loader Plugins

var fooModule = require('./foo');
var htmlString = require('text!./foo.html');
var localizedDomNode = require('tmpl!./foo.html');

The standard Common JS loader semantics used by node.js and io.js are the first one you see here.  Load the module, return its exports.

But RequireJS loader plugins also allow us to do things like the second line where the exclamation point indicates that the load should occur using a loader plugin, which is itself a module that conforms to the loader plugin contract.  In this case it’s saying load the file foo.html as raw text and return it as a string.

But, wait, there’s more!  loader plugins can do more than that.  The third example uses a loader that loads the HTML file using the ‘text’ plugin under the hood, creates an HTML document fragment, and pre-localizes it using our localization library.  And this works un-optimized in a browser, no compilation step needed, but it can also be optimized.

So when our optimizer runs, it bundles up the core modules we use, plus, the modules for our “message list” card that displays the inbox.  And the message list card loads its HTML snippets using the template loader plugin.  The r-dot-js optimizer then locates these dependencies and the loader plugins also have optimizer logic that results in the HTML strings being inlined in the resulting optimized file.  So there’s just one single javascript file to load with no extra HTML file dependencies or other loads.

We then also run the optimizer against our other important cards like the “compose” card and the “message reader” card.  We don’t do this for all cards because it can be hard to carve up the module dependency graph for optimization without starting to run into cases of overlap where many optimized files redundantly include files loaded by other optimized files.

Plus, we have another trick up our sleeve:

Seeming Fast: Preloading

Preloading.  Our cards optionally know the other cards they can load.  So once we display a card, we can kick off a preload of the cards that might potentially be displayed.  For example, the message list card can trigger the compose card and the message reader card, so we can trigger a preload of both of those.

But we don’t go overboard with preloading in the frontend because we still haven’t actually loaded the back-end that actually does all the emaily email stuff.  The back-end is also chopped up into optimized layers along account type lines and online/offline needs, but the main optimized JS file still weighs in at something like 17 thousand lines of code with newlines retained.

So once our UI logic is loaded, it’s time to kick-off loading the back-end.  And in order to avoid impacting the responsiveness of the UI both while it loads and when we’re doing steady-state processing, we run it in a DOM Worker.

Being Responsive: Workers and SharedWorkers

DOM Workers are background JS threads that lack access to the page’s DOM, communicating with their owning page via message passing with postMessage.  Normal workers are owned by a single page.  SharedWorkers can be accessed via multiple pages from the same document origin.

By doing this, we stay out of the way of the main thread.  This is getting less important as browser engines support Asynchronous Panning & Zooming or “APZ” with hardware-accelerated composition, tile-based rendering, and all that good stuff.  (Some might even call it magic.)

When Firefox OS started, we didn’t have APZ, so any main-thread logic had the serious potential to result in janky scrolling and the impossibility of rendering at 60 frames per second.  It’s a lot easier to get 60 frames-per-second now, but even asynchronous pan and zoom potentially has to wait on dispatching an event to the main thread to figure out if the user’s tap is going to be consumed by app logic and preventDefault called on it.  APZ does this because it needs to know whether it should start scrolling or not.

And speaking of 60 frames-per-second…

Being Fast: Virtual List Widgets

…the heart of a mail application is the message list.  The expected UX is to be able to fling your way through the entire list of what the email app knows about and see the messages there, just like you would on a native app.

This is admittedly one of the areas where native apps have it easier.  There are usually list widgets that explicitly have a contract that says they request data on an as-needed basis.  They potentially even include data bindings so you can just point them at a data-store.

But HTML doesn’t yet have a concept of instantiate-on-demand for the DOM, although it’s being discussed by Firefox layout engine developers.  For app purposes, the DOM is a scene graph.  An extremely capable scene graph that can handle huge documents, but there are footguns and it’s arguably better to err on the side of fewer DOM nodes.

So what the email app does is we create a scroll-region div and explicitly size it based on the number of messages in the mail folder we’re displaying.  We create and render enough message summary nodes to cover the current screen, 3 screens worth of messages in the direction we’re scrolling, and then we also retain up to 3 screens worth in the direction we scrolled from.  We also pre-fetch 2 more screens worth of messages from the database.  These constants were arrived at experimentally on prototype devices.

We listen to “scroll” events and issue database requests and move DOM nodes around and update them as the user scrolls.  For any potentially jarring or expensive transitions such as coordinate space changes from new messages being added above the current scroll position, we wait for scrolling to stop.

Nodes are absolutely positioned within the scroll area using their ‘top’ style but translation transforms also work.  We remove nodes from the DOM, then update their position and their state before re-appending them.  We do this because the browser APZ logic tries to be clever and figure out how to create an efficient series of layers so that it can pre-paint as much of the DOM as possible in graphic buffers, AKA layers, that can be efficiently composited by the GPU.  Its goal is that when the user is scrolling, or something is being animated, that it can just move the layers around the screen or adjust their opacity or other transforms without having to ask the layout engine to re-render portions of the DOM.

When our message elements are added to the DOM with an already-initialized absolute position, the APZ logic lumps them together as something it can paint in a single layer along with the other elements in the scrolling region.  But if we start moving them around while they’re still in the DOM, the layerization logic decides that they might want to independently move around more in the future and so each message item ends up in its own layer.  This slows things down.  But by removing them and re-adding them it sees them as new with static positions and decides that it can lump them all together in a single layer.  Really, we could just create new DOM nodes, but we produce slightly less garbage this way and in the event there’s a bug, it’s nicer to mess up with 30 DOM nodes displayed incorrectly rather than 3 million.

But as neat as the layerization stuff is to know about on its own, I really mention it to underscore 2 suggestions:

1, Use a library when possible.  Getting on and staying on APZ fast-paths is not trivial, especially across browser engines.  So it’s a very good idea to use a library rather than rolling your own.

2, Use developer tools.  APZ is tricky to reason about and even the developers who write the Async pan & zoom logic can be surprised by what happens in complex real-world situations.  And there ARE developer tools available that help you avoid needing to reason about this.  Firefox OS has easy on-device developer tools that can help diagnose what’s going on or at least help tell you whether you’re making things faster or slower:

– it’s got a frames-per-second overlay; you do need to scroll like mad to get the system to want to render 60 frames-per-second, but it makes it clear what the net result is

– it has paint flashing that overlays random colors every time it paints the DOM into a layer.  If the screen is flashing like a discotheque or has a lot of smeared rainbows, you know something’s wrong because the APZ logic is not able to to just reuse its layers.

– devtools can enable drawing cool colored borders around the layers APZ has created so you can see if layerization is doing something crazy

There’s also fancier and more complicated tools in Firefox and other browsers like Google Chrome to let you see what got painted, what the layer tree looks like, et cetera.

And that’s my spiel.

Links

The source code to Gaia can be found at https://github.com/mozilla-b2g/gaia

The email app in particular can be found at https://github.com/mozilla-b2g/gaia/tree/master/apps/email

(I also asked for questions here.)

W3C Team blogDigital Signage and HTML5: Seminar Report

On March 26, W3C Keio host held a “Digital Signage and HTML5” seminar with the support of the Japanese Digital Signage Consortium. With a capacity crowd of 115 attendees it was a full afternoon of presentations, discussions and high expectations.

As well as an introduction to the work towards the 2020 Tokyo Olympic and Paralympic Games by the Ministry of Internal Affairs and Communications’ Mr. Ogasawara (Director of the ICT Strategy Policy Division) following a message by Mr. Suzuki (Director-General of the Global ICT Strategy Bureau), there was also an explanation of the current digital signage landscape by Ms. Iyoku of the Digital Signage Consortium.

As an example of HTML5-based signage, Mr. Watanabe and Mr. Yamada from NTT, and Mr. Kato from Recruit Technologies showed examples of their research and development into linking smartphones and beacons. There were also demos making use of beacons (Physical Web) which caught the eye of attendees.

There then followed an introduction to functional requirements in HTML5-based signage by Newphoria’s Mr. Hatano, co-chair of the Web-based Signage Business Group.

Representing the Web Platform Performance Benchmarking Focus Group, Mr. Hatano, Mr. Hoshino from ACCESS and Dr. Kawada from W3C/Keio presented terminal performance requirements, a terminal performance test specification, and development guidelines for web-based signage contents.

Finally there was a panel discussion focusing on the future of digital signage. The importance of HTML5 going forward was discussed by Ms. Iyoku, Mr. Kato, Dr. Kishigami (W3C Advisory Board) and Mr. Hatano, with Keio University’s Mr. Kikuchi moderating.

The seminar was closed with a reception where W3C members and Digital Signage Consortium members were able to deepen ties going forward.

W3C慶應では、去る3月26日に、「デジタルサイネージとHTML5」セミナーを、デジタルサイネージコンソーシアム後援のもと、実施しました。総勢115名という多数の参加者で会場は満席、熱気の中で、午後いっぱいを使ってセミナー・ディスカッションが行われました。

本セミナーでは、総務省情報通信国際戦略局の鈴木局長からご挨拶頂き、同情報通信政策課の小笠原課長より2020年のオリンピック・パラリンピックに向けた取組が紹介された後、デジタルサイネージの現在の状況についてデジタルサイネージコンソーシアム(DSC)の伊能氏から解説がありました。

また、HTML5ベースのサイネージの事例として、NTT(株)渡邉氏・山田氏とリクルートテクノロジーズ(株)加藤氏からそれぞれ、スマホ連携とビーコン連携の研究開発例が紹介されました。ビーコン(Physical Web)活用についてはデモも行われ、参加者の注目を集めました。

次にW3C Web-based Signage BG議長の(株)ニューフォリア羽田野氏より、HTML5ベースサイネージの機能要求条件が紹介されました。

続いて、Webプラットフォーム性能ベンチマーク検討会を代表して、ニューフォリア羽田野氏、(株)ACCESS星野氏、W3C/慶應の川田より、Web-based Signage端末の性能要求条件、端末性能テスト仕様、およびコンテンツ開発ガイドラインの説明がありました。

最後にデジタルサイネージの将来について、パネルディスカッションが行われました。モデレータが慶應義塾大学の菊池氏、パネリストはDSC伊能氏、リクルートテクノロジーズ加藤氏、W3Cアドバイザリーボードの岸上氏、ニューフォリア羽田野氏が務め、今後のHTML5の重要性が語られました。

セミナー後は懇親会が行われ、W3C会員相互間、またDSC会員とW3C会員間の交流を深め、今後なお一層の連携を誓っての終了となりました。

Jeremy Keith100 words 035

I went spelunking down the caves of the web’s history today. I went digging through the original proposal that Tim Berners-Lee presented to his boss, the late Mike Sendall. I explored the design issues that were raised in the creation the World Wide Web. I rifled through the initial vocabulary of HTML. I watched a video from the web’s twentieth anniversary featuring Tim Berners-Lee, Robert Cailliau, and Jean-François Groff.

I was able to do all this because my computer was connected to the internet and I was running a piece of software that understood HTTP, URLs, and HTML’s A element.

Planet MozillaWhys and Hows of Suggested Tiles

As Darren discussed on Monday’s project call [wiki.mozilla.org], Suggested Tiles has been on track to go live to a larger audience next week for wider beta testing. I’ll provide some context around why we implemented this feature, details of how it works, and open questions of how to make it even better.

A suggestion made for people who visit Mozilla sites

A suggestion made for people who visit Mozilla sites

We’ve been looking for ways to improve the user experience within Firefox by combining data that Firefox knows about with data Mozilla can provide. We’re also in a good position to work with the advertising ecosystem so that we can change it to care more about the values Mozilla cares about. We want to create advertising products that respect user’s choice/control over their data and get others interested by showing money can be made.

Last November, we launched Tiles with a framework to show external content within Firefox’s new tab page. A relatively small portion of Firefox users saw these because they only showed up if there would have been empty tiles, i.e., new users with little history or existing users who cleared history. Suggested Tiles expands on this to be a little bit smarter by showing content based on the user’s top sites. For example, if a user tends to visit sites about mobile phones, Firefox can now decide to show a suggestion for Firefox for Android.

We intend to bring value to users by showing them content that they would be interested in and engage with. On the flip side, this means we purposefully hold back on showing content that users might get annoyed with and block. We do this in a way that requires using a minimal amount of the user’s data, and as usual, we provide controls to the user to turn things off if that’s desired. For the initial release of Suggested Tiles, we plan to show content from Mozilla such as mobile Firefox, MDN, and HTML5 gaming.

The Tiles framework has been built in ways that are different from traditional web advertising in both how it gets data into Firefox [blog.mozilla.org] and how it reports on Tiles performance. The two linked posts have quite a bit of detail, but to summarize, Firefox makes generic encrypted cookieless requests to get enough data to decide locally in Firefox whether content should be added to the new tab page. In order for us to have data on how to improve the experience, Firefox reports back when users see and click on these tiles and includes tile data such as IDs of added content but no URLs. We have aggressive data deletion policies and don’t keep any unique identifiers that can be associated with our users.

The technical changes to support Suggested Tiles are not overly complex as the server provides one additional value specifying when a suggestion should be shown. This value informs Firefox which sites need to be in the user’s top sites before showing the tile. The reporting mechanism is unchanged, so if a Suggested Tile is shown or clicked, Firefox reports back the tile’s ID and no URLs just as before.

Even though the technical changes are not too complicated, the effect of this can be significant. In particular, Firefox reports back if it shows a Suggested Tile, and if that tile is only shown when the user has been to one of various news sites, Firefox reveals to our servers that this user reads news. Our handling of the data is no different from before.

Because we care about user privacy, we have policies around how/what suggestions can be made. For example, to match on news sites, we make sure there’s at least 5 popular news sites for Firefox to check against. This makes it so there’s user deniability of which site the user actually visited. We also focus on broad uncontroversial topics, so we don’t make suggestions based on adult content or illegal gambling sites.

An additional layer of protection is built into Firefox by only allowing predefined sets of sites [hg.mozilla.org] for making suggestions. These include mobile phones, technology news, web development, and video games. This rigidness protects Firefox from accepting fake/malicious suggestions that could reveal data Mozilla doesn’t want to collect.

This last point is important to highlight because we want to have a discussion around how we can be more flexible in showing more relevant content and fixing mistakes. For example, people who care about video games might only care about a specific gaming platform, but because Firefox only allows for the predefined sites, we would end up suggesting content that many users didn’t actually want to see.

Feel free to respond with comments about Suggested Tiles or to join in on the discussion about various topics on dev.planning [groups.google.com]:

As usual, we have the source code available for Firefox [hg.mozilla.org] and our servers that send/receive tiles data [github.com], create tiles, and process tiles data. You can also find additional details in the Directory Links Architecture and Data Formats documentation [people.mozilla.org].

No comments

Bruce LawsonWeb Components, accessibility and the Priority of Constituencies

Gosh, what a snappy title. I’m not expecting a job offer from Buzzfeed any time soon.

Today, Apple sent their consolidated feedback on Web Components to the webapps Working Group. The TL;DR: they like the concept, are “considering significant implementation effort”, but want lots of changes first including removal of subclassing, eg <button is=”my-button”>.

I think this is bad; this construct means existing HTML elements can be progressively enhanced – in the example above, browsers that don’t support components or don’t support JavaScript get a functional HTML <button> element. It also means that, by enhancing existing HTML elements, your components get the default browser behaviour for free – so, in this example, your snazzy my-button element inherits focussability and activation with return or spacebar withut you having to muck about with tabindex or keyboard listeners. (I wrote about this in more detail last year in On the accessibility of web components. Again.)

Apple raised a bug Remove the support for inherting from builtin subclasses of HTMLElement and SVGElement and notes “without this hack accessibility for trivial components is harder as more things have to be done by hand” (why “this hack”? A loaded term). However, it calls for removal because “Subclassing existing elements is hard as implementation-wise identity is both object-based and name / namespace based.”

Implementation is hard. Too hard for the developers at Apple, it appears. So Web developers must faff around adding ARIA and tabindex and keyboard listeners (so most won’t) and the inevitable consequence of making accessibility hard is that assistive technology users will suffer.

HTML has a series of design principles, co-edited by Maciej Stachowiak who sent Apple’s feedback. One of those is called “Priority of Constituencies” which says

In case of conflict, consider users over authors over implementors over specifiers over theoretical purity. In other words costs or difficulties to the user should be given more weight than costs to authors; which in turn should be given more weight than costs to implementors; which should be given more weight than costs to authors of the spec itself, which should be given more weight than those proposing changes for theoretical reasons alone.

Fine words. What changed?

Planet MozillaThings I’ve Learned This Week (April 13 – April 17, 2015)

When you send a sync message from a frame script to the parent, the return value is always an array

Example:

// Some contrived code in the browser
let browser = gBrowser.selectedBrowser;
browser.messageManager.addMessageListener("GIMMEFUE,GIMMEFAI", function onMessage(message) {
  return "GIMMEDABAJABAZA";
});

// Frame script that runs in the browser
let result = sendSendMessage("GIMMEFUE,GIMMEFAI");
console.log(result[0]);
// Writes to the console: GIMMEDABAJABAZA

From the documentation:

Because a single message can be received by more than one listener, the return value of sendSyncMessage() is an array of all the values returned from every listener, even if it only contains a single value.

I don’t use sync messages from frame scripts a lot, so this was news to me.

You can use [cocoaEvent hasPreciciseScrollingDeltas] to differentiate between scrollWheel events from a mouse and a trackpad

scrollWheel events can come from a standard mouse or a trackpad1. According to this Stack Overflow post, one potential way of differentiating between the scrollWheel events coming from a mouse, and the scrollWheel events coming from a trackpad is by calling:

bool isTrackpad = [theEvent hasPreciseScrollingDeltas];

since mouse scrollWheel is usually line-scroll, whereas trackpads (and Magic Mouse) are pixel scroll.

The srcdoc attribute for iframes lets you easily load content into an iframe via a string

It’s been a while since I’ve done web development, so I hadn’t heard of srcdoc before. It was introduced as part of the HTML5 standard, and is defined as:

The content of the page that the embedded context is to contain. This attribute
is expected to be used together with the sandbox and seamless attributes. If a
browser supports the srcdoc attribute, it will override the content specified in
the src attribute (if present). If a browser does NOT support the srcdoc
attribute, it will show the file specified in the src attribute instead (if
present).

So that’s an easy way to inject some string-ified HTML content into an iframe.

Primitives on IPDL structs are not initialized automatically

I believe this is true for structs in C and C++ (and probably some other languages) in general, but primitives on IPDL structs do not get initialized automatically when the struct is instantiated. That means that things like booleans carry random memory values in them until they’re set. Having spent most of my time in JavaScript, I found that a bit surprising, but I’ve gotten used to it. I’m slowly getting more comfortable working lower-level.

This was the ultimate cause of this crasher bug that dbaron was running into while exercising the e10s printing code on a debug Nightly build on Linux.

This bug was opened to investigate initializing the primitives on IPDL structs automatically.

Networking is ultimately done in the parent process in multi-process Firefox

All network requests are proxied to the parent, which serializes the results back down to the child. Here’s the IPDL protocol for the proxy.

On bi-directional text and RTL

gw280 and I noticed that in single-process Firefox, a <select> dropdown set with dir=”rtl”, containing an <option> with the value “A)” would render the option as “(A”.

If the value was “A) Something else”, the string would come out unchanged.

We were curious to know why this flipping around was happening. It turned out that this is called “BiDi”, and some documentation for it is here.

If you want to see an interesting demonstration of BiDi, click this link, and then resize the browser window to reflow the text. Interesting to see where the period on that last line goes, no?

It might look strange to someone coming from a LTR language, but apparently it makes sense if you’re used to RTL.

I had not known that.

Some terminal spew

Some terminal spew

Now what’s all this?

My friend and colleague Mike Hoye showed me the above screenshot upon coming into work earlier this week. He had apparently launched Nightly from the terminal, and at some point, all that stuff just showed up.

“What is all of that?”, he had asked me.

I hadn’t the foggiest idea – but a quick DXR showed basic_code_modules.cc inside Breakpad, the tool used to generate crash reports when things go wrong.

I referred him to bsmedberg, since that fellow knows tons about crash reporting.

Later that day, mhoye got back to me, and told me that apparently this was output spew from Firefox’s plugin hang detection code. Mystery solved!

So if you’re running Firefox from the terminal, and suddenly see some basic_code_modules.cc stuff show up… a plugin you’re running probably locked up, and Firefox shanked it.


  1. And probably a bunch of other peripherals as well 

Footnotes

Updated: .  Michael(tm) Smith <mike@w3.org>