IEBlogMoving to HTML5 Premium Media

The commercial media industry is undergoing a major transition as content providers move away from proprietary web plug-in based delivery mechanisms (such as Flash or Silverlight), and replace them with unified plug-in free video players that are based on HTML5 specifications and commercial media encoding capabilities.  Browsers are moving away from plug-ins as well, as Chrome is with NPAPI and Microsoft Edge is with ActiveX, and toward more secure extension models.

The transition to plug-in free media has been enabled through the recent development of new specifications:

These specs were designed and developed to enable interoperable streaming to a variety of media platforms and devices.  By focusing on interoperable solutions, content providers are able to reduce costs and at the same time users are able to access the content they want on the device they prefer using the app or web browser of their choice. Microsoft believes that this is a huge benefit to both content producers and consumers, and is committed to supporting companies that make this transition.

This is a long blog, and we don’t want you to miss a topic that interests you.  Here’s a glimpse of what we’ll cover:

  • Some information on Microsoft Edge and Silverlight
  • An overview of interoperable web media.
  • Some challenges and options to address them:
    • The simplest form of DASH streaming.
    • A website demo that uses a library to play Smooth content.
    • Services from Azure Media Services that can help.
    • A simple method for creating a Universal Windows Platform (UWP) app based on website code.
    • A demo UWP that integrates video playback with Cortana voice commands.

Microsoft Edge and Silverlight

Support for ActiveX has been discontinued in Microsoft Edge, and that includes removing support for Silverlight.  The reasons for this have been discussed in previous blogs and include the emergence of viable and secure media solutions based on HTML5 extensions.  Microsoft continues to support Silverlight, and Silverlight out-of-browser apps can continue to use it.  Silverlight will also continue to be supported in Internet Explorer 11, so sites continue to have Silverlight options in Windows 10.  At the same time, we encourage companies that are using Silverlight for media to begin the transition to DASH/MSE/CENC/EME based designs and to follow a single, DRM-interoperable encoding work flow enabled by CENC.  This represents the most broadly interoperable solution across browsers, platforms, content and devices going forward.

Interoperable Media across Browsers

Plug-ins like Silverlight were intended to support interoperable media by providing versions for every browser they supported.  This became more difficult as devices and platforms that support browsers multiplied.  Now, as the old plug-in models are being removed, replacements for them are needed.  For media, a great forward looking replacement can be based on DASH, MSE, EME and CENC.

Windows 10 and Microsoft Edge support DASH, MSE, EME and CENC natively, and other major browsers ship implementations of MSE and CENC compliant EME. This support allows developers to build plug-in free web video apps that runs across a huge range of platforms and devices, with each MSE/EME implementation built on top of a different media pipeline and DRM provider.

DRM Providers Can Differ by Browser

DRM Providers Can Differ by Browser (Click to enlarge)

In the days when DRM systems used proprietary file formats and encryption methods, this variation in DRM providers by browser would have presented a significant issue.  With the development and use of Common Encryption (CENC), the problem is substantially reduced because the files are compressed in standard formats and encrypted using global industry standards.  The service provider issues the keys and licenses necessary to consume the content in a given browser, but the website code, content and encryption keys are common across all of them, regardless of which DRM is in use. An example of such an implementation is DASH.js, the open source industry reference player used to prove out these technologies and which serves as the basis for many players across the web today.

As shown above, Microsoft’s PlayReady DRM supports two modes of DRM protection:  “Software DRM”, which uses our traditional software protected media path, and “Hardware DRM”, which moves the media path into secured hardware when available on the device.  Hardware DRM was designed to meet the highest requirements for commercial media content protection, and will allow streaming of the highest quality content available.  Not all devices will be Hardware DRM capable, but sites built on MSE/EME can accommodate the difference and stream the best possible content quality supported by the browser or device.

Support from Microsoft

Like any new technology, the transition to DASH/MSE/EME/CENC can present some challenges.  These include:

  • MSE works by allowing a JavaScript client to set one or more sourceBuffer(s) as the source on a media element, and dynamically download and append media segments to the sourceBuffer. This provides sites precise control over the experience, but also requires a larger investment in site development.
  • Large existing media libraries have been encoded in formats which are not directly supported by MSE/EME. These libraries must be either supported in some way on the new APIs or re-encoded. Silverlight Smooth Streaming is an example format which was designed and built for sites using Silverlight plug-ins.  A solution that could play this content directly would be useful for any Silverlight replacement technology.
  • MSE/EME are maturing, but still undergoing change that may present interop challenges working with different media formats and across browsers.

As part of helping the industry move to interoperable media delivery solutions, we are investing in technologies to address these challenges.

DASH Type 1:  MSE Made Easy

DASH content usually consists of media files encoded at various quality levels and a manifest that provides information on the files to the media application.  An MSE based player is then responsible for parsing these files, downloading the most appropriate content and feeding it into the media element’s sourceBuffer(s). This is very flexible, but requires either investment in authoring site MSE implementations or use of a reference MSE implementation such as the aforementioned DASH.js.

There is another much easier option:  native DASH streaming, where the site code simply sets the manifest as the media element source, and playback is automatically managed by the browser’s built-in streaming engine.  This approach allows web developers to leverage the learnings and investments made by browser makers and easily provide a premium media experience on their site. We have added support for native DASH streaming to Windows 10 and Microsoft Edge, and more details on this are available in our previous blog: Simplified Adaptive Video Streaming: Announcing support for HLS and DASH in Windows 10.

A DASH JavaScript Library That Plays Smooth Content

A number of websites have large media libraries encoded in the Smooth Streaming format and are looking to move to an interoperable HTML5-based solution. One possible solution is to use a library that supports their current content through MSE/EME without re-encoding.  Libraries are available now that are capable of playing back Smooth Streaming content using MSE and EME. For example, a version of the “hasplayer.js” library can do just that and is available for download at https://github.com/eiximenis/hasplayer.js.

This hasplayer.js library is based on dash.js, and enables playback of both clear and protected Smooth content using PlayReady on Microsoft Edge. It is a client-side JavaScript library that provides content and manifest translations and is cross browser compatible. Thanks to the inclusion of EME poly-fill support, it may be easily extended to support other browser’s DRM solutions as well.

Here’s a sample of JavaScript that uses hasplayer.js to retrieve and play a DASH or Smooth media file:

<style>.gist table { margin-bottom: 0; }</style>

<script src="https://gist.github.com/kypflug/654574c120e7ad66fa70.js"></script>

That makes it very simple to support streaming Smooth content on a website.  We’ve provided a sample in the Contoso Video Sample GitHub Repository that uses this library to play video.  You can try the webpage yourself at the Contoso Video Demo Website.

Screenshot of the Contoso Video Demo Website

• Microsoft Edge Rendering
• Chakra JavaScript Engine
• HTML/CSS/JS from server

It is possible to provide a client-side translation library for Smooth Streaming content because the Protected Interoperable File Format (PIFF) underlying the Smooth Streaming protocol was a primary input into the ISO Base Media File Format (ISOBMFF) spec used with DASH and because PIFF introduced the multi-DRM protocol which was standardized as ISO MPEG Common Encryption (CENC). Two PIFF formats are in broad use today – PIFF 1.1 and PIFF 1.3 – and the hasplayer.js Smooth Streaming MSE/EME library supports both formats. These formats are supported by on-the-fly conversion from the PIFF format to the Common Media Format (CMF) used with DASH. This ensures that all browser content played back by the library is conformant with DASH CMF and capable of playback in all MSE-capable browsers.

Media Services

Some content owners would prefer to focus on producing great content, not the technical details required to deliver their content to consumers. These companies may wish to leverage hosted media services that prepare content for web delivery, handle the streaming logic and player UI, and manage the DRM license servers.  Azure Media Services offers this capability today, with support for both PlayReady and Widevine DRM systems.  This service provides support for both Video on Demand (VoD) and live streaming.  A single high quality file/stream is provided to Azure, where it undergoes dynamic compression and encryption into CENC protected content that can be streamed to end devices, and a complete player solution is available for developers to simply add to their website. Some details of this service are newly announced in Azure Media Services delivers Widevine encrypted stream by partnering with castLabs.

Hosted Web Apps

One powerful advantage of moving to DASH/MSE/EME/CENC streaming is that the same code running on your website can be packaged as a Universal Windows Platform (UWP) app. UWP applications run on all devices with Windows 10.   A website developer can create both an interoperable website based player and a Windows app that uses the same code, but offers a richer experience using Windows platform capabilities.  Their common code will thus be able to handle UI and media streaming details, AND take advantage of capabilities only available to apps through WinRT APIs.

These hosted web apps:

  • Are offered through the Windows Store
  • Are able to interact with Cortana (“Contoso Video play elephants”)
  • Are able to offer notifications (“NBA Finals streaming starts in 15 minutes”)
  • Have access to advanced adaptive streaming support
  • Have access to enhanced content protection for Full HD and Ultra-HD video playback
  • Are able to light up live tiles
  • And more!

We talked about the power of hosted apps in our talk on Hosted web apps and web platform innovations at the Microsoft Edge Web Summit 2015. You can also find reference documentation on MSDN.

Hosted App Demo

We’ve taken the demo Contoso Video website shown above and packaged it as a demo UWP app that takes advantage of Windows Platform APIs.  This demo shows how simple it is to take the basic video player and integrate Cortana Voice Commands.  The demo also customizes the colors used in the app’s title bar.  All of the JavaScript code is part of the HTML website which is deployed as part of the standard web development workflow.

Screenshot of the Contoso Video website packaged as a UWP app

• Retain Microsoft Edge Rendering
• Retain Chakra JavaScript Engine
• Retain HTML/CSS/JS from server or local
• Add access native Windows APIs – Cortana, notifications, customizations & more…
• Offered in the Windows Store Catalog

Three files are needed to integrate Cortana in your Hosted Web Applications (HWA): A Voice Command Definition (VCD) file, a JavaScript file, and an HTML file.

Voice Command Definition (VCD) file

The Voice Command Definition (VCD) File specifies the actions you want to support with voice commands.   The code below informs Cortana of the app name (Contoso Video), that the app supports a “play” command, and how the “playing” state should be represented in the app UI.

<style>.gist table { margin-bottom: 0; }</style>

<script src="https://gist.github.com/kypflug/6d1e47aec679fa0b38fe.js"></script>

The JavaScript File

JavaScript must listen for the Windows Activation event and check for VoiceCommands.

<style>.gist table { margin-bottom: 0; }</style>

<script src="https://gist.github.com/kypflug/bafa784fb0995437e91b.js"></script>

The HTML File

The HTML file must add a meta element pointing to a VCD file on your server.

<style>.gist table { margin-bottom: 0; }</style>

<script src="https://gist.github.com/kypflug/5d25e2a9508e238dabe0.js"></script>

With the addition of a VCD, and updates to the website HTML and JavaScript, our Contoso Video website demo can now be packaged as a Universal Windows Platform application that will run across every device that runs Windows 10.   Further, users can launch the app to play a video by saying “Contoso, play Tears of Steel”.  Cortana will understand the command, launch the Contoso Video app and start playing the video “Caves of Steel”.  The app also has a customized view on the app bar.

Screenshot showing the Contoso Video UWP app being launched via Cortana

Contoso Video in Cortana

Screenshot showing Contoso Video as a search result in the App Menu

Contoso Video in the App Menu

The full source for the Contoso Video website and demo app is available in the Contoso Video Sample GitHub Repository.

Conclusion

DASH/MSE/EME/CENC offer compelling advantages over plug-in based solutions, and we are quickly moving towards an era of broadly interoperable streaming media. Both content providers and consumers will benefit from this transition. While the adoption of these technologies may present short-term challenges, the features and options discussed in this blog are provided to assist companies as they make this change.

We’re eager for your feedback so we can further improve our streaming media offerings, and are looking forward to seeing what you do with the tools and approaches we’ve discussed here!

– David Mebane, Program Manager, Windows Media
– Jerry Smith, Senior Program Manager, Microsoft Edge
– Kevin Hill, Senior Program Manager, Microsoft Edge

Planet MozillaWeb Components, Stories Of Scars

Chris Heilmann has written about Web Components.

If you want to see the mess that is the standardisation effort around web components right now in all its ugliness, Wilson Page wrote a great post on that on Mozilla Hacks. Make sure to also read the comments – lots of good stuff there.

Indeed a very good blog post to read. Then Chris went on saying:

Web Components are a great idea. Modules are a great idea. Together, they bring us hours and hours of fun debating where what should be done to create a well-performing, easy to maintain and all around extensible complex app for the web.

This is twitching in the back of my mind for the last couple of weeks. And I kind of remember a wicked pattern from 10 years ago. Enter Compound Document Formats (CDF) with its WICD (read wicked) specifications. If you think I'm silly, check the CDF FAQ:

When combining content from arbitrary sources, a number of problems present themselves, including how rendering is handled when crossing from one markup language to another, or how events propagate across the same boundaries, or how to interpret the meaning of a piece of content within an unanticipated context.

and

Simply put, a compound document is a mixture of content in any number of formats. Compound documents range from static (say, XHTML that includes a simple SVG illustration) to very dynamic (a full-fledged Web Application). A compound document may include its parts directly (such as when you include an SVG image in an XHTML file) or by reference (such as when you embed a separate SVG document in XHTML using an <object> element. There are benefits to both, and the application should determine which one you use. For instance, inclusion by reference facilitates reuse and eases maintenance of a large number of resources. Direct inclusion can improve portability or offline use. W3C will support both modes, called CDR ("compound documents by reference") and CDI ("compound documents by inclusion").

At that time, the Web and W3C, where full throttle on XML and namespaces. Now, the cool kids on the block are full HTML, JSON, polymers and JS frameworks. But if you look carefully and remove the syntax, architecture parts, the narrative is the same. And with the narratives of the battle and its scars, the Web Components sound very familiar to the Coupound Document Format.

Still by Chris

When it comes to componentising the web, the rabbit hole is deep and also a maze.

Note that not everything was lost from WICD. It helped develop a couple of things, and reimagine the platform. Stay tune, I think we will have surprises on this story. Not over yet.

Modularity has already a couple of scars when thinking about large distribution. Remember Opendoc and OLE. I still remember using Cyberdog. Fun times.

Otsukare!

Planet MozillaOver the Edge: Web Components are an endangered species

Last week I ran the panel and the web components/modules breakout session of the excellent Edge Conference in London, England and I think I did quite a terrible job. The reason was that the topic is too large and too fragmented and broken to be taken on as a bundle.

If you want to see the mess that is the standardisation effort around web components right now in all its ugliness, Wilson Page wrote a great post on that on Mozilla Hacks. Make sure to also read the comments – lots of good stuff there.

Web Components are a great idea. Modules are a great idea. Together, they bring us hours and hours of fun debating where what should be done to create a well-performing, easy to maintain and all around extensible complex app for the web. Along the way we can throw around lots of tools and ideas like NPM and ES6 imports or – as Alex Russell said it on the panel: “tooling will save you”.

It does. But that was always the case. When browsers didn’t support CSS, we had Dreamweaver to create horribly nested tables that achieved the same effect. There is always a way to make browsers do what we want them to do. In the past, we did a lot of convoluted things client-side with libraries. With the advent of node and others we now have even more environments to innovate and release “not for production ready” impressive and clever solutions.

When it comes to componentising the web, the rabbit hole is deep and also a maze. Many developers don’t have time to even start digging and use libraries like Polymer or React instead and call it a day and that the “de facto standard” (a term that makes my toenails crawl up – layout tables were a “de facto standard”, so was Flash video).

React did a genius thing: by virtualising the DOM, it avoided a lot of the problems with browsers. But it also means that you forfeit all the good things the DOM gives you in terms of accessibility and semantics/declarative code. It simply is easier to write a <super-button> than to create a fragment for it or write it in JavaScript.

Of course, either are easy for us clever and amazing developers, but the fact is that the web is not for developers. It is a publishing platform, and we are moving away from that concept at a ridiculous pace.

And whilst React gives us all the goodness of Web Components now, it is also a library by a commercial company. That it is open source, doesn’t make much of a difference. YUI showed that a truckload of innovation can go into “maintenance mode” very quickly when a company’s direction changes. I have high hopes for React, but I am also worried about dependencies on a single company.

Let’s rewind and talk about Web Components

Let’s do away with modules and imports for now, as I think this is a totally different discussion.

I always loved the idea of Web Components – allowing me to write widgets in the browser that work with it rather than against it is an incredible idea. Years of widget frameworks trying to get the correct performance out of a browser whilst empowering maintainers would come to a fruitful climax. Yes, please, give me a way to write my own controls, inherit from existing ones and share my independent components with other developers.

However, in four years, we haven’t got much to show.. When we asked the very captive and elite audience of EdgeConf about Web Components, nobody raised their hand that they are using them in real products. People either used React or Polymer as there is still no way to use Web Components in production otherwise. When we tried to find examples in the wild, the meager harvest was GitHub’s time element. I do hope that this was not all we wrote and many a company is ready to go with Web Components. But most discussions I had ended up the same way: people are interested, tried them out once and had to bail out because of lack of browser support.

Web Components are a chicken and egg problem where we are currently trying to define the chicken and have many a different idea what an egg could be. Meanwhile, people go to chicken-meat based fast food places to get quick results. And others increasingly mention that we should hide the chicken and just give people the eggs leaving the chicken farming to those who also know how to build a hen-house. OK, I might have taken that metaphor a bit far.

We all agreed that XHTML2 sucked, was overly complicated, and defined without the input of web developers. I get the weird feeling that Web Components and modules are going in the same direction.

In 2012 I wrote a longer post as an immediate response to Google’s big announcement of the foundation of the web platform following Alex Russell’s presentation at Fronteers 11 showing off what Web Components could do. In it I kind of lamented the lack of clean web code and the focus on developer convenience over clarity. Last year, I listed a few dangers of web components. Today, I am not too proud to admit that I lost sight of what is going on. And I am not alone. As Wilson’s post on Mozilla Hacks shows, the current state is messy to say the least.

We need to enable web developers to use “vanilla” web components

What we need is a base to start from. In the browser and in a browser that users have and doesn’t ask them to turn on a flag. Without that, Web Components are doomed to become a “too complex” standard that nobody implements but instead relies on libraries.

During the breakout session, one of the interesting proposals was to turn Bootstrap components into web components and start with that. Tread the cowpath of what people use and make it available to see how it performs.

Of course, this is a big gamble and it means consensus across browser makers. But we had that with HTML5. Maybe there is a chance for harmony amongst competitors for the sake of an extensible and modularised web that is not dependent on ES6 availability across browsers. We’re probably better off with implementing one sci-fi idea at a time.

I wished I could be more excited or positive about this. But it left me with a sour taste in my mouth to see that EdgeConf, that hot-house of web innovation and think-tank of many very intelligent people were as confused as I was.

I’d love to see a “let’s turn it on and see what happens” instead of “but, wait, this could happen”. Of course, it isn’t that simple – and the Mozilla Hacks post explains this well – but a boy can dream, right? Remember when using HTML5 video was just a dream?

Planet MozillaCSS Working Group's future

Hello everyone.

Back in March 2008, I was extremely happy to announce my appointment as Co-chairman of the CSS Working Group. Seven years and a half later, it's time to move on. There are three main reasons to that change, that my co-chair Peter and I triggered ourselves with W3C Management's agreement:

  1. We never expected to stay in that role 7.5 years. Chris Lilley chaired the CSS Working Group 1712 days from January 1997 (IIRC) to 2001-oct-10 and that was at that time the longest continuous chairing in W3C's history. Bert Bos chaired it 2337 days from 2001-oct-11 to 2008-mar-05. Peter and I started co-chairing it on 2008-mar-06 and it will end at TPAC 2015. That's 2790 days so 7 years 7 months and 20 days! I'm not even sure those 2790 days hold a record, Steven Pemberton probably chaired longer. But it remains that our original mission to make the WG survive and flourish is accomplished, and we now need fresher blood. Stability is good, but smart evolution and innovation are better.
  2. Co-chairing a large, highly visible Working Group like the CSS Working Group is not a burden, far from it. But it's not a light task either. We start feeling the need for a break.
  3. There were good candidates for the role, unanimously respected in the Working Group.

So the time has come. The new co-chairs, Rossen Atanassov from Microsoft and Alan Stearns from Adobe, will take over during the Plenary Meeting of the W3C held in Sapporo, japan, at the end of October and that is A Good Thing™. You'll find below a copy of my message to W3C.

To all the people I've been in touch with while holding my co-chair's hat: thank you, sincerely and deeply. You, the community around CSS, made everything possible.

Yours truly.

Dear Tim, fellow ACs, fellow Chairs, W3C Staff, CSS WG Members,

After seven years and a half, it's time for me to pass the torch of the CSS Working Group's co-chairmanship. 7.5 years is a lot and fresh blood will bring fresh perspectives and new chairing habits. At a time the W3C revamps its activities and WGs, the CSS Working Group cannot stay entirely outside of that change even if its structure, scope and culture are retained. Peter and I decided it was time to move on and, with W3M's agreement, look for new co-chairs.

I am really happy to leave the Group in Alan's and Rossen's smart and talented hands, I'm sure they will be great co-chairs and I would like to congratulate and thank them for accepting to take over. I will of course help the new co-chairs on request for a smooth and easy transition, and I will stay in the CSS WG as a regular Member.

I'd like to deeply thank Tim for appointing me back in 2008, still one of the largest surprises of my career!

I also wish to warmly thank my good friends Chris Lilley, Bert Bos and Philippe Le Hégaret from W3C Staff for their crucial daily support during all these years. Thank you Ralph for the countless transition calls! I hope the CSS WG still holds the record for the shortest positive transition call!

And of course nothing would have been possible without all the members of the CSS Working Group, who tolerated me for so long and accepted the changes we implemented in 2008, and all our partners in the W3C (in particular the SVG WG) or even outside of it, so thank you all. The Membership of the CSS WG is a powerful engine and, as I often say, us co-chairs have only been a drop of lubricant allowing that engine to run a little bit better, smoother and without too much abrasion.

Last but not least, deep thanks to my co-chair and old friend Peter Linss for these great years; I accepted that co-chair's role to partner with Peter and enjoyed every minute of it. A long ride but such a good one!

I am confident the CSS Working Group is and will remain a strong and productive Group, with an important local culture. The CSS Working Group has both style and class (pun intended), and it has been an honour to co-chair it.

Thank you.

</Daniel>

Planet MozillaHello, 38 beta, it's nice to meet you

And at long last the 38 beta is very happy to meet you too (release notes, downloads, hashes). Over the next few weeks I hope you and the new TenFourFox 38 will become very fond of each other, and if you don't, then phbbbbt.

There are many internal improvements to 38. The biggest one specific to us, of course, is the new IonPower JavaScript JIT backend. I've invested literally months in making TenFourFox's JavaScript the fastest available for any PowerPC-based computer on any platform, not just because every day websites lard up on more and more crap we have to swim through (viva Gopherspace) but also because a substantial part of the browser is written in JavaScript: the chrome, much of the mid-level plumbing and just about all those addons you love to download and stuff on in there. You speed up JavaScript, you speed up all those things. So now we've sped up many browser operations by about 11 times over 31.x -- obviously the speed of JavaScript is not the only determinant of browser speed, but it's a big part of it, and I think you'll agree that responsiveness is much improved.

JavaScript also benefits in 38 from a compacting, generational garbage collector (generational garbage collection was supposed to make 31 but was turned off at the last minute). This means recently spawned objects will typically be helplessly slaughtered in their tender youth in a spasm of murderous efficiency based on the empiric observation that many objects are created for brief usage and then never used again, reducing the work that the next-stage incremental garbage collector (which we spent a substantial amount of time tuning in 31 as you'll recall, including backing out background finalization and tweaking the timeslice for our slower systems) has to do for objects that survive this pediatric genocide. The garbage collector in 38 goes one step further and compacts the heap as well, which is to say, it moves surviving objects together contiguously in memory instead of leaving gaps that cannot be effectively filled. This makes both object cleanup and creation much quicker in JavaScript, which relies heavily on the garbage collector (the rest of the browser uses more simplistic reference counting to determine object lifetime), to say nothing of a substantial savings in memory usage: on my Quad G5 I'm seeing about 200MB less overhead with 48 tabs open.

I also spent some time working on font enumeration performance because of an early showstopper where sites that loaded WOFF fonts spun and spun and spun. After several days of tearing my hair out in clumps the problem turned out to be a glitch in reference counting caused by the unusual way we load platform fonts: since Firefox went 10.6+ it uses CoreText exclusively, but we use almost completely different font code based on the old Apple Type Services which is the only workable choice on 10.4 and the most stable choice on 10.5. ATS is not very fast at instantiating lots of fonts, to say the least, so I made the user font cache stickier (please don't read that as "leaky" -- it's sticky because things do get cleaned up, but less aggressively to improve cache hit percentage) and also made a global font cache where the font's attribute tag directory is cached browser-wide to speed up loading font tables from local fonts on your hard disk. Previously this directory was cached per font entry, meaning if the font entry was purged for re-enumeration it had to be loaded all over again, which usually happened when the browser was hunting for a font with a particular character. This process used to take about fifteen to twenty seconds for the 700+ font faces on my G5. With the global font cache it now takes less than two.

Speaking of showstoppers, here's an interesting one which I'll note here for posterity. nsChildView, the underlying system view which connects Cocoa/Carbon to Gecko, implements the NSTextInput protocol which allows it to accept Unicode input without (as much) mucking about with the Carbon Text Services Manager (Firefox also implements NSTextInputClient, which is the new superset protocol, but this doesn't exist in 10.4). To accept Unicode input, under the hood the operating system actually manipulates a special undocumented TSM input context called, surprisingly, NSTSMInputContext (both this and the undocumented NSInputContext became the documented NSTextInputContext in 10.6), and it gets this object from a previously undocumented method on NSView called (surprise again) inputContext. Well, turns out if you override this method you can potentially cause all sorts of problems, and Mozilla had done just that to handle complex text input for plugins. Under the 10.4 SDK, however, their code ended up returning a null input context and Unicode input didn't work, so since we don't support plugins anyhow the solution was just to remove it completely ... which took several days more to figure out. The moral of the story is, if you have an NSView that is not responding to setMarkedText or other text input protocol methods, make sure you haven't overridden inputContext or screwed it up somehow.

I also did some trivial tuning to the libffi glue library to improve the speed of its calls and force it to obey our compiler settings (there was a moment of panic when the 7450 build did not start on the test machines because dyld said XUL was a 970 binary -- libffi had seen it was being built on a G5 and "helpfully" compiled it for that target), backed out some portions of browser chrome that were converted to CoreUI (not supported on 10.4), and patched out the new tab tile page entirely; all new tabs are now blank, like they used to be in previous versions of Firefox and as intended by God Himself. There are also the usual cross-platform HTML5 and CSS improvements you get when we leap from ESR to ESR like this, and graphics are now composited off-main-thread to improve display performance on multiprocessor systems.

That concludes most of the back office stuff. What about user facing improvements? Well, besides the new blank tabs "feature," we have built-in PDF viewing as promised (I think you'll find this more useful to preview documents and load them into a quicker viewer to actually read them, but it's still very convenient) and Reader View as the biggest changes. Reader View, when the browser believes it can attempt it, appears in the address bar as a little book icon. Click on it and the page will render in a simplified view like you would get from a tool such as Readability, cutting out much of the extraneous formatting. This is a real godsend on slower computers, lemme tell ya! Click the icon again to go back. Certain pages don't work with this, but many will. I have also dragged forward my MP3 decoder support, but see below first, and we have prospectively landed Mozilla bug 1151345 to fix an issue with the application menu (modified for the 10.4 SDK).

You will also note the new, in-content preferences (i.e., preferences appears in a browser tab now instead of a window, a la, natch, Chrome), and that the default search engine is now Yahoo!. I have not made this default to anything else since we can still do our part this way to support MoCo (but you can change it from the preferences, of course).

I am not aware of any remaining showstopper bugs, so therefore I'm going ahead with the beta. However, there are some known issues ("bugs" or "features" mayhaps?) which are not critical. None of these will hold up final release currently, but for your information, here they are:

  • If you turn on the title bar, private browsing windows have the traffic light buttons in the wrong position. They work; they just look weird. This is somewhat different than issue 247 and probably has a separate, though possibly related, underlying cause. Since this is purely cosmetic and does not occur in the browser's default configuration, we can ship with this bug present but I'll still try to fix it since it's fugly (plus, I personally usually have the title bar on).

  • MP3 support is still not enabled by default because seeking within a track (except to the beginning) does not work yet. This is the last thing to do to get this support off the ground. If you want to play with it in its current state, however, set tenfourfox.mp3.enabled to true (you will need to create this pref). If I don't get this done by 38.0.2, the pref will stay off until I do, but the rest of it is working already and I have a good idea how to get this last piece functional.

  • I'm not sure whether to call this a bug or a feature, but scaling now uses a quick and dirty algorithm for many images and some non-.ico favicons apparently because we don't have Skia support. It's definitely lower quality, but it has a lot less latency. Images displayed by themselves still use the high-quality built-in scaler which is not really amenable to the other uses that I can tell. Your call on which is better, though I'm not sure I know how to go back the old method or if it's even possible anymore.

  • To reduce memory pressure, 31 had closed tab and window undos substantially reduced. I have not done that yet for 38 -- near as I can determine, the more efficient memory management means it is no longer worth it, so we're back to the default 10 and 3. See what you think.

Builders: take note that you will need to install a modified strip ("strip7") if you intend to make release binaries due to what is apparently a code generation bug in gcc 4.6. If you want to use a different (later) compiler, you should remove the single changeset with the gcc 4.6 compatibility shims -- in the current changeset pack it's numbered 260681, but this number increments in later versions. See our new HowToBuildRightNow38 for the gory details and where to get strip7.

Localizers: strings are frozen, so start your language pack engines one more time in issue 42. We'd like to get the same language set for 38 that we had for 31, and your help makes it possible. Thank you!

As I mentioned before, it's probably 70-30 against there being a source parity version after 38ESR because of the looming threat of Electrolysis, which will not work as-is on 10.4 and is not likely to perform well or even correctly on our older systems. (If Firefox 45, the next scheduled ESR, still allows single process operation then there's a better chance. We still need to get a new toolchain up and a few other things, though, so it won't be a trivial undertaking.) But I'm pleased with 38 so far and if we must go it means we go out on a high note, and nothing says we can't keep improving the browser ourselves separate from Mozilla after we split apart (feature parity). Remember, that's exactly what Classilla does, except that we're much more advanced than Classilla will ever be, and in fact Pale Moon recently announced they're doing the same thing. So if 38 turns out to be our swan song as a full-blooded Mozilla tier 3 port, that doesn't mean it's the end of TenFourFox as a browser. I promise! Meanwhile, let's celebrate another year of updates! PowerPC forever!

Finally, looking around the Power Mac enthusiast world, it appears that SeaMonkeyPPC has breathed its last -- there have been no updates in over a year. We will pour one out for them. On the other hand, Leopard Webkit continues with regular updates from Tobias, and our friendly builder in the land of the Rising Sun has been keeping up with Tenfourbird. We have the utmost confidence that there will be a Tenfourbird 38 in your hands soon as well.

Some new toys to play with are next up in a couple days.

Planet MozillaAnnouncing the Content Performance program

Introduction

Aaron Klotz, Avi Halachmi and I have been studying Firefox’s performance on Android & Windows over the last few weeks as part of an effort to evaluate Firefox “content performance” and find actionable issues. We’re analyzing and measuring how well Firefox scrolls pages, loads sites, and navigates between pages. At first, we’re focusing on 3 reference sites: Twitter, Facebook, and Yahoo Search.

We’re trying to find reproducible, meaningful, and common use cases on popular sites which result in noticeable performance problems or where Firefox performs significantly worse than competitors. These use cases will be broken down into tests or profiles, and shared with platform teams for optimization. This “Content Performance” project is part of a larger organizational effort to improve Firefox quality.

I’ll be regularly posting blog posts with our progress here, but you can can also track our efforts on our mailing list and IRC channel:

Mailing list: https://mail.mozilla.org/listinfo/contentperf
IRC channel: #contentperf
Project wiki page: Content_Performance_Program

Summary of Current Findings (June 18)

Generally speaking, desktop and mobile Firefox scroll as well as other browsers on reference sites when there is only a single tab loaded in a single window.

  • We compared Firefox vs Chrome and IE:
    • Desktop Firefox scrolling can badly deteriorate when the machine is in power-saver mode1 (Firefox performance relative to other browsers depends on the site)
    • Heavy activity in background tabs badly affects desktop Firefox’s scrolling performance1 (much worse than other browsers — we need E10S)
    • Scrolling on infinitely-scrolling pages only appears janky when the page is waiting on additional data to be fetched
  • Inter-page navigation in Firefox can exhibit flicker, similar to other browsers
  • The Firefox UI locks up during page loading, unlike other browsers (need E10S)
  • Scrolling in desktop E10S (with heavy background tab activity) is only as good as the other browsersn1 when Firefox is in the process-per-tab configuration (dom.ipc.processCount >> 1)

1 You can see Aaron’s scrolling measurements here: http://bit.ly/1K1ktf2

Potential scenarios to test next:

  • Check impact of different Firefox configurations on scrolling smoothness:
    • Hardware acceleration disabled
    • Accessibility enabled & disabled
    • Maybe: Multiple monitors with different refresh rate (test separately on Win 8 and Win 10)
    • Maybe: OMTC, D2D, DWrite, display & font scaling enabled vs disabled
      • If we had a Telemetry measurement of scroll performance, it would be easier to determine relevant characteristics
  • Compare Firefox scrolling & page performance on Windows 8 vs Windows 10
    • Compare Firefox vs Edge on Win 10
  • Test other sites in Alexa top 20 and during random browsing
  • Test the various scroll methods on reference sites (Avi has done some of this already): mouse wheel, mouse drag, arrow key, page down, touch screen swipe and drag, touchpad drag, touchpad two finger swipe, trackpoints (special casing for ThinkPads should be re-evaluated).
    • Check impact of pointing device drivers
  • Check performance inside Google web apps (Search, Maps, Docs, Sheets)
    • Examine benefits of Chrome’s network pre-fetcher on Google properties (e.g. Google search)
    • Browse and scroll simple pages when top Google apps are loaded in pinned tabs
  • Compare Firefox page-load & page navigation performance on HTTP/2 sites (Facebook & Twitter, others?)
  • Check whether our cache and pre-connector benefit perceived performance, compare vs competition

Issues to report to Platform teams

  • Worse Firefox scrolling performance with laptop in power-save mode
  • Scrolling Twitter feed with YouTube HTML5 videos is jankier in Firefox
  • bug 1174899: Scrolling on Facebook profile with many HTML5 videos eventually causes 100% CPU usage on a Necko thread + heavy CPU usage on main thread + the page stops loading additional posts (videos)

Tooling questions:

  • Find a way to to measure when the page is “settled down” after loading, i.e. time until last page-loading event. This could be measured by the page itself (similar to Octane), which would allow us to compare different browsers
  • How to reproduce dynamic websites offline?
  • Easiest way to record demos of bad Firefox & Fennec performance vs other browsers?

Decisions made so far:

  • Exclusively focus on Android 5.0+ and Windows 7, 8.1 & 10
  • Devote the most attention to single-process Nightly on desktop, but do some checks of E10S performance as well
  • Desktop APZC and network pre-fetcher are a long time away, don’t wait

IEBlogBuilding Flight Arcade: Behind the scenes with WebGL, WebAudio and GamePad API

During Microsoft’s //build conference day 2 keynote, we demonstrated some of the advancements in Microsoft Edge’s platform features with a new demo from Pixel Lab, Flight Arcade. Today I’d like to highlight some of the new Web technologies behind Flight Arcade and show developers how the demo came together behind the scenes.

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="390" src="http://www.youtube.com/embed/xyaq9TPmXrA?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="640"></iframe>

Flight Simulator as Inspiration

A flight demo seemed a natural fit to showcase the new platform features, bringing together 3D graphics, audio modulation and using a game pad to control the plane. Flight Simulator had a huge impact on the PC hardware ecosystem starting 30+ years ago, pushing the boundaries of early personal computers like the Apple II and Commodore 64 and demonstrating how the PC could be used for more than just spreadsheets.

When our team began development on Flight Arcade, they originally hoped to recreate as much of the Flight Simulator experience as possible, even going to the effort of accessing the original code and 3D assets. It’s fair to say that ambition far outstripped the time and resources we had to bring the demo together in time, as we realized the complexity of the original simulator code base. The original goal of complete simulation included modelling complex factors like wind simulation, voltage drop across on-board circuits, weight and many other aspects that impact real-world flight.

After evaluating the scope of the simulation problem, we decided instead to hone our focus to exhibit three major new features of the platform, and simplify gameplay to demonstrate those features more effectively.

Web Platform Features Highlighted

WebGL

The team used the popular 3D framework Babylon.JS to build the visual components of Flight Arcade. Some of the challenges they faced included how to lay out terrain,  build in heightmaps for relief, and lay down textures on top of the terrain map to make it look as realistic as possible while still working well in the browser. You can learn more about this process on the Flight Arcade detail page.

WebAudio API

Prior to the WebAudio API, developers and content authors were limited to using HTML audio tags to embed a sound file for playback within their page and then layout around the control.

The HTML5 WebAudio API takes audio on the web to another level, with powerful and easy to use APIs providing a broad spectrum of audio manipulation techniques.

Check out the team’s detailed breakdown of how the team used the WebAudio API to modulate engine sound and distort the flight instructors voice, with code samples so you can try out WebAudio for yourself.

GamePad API

To add more fidelity to the Arcade flying experience and showcase another web platform feature, the team used the GamePad API and wrote a helper class for other developers to use that maps the button and axis indices to the more familiar names as labeled on the Xbox controller. You can grab the open-sourced code and get a full breakdown of how we used the API with FlightArcade here.

More to Come

We’ll be bringing more tech demos to the web on the new Microsoft Edge Dev Test Drive site over the coming months as we continue to add more web platform features to Microsoft Edge. You can keep in touch on Twitter to find out about new projects, and check out our community projects on the Microsoft Edge Dev Site where we’re working to build a better, more interoperable web.

– Jason McConnell, Microsoft Edge Developer Relations

Steve Faulkner et alCurrent Standards Work at W3C

People regularly ask me what I am working on at the W3C, here is a run down of standards/guidance documents I am editing/co-editing, contributing to (note: I am only 1 of the people at The Paciello Group involved directly in standards development at the W3C)

HTML 5.1

This specification defines the 5th major version, first minor revision of the core language of the World Wide Web: the Hypertext Markup Language (HTML). Editing updates to and maintenance of (mainly) accessibility related advice and requirements, with an emphasis on information for web developers.

ARIA in HTML

HTML5.1 specification module defining the web developer rules  for the use of ARIA attributes on HTML 5.1 elements. It also defines requirements for Conformance Checking tools. In HTML 5.1 this spec replaces the web developer (author) conformance requirements in section 3.2.7 WAI-ARIA of the HTML5 spec (titled 3.2.7 WAI-ARIA and HTML Accessibility API Mappings in HTML 5.1).

HTML Accessibility API Mappings 1.0

Defines how user agents map HTML 5.1 elements and attributes to platform accessibility application programming interfaces (APIs).  this spec replaces (and extends to all HTML elements/attributes) the user agent implementation requirements in section 3.2.7 WAI-ARIA of the HTML5 Recommendation (titled 3.2.7 WAI-ARIA and HTML Accessibility API Mappings in HTML 5.1).

ARIA in SVG1.1

This specification defines the web developer (author) rules (conformance requirements) for the use of ARIA attributes on SVG 1.1 elements. It also defines the requirements for Conformance Checking tools.

Custom Elements

This specification describes the method for enabling the author to define and use new types of DOM elements in a document.

Editing the Custom Element Semantics section of the specification.

Best Practice

HTML5: Techniques for providing useful text alternatives

This document contains best practice guidance for authors of HTML documents on providing text alternatives for images.  Edited until October 2014, the bulk of this document is  included in the HTML5 and HTML 5.1 specifications under section 4.8.5.1 Requirements for providing text to act as an alternative for images where I continue to update and maintain.

Notes on using ARIA in HTML

This document is a practical guide for developers on how to add accessibility information to HTML elements using the Accessible Rich Internet Applications (ARIA) specification.

 

 

Planet MozillaMozilla Tech Speakers: A pilot for technical evangelism

The six-week pilot version of the Mozilla Tech Speakers program wrapped up at the end of May. We learned a lot, made new friends on several continents, and collected valuable practical feedback on how to empower and support volunteer Mozillians who are already serving their regional communities as technical evangelists and educators. We’ve also gathered some good ideas for how to scale a speaker program that’s relevant and accessible to technical Mozillians in communities all over the world. Now we’re seeking your input and ideas as well.

During the second half of 2015, we’ll keep working with the individuals in our pilot group (our pilot pilots) to create technical workshops and presentations that increase developer awareness and adoption of Firefox, Mozilla, and the Open Web platform. We’ll keep in touch as they submit talk proposals and develop Content Kits during the second half of the year, work with them to identify relevant conferences and events, fund speaker travel as appropriate, make sure speakers have access to the latest information (and the latest swag to distribute), and offer them support and coaching to deliver and represent!

Why we did it

Our aim is to create a strong community-driven technical speaker development program in close collaboration with Mozilla Reps and the teams at Mozilla who focus on community education and participation. From the beginning we benefited from the wisdom of Rosana Ardila, Emma Irwin, Soumya Deb, and other Mozillian friends. We decided to stand up a “minimum viable” program with trusted, invited participants—Mozillians who are active technical speakers and are already contributing to Mozilla by writing about and presenting Mozilla technology at events around the world. We were inspired by the ongoing work of the Participation Team and Speaker Evangelism program that came before us, thanks to the efforts of @codepo8, Shezmeen Prasad, and many others.

We want this program to scale and stay sustainable, as individuals come and go, and product and platform priorities evolve. We will incorporate the feedback and learnings from the current pilot into all future iterations of the Mozilla Tech Speaker program.

What we did

Participants met together weekly on a video call to practice presentation skills and impromptu storytelling, contributed to the MDN Content Kit project for sharing presentation assets, and tried out some new tools for building informative and inspiring tech talks.

Each participant received one session of personalized one-to-one speaker coaching, using “techniques from applied improvisation and acting methods” delivered by People Rocket’s team of coaching professionals. For many participants, this was a peak experience, a chance to step out of their comfort zone, stretch their presentation skills, build their confidence, and practice new techniques.

In our weekly meetings, we worked with the StoryCraft technique, and hacked it a little to make it more geek- and tech speaker-friendly. We also worked with ThoughtBox, a presentation building tool to “organize your thoughts while developing your presentation materials, in order to maximize the effectiveness of the content.” Dietrich took ThoughtBox from printable PDF to printable web-based form, but we came to the conclusion it would be infinitely more usable if it were redesigned as an interactive web app. (Interested in building this? Talk to us on IRC. You’ll find me in #techspeakers or #devrel, with new channels for questions and communication coming soon.)

We have the idea that an intuitive portable tool like ThoughtBox could be useful for any group of Mozillians anywhere in the world who want to work together on practicing speaking and presentation skills, especially on topics of interest to developers. We’d love to see regional communities taking the idea of speaker training and designing the kind of programs and tools that work locally. Let’s talk more about this.

What we learned

The pilot was ambitious, and combined several components—speaker training, content development, creating a presentation, proposing a talk—into an aggressive six-week ‘curriculum.’ The team, which included participants in eight timezones, spanning twelve+ hours, met once a week on a video call. We kicked off the program with an introduction by People Rocket and met regularly for the next six weeks.

Between scheduled meetings, participants hung out in Telegram, a secure cross-platform messaging app, sharing knowledge, swapping stickers (the virtual kind) and becoming friends. Our original ambitious plan might have been feasible if our pilots were not also university students, working developers, and involved in multiple projects and activities. But six weeks turned out to be not quite long enough to get it all done, so we focused on speaking skills—and, as it turned out, on building a global posse of talented tech speakers.

What’s next

We’re still figuring this out. We collected feedback from all participants and discovered that there’s a great appetite to keep this going. We are still fine-tuning some of the ideas around Content Kits, and the first kits are becoming available for use and re-use. We continue to support Tech Speakers to present at conferences organize workshops and trainings in their communities. And create their own Mozilla Tech Speakers groups with local flavor and focus.

Stay tuned: we’ll be opening a Discourse category shortly, to expand the conversation and share new ideas.

And now for some thank yous…

I’d like to quickly introduce you to the Mozilla Tech Speakers pilot pilots. You’ll be hearing from them directly in the days, weeks, months ahead, but for today, huge thanks and hugs all around, for the breadth and depth of their contributions, their passion, and the friendships we’ve formed.

Adrian Crespo, Firefox Marketplace reviewer, Mozilla Rep, student, and technical presenter from Madrid, Spain, creator of the l10n.js Content Kit, for learning and teaching localization through the native JavaScript method.

Ahmed Nefzaoui, @AhmedNefzaoui, recent graduate and active Mozillian, Firefox OS contributor, Arabic Mozilla localizer, RTL (right-to-left) wizard, and web developer from Tozeur, Tunisia.

Andre Garzia, @soapdog, Mozilla Rep from Rio de Janeiro, Brazil, web developer, app developer and app reviewer, who will be speaking about Web Components at Expotec at the end of this month. Also, ask him about the Webmaker team LAN Houses program just getting started now in Rio.

Andrzej Mazur, @end3r, HTML5 game developer, active Hacks blog and MDN contributor, creator of a content kit on HTML5 Game Development for Beginners, active Firefox app developer, Captain Rogers creator, and frequent tech speaker, from Warsaw, Poland.

István “Flaki” Szmozsánszky, @slsoftworks, Mozillian and Mozilla Rep, web and mobile developer from Budapest, Hungary. Passionate about Rust, Firefox OS, the web of things. If you ask him anything “mildly related to Firefox OS, be prepared with canned food and sleeping bags, because the answer might sometimes get a bit out of hand.”

Kaustav Das Modak, @kaustavdm, Mozilla Rep from Bengalaru, India; web and app developer; open source evangelist; co-founder of Applait. Ask him about Grouphone. Or, catch his upcoming talk at the JSChannel conference in Bangalore in July.

Michaela R. Brown, @michaelarbrown, self-described “feisty little scrapper,” Internet freedom fighter, and Mozillian from Michigan. Michaela will share skills in San Francisco next week at the Library Freedom Project: Digital Rights in Libraries event.

Rabimba Karanjai, @rabimba, a “full-time graduate researcher, part-time hacker and FOSS enthusiast,” and 24/7 Mozillian. Before the month is out, Rabimba will speak about Firefox OS at OpenSourceBridge in Portland and at the Hong Kong Open Source conference.

Gracias. شكرا. धन्यवाद. Köszönöm. Obrigada. Dziękuję. Thank you. #FoxYeah.

Bruce LawsonReading List

Planet MozillaSource Maps Are An Insufficient Debugging Format For The Web

I want exceptional debugging support for programs compiled to JavaScript (or wasm) on the web. I want first class developer tools support for these programs. Unfortunately, this isn't the case right now because source maps are currently insufficient.

When boiled down to their essence, stepping debuggers enable users to do two things:

  1. set breakpoints and incrementally step through their program's execution, and
  2. inspect the values of variables and function parameters as they step through their program's execution.

A debugging format encodes information about a program that is lost at compile time, enabling source-level debugging of the compiled program. Source maps — the sole debugging format for compilers targeting JavaScript — can reconstruct source-level location information (filename, line, and column) which enables (1). However, that is the only data source maps encode, and the source map format does not attempt to solve (2). The result is that debugging programs compiled to JavaScript is positively maddening.

The good news is that we can overcome this limitation and extend the source map format to support reconstructing source-level scopes, bindings, and values. Before we do that, we should understand how mature debugging formats encode this information and how it is in turn consumed by debuggers. The prevailing debugging data format for *nix-based systems today is DWARF. It supersedes earlier ad-hoc formats, such as stabs. DWARF's designers worked with earlier formats for years, and used that experience as the foundation upon which DWARF was built. What follows is an exploration of how DWARF encodes scope and binding debugging information while keeping an eye out for lessons we can learn and apply when extending source maps.

Compact Binary Format

DWARF uses a compact binary record format to store structured information, conceptually similar to Cap'n Proto or Message Pack serialization formats. These Debugging Information Entries, or DIEs, are made up of a type tag, a set of attribute type and attribute value pairs, and an optional set of child DIEs. With these children, DIEs form a tree structure.

<DIE tag 1>
    <attribute name and type 1>
    <attribute value 1>
    <attribute name and type 2>
    <attribute value 2>
    <attribute name and type 3>
    <attribute value 3>

<DIE tag 1>
    <attribute name and type 1>
    <attribute value 1>
    <attribute name and type 2>
    <attribute value 2>
    <attribute name and type 3>
    <attribute value 3>

<DIE tag 1>
    <attribute name and type 1>
    <attribute value 1>
    <attribute name and type 2>
    <attribute value 2>
    <attribute name and type 3>
    <attribute value 3>

        .
        .
        .

Furthermore, the metadata describing common attribute shapes can be factored out and de-duplicated with abbreviations. By using abbreviations, that previous series of DIEs can be compressed into this:

<abbreviation definition 1>
    <attribute name and type 1>
    <attribute name and type 2>
    <attribute name and type 3>

<DIE abbreviation tag 1>
    <attribute value 1>
    <attribute value 2>
    <attribute value 3>

<DIE abbreviation tag 1>
    <attribute value 1>
    <attribute value 2>
    <attribute value 3>

<DIE abbreviation tag 1>
    <attribute value 1>
    <attribute value 2>
    <attribute value 3>

        .
        .
        .

With abbreviations, none of the attribute name and type metadata is repeated. This helps keep the debugging data compact. Debugging data can be huge in practice. Source maps can already be many megabytes in size, and the format already strives for as much de-duplication of data it can and only encodes limited debugging information as it is. This issue is exacerbated for source maps, because they are sent over the network and downloaded by debuggers. We must go to great lengths to keep debugging formats compact.

Additionally, because the set of attributes a DIE contains is not fixed, the format is future extensible. New attributes can be added to provide new debugging information that wasn't previously encoded. If a debugger does not know how to interpret a given attribute, it can skip and ignore it. This is a huge deal for the web, where we tend to take many baby steps rather than a few giant leaps at a time. Any extension to the source map format must strive to remain future extensible.

Lexical Scopes and Bindings

DWARF encodes data about the program's source-level scopes, bindings, and values in a DIE tree.

Consider this example program, fact.c:

#include "stdio.h"

static const int FACT_N = 6;

int main(int argc, char **argv) {
    int i = FACT_N;
    int result = 1;

    while (i > 0) {
        result *= i;

        int decrement = 1;
        i -= decrement;
    }

    printf("%d\n", result);

    return 0;
}

Within this source, there are three scopes we are interested in: the top level, the main function, and the block within the while loop. These scopes nest to form a tree (it would be a more interesting tree with a more complicated example program). DWARF encodes this tree directly with DIEs: scopes have their variables as child DIEs, and nested scopes are also child DIEs of their parent scope's DIE. Different kinds of bindings, such as formal parameters vs. local variables vs. constants, are declared with separate DIE type tags.

Below is a trimmed down subset of the information clang -g fact.c provides to debuggers via DWARF. It depicts the outline of the scope tree and the various bindings within those scopes. You can use the dwarfdump command line tool to view the text representation of the binary DIEs.

0x00000026:     TAG_variable [2]
                 AT_name( "FACT_N" )

0x0000003e:     TAG_subprogram [5] *
                 AT_name( "main" )

0x0000005e:         TAG_formal_parameter [6]
                     AT_name( "argc" )

0x0000006c:         TAG_formal_parameter [6]
                     AT_name( "argv" )

0x0000007a:         TAG_variable [7]
                     AT_name( "i" )

0x00000088:         TAG_variable [7]
                     AT_name( "result" )

0x00000096:         TAG_lexical_block [8] *

0x000000a7:             TAG_variable [7]
                         AT_name( "decrement" )

0x000000b5:             NULL

0x000000b6:         NULL

0x000000c8:     NULL

Additionally, the start and end target location bounds of these scopes are embedded as well. These bounds are denoted with the AT_low_pc and AT_high_pc DIE attributes. When the debuggee pauses at a given target location, the debugger finds the innermost scope containing the current target location, and can walk up the DIE tree to find every binding that is in scope.

Here is the DWARF data describing the bounds for the main function, and the while loop's block:

0x0000003e:     TAG_subprogram [5] *
                 AT_name( "main" )
                 AT_low_pc( 0x0000000100000f00 )
                 AT_high_pc( 0x0000000100000f75 )

0x00000096:         TAG_lexical_block [8] *
                     AT_low_pc( 0x0000000100000f31 )
                     AT_high_pc( 0x0000000100000f54 )

Locating a Binding's Value

It is not enough to list the names of bindings that are in scope when paused during the program's execution. A debugger must also provide the current values of those bindings so that the user may inspect them for irregularities.

Often, a binding's value is simply at some constant offset from the frame pointer. Other times it is in a specific register. Other times, the situation is a more complicated combination of these cases: for the first n instructions of this function, the parameter's value is located in this register, for the rest it is at that constant offset from the frame pointer. Even other times, a value is not located in one contiguous location! An optimizing compiler is free to explode a struct's members and store one member in a register, another member at this offset from the stack pointer, and to avoid even instantiating a value for the last member because that member is never used by the program.

To handle all of these cases, DWARF employs "location descriptions". These descriptions are expressed in stack-based operations that form a Turing-complete language.

JavaScript does not have frame pointers or registers. It has local, global, lexical, and dynamic (via this) bindings. It has objects and arrays with properties and indices. Because of that mismatch, it doesn't make sense for the source map format to adopt DWARF's location descriptions. Luckily, JavaScript engines already implement a sandboxed language for accessing and manipulating JavaScript values: JavaScript. We can embed snippets of JavaScript within the debugging data to locate a binding's value.

Conclusions

We can extend the source map format to encode the source-level lexical environment, its scopes and bindings, and a way to locate those binding's values. This enables users compiling their programs to JavaScript to do the other half of their debugging: inspecting variables and values as they step through their programs instead of stepping blindly.

However, we must ensure that source maps remain compact for quick downloads and parse times. The format should also be future extensible so we can add additional features down the line and don't need to reach consensus on everything all at once.

Finally, we should follow DWARF's lead and encode the scopes as a tree and embed within each scope its set of bindings, their symbol names, the scope's location boundaries, types, etc. We can reuse JavaScript as our language for describing the location of a binding's value.

If you are interested in being a part of these discussions, please join in at the Source Map RFC Repository.

Thanks to Tom Tromey and Dave Herman for reading drafts.

References

IEBlogNew Microsoft Edge Test Drive web demos go open-source

We first launched the Internet Explorer Test Drive site in 2010 during the Internet Explorer 9 Preview and regularly added demos through IE10 and IE11 development as a way to help web developers get hands-on with new features and performance improvements that were being added to the browser. Test Drive also exhibited polished end-to-end experiences showing the best of what the modern web is capable of with showcase sites like Atari Arcade, CyArk, 22Tracks, and Assassin’s Creed Pirates.

Screenshot of the Internet Explorer 9 Test Drive site

Internet Explorer 9 Test Drive circa 2010

As we rapidly approach the July 29th release of Windows 10 and Microsoft Edge, we wanted to get in the spirit of “a break from the past” by launching a new Test Drive as part of the new Microsoft Edge Developer Site.

Screenshot of the new Microsoft Edge Test Drive Sites and Demos

The new Microsoft Edge Test Drive Sites and Demos

A core focus of Microsoft Edge is delivering greater interoperability across browsers, so the web just works for users on any device and on any platform. The new Test Drive supports this goal by providing web developers with feature demos that demonstrate how to build interoperable code based on new web platform features. To make this process as easy as possible, we are excited to open-source all our feature demos on GitHub, so anyone can learn and reuse this code in any website (or contribute fixes!).

The new Test Drive site has a few of the most popular demos migrated from the old site (such as Touch Effects and Mandlebrot) but also introduces a number of new demos:

  • CSS3 Filters: Learn the different types of filters and even how to animate them using @keyframes
  • Music Lounge: Create your own composition using Web Audio and touch input, using BabylonJS
  • Web Audio tuner: Chromatic tuner built using Web Audio API and Media Capture API
  • Math APIs: Bypass the limits of number precision in JavaScript with the new precision API
  • PhotoCapture: Learn how to select cameras and take pictures with the browser using the Media Capture API
  • Microphone: Record, play, save and load streaming audio acquired using Media Capture API; then apply Web Audio visualizations, effects, and filters to stream data
  • @Supports: Test whether certain CSS properties and values are supported

The site also features links to a number of showcase sites that we have collaborated with partners to build, including Flight Arcade and the recently announced Dolby Audio Experience.

This is just the beginning – in the next few months you can expect to see more demos lighting up on Test Drive as we add new features to Microsoft Edge, so be sure to check the website often or star the project to be notified on GitHub.

If you are curious about the demos or interested in contributing, please reach out to us on GitHub or Twitter.

Antón Molleda, Program Manager, Microsoft Edge

Bruce LawsonProgressive Apps and bikeshedding with Alex Russell

In the corridor track at Velocity Conference, Santa Clara, I had a natter with international glamourpuss Alex Russell about Installable Web Apps/ HTML Manifest and Opera’s forthcoming implementation on Android. Alex has elegantly written his thoughts down in Progressive Apps: Escaping Tabs Without Losing Our Soul. Go and read it, because he’s absolutely right, except for one thing.

The name. He was so close:

Frances called them “Progressive Open Web Apps” and we both came around to just “Progressive Apps”. They existed before, but now they have a name.

“Progressive Open Web Apps” makes the satisfying acronym “POWA”. Enthused and infused with the mightiness of HTML App Manifest, Add to Homescreen and Web App Install banners, I went to the Marvel comics’ Superhero generator and gave them a mascot for the collective pusissance. Meet…The POWA-arranger:

POWA arranger

I jest, of course. Doesn’t matter to me what they’re called. A good user experience, and all the power of URLs and the Web is what matters to me. As Alex writes

Building immersive apps using web technology no longer requires giving up the web itself.

Stay tuned for an announcement about this functionality in Opera for Android. Product Manager Andreas Bovens will be leading a breakout session about installable web apps at Edge Conference London on 27th July. I’ll be there too, to straighten his tie, warm his microphone and shake my pom-poms.

Whether or not you’ll attend in person (it’ll be streamed and recorded) feel free to add questions or discussion points.

Planet MozillaJust Patch This! (or 101 for patching Firefox)

Contributing to Mozilla projects can be done in many ways. This week, a Webcompat bug about behavior differences in between browsers led me to propose a patch on Firefox and get it accepted. I learned a couple of things along the way. That might be useful for others.

The Webcompat Bug - Search Placeholder On Mobile Sites

It started with a difference of behavior in between Chrome and Firefox on Android devices for Nikkei Web site reported by kudodo from Mozilla Japan. I looked at the bug and checked the differences in Firefox 40 (Gecko 40) and Opera 30 (Blink/Chrome 43).

nikkei site in Gecko and Blink

Indeed the placeholder text appears in Gecko and not in Blink.

<form id="searchBlank" action="" method="get">
    <p><input id="searchBox" placeholder="検索" name="" type="search"></p>
</form>

What was happening? I started the inspectors in Gecko and in Blink looking at the CSS properties used in both cases. The CSS didn't have any WebKit differences (for once).

On the Opera/Blink inspector, I noticed in the inherited properties from the user agent stylesheet (aka the default style that any Web page inherits from browsers).

Blink inspector

Gecko didn't have the box-sizing: border-box; for input[type=search] in its default CSS.

input[type=search] {box-sizing: border-box;}

I tested in the inspector adding it to the stylesheet of the Web site, and noticed it was indeed fixing the issue in Gecko without creating an issue in Blink. Good! We need to contact the Web site so they can modify their CSS.

Gecko/Blink Default Rendering Differences

That said, fixing this particular Web site will not fix all the other sites on the Web which might exhibit the same issue. We also need to advocate for the same default rendering in Gecko (Firefox), Blink (Chrome, Opera, UCWeb, …), WebKit (Safari), Spartan (IE) and Edge (new MS browser). This led me to two questions:

  1. What is the most common behavior across browsers?
  2. Do Mozilla have a bug in bugzilla about this?

After testing, I found out that box-sizing: border-box; was the most common behavior for input[type=search] and that there was an open bug about input[type=search] rendering.

Patching Firefox User-Agent StyleSheet

I added a comment on the bug that it was creating a Web compatibility issue. Then I went on a journey to find where the user-agent stylesheet was in the source code repository of Mozilla (Quick Tip: You may want to search old bugs related to the same area in bugzilla, then you might find where you need to look at in the source code with previous patches). So I finally found that the user-agent stylesheet was managed in layout/style/ and that the file forms.css seemed to be pretty close from what I needed. I proposed a patch which was fixing the issue but was not perfect. I made mistakes and it's ok. Boris Zbarski and David Baron helped me to get it straight.

  • The forms.css file had trailing white-spaces. My first patched fixed both the spaces issue and added the missing property. Boris invited my to make two separate patches.
  • My first patch didn't contain a summary (aka commit message). I had done in the past patches with Mercurial for the UA override but I didn't know you could add a message at this stage. So David invited me to add proper messages on the patches.
  • One another mistake I did was to use "Removing" instead of "Remove". David said: We usually write commit messages with verbs rather than nouns, so "Remove trailing spaces".

Creating A Patch On Mozilla Central

I'm assuming here that

  1. you have already a copy of mozilla-central on your computer. See Get the source code.
  2. you have properly configured Mercurial for Mozilla.

These are the steps on the command line.

cd /your/path/mozilla-central
hg status
hg qpop -a
hg pull -u
cd layout/style
hg qnew bug1164783-spaces
# I'm using sublimetext. Choose a text editor.
subl forms.css
# checking if the diff makes sense
hg diff
# Adding a message to the patch
hg qrefresh -m "Bug 1164783 - Remove trailing spaces."
hg export qtip > ~/bug-1164783.patch
# creating another patch
hg qnew bug1164783-input
# doing the edits
hg diff
hg qrefresh -m "Bug 1164783 - Change default style for input type=search to box-sizing: border-box."
hg export qtip > ~/bug-1164783-input.patch

Once the modifications on the code are done and the patches are ready, you can attach the patch the bug on bugzilla.

Window for attaching bugs in bugzilla

  • Browse File for choosing the patch you created
  • select r? for the review and if you do not know who to ask for the review, there is usually a list of suggested reviewers.

If the the review is positive with r+ you can go on and add checkin-needed in the keywords of the bug. A committer will add your modifications to the next release of Firefox.

If the review is negative r-, read properly the comments which have been given in the review. Create another patch and attach it to the bug. At the same time, tick the box for the obsolete option on the previous patch.

The patch for input[search] should appear in Firefox 41.

Opening A Bug On HTML Specification

The input[type=search] default rendering was not defined on HTML specification. I opened a bug on W3C issue tracker so the specification could be fixed and if a group was implementing a new rendering engine, they would not have to reverse engineer the behavior of other browsers.

Otsukare!

Bruce LawsonIs HTML ‘complete’?

Tim Bray recently wrote an interesting post called </html> in which he stated

in­ter­est in work on “vocabulary” (by which they mean the ac­tu­al angle-bracketed thin­gies that go in­to HTML) seems pret­ty lack­ing.

Me, I think HTML is done. Which doesn’t mean I think that the whole Web-programming plat­form is in a good state…

Let’s down tools and fo­cus on more im­por­tant prob­lem­s.

I agree with Tim that fixing the web platform is more important right now than adding more elements to HTML. He cites fixing the things that jQuery, Backbone, Angular, Less, DART et al are trying to fix. Those too, but I’d also cite Service Workers, Web Manifest, device APIs as things that are urgently required to bring some level of feature-parity between web and native.

But I think it’s incorrect to claim HTML is finished. We’ve been there before, with HTML 4.01 published in December 1999 and then considered “finished”. HTML5 added lots more elements, some of which are well-used (20% of the top 100,000 sites use the HTML5 doctype, 12.% of those use <header>, for example).

Some of the HTML5 elements haven’t gained good traction. I’m inclined to agree with Matthew Thomas, who wrote (in 2004!) that new elements need to have some form of User Interface:

One way of improving this situation would be to reduce the number of new elements — forget about <article> and <footer>, for example.

Another way would be to recommend more distinct default presentation for each of the elements — for example, default <article> to having a drop cap, default <sidebar> to floating right, default <header>, <footer>, and <navigation> to having a slightly darker background than their parent element, and default <header>…<li> and <footer>…</li> to inline presentation. This would make authors more likely to choose the appropriate element.

Not every manifestation of UI is visual, however. There are still many gaps in the language that have to be patched with WAI-ARIA; on the webkit blog, James Craig writes

Many of the original features of ARIA (such as dialogs, landmarks, and menus) have been adopted into recent versions of HTML as elements. However, there are interaction patterns, common on the Web since the 1990s, that still have no native support or unreliable rendering support in HTML, such as date pickers, combo boxes, tab sets, data grids, tree controls, and more. Web developers must render all these controls themselves, using custom code or a JavaScript framework.

Steve Faulkner has a list of Aria roles and properties not available in HTML5. I’m not suggesting all should become elements or attributes, but it shows that what people make constantly outpaces the semantics available in HTML.

Since the WHATWG stopped adding new elements, we’ve seen the <main> element added to the language, which —although comparatively new— is used on 5% of the HTML5 sites in the top 100,000. Although it has no visual UI, it hooks into assistive technologies so that users can quickly get to the main content on a page.

Similarly, <picture> and associated responsive images attributes (srcset, sizes, x and w descriptors) have been added to the language.

Brian Kardell, Léonie Watson, and Steve Faulkner are working on a spec for a Panels and Panel Sets Extension that “defines elements and attributes for constructing a panel or collection of panels based on a single interaction paradigm.”

Elsewhere, attempts at adding other useful declarative features to the language are rebuffed. For example, <table sortable> was specified (with 9 years of anecdata from Stuart Langridge) to allow data tables to be natively sorted in the browser (a very common use-case), but implementation was rejected because “Instead of trying to bake so much into the platform someone should create a web components library that supports Hixie’s spec”.

Ah, Web Components – not ready for production any time soon, totally reliant on JavaScript, and so much harder for authors to implement than adding a single attribute.

I’m encouraged by the Extensible Web Manifesto which states

We prefer to enable feature development and iteration in JavaScript, followed by implementation in browsers and standardization… We want web developers to write more declarative code, not less.

Trialling stuff out in JavaScript, and then standardising it and adding it to core HTML/ CSS is the future of standardisation, I think. So, to declare HTML “finished” is, in my opinion, premature.

(Thanks to Steve Faulkner for crunching his dataset —ooh, matron— to provide use stats for <header> and <main>.)

W3C Team blogWeb and Digital Publishing Experts Converge at Digital Book Event

The Digital Publishing industry convened at the Javits Center in New York City last week for the Bookexpo America (BEA) trade show and the International Digital Publishing Forum’s (IDPF) Digital Book conference.

At Digital Book, IDPF Executive Director Bill McCoy and Book Industry Study Group (BISG) out-going Executive Director Len Vlahos welcomed more than 400 participants for plenary sessions the morning of 27 May. Over the next two days, representatives from the publishing community, including those from a dozen W3C member organizations and staff, participated in numerous Digital Book track sessions addressing current and future industry challenges and opportunities in business, education and technology.

W3C CEO Dr. Jeff Jaffe spoke on a panel, “The Current State of Book Industry Standards,” with standards organization executives Rob Abel, IMS Global Learning, Graham Bell, EDItEUR, Bill McCoy, IDPF, and Len Vlahos, BISG, moderated by Bill Kasdorf, Apex Covantage.

Jeff Jaffe in panel

Jeff Jaffe speaking on Executive Panel on “The Current State of Book Industry Standards”

In response to the moderator’s question why there are so many standards bodies in digital publishing, Jaffe first clarified that W3C’s mission is to develop global Web standards, and then added that the different standards organizations bring different models and perspectives that are healthy to have because they are indicators of the extensive transformation and innovation taking place in the publishing industry as a result of the Open Web Platform.

He suggested “A more exciting question is what is going to be the impact of these standards on new kinds of book forms, content innovation and business opportunities, similar to what we have seen in the entertainment industry.”

McCoy concurred that the convergence of publishing and the web provides a new rich environment and emphasized that “all this content has to interoperate in a way that it did not have to before.”

The vision for how this interoperability and new features are being addressed was the subject of the session, “The Convergence of EPUB and the Web,” moderated by Tzviya Siegman, Wiley. Siegman, who together with Markus Gylling, IDPF CTO and Daisy Consortium co-chair the W3C Digital Publishing Interest Group, were joined on the panel by Ivan Herman, W3C Digital Publishing Activity Lead.

Ivan Herman speaking

Ivan Herman speaking on “The Convergence of EPUB and the Web” panel with Markus Gylling and Tzviya Siegman

During the presentation Siegman explained that the current publishing standards format EPUB3, while based on foundational W3C technologies such as HTML5, CSS and SVG, does not yet have the full feature set of capabilities of the Web. Conversely the Web does not have the presentation features of EPUB such as pagination and other layout features. She explained the progress of the W3C’s Digital Publishing Interest Group to date in identifying specific use cases and requirements through the work of various task forces.

Gylling and Herman further explored the technical aspects of what is needed to achieve the vision of offline and online states of packaged or portable documents which they have begun to document in a white paper. They invited more industry participation in this conversation about EPUB and Web convergence.

The W3C Digital Publishing Interest Group is open to W3C member participation, but others may follow and comment on the work by joining the public mailing list: public-digipub-ig@w3.org

For more information about W3C’s standards activities in Digital Publishing, contact Ralph Swick, Ivan Herman or Karen Myers.

Planet MozillaThat one tweet…

One simple tweet made me feel terrible. One simple tweet made me doubt myself. One simple tweet – hopefully not meant to be mean – had a devastating effect on me. Here’s how and why, and a reminder that you should not be the person that with one simple tweet causes anguish like that.

you don't know the struggle someone had to go through to get where they are

Beep beep, I’m a roadrunner

As readers of this blog, you know that the last weeks have been hectic for me:

I bounced from conference to conference, delivering a new talk at each of them, making my slides available for the public. I do it because I care about people who can not get to the conference and for those I coach about speaking so they can re-use the decks if they wanted to. I also do a recording of my talks and publish them on YouTube so people can listen. Mostly I do these for myself, so I can get better at what I do. This is a trick I explained in the developer evangelism handbook – another service I provide for free.

Publishing on the go is damn hard:

  • Most Wi-Fi at events is flaky or very slow.
  • I travel world-wide which means I have no data on my phone without roaming and such.
  • Many times uploading my slides needs four to five attempts
  • Creating the screencast can totally drain the battery of my laptop with no power plug in sight.
  • Uploading the screencast can mean I do it over night.

The format of your slides are irellevant to these issues. HTML, Powerpoint, Keynote – lots of images means lots of bytes.

Traveling and presenting is tough – physical space still matters

Presenting and traveling is both stressful and taxing. Many people ask me how I do it and my answer is simply that: the positive feedback I get and seeing people improve when they get my advice is a great reward and keeps me going. The last few weeks have been especially taxing as I also need to move out of my flat. I keep getting calls by my estate agent that I need to wire money or be somewhere I can not. I also haven’t seen my partner more than a few hours because we are both busy.

I love my job. I still get excited to go to conferences, hear other presenters, listen to people’s feedback and help them out. A large part of my career is based on professional relationships that formed at events.

The lonely part of the rockstar life

Bill Murray in Lost in Translation

Being a public speaker means you don’t spend much time for yourself. At the event you sleep on average 4-5 hours as you don’t want to be the rockstar presenter that arrives, delivers a canned talk and leaves. You are there for the attendees, so you sacrifice your personal time. That’s something to prepare for. If you make promises, you also need to deliver them immediately. Any promise of you to look into something or contact someone you don’t follow up as soon as you can piles up to a large backlog you have a hard time remembering what it is you wanted to find out.

It also can make you feel very lonely. I’ve had many conversations with other presenters who feel very down as you are not with the people you care about, in the place you call home or in an environment you understand and feel comfortable in. Sure, hotels, airports and conference venues are all lush and have a “jet set” feel to them. They are also very nondescript and make you feel like a stranger.

Progressive Enhancement discussions happen and I can’t be part of it!

I care deeply about progressive enhancement. To me, it means you care more for the users of your products than you care about development convenience. It is a fundamental principal of the web, and – to me – the start of caring about accessibility.

In the last few weeks progressive enhancement was a hot topic in our little world and I wanted to chime in many a time. After all, I wrote training materials on this 11 years ago, published a few books on it and keep banging that drum. But, I was busy with the other events on my backlog and the agreed topics I would cover.

That’s why I was very happy when “at the frontend” came up as a speaking opportunity and I submitted a talk about progressive enhancement. In this talk, I explain in detail that it is not about the JavaScript on or off case. I give out a lot of information and insight into why progressive enhancement is much more than that.

That Tweet

Just before my talk, I uploaded and tweeted my deck, in case people are interested. And then I get this tweet as an answer:

Yehuda Katz: @codepo8, I can't see your slides without JavaScript

It made me angry – a few minutes before my talk. It made me angry because of a few things:

  • it is insincere – this is not someone who has trouble accessing my content. It is someone who wants to point out one flaw to have a “ha-ha you’re doing it wrong” moment.
  • the poster didn’t bother to read what I wrote at all – not even the blog post I wrote a few days before covering exactly the same topic or others explaining that the availability of JS is not what PE is about at all.
  • it judges the content of a publication by the channel it was published on. Zeldman wrote an excellent piece on this years ago how this is a knee-jerk reaction and an utter fallacy.
  • it makes me responsible for Slideshare’s interface – a service used my many people as a great place to share decks
  • it boils the topic I talked about and care deeply for down to a simple binary state. This state isn’t even binary if you analyse it and is not the issue. Saying something is PE because of JavaScript being available or not is the same technical nonsense that is saying a text-only version means you are accessible.

I posted the Zeldman article as an answer to the tweet and got reprimanded for not using any of the dozens available HTML slide deck versions that are progressively enhancing a document. Never mind that using keynote makes me more effective and helps me with re-use. I have betrayed the cause and should do better and feel bad for being such a terrible slide-creator. OK then. I shrugged this off before, and will again. But, this time, I was vulnerable and it hurt more.

Siding with my critics

I addition to me having lot of respect of what Yehuda achieved other people started favouriting the tweet. People I look up to, people I care about:

  • Alex Russel, probably one of the most gifted engineers I know with a vocabulary that makes a Thesaurus blush.
  • Michael Mahemoff, always around with incredibly good advice when HTML5 and apps where the question.
  • My colleague Jacob Rossi, who blows me away every single day with his insight and tech knowledge

And that’s when my anger turned inward and the ugly voice of impostor syndrome reared its head. Here is what it told me:

  • You’re a fool. You’re making a clown of yourself trying to explain something everyone knows and nobody gives a shit about. This battle is lost.
  • You’re trying to cover up your loss of reality of what’s needed nowadays to be a kick-ass developer by releasing a lot of talks nobody needs. Why do you care about writing a new talk every time? Just do one, keep delivering it and do some real work instead
  • Everybody else moved on, you just don’t want to admit to yourself that you lost track.
  • They are correct in mocking you. You are a hypocrite for preaching things and then violating them by not using HTML for a format of publication it wasn’t intended for.

I felt devastated, I doubted everything I did. When I delivered the talk I had so looked forward to and many people thanked me for my insights I felt even worse:

  • Am I just playing a role?
  • Am I making the lives of those who want to follow what I advocate unnecessarily hard?
  • Shouldn’t they just build things that work in Chrome now and burn them in a month and replace them with the next new thing?

Eventually, I did what I always do and what all of you should: tell the impostor syndrome voice in your head to fuck off and let my voice of experience ratify what I am doing. I know my stuff, I did this for a long time and I have a great job working on excellent products.

Recovery and no need for retribution

I didn’t have any time to dwell more on this, as I went to the next conference. A wonderful place where every presentation was full of personal stories, warmth and advice how to be better in communicating with another. A place with people from 37 countries coming together to celebrate their love for a product that brings them closer. A place where people brought their families and children although it was a geek event.

I’m not looking for pity here. I am not harbouring a grudge against Yehuda and I don’t want anyone to reprimand him. My insecurities and how they manifest themselves when I am vulnerable and tired are my problem. There are many other people out there with worse issues and they are being attacked and taken advantage of and scared. These are the ones we need to help.

Think before trying to win with a tweet

What I want though is to make you aware that everything you do online has an effect. And I want you to think next time before you post the “ha-ha you are wrong” tweet or favourite and amplify it. I want you to consider to:

  • read the whole publication before judging it
  • question if your criticism really is warranted
  • wonder how much work it was to publish the thing
  • consider how the author feels when the work is reduced to one thing that might be wrong.

Social media was meant to make media more social. Not to make it easier to attack and shut people up or tell them what you think they should do without asking about the how and why.

I’m happy. Help others to be the same.

Planet MozillaFirefox Platform Rendering – Current Work

I’m often asked “what are you working on?” Here’s a snapshot of some of the things currently on my teams’ front burners:

I’m surely forgetting a few things, but that’s a quick snapshot for now. Do you have suggestions for what Platform Rendering features we should pick up next? Add your comments below…

IEBlogAnnouncing Dolby Audio for high performance audio in Microsoft Edge

Web browser media experiences have advanced significantly in recent years, from the first releases of HTML5 media, through the subsequent additions of Media Source Extensions for adaptive streaming and Encrypted Media Extensions for content protection.  And formats have been similarly advanced by the development of MPEG DASH and Common Encryption (CENC).  These important specifications offered the potential for interoperable, plug-in free web video.  We referred to the resulting experience as Professional Quality Video when we first shipped support for these specs in Internet Explorer 11 on Windows 8.1.

Advancement in media experiences continues in Windows 10 and Microsoft Edge. Today, we are announcing that Microsoft Edge includes HTML5, MSE and EME support for a new audio format:  Dolby Audio™.  It allows websites to match the compelling visuals of H.264 video with equally compelling multi-channel audio.  It works well with AVC/H.264 video and also with our previously announced HLS and MPEG DASH Type 1 streaming features, which both support integrated playback of an HLS or DASH manifest.

Dolby Audio logo

High Performance Media Experiences

We added Dolby Audio to Microsoft Edge so that web apps and sites can deliver the richest and most compelling media experiences with Windows 10.  In particular, we wanted Microsoft Edge to support the most demanding requirements from media websites.  Supporting Dolby Audio fits with this goal.

Dolby Audio

Dolby Audio supports the multi-channel Dolby Digital Plus™ audio format that provides crisp, clear audio in a broad range of channel configurations.  Microsoft Edge is the first browser to include support for Dolby Digital Plus.  Websites will now be able to selectively support Dolby Audio as a premium sound experience, and use HTML5, MSE and EME capability checks to fall back to other supported formats on browsers other than Microsoft Edge.

Windows 10 includes support for the Dolby Digital Plus Portable Mode, which is a dynamic range control setting that is optimized for playback over built-in speakers and headphones. Dolby content will play back louder and clearer with a dynamic range more appropriate for portable devices. This improves the experience from Windows 8 where content could occasionally render softly over built-in speakers and headphones, and the advantages will apply equally to websites running in Microsoft Edge.

Dolby Audio is compatible with current DASH (MP4) and HLS (M2TS) file formats, and so can be included with progressive download HTML5 content, adaptive streaming HLS or MSE/EME MP4 content.

Usage Guidance

There are two adaptive streaming approaches for websites to take advantage of these new formats:

  1. Use either DASH Type 1 streaming (where the DASH manifest (.mpd) or HLS manifest (.m3u8) is set as the src for an HTML 5 media element). Using our DASH Type 1 implementation, it’s important to use the DASH live profile with MPD Type=”static”.
  2. Build a custom adaptive streaming solution using MSE APIs directly. This is more work, but provides the most control over streaming details.

Both the DASH Type 1 and full custom MSE approaches may be combined with Encrypted Media Extensions apply DRM protection.  In Microsoft Edge, this would require supporting PlayReady DRM.

Capabilities

Dolby Audio offers superior performance, but isn’t currently supported by other browsers.  Websites that use it should feature detect on the format and be prepared to stream alternative audio formats on systems that don’t support Dolby Audio.  Javascript examples to check for format support are listed below.

For HTML5:

  • Dolby EC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = myvideo.canPlayType(‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = myvideo.canPlayType(‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = myvideo.canPlayType(‘video/mp4; codecs=”avc3”’);

For MSE:

  • Dolby EC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MediaSource.isTypeSupported (‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MediaSource.isTypeSupported (‘video/mp4; codecs=”avc3″‘);

For EME:

  • Dolby EC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ec-3″‘);
  • Dolby AC-3: test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘audio/mp4; codecs=”ac-3″‘);
  • 264(AVC1): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc1.42E01E”‘);
  • 264(AVC3): test = MSMediaKeys.isTypeSupported (‘com.microsoft.playready’, ‘video/mp4; codecs=”avc3″‘);

Dolby Audio Experience Demo

Dolby has created a Dolby Audio Experience demo using WebGL to provide an interactive environment where users can explore and play content on a variety of media devices (a media PC, desktop, laptop and tablet).   Each device demonstrates Dolby Audio with a different streaming video, either MPEG DASH H.264 or HLS.  The demo currently requires Microsoft Edge, since it is the first browser to include native support for Dolby Audio.

Conclusion

The addition of Dolby Audio to Microsoft Edge is intended to open new opportunities for website media experiences.  It is supported only in Microsoft Edge now, but fallback strategies are possible that allow sites to provide the highest quality media experience when supported.

We look forward to your feedback, and hope to see websites using Dolby Audio soon!

– Jerry Smith, Senior Program Manager, Microsoft Edge

Footnotes

Updated: .  Michael(tm) Smith <mike@w3.org>