Abstract

This specification defines an API extending the HTMLMediaElement that enables controlling remote playback of media from a web page.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document was published by the Second Screen Presentation Working Group as a First Public Working Draft. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to public-secondscreen@w3.org (subscribe, archives). All comments are welcome.

This document is a work in progress and is subject to change. It builds on the group's experience on presenting web content on external presentation-type displays, and re-uses patterns and design considerations from the Presentation API specification whenever appropriate [PRESENTATION-API]. In case of issues or concerns, it is possible to file a bug or send an email to the mailing list. For small editorial changes like typos, sending a pull request is appreciated.

Publication as a First Public Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 1 September 2015 W3C Process Document.

1. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY, MUST, MUST NOT, and SHOULD NOT are to be interpreted as described in [RFC2119].

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Implementations that use ECMAScript to expose the APIs defined in this specification MUST implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL].

2. Introduction

This section is non-normative.

This specification aims to make remote playback devices such as connected TVs, projectors or audio-only speakers, available to the Web and takes into account playback devices that are attached using wired (HDMI, DVI, or similar) and wireless technologies (Miracast, Chromecast, DLNA, AirPlay, or similar).

Devices with limited screen size or quiet speakers lack the ability to playback media content to a larger audience, for example, a group of colleagues in a conference room, or friends and family at home. Playing media content on an external larger and/or louder remote playback device helps to improve the perceived quality and impact of the played media.

At its core, this specification enables a page that acts as the browsing context to initiate and control remote playback of a particular media element on a selected remote playback device. How the remoting is initiated and controlled is left to the UA in order to allow the use of remote playback devices that can be attached in a wide variety of ways. For example, when a remote playback device is attached using HDMI or Miracast, the same UA that acts as the browsing context renders the remote media. Instead of playing the media on the same device, however, it can use whatever means the operating system provides for using the external remote playback device. In such a case, both the browsing context and the media player run on the same UA and the operating system is used to route the player output to the remote playback device. This is commonly referred to as the media mirroring case. This specification imposes no requirements on the remote playback devices connected in such a manner.

If the remote playback device is able to play the media and communicate with the browsing context but is unable to fetch the media, the browsing context needs to fetch the media data and pass it on to the remote playback device for rendering. This is commonly referred to as media remoting case.

If the remote playback device is able to fetch and play the media and communicate with the browsing context, the browsing context does not need to fetch or render the remoted media. In this case, the UA acts as a proxy that requests the remote playback device to play the media itself by passing the necessary data like the media source. This is commonly referred to as the media flinging case. This way of attaching to displays could be enhanced in the future by defining a standard protocol for delivering these types of messages that remote playback devices could choose to implement.

The API defined here is intended to be used with UAs that attach to remote playback device devices through any of the above means.

3. Use cases and requirements

This section is non-normative.

The use cases and requirements of this specification are captured in a separate document available here.

4. Dependencies

The following concepts and interfaces are defined in [HTML]:

The term URL is defined in the WHATWG URL standard [URL].

The term throw in this specification is used as defined in [ WEBIDL].

The following exception names are defined by [WEBIDL] and used by this specification:

5. Examples

This section shows code examples that highlight the usage of the main features of the Remote Playback API. In these examples, player.html implements the player page controlling the remote playback and media.ext is the media file to be played remotely. Both the page and the media are served from the domain https://example.org. Please refer to the comments in the code examples for further details.

5.1 Monitor availability of remote playback devices example

Example 1
<!-- player.html -->
<!-- The video element with custom controls that supports remote playback. -->
<video id="videoElement" src="https://example.org/media.ext" />
<button id="deviceBtn" style="display: none;">Pick device</button>
<script>
  // The "Pick device" button is visible if at least one remote playback device is available.
  var deviceBtn = document.getElementById("deviceBtn");
  var videoElem = document.getElementById("videoElement");

  function availabilityCallback(available) {
    // Show or hide the device picker button depending on device availability.
    deviceBtn.style.display = available ? "inline" : "none";
  }

  videoElem.watchAvailability(availabilityCallback).catch(function() {
    // Availability monitoring is not supported by the platform, so discovery of
    // remote playback devices will happen only after remote.connect() is called.
    // Pretend the devices are available for simplicity; or, one could implement
    // a third state for the button.
    deviceBtn.style.display = "inline";
  });
</script>

5.2 Starting remote playback of a video example

Example 2
<!-- player.html -->
<script>
  devicesBtn.onclick = function() {
    // Request the user to select a remote playback device.
    videoElem.remote.prompt()
      // Update the UI and monitor the connected state.
      .then(updateRemotePlaybackState);
      // Otherwise, the user cancelled the selection UI or no screens were found.
  };
<script>

5.3 Monitoring remote playback state changes

Example 3
<!-- player.html -->
<script>
  // The remote playback may be initiated by the user agent,
  // so check the initial state to sync the UI with it.
  if (videoElem.remote.state == "disconnected")
    switchToLocalUI();
  else
    switchToRemoteUI();

  videoElem.remote.onconnecting = switchToRemoteUI;
  videoElem.remote.onconnect = swithToRemoteUI;
  videoElem.remote.ondisconnect = switchToLocalUI;

  // Handles both 'connecting' and 'connected' state. Calling more than once
  // is a no-op.
  function switchToRemoteUI() {
    // Indicate that the state is 'connecting' or 'connected' to the user.
    // For example, hide the video element as only controls are needed.
    videoElem.style.display = "none";

    // Stop monitoring the availability of remote playback devices.
    videoElem.remote.cancelWatchAvailability();
  };

  function switchToLocalUI() {
    // Show the video element.
    videoElem.style.display = "inline";
    // Start watching the device availability again.
    videoElem.remote.watchAvailability(availabilityCallback);
  };
<script>

6. API

6.1 Common idioms

A local playback device is the device the browsing context is running on along with the default video/audio outputs the device has.

Note
A local playback device might have extra outputs, like an external display or speakers/headphones. As long as the switch of what output to use happens outside of the user agent on the system level, the playback is considered to happen on a local playback device for the purpose of this spec.

A remote playback device is any other device but the local playback device that the browsing context can use to play media on.

A media element state is the set of all single media element properties observable by the page and/or the user via the user agent implementation. The new properties introduced by this spec are not considered part of the media element state for convenience.

For example, the paused attribute or the pause/resume button reflecting that state on the default controls of the media element would be a part of media element state.

A local playback state is the user agent implementation of media element state for the particular media element for playback on the local playback device.

A remote playback state is the user agent implementation of media element state for the particular media element for playback on the certain remote playback device.

Note

For a good user experience it is important that the media element state doesn't change unexpectedly when the state changes. It is also important that remote playback state is in sync with the media element state so when the media is paused on the remote playback device it looks paused to both the user and the page.

6.2 RemotePlayback interface

interface RemotePlayback : EventTarget {
    Promise<long> watchAvailability(RemotePlaybackAvailabilityCallback callback);
    Promise<void> cancelWatchAvailability(optional long id);

    readonly attribute RemotePlaybackState state;

             attribute EventHandler        onconnecting;
             attribute EventHandler        onconnect;
             attribute EventHandler        ondisconnect;

    Promise<void> prompt();
};

enum RemotePlaybackState {
    "connecting",
    "connected",
    "disconnected"
};

callback RemotePlaybackAvailabilityCallback = void (boolean available);

6.2.1 Observing remote playback device availability

A RemotePlaybackAvailabilityCallback is the way for the page to obtain the remote playback device availability for the correspodning media element. If the user agent can monitor the list of available remote playback devices in the background (without a pending request to prompt()), the RemotePlaybackAvailabilityCallback behavior defined below MUST be implemented by the user agent. Otherwise, the promise returned by watchAvailability() MUST be rejected with NotSupportedError.

6.2.1.1 The set of availability callbacks

The user agent MUST keep track of the set of availability callbacks registered with each media element through the watchAvailability() method. The set of availability callbacks for each RemotePlayback object is represented as a set of tuples (callbackId, callback), initially empty, where:

  1. callbackId is an id unique to the RemotePlayback object;
  2. callback is a RemotePlaybackAvailabilityCallback object;

Since there's one and only one RemotePlayback object per each media element, set of availability callbacks of a media element is the same set as the set of availability callbacks of the RemotePlayback object referred to by the element's remote property.

The combined set of all sets of availability callbacks of all RemotePlayback objects known to the browsing context is referred to as global set of availability callbacks.

6.2.1.2 The list of available remote playback devices

The user agent MUST keep a list of available remote playback devices. This list contains remote playback devices and is populated based on an implementation specific discovery mechanism. It is set to the most recent result of the algorithm to monitor the list of available remote playback devices.

The user agent MAY not support running the algorithm to monitor the list of available remote playback devices continuously, for example, because of platform or power consumption restrictions. In this case the promise returned by watchAvailability() MUST be rejected with NotSupportedError, the global set of availability callbacks will be empty and the algorithm to monitor the list of available remote playback devices will only run as part of the initiate remote playback algorithm.

When the global set of availability callbacks is not empty, the user agent MUST monitor the list of available remote playback devices continuously, so that pages can keep track of the last value received via the registered callbacks to offer remote playback only when there are available devices.

Note

User agents SHOULD NOT monitor the list of available remote playback devices when possible, to satisfy the power saving non-functional requirement. For example, the user agent MAY not run the monitoring algorithm when the global set of availability callbacks is empty or each page that has media elements with non-empty set of availability callbacks is not in the foreground.

Some remote playback devices may only be able to play a subset of media resources because of functional, security or hardware limitations. Examples are set-top boxes, smart TVs or networked speakers capable of rendering only certain formats of video and/or audio. We say that such a device is a compatible remote playback device for a media resource if the user agent can reasonably guarantee that the remote playback of the media specified by the resource will succeed on that device.

The media resources of a media element, that were considered by the user agent to find a compatible remote playback device, are called the availability sources set.

The media resource of a media element, that is used to initiate remote playback on the selected remote playback device is called remote playback source. Remote playback source MUST belong to availability sources set.

The mechanism of picking the availability sources set and the remote playback source is implementation-specific. For example, the user agent MUST either use the currentSrc of the media element for both availability monitoring and remote playback or use all the media resources associated with the media element as the availability sources set and pick one of the resources as the remote playback source after user selects the remote playback device.

Remote playback is said to be unavailable for the media element if the list of available remote playback devices is empty or none of them is compatible with any source from availability sources set for the media element. The remote playback is said to be available otherwise. A boolean set to false if the remote playback is unavailable for the media element or true if it is available is called availability for the media element.

6.2.1.3 Getting the remote playback devices availability information

When the watchAvailability() method is called, the user agent MUST run the following steps:

Input
callback, the callback that will get fired with availability information.
Output
promise, a promise.
  1. Let promise be a new promise.
  2. Return promise, and run the following steps below:
  3. If the disableRemotePlayback attribute is present for the media element, reject the promise with InvalidStateError and abort all the remaining steps.
  4. If the user agent is unable to monitor the list of available remote playback devices for the entire lifetime of the browsing context (for instance, because the user has disabled this feature), then run the following steps in parallel:
    1. Fulfill promise.
    2. Queue a task to invoke the callback with false as its argument.
    3. Abort all remaining steps.
  5. If the user agent is unable to continuously monitor the list of available remote playback devices but can do it for a short period of time when initiating remote playback, then:
    1. Reject promise with a NotSupportedError exception.
    2. Abort all remaining steps.
  6. Let callbackId be a number unique to the media element that will identify the callback.
  7. Create a tuple (callbackId, callback) and add it to the set of availability callbacks for this media element.
  8. Fulfill promise with the callbackId and run the following steps in parallel:
    1. Queue a task to invoke the callback with the current availability for the media element.
    2. Run the algorithm to monitor the list of available remote playback devices.
6.2.1.4 Monitoring the list of available remote playback devices

If the set of availability callbacks is non-empty, or there is a pending request to initiate remote playback, the user agent MUST monitor the list of available remote playback devices by running the following steps:

  1. Retrieve available remote playback devices (using an implementation specific mechanism) and let newDevices be this list.
  2. For each media element known to the browsing context:
    1. If the disableRemotePlayback attribute is present for mediaElement, abort all the remaining steps for this tuple and continue to the next one.
    2. Set newAvailabilityValue to the value of availability for the media element calculated using the newDevices list instead of the list of available remote playback devices.
    3. If the current availability is not equal to newAvailabilityValue, then for each (callbackId, callback) of the element's set of availability callbacks:
      1. Queue a task to invoke callback with newAvailabilityValue as its argument.
  3. Set the list of available remote playback devices to the value of newDevices.
6.2.1.5 Stop observing remote playback devices availability

When a cancelWatchAvailability() method is called, the user agent MUST run the following steps:

Input
id, the callback identifier.
Output
promise, a promise.
  1. Let promise be a new promise.
  2. Return promise, and run the following steps below:
  3. If the disableRemotePlayback attribute is present for the media element, reject promise with InvalidStateError and abort all the remaining steps.
  4. If the parameter id is undefined, clear the set of availability callbacks.
  5. Otherwise, if id matches the callbackId for any entry in the set of availability callbacks, remove the entry from the set.
  6. Otherwise, reject promise with NotFoundError and abort all the remaining steps.
  7. If the set of availability callbacks is now empty and there is no pending request to initiate remote playback, cancel any pending task to monitor the list of available remote playback devices for power saving purposes.
  8. Fulfill promise.
Note
The mechanism used to monitor remote playback devices availability and determine the compatibility of a remote playback device with the selected availability sources set is left to the user agent.

6.2.2 Prompt user for changing remote playback statee

When the prompt() method is called, the user agent MUST run the following steps:

Input
None, but the algorithm references the media element, its remote property and its availability sources set.
Output
A promise.
  1. Let promise be a new promise.
  2. Return promise and continue running these steps in parallel.
  3. If the disableRemotePlayback attribute is present for the media element, reject the promise with InvalidStateError and abort all the remaining steps.
  4. If there is already an unsettled promise from a previous call to prompt for the same media element or even for the same browsing context, the user agent MAY reject promise with an OperationError exception and abort all remaining steps.
    Note
    The rationale here is that the user agent might use the UI that's modal to either the media element or the browsing context. In such a case, the second call to prompt() would not be able to show any UI.
  5. OPTIONALLY, if the user agent knows a priori that showing the UI for this particular media element is not feasible, reject promise with a NotSupportedError and abort all remaining steps.
    Note
    An example of such scenario could be when the user agent only supports media flinging case while the media element's source is not a URL that could be passed over to any remote playback device.
  6. If the algorithm isn't allowed to show a popup, reject promise with an InvalidAccessError exception and abort these steps.
  7. If the user agent needs to show the list of available remote playback devices and is not monitoring the list of available remote playback devices, run the steps to monitor the list of available remote playback devices in parallel.
  8. If the list of available remote playback devices is empty and will remain so before the request for user permission is completed, reject promise with a NotFoundError exception and abort all remaining steps.
  9. If the state is disconnected and availability for the media element is false, reject promise with a NotSupportedError exception and abort all remaining steps.
  10. If the user picked a remote playback device device to initiate remote playback with, the user agent MUST run the following steps:
    1. Set the state of the remote object to connecting.
    2. Fulfill promise.
    3. Queue a task to fire a simple event with the name connecting at the remote property of the media element. The event must not bubble, must not be cancelable, and has no default action.
    4. Establish a connection with the remote playback device device for the media element.
    Note

    By picking a remote playback device the user grants permission to use the device.

  11. Otherwise, if the user chose to disconnect from the remote playback device device, the user agent MUST run the following steps:
    1. Fulfill promise.
    2. Run the disconnect from remote playback device algorithm for the device.
  12. Otherwise, the user is considered to deny permission to use the device, so reject promise with NotAllowedError exception and hide the UI shown by the user agent
Note
The details of implementing the UI and device selection are left to the user agent; for example it MAY show the user a dialog and allow the user to select an available device (granting permission), or cancel the selection (denying permission).
Note
The algorithm to select the remote playback source for a selected device depends on the user agent and supported remote playback device types. For example, in case of media mirroring the user agent can simply follow the HTMLMediaElement's resource selection algorithm. However, if media remoting or media flinging is used, the best media source can depend on the selected remote playback device fetch and playback capabilities.

6.2.3 The state attribute

The state attribute represents the RemotePlayback connection's current state. It can take one of the values of RemotePlaybackState depending on the connection state:

  • connecting means that the user agent is attempting to initiate remote playback with the selected remote playback device. This is the initial state when the promise returned by prompt() is fulfilled with true. The local playback of the media element continues in this state and media commands still take effect on the local playback state.
  • connected means that the transition from local to remote playback has finished and all media commands now take effect on the remote playback state.
  • disconnected means that the remote playback has not been initiated, has failed to initiate or has been stopped. All media commands will take effect on the local playback state. The remote playback can be initiated through a call to prompt().

6.2.4 Establishing a connection with a remote playback device

When the user agent is to establish a connection with the remote playback device, it MUST run the following steps:

Input
remote, the RemotePlayback object that is to be connected.
device, the remote playback device to connect to.
  1. If the state of remote is not equal to connecting, abort all the remaining steps.
  2. Request connection of remote to device. The implementation of this step is specific to the user agent.
  3. If connection completes successfully, queue a task to run the following steps:
    1. Set the state of remote to connected.
    2. Fire a simple event named connect at remote.
    3. Synchronize the current media element state with the remote playback state. Implementation is specific to user agent.
  4. If connection fails, queue a task to run the following steps:
    1. Set the remote playback state of remote to disconnected.
    2. Fire a simple event named disconnect at remote.
Note
The mechanism that is used to connect the user agent with the remote playback device and play the remote playback source is an implementation choice of the user agent. The connection will likely have to provide a two-way messaging abstraction capable of carrying media commands to the remote playback device and receiving media playback state in order to keep the media element state and remote playback state in sync (unless media mirroring is used).

6.2.5 Media commands and media playback state

The HTMLMediaElement interface interacts with the remotely played media as soon as the connection with the remote playback device is established.

In particular, as soon as the state of a RemotePlayback object has changed to connected, the user agent MUST send all the media commands issued on the HTMLMediaElement object with which the RemotePlayback object is associated to the remote playback device in order to change the remote playback state vs the local playback state.

Similarly, the user agent MUST reflect all updates of the remote playback state received from the remote playback device on the media element state.

If sending any command fails, the user agent MAY disconnect from remote playback device.

6.2.6 Disconnecting from remote playback device

When the user agent is to disconnect from remote playback device, it MUST do the following:

Input
remote, the RemotePlayback object representing the playback to be stopped.
device, the remote playback device to disconnect from.
  1. If the state of remote is disconnected, abort all remaining steps.
  2. Queue a task to run the following steps:
    1. Request disconnection of remote from the device. Implementation is user agent specific.
    2. Change the remote's state to disconnected.
    3. Fire an event with the name disconnect at remote.
    4. Synchronize the current media element state with the local playback state. Implementation is specific to user agent.
Note
The remote playback device might not actually stop playback of the media when requested by the user agent since it's implementation specific and depends on the device in question. In such case stopping remote playback merely means the user agent literally only disconnecting from the remote playback device and the media element switching to the disconnected state.

6.2.7 Event Handlers

The following are the event handlers (and their corresponding event handler event types) that must be supported, as event handler IDL attributes, by objects implementing the RemotePlayback interface:

Event handler Event handler event type
onconnecting connecting
onconnect connect
ondisconnect disconnect

7. Extension to the HTMLMediaElement

partial interface HTMLMediaElement {
    readonly attribute RemotePlayback remote;

    [CEReactions]
             attribute boolean        disableRemotePlayback;
};

The remote attribute MUST return the RemotePlayback instance associated with the media element.

The disableRemotePlayback IDL attribute MUST reflect the content attribute of the same name.

7.1 Disabling remote playback

If the disableRemotePlayback attribute is present on the media element, the user agent MUST NOT play the media remotely or present any UI to do so.

When the disableRemotePlayback attribute is added to the media element and its state is not disconnected, the user agent MUST run the disconnect from remote playback device algorithm for the remote playback device the media element is connected or connecting to.

A. References

A.1 Normative references

[HTML]
Ian Hickson. WHATWG. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[RFC2119]
S. Bradner. IETF. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119
[URL]
Anne van Kesteren; Sam Ruby. W3C. URL. 9 December 2014. W3C Working Draft. URL: https://www.w3.org/TR/url-1/
[WEBIDL]
Cameron McCormack; Boris Zbarsky. W3C. WebIDL Level 1. 8 March 2016. W3C Candidate Recommendation. URL: https://www.w3.org/TR/WebIDL-1/

A.2 Informative references

[PRESENTATION-API]
Mark Foltz; Dominik Röttsches. W3C. Presentation API. 10 May 2016. W3C Working Draft. URL: https://www.w3.org/TR/presentation-api/