Main Page/Cloud Browser TF/UseCases/MSE

From Web and TV IG

Media Source Extension

How these fit into cloud browsing

These use cases are part of a larger set of Cloud Browser use Cases being worked on by the Cloud Browsing taskforce. This page covers use case for the Media source extension

Abstract

This document describes use cases for the Cloud Browser MSE

Status of this document

This is a public working in draft collection of use cases that the Cloud Browsing task force is discussing and exploring. It has no official standing of any kind and note represent the support or consensus of any standards organization or contributor. Cloud Browsing is a taskforce of the W3C Public Web & TV Interest Group

Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].


Use Cases

UC-1 video playback quality

Extends

Nothing

Description

A cloud browser is controlling the video playback for out-of-band media. The cloud browser playback qualities (such as dropped frames) are requested by the browsing context. The video playback quality properties are exposed by the rte.
  1. A cloud browser is controlling the out-of-band video playback
  2. The cloud browser playback qualities are requested by the browsing context
  3. the cloud browser keeps in sync with the rte regarding quality playback

Actors

  • cloud browser
  • browsing context
  • rte

Dependecies

Notes

This use case assume that the video playback qualities need to be obtained by the rte. i.e. the cloud browser should expose the video quality as it is visual perceivable. Therefore there is a need for communication between the rte and the cloud browser.

requirements

the cloud browser should provide playback qualities as the end user would perceive

Authors

Colin Meerveld (talk)

History

1-21-2016 Colin Meerveld (talk) created
1-26-2016 Colin Meerveld (talk) added some additional notes
7-09-2016 Colin Meerveld (talk) added requirement

UC-2 MSE termination by Cloud Browser

Extends

Nothing

Description

fully Cloud-based MSE support

  1. Cloud Browser initiates a session with the Cloud Browser client
  2. Cloud Browser client requests a web page that uses MSE for video delivery
  3. Video is being delivered to the MSE component of Cloud Browser
  4. Cloud Browser terminates MSE and delivers the video to the client
  • Delivery to the Cloud Browser client:
    • The video stream and the UI/Apps might be combined into one single video stream that is then delivered to the client;
    • The Video and UI/Apps streams might be also delivered separately to the client which then has to combine both these streams itself to present them to the end user in a unified form.
  • Player
    • Player functionality is part of Cloud Browser

Actors

Server:
  • cloud browser
  • MSE
  • Graphics Lib
Client:
  • rte

Dependecies

none

Notes

Addressed devices: Legacy STBs, low-end STBs, STBs with no MSE support

Gaps

none

Architectures

  • Single Stream Cloud Player

Authors

Alexandra Mikityuk (talk)

History

4-12-2016 Alexandra Mikityuk (talk) created
4-12-2016 Ronen Mizrahi (talk) reviewed
4-17-2016 Colin Meerveld (talk) reviewed


UC-3 Cloud-based MSE support

Extends

Nothing

Precondition

  • Cloud Browser (CB) client does not support MSE
  • Double stream is required

Description

fully Cloud-based MSE support

UC-3.1 Execution of XHRs
  1. CB initiates a session with the CB client (session id, user id)
  2. The CB client requests a web application that uses MSE for video delivery
  3. CB executes the web application: CB parses the html and css data, web app sets the HTMLMediaElement
  4. The web application requests the manifest (mpd, etc.) file and the CB parses it
  5. The web application creates mediaSource objects and associates it with HTMLMediaElement
  6. mediaSource creates sourceBuffer objects that in turn append media segments into the SourceBuffer array with the appenBuffer method
  7. The web application defines the media segment URLs (with byte range params, video chunks id, etc.)
  8. The web application sends the XMLHTTP Requests towards the media host to request these media segment
  9. As the media segments are downloaded by the client, these XMLHTTP Requests are forwarded to the CB client by the CB.

In this Use Case the CB currently executes a variety of different XHRs and does not have a notion about the art of XMLHTTP Requests and can not choose the required ones.

Here, the Gap1 arises: in MSE the Web browser currently does not have any metadata information about the type of XHRs and therefore does not have any mechanisms to select the required ones.

Requirenemnt1: The XHRs must be made distinguishable for the CB through identifier that would uniquely identify the MSE-Request type.

Accessibility

XHRs resulting in dynamic changes to user interfaces must inform assistive technology of that change. For example, if a menu system made up of media titles is refreshed on screen, a screen reader user would require a notification that the contents of the screen has changed.

ARIA live regions can be leveraged to provide the proper notifications

UC-3.2 Loading of media segments
  1. The CB client loads the data on behalf of CB.

The CB client should download the media segments according to the ordering required by the manifest. However, the manifest is available only at the CB. Ideally, the web application executed by the CB should send XHRs sequentially towards the media host requesting the media segments: the order of the requests is a fixed sequence and should be simply repeated by the CB client. However, in praxis, the web application can load the segments in any order.

Gap2: the ordering of the data is impacted, because the web application loads the segments in any order. Therefore, the web application, i.e. the appendbuffer method, might have a different order than the CB client. This might be also important when some of the packets have not been downloaded due to bad network conditions.

Requirenemnt2: A controllable execution of loading tasks on the SCB client must be established through unique ordering reference. This reference must be generated at the web application and made available to the SCB client. The CB client must be able to exactly rebuild the media stream that has been virtually created by the SCB.

UC-3.3 Buffering while handling the media segments
  1. The CB client requests video at media hosts and buffers it in memory.
  2. The CB client renders and displays the video
  3. CB delivers UI/Apps the stream to the CB client
  4. The CB client overlaps the UI/Apps video stream with ABR video, being played back locally

However, appendbuffer does not append the whole data that is being received after XHR. The client does not know when to clean the buffer. The client also can not differentiate between different resources as audio and video.

The Gap3 arises: there is no way for the SCB client to know which media segment is currently being appended. In other words, the client does not know exactly which segments must be buffered at a certain point in time.

This also leads to the following implications. The appendBuffer method at the SCB does not append the entire amount of data that is being received in one XMLHTTP response. The SCB client also cannot differentiate between different resources like audio and video. Also, if the pre-buffered segments must be removed from the buffer due to changed network conditions, the SCB client also does not know when to clean the buffer, i.e. which segments must be removed.

Requirenemnt3: The media segments as well as their types must be uniquelly identified to provide the CB client the means for rebuilding the media stream that has been virtually created by the CB.

UC-3.4 Control
  1. The CB client processes video on its own.

This can put the web app and the CB client out of sync as the web app does not have a notion of what the client is doing. For the web application it means that no data is actually available on the Server, the data is physically available on the CB client only. E.g. the web application might want to modify or manipulate the video data (timestamps manipulations, current video positions, video time, etc.).

The virtually doubled execution of MSE, where the SCB is aware of the MSE processing, but its client is not, causes Gap4: the CB client processes media on its own, which can spin the CB client out of control as the web app does not have any response for the state of current execution tasks on the SCB client. No data is available for the web app on the server, the data is physically available on the CB client only.

For example, any manipulation of the data as trick functions, e.g. play forward, are not enabled for the web app, as none of the data exist on the server. The data might also be appended before it is manipulated at the CB. The web app might also want to signal to the CB client only the changes that must be applied to the data.

Requirenemnt4: The reference data must be made available to the web app executed in the CB context.

Herewith, the CB client returns to the CB the reference data required by the web app. These reference data are then adjusted by the web app and delivered to the SCB client, where the required changes would be applied.

Actors

Server:
  • Cloud Browser
  • Web Application
Client:
  • rte
  • Graphics Lib

Dependecies

MSE

Notes

Addressed devices: Legacy STBs, low-end STBs, STBs with no MSE support

Architectures

  • Double Stream Local Player

Authors

Alexandra Mikityuk (talk)

History

4-14-2016 Alexandra Mikityuk (talk) created
4-18-2016 Ronen Mizrahi (talk) reviewed
4-23-2016 Colin Meerveld (talk) reviewed
6-28-2017 Alexandra Mikityuk (talk) added the sub-use-cases