SVGStreaming

Streaming of SVG graphics

This document, intended as input to the SVG Seattle/Paris F2F meeting, discusses how streaming concepts can apply to SVG.

Background terminology

Progressive loading is usually understood as the ability to consume a data (a video, an XML file ...) before it is fully received. Currently, all browsers support such a loading mode for SVG (and HTML).

Streaming is usually understood as progressive loading with some control of the timing at which chunk of media data is received and consumed. A chunk of media data is associated with a timestamp which indicates at which time, on a given timeline, the media data should be processed by the client. Streaming has been used in the past on unreliable protocols such as UDP with protocols such as RTP, but can also be used over HTTP, more recently with approaches like IceCast (Web Radios) or MPEG-DASH (Dynamic Streaming over HTTP).

In general streaming, it is sometimes interesting to jump at some specific position of a stream without having to process the data from the beginning of the stream (which might not be available) but only some global initialization information. Such positions are called Random Access Points.

Use cases and considerations for streaming of SVG

Long running animations (cartoons, ads...)

It is possible to design frame-based SVG content similar to Flash, to represent cartoons or ads (see Representing 2D Cartoons using SVG, SMIL Europe, January 2003 ), for instance as follows:

<g id="frame_N" display="none">
 <animate begin="N" attributeName="display" to="inline" dur="1"/>
</g>

It is possible to design long-running SVG animations, especially using the <discard> element.

<g id="frame_N" display="none">
 <animate begin="N" attributeName="display" to="inline" dur="1"/>
 <discard begin="N+1"/>
</g>

Some experiments (see Timed-fragmentation of SVG documents to control the playback memory usage, DocEng, August 2007) show that when the network bandwidth is higher than the bitrate of the SVG animation (size of the file divided by duration), progressive loading of SVG leads to unnecessary high memory consumption peaks. This is a problem when the SVG content is large. Controlling the time at which the XML fragments are passed to the SVG engine reduces the peak (when <discard> elements are used). So, for some cases, there is a need to control the timing at which SVG content is loaded by defining an SVG stream.

In general XML processing, it is not possible to define a Random Access Point. However, for some SVG streams, such as the one described above, in some cases, it is possible to ignore the SVG content prior to a <g id="frame"> up to some element before (e.g. the <svg> root element). In such cases, it is possible to define Random Access Points in an SVG stream and to define the initialization information as the beginning of the SVG document up to the first Random Access Point.

Overlaying SVG synchronously on top of a video

SVG can be used to represent animated overlays on top of a video. Examples of such overlays are:

• highlights or clickable areas of interest in the video
• subtitles made of text, graphics, images, possibly using SVG fonts
• graphics representing additional information (e.g. augmented reality graphical objects to be displayed on top of a video, or a skeleton extracted from a video of a person for example using the Microsoft Kinect)

One way is to achieve that synchronized overlay of SVG content is to control the playback of the video and of the SVG in the same way, ie. to start/stop them at the same time, to set the seek time and playback speed with the same values. There are 2 problems with this approach:

• the SVG API is not aligned with the HTML5 Media Element API: missing playbackRate, and media events.
• When video streams are processed by video decoders, some frames may be dropped (e.g. because of delivery errors or of lack of CPU power), so the playback frame rate may not be constant. For these reasons, if you want frame-accuracy in the overlay, the synchronization of SVG with video needs to be maintained throughout the playback of the video.

To monitor the playback of the video and to update the playback of the SVG accordingly, one can use JavaScript, the HTML 5 Media Element "timeupdate" event and the SVG setCurrentTime method. Some experiments (see examples) show that it's not good enough. "timeupdate" events are not generated at the right time, not at a sufficient frequency and the currentTime value is currently not implemented reliably (precision problems).

Proposed actions

First, SVG Timing API should be aligned with the HTML 5 Media API and Events.

Then, the best solution to enable synchronized playback of SVG and other media streams would be to signal to the browser that an SVG content needs to be synchronized with a stream and let the browser maintain the synchronization natively.

This can be done at different level:

• at the SVG level, if a @mediagroup attribute is allowed on the SVG root element, when embedded in HTML.
• at the HTML level, by allowing animated SVG content to be used as a track source (or probably as a video source). The SVG WG should coordinate with the HTML WG.
• at the delivery level (MP4, WebM, RTP, DASH). The definition of how streaming technologies are used for SVG is probably outside of the scope of the SVG WG. SVG won't define how to use RTP or DASH or MP4 or WebM to deliver SVG stream. Other groups are working on that. See for instance the proposal to MPEG to store SVG streams in MP4 as media tracks (see MPEG contribution)

The work of the other groups doing that could be facilitated here. The SVG WG should just make it clearer and more visible (maybe with an annex) that the streaming of SVG content is possible by:

• documenting that SVG rendering is compatible with streaming (as an extension of progressive loading)
• giving examples of what SVG stream can be (for example, subtitles, cartoons or more generally content either frame-based or splittable into timed fragments)
• documenting good practices for the fragmentation of SVG document into SVG streams (eg. not fragmenting some elements, not fragmenting start tags, ...) and on how to create streams with Random Access Points.