W3C

Use cases and requirements for Media Fragments

W3C Working Draft 10 March 2010

This version:
http://www.w3.org/TR/2010/WD-media-frags-reqs-20100310
Latest version:
http://www.w3.org/TR/media-frags-reqs
Previous version:
http://www.w3.org/TR/2009/WD-media-frags-reqs-20091217
Editors:
Raphaël Troncy , EURECOM
Erik Mannens , IBBT Multimedia Lab, University of Ghent
Contributors:
Jack Jansen , Center for Mathematics and Computer Science (CWI Amsterdam)
Yves Lafon , W3C
Silvia Pfeiffer , W3C Invited Expert
Davy Van Deursen , IBBT Multimedia Lab, University of Ghent

Copyright © 2010 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.

Abstract

This document describes use cases and requirements for the development of the Media Fragments 1.0 specification. It includes a technology survey for addressing fragments of multimedia document.

Status of this Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This is the First Public Working Draft of the Use cases and requirements for Media Fragments specification. It has been produced by the Media Fragments Working Group, which is part of the W3C Video on the Web Activity.

A list of changes is available in E Change Log.

A diff-marked version against the previous version of this document is available.

Please send comments about this document to public-media-fragment@w3.org mailing list (public archive).

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. The group does not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

1 Introduction
2 Terminology
3 Side Conditions
    3.1 Single Media Resource Definition
    3.2 Existing Standards
    3.3 Unique Resource
    3.4 Valid Resource
    3.5 Parent Resource
    3.6 Single Fragment
    3.7 Relevant Protocols
    3.8 No Recompression
    3.9 Minimize Impact on Existing Infrastructure
    3.10 Focus for Changes
    3.11 Browser Impact
    3.12 Fallback Action
4 Use Cases
    4.1 Linking to and Display of Media Fragments
        4.1.1 Scenario 1: Retrieve only segment of a video
        4.1.2 Scenario 2: Region of an Image
        4.1.3 Scenario 3: Portion of Music
        4.1.4 Scenario 4: Image Region of video over time
    4.2 Browsing and Bookmarking Media Fragments
        4.2.1 Scenario 1: Temporal Video Pagination
        4.2.2 Scenario 2: Audio Passage Bookmark
        4.2.3 Scenario 3: Audio Navigation
        4.2.4 Scenario 4: Caption and chapter tracks for browsing Video
        4.2.5 Scenario 5: Jumping back in time during live streaming
        4.2.6 Scenario 6: Jumping to a particular event in a live stream
    4.3 Recompositing Media Fragments
        4.3.1 Scenario 1: Reframing a photo in a slideshow
        4.3.2 Scenario 2: Mosaic
        4.3.3 Scenario 3: Video Mashup
        4.3.4 Scenario 4: Spatial Video Navigation
        4.3.5 Scenario 5: Selective previews
        4.3.6 Scenario 6: Music Samples
        4.3.7 Scenario 7: Highlighting regions (out-of-scope)
    4.4 Annotating Media Fragments
        4.4.1 Scenario 1: Spatial Tagging of Images
        4.4.2 Scenario 2: Temporal Tagging of Audio and Video
        4.4.3 Scenario 3: Named Anchors
        4.4.4 Scenario 4: Spatial and Temporal Tagging
        4.4.5 Scenario 5: Search Engine
    4.5 Adapting Media Resources
        4.5.1 Scenario 1: Changing Video quality (out-of-scope)
        4.5.2 Scenario 2: Selecting Regions in Images
        4.5.3 Scenario 3: Selecting an Image from a multi-part document (out-of-scope)
        4.5.4 Scenario 4: Retrieving an Image embedded thumbnail (out-of-scope)
        4.5.5 Scenario 5: Switching of Video Transmission
        4.5.6 Scenario 6: Toggle All Audio OFF
        4.5.7 Scenario 7: Toggle specific Audio tracks
        4.5.8 Scenario 8: Video aspect ratio (out-of-scope)
5 Requirements for Media Fragment URIs
    5.1 Requirement r01: Temporal fragments
    5.2 Requirement r02: Spatial fragments
    5.3 Requirement r03: Track fragments
    5.4 Requirement r04: Named fragments
    5.5 Fitness Conditions on Media Containers/Resources

Appendices

A References
B Evaluation of fitness per media formats
C Technologies Survey
    C.1 Existing URI fragment schemes
        C.1.1 General specification of URI fragments
        C.1.2 Fragment specifications not for audio/video
        C.1.3 Fragment specifications for audio/video
    C.2 Existing applications using proprietary temporal media fragment URI schemes
    C.3 Media fragment specification approaches
        C.3.1 URI based
            C.3.1.1 SVG
                C.3.1.1.1 Spatial
            C.3.1.2 Temporal URI/Ogg technologies
                C.3.1.2.1 Temporal
                C.3.1.2.2 Track
                C.3.1.2.3 Named
            C.3.1.3 MPEG-21
                C.3.1.3.1 Temporal
                C.3.1.3.2 Spatial
                C.3.1.3.3 Track
                C.3.1.3.4 Named
        C.3.2 Non-URI-based
            C.3.2.1 SMIL
                C.3.2.1.1 Temporal
                C.3.2.1.2 Spatial
                C.3.2.1.3 Track
                C.3.2.1.4 Named
            C.3.2.2 MPEG-7
                C.3.2.2.1 Temporal
                C.3.2.2.2 Spatial
                C.3.2.2.3 Track
                C.3.2.2.4 Named
            C.3.2.3 SVG
                C.3.2.3.1 Temporal
                C.3.2.3.2 Spatial
            C.3.2.4 TV-Anytime
                C.3.2.4.1 Temporal
                C.3.2.4.2 Named
            C.3.2.5 ImageMaps
                C.3.2.5.1 Spatial
            C.3.2.6 HTML 5
D Acknowledgements (Non-Normative)
E Change Log (Non-Normative)

1 Introduction

Audio and video resources on the World Wide Web are currently treated as "foreign" objects, which can only be embedded using a plugin that is capable of decoding and interacting with the media resource. Specific media servers are generally required to provide for server-side features such as direct access to time offsets into a video without the need to retrieve the entire resource. Support for such media fragment access varies between different media formats and inhibits standard means of dealing with such content on the Web.

This document collects background information to the Media Fragment URI specification Media Fragments URI 1.0. It contains a collection of side conditions under which the specification was developed. It further contains a large collection of use cases that are either regarded as relevant to this specification or as out-of-scope. From these use cases, it deducts the different required dimensions for the Media Fragment URI specification. Finally, this document finishes with a survey of existing media fragment addressing approaches.

2 Terminology

The keywords MUST, MUST NOT, SHOULD and SHOULD NOT are to be interpreted as defined in RFC 2119.

According to RFC 3986, URIs that contain a fragment are actually not URIs, but URI references relative to the namespace of another URI. In this document, when the term 'media fragment URIs' is used, it actually means 'media fragment URI references'.

3 Side Conditions

This section lists a number of conditions which have directed the development of this specification. These conditions help clarify some of the decisions made, e.g. about what types of use cases are within the realm of this specification and which are outside. Spelling out these side conditions should help increase transparency of the specifications.

3.1 Single Media Resource Definition

The following picture explains the generic composition of a media resource: Model of a media resource

A media resource for the purposes of this Working Group is defined along a single timeline. It can consist of multiple tracks of data that are parallel along this timeline. These tracks can be audio, video, images, text or any other time-aligned data. The main interest of this group is in audio and video. A media resource also typically has some control information in data headers. These may be located at a particular position in the resource, e.g. the beginning or the end, or spread throughout the data tracks as headers for data packets. There is possibly also a general header for the complete media resource. The data tracks are typically encoded in an interleaved fashion, which allows for progressive decoding. All of this is provided in a single file.

3.2 Existing Standards

Media fragment URIs will work within the boundaries of existing standards as much as possible, in particular within the URI specification RFC 3986.

3.3 Unique Resource

Media fragments are a representation of the parent resource and should not create a new resource, in particular not a new resource of a different Internet media type (or MIME type). Note that there are use cases for creating a new resource, such as the extraction of a thumbnail from a video. These are currently outside the scope of this document.

3.4 Valid Resource

Resources delivered as a response to a media fragment URI request should be valid media resources by themselves and thus be playable by existing media players / image viewers.

3.5 Parent Resource

The entire resource should be accessible as the "context" of a fragment via a simple change of the URI. The media fragment URI - as a selective view of the resource - provides a mechanism to focus on a fragment whilst hinting at the wider media context in which the fragment is included.

3.6 Single Fragment

Editorial note: Werner Bailer  
Not sure that the term 'mask' is the best choice here, e.g. in MPEG-7 mask is used for the opposite, i.e. not a single segment but a segment composed of several unconnected parts.

A media fragment URI should create only a single "mask" onto a media resource and not a collection of potentially overlapping fragments.

3.7 Relevant Protocols

The main protocols we are concerned with are HTTP and RTSP, since they are open protocols for media delivery.

3.8 No Recompression

Media fragments should preferably be delivered as byte-range subparts of the media resource such as to make the fragments an actual subresource of the media resource. The advantage of this is that such fragments are cachable as byte ranges in existing caching Web proxies. This implies that we should avoid to decode and recompress a media resource to create a fragment.

3.9 Minimize Impact on Existing Infrastructure

The necessary changes to all software in the media delivery chain should be kept to a minimum: User Agents, Proxies, Media Servers.

3.10 Focus for Changes

Focus for necessary changes should be as much as possible on the media servers because in any case they have to implement fragmentation support for the media formats as the most fundamental requirement for providing media fragment addressing.

3.11 Browser Impact

Changes to the user agent need to be a one-off and not require adaptation per media encapsulation/encoding format.

3.12 Fallback Action

If a User Agent connects with a media fragment URI to a media server that does not support media fragments, the media server should reply with the full resource. The User Agent will then have to take action to either cancel this connection (if e.g. the media resource is too long) or do a fragment offset locally.

A User Agent that does not understand media fragment URIs will simply hand on the URI (potentially with a stripped off fragment part) to the server and receive the full resource in lieu of the fragment. This may lead to unexpected behaviour with media fragment URIs in non-conformant User Agents, e.g. where a mash-up of media fragments is requested, but a sequence of the full files is played. This is acceptable during a transition phase.

Editorial note: David Singer 
The fallback plan needs to be clarified. We must be able to handle the way the # is already used, e.g. in YouTube, without breaking what is already working.

4 Use Cases

In which situations do users need media fragment URIs? This section explains the types of user interactions with media resources that media fragment URIs will enable. For each type it shows how media fragment URIs can improve the usefulness, usability, and functionality of online audio and video.

4.1 Linking to and Display of Media Fragments

In this use case, a user is only interested in consuming a fragment of a media resource rather than the complete resource. A media fragment URI allows addressing this part of the resource directly and thus enables the User Agent to receive just the relevant fragment.

4.1.1 Scenario 1: Retrieve only segment of a video

Tim does a keyword search on a video search service. That keyword is found in several videos in the search service's collection and it relates to clips inside the videos that appear at a time offset. Tim would like the search result to point him to just these media fragments so he can watch the relevant clips rather than having to watch the full videos and manually scroll for the relevant clips.

4.1.2 Scenario 2: Region of an Image

Tim has discovered on an image hosting service a photo of his third school year class. He is keen to put a link to his own face inside this photo onto his private Web site where he is collecting old photos of himself. He does not want the full photo to be displayed and he does not want to have to download and crop the original image since he wants to reference the original resource.

4.1.3 Scenario 3: Portion of Music

Tim is a Last.fm user. He wants his friend Sue to listen to a cool song, Gypsy Davy. However, not really the entire song is worth it, Tim thinks. He wants Sue to listen to the last 10 seconds only and sends her an email with a link to just that subpart of the media resource.

4.1.4 Scenario 4: Image Region of video over time

Tim is now creating an analysis of the movements of muscles of horses during trotting and finds a few relevant videos online. His analysis is collected on a Web page and he'd like to reference the relevant video sections, cropped both in time and space to focus his viewers' attention on specific areas of interest that he'd like to point out.

4.2 Browsing and Bookmarking Media Fragments

Media resources - audio, video and even images - are often very large resources that users want to explore progressively. Progressive exploration of text is well-known in the Web space under the term "pagination". Pagination in the text space is realized by creating a series of Web pages and enabling paging through them by scripts on a server, each page having their own URI. For large media resources, such pagination can be provided by media fragment URIs, which enable direct access to media fragments.

4.2.1 Scenario 1: Temporal Video Pagination

Michael has a Website that collects recordings of the sittings of his government's parliament. These recordings tend to be very long - generally on the order of 7 hours in duration. Instead of splitting up the recordings into short files by manual inspection of the change of topics or some other segmentation approach, he prefers to provide many handles to a unique video resource. As he publishes the files, however, he provides pagination on the videos such that people can watch them 20 min at a time.

4.2.2 Scenario 2: Audio Passage Bookmark

Users not only want to receive links to highlights in media resources, but also like to bookmark them in their browsers to be able to get back to them.

Sue likes the song segment that Tim has sent her and decides to add this specific segment to her bookmarks.

4.2.3 Scenario 3: Audio Navigation

When regarding media resources (in particular audio and video) as monolithic blocks, they are very inaccessible. For example, it is difficult to find out what they are about, where the highlights are, or what the logical structure of the resources are. Lack of these features, in particular lack of captions and audio annotations, further make the resources inaccessible to disabled people. Introducing an ability to directly access highlights, fragments, or the logical structure of a media resource will provide a big contribution towards making a media resource more accessible.

Lena would like to browse the descriptive audio tracks of a video as she does with Daisy audio books, by following the logical structure of the media. Audio descriptions and captions generally come in blocks either timed or separated by silences. Chapter by chapter and then section by section she eventually jumps to a specific paragraph and down to the sentence level by using the "tab" control as she would normally do in audio books. The descriptive audio track is an extra spoken track that provides a description of scenes happening in a video. When the descriptive audio track is not present, Lena can similarly browse through captions and descriptive text tracks which are either rendered through her braille reading device or through her text-to-speech engine.

4.2.4 Scenario 4: Caption and chapter tracks for browsing Video

Silvia has a deaf friend, Elaine, who would like to watch the holiday videos that Silvia is publishing on her website. Silvia has created subtitle tracks for her videos and also a segmentation (e.g. using CMML CMML) with unique identifiers on the clips that she describes. The clips were formed based on locations that Silvia has visited. In this way, Elaine is able to watch the videos by going through the clips and reading the subtitles for those clips that she is interested in. She watches the sections on Korea, Australia, and France, but jumps over the ones of Great Britain and Holland.

4.2.5 Scenario 5: Jumping back in time during live streaming

A URL to a live video stream may look no different than a URL to a canned video file, e.g. http://www.example.com/video.ogv . However, in contrast to canned video file URLs, this URL always points to the live video data, i.e. what is transmitted "now". This leads to different requirements on fragment addressing than with canned video files.

Thomas is watching a live video stream, but has to take a business call right in the middle. He stops his video player to take the call. As he reconnects, he gets connected back with the live stream, but has missed the last 5 min. He would like to rewind to 5 min ago. A URL scheme that can capture the time at which a live video stream was transmitted and allow for direct access to any time within that real-world clock time will allow such direct access.

4.2.6 Scenario 6: Jumping to a particular event in a live stream

Thomas is watching a Formula 1 race on a Website that is streaming live video with a real-time commentary and interactive textual descriptions of particular events that are happening. Thomas wants to directly jump to the 'Alonso accident' listed next to the video as a section of interest in the video.

4.3 Recompositing Media Fragments

As we enable direct linking to media fragments in a URI, we can also enable simple recompositing of such media fragments. Note that because the media fragments in a composition may possibly originate from different codecs and very different files, we can not realistically expect smooth playback between the fragments.

4.3.1 Scenario 1: Reframing a photo in a slideshow

Erik has a collection of photos and wants to create a slide show of some of the photos and wants to highlight specific areas in each image. He uses xspf to define the slide show (playlist) using spatial fragment URIs to address the photo fragments.

4.3.2 Scenario 2: Mosaic

Jack wants to create a mosaic for his website with all the image fragments that Erik defined collated together. He uses SMIL 3.0 Tiny Profile and the spatial fragment URIs to layout the image fragments and stitch them together as a new "image".

4.3.3 Scenario 3: Video Mashup

Jack has a collection of videos and wants to create a mashup from segments out of these videos without having to manually edit them together. He uses SMIL 3.0 Tiny Profile and temporal fragment URIs to address the clips out of the videos and sequence them together.

4.3.4 Scenario 4: Spatial Video Navigation

Elaine has recorded a video mosaic of all her 4 TV channels of an international election day in a single video. She wants to keep the original synchronised file, but now she wants to be able to play back each of the four channels' recordings separately and in sequence. She creates a playlist of media fragments URIs that each select a specific channel in the mosaic to play each channel after one another.

4.3.5 Scenario 5: Selective previews

Given an ability to link to media fragments through URIs, people will want to decide whether they receive the full resource or just the data that relates to the media fragment. This is particularly the case where the resource is large, where the bandwidth is scarce or expensive, and/or where people have limited time/patience to wait until the full resource is loaded.

Yves is a busy person. He doesn't have time to attend all meetings that he is supposed to attend. He also uses his mobile device for accessing Web resources while traveling, to make the most of his time. Some of the recent meetings that Yves was supposed to attend have been recorded and published on the Web. A colleague points out to Yves in an email which sections of the meetings he should watch. While on his next trip, Yves goes back to this email and watches the highlighted sections by simply clicking on them. The media server of his company dynamically composes a valid media resource from the URIs that Yves is sending it such that Yves' video player can play just the right fragments.

4.3.6 Scenario 6: Music Samples

Erik has a music collection. He creates an "audio podcast" in the form of an RSS feed with URIs that link to samples from his music files. His friends can play back the samples in their Web-attached music players.

4.3.7 Scenario 7: Highlighting regions (out-of-scope)

Tim has discovered yet another alumni photo of his third school year class. This time he doesn't want to crop his face but he wants to keep the photo in the context of his classmates. He wants his region of the photo highlighted and the rest grey scaled.

This scenario is out of scope for this Working Group because the display of the highlighted region is up to the user agent and is not relevant to the network interaction. This particular scenario is already possible with image maps in HTML.

4.4 Annotating Media Fragments

Media resources typically don't just consist of the binary data. There is often a lot of textual information available that relates to the media resource. Enabling the addressing of media fragments ultimately creates a means to attach annotations to media fragments, for example, using the Ontology for Media Resource 1.0.

4.4.1 Scenario 1: Spatial Tagging of Images

Raphael systematically annotates some highlighted regions in his photos that depict his friends, families, or the monuments he finds impressive. This knowledge is represented by RDF descriptions that use spatial fragment URIs to relate to the image fragments in his annotated collection. It makes it possible later to search and retrieve all these media fragment URIs that relate to one particular friend or monument.

4.4.2 Scenario 2: Temporal Tagging of Audio and Video

Raphael also has a collection of audio and video files of all the presentations he ever made. His RDF description collection extends to describing all the temporal segments where he gave a demo of a software system with structured details on the demo.

Editorial note: Silvia 
Time-aligned text such as captions, subtitles in multiple languages, and audio descriptions for audio and video don't have to be created as separate documents and link to each segment through a temporal URI. Such text can be made part of the media resource by the media author or delivered as a separate, but synchronised data stream to the media player. In either case, when it comes to using these with the HTML5 <video> tag, they should be made accessible to the Web page through a javascript API for the video/audio/image element. This needs to be addressed in the HTML5 working group.

4.4.3 Scenario 3: Named Anchors

Annotating media resources at the level of a complete resource is in certain circumstances not enough. Support for annotating multimedia on the level of fragments is often desired. The definition of "anchors" (or id tags) for fragments of media resources will allow us to identify fragments by name. It allows the creation of an author-defined segmentation of the resource - an author-provided structure.

Raphael would like to attach an RDF-based annotation to a video fragment that is specified through an "anchor". Identifying the media fragment by name instead of through a temporal video fragment URI allows him to create a more memorable URI than having to remember the time offsets.

4.4.4 Scenario 4: Spatial and Temporal Tagging

Guillaume uses video fragment URIs in an MPEG-7 sign language profile to describe a moving point of interest: he wants the focus region to be the dominant hand in a Sign Language video. The series of video fragment URIs gives the coordinates and timing of the trajectory followed by the hand, and by naming them, can also describe the areas of changing hand-shapes.

4.4.5 Scenario 5: Search Engine

Guillaume wants to retrieve the images of each bike present at a recent cycling event. Group photos and general shots of the event have been published online together with detailed RDF annotations. Thanks to a query in a search engine that is able to parse the RDF annotations, Guillaume can now retrieve multiple individual shots of each bike in the collection, where the URI is created based on the RDF annotations.

4.5 Adapting Media Resources

When addressing a media resource as a user, one often has the desire not to retrieve the full resource, but only a subpart of interest. This may be a temporally or spatially consecutive subpart, but could also be e.g. a smaller bandwidth version of the same resource, a lower framerate video, a image with less colour depth or an audio file with a lower sampling rate. Media adaptation is the general term used for such server-side created versions of media resources.

4.5.1 Scenario 1: Changing Video quality (out-of-scope)

Davy is looking for videos about allergies and would like to get previews at a lower frame rate to decide whether to download and save them in his collection. He would like to be able to specify in the URI a means of telling the media server the adaptation that he is after. For video he would like to adapt width, height, frame rate, colour depth, and temporal subpart selection. Alternatively, he may want to get just a thumbnail of the video.

This scenario is out of scope for this Working Group because it requires changes be made to the actual encoded data to retrieve a "fragment". URI based media fragments should basically be achieved through cropping of one or more byte sections. It is possible to develop in future a scheme for such transcoded resources using a URI query (?) specification.

4.5.2 Scenario 2: Selecting Regions in Images

Davy is interested to have precise coordinates on his browser address bar to see and pan over large-size images maps. Through the same URI scheme he can now generically address and locate different image subparts on his User Agent for all image types.

4.5.3 Scenario 3: Selecting an Image from a multi-part document (out-of-scope)

Davy is now interested in multi-resolution, multi-page medical images. He wants to select the detailed image of the toe X-rays which appear on page 7 of the TIFF document.

The support of particular media formats such as TIFF is out of scope - the Working Group only deals with the specification of generic addressing approaches, but support of particular file formats needs to be implemented by the format developers. A spatial fragment URI to an image is, however, in scope.

4.5.4 Scenario 4: Retrieving an Image embedded thumbnail (out-of-scope)

Davy is also interested to have the kind of preview functionality for pictures, in particular these large 10 mega-pixel JPEG files that have embedded thumbnails in them. He can now provide a fast preview by selecting the embedded thumbnail in the original image without even having to resize or create a new separate file!

This particular scenario is out of scope for a media fragment URI, since it creates a resource of a different mime type to the original resource. This cannot be done using the URI fragment specifier, but only using the query specifier. This is left as a future exercise.

4.5.5 Scenario 5: Switching of Video Transmission

Davy has a blind friend called Katrina. Katrina would also like to watch the videos that Davy has found, and is lucky that the videos have additional alternative audio tracks, which describe to blind users what is happening in the videos. Her Internet connection is of lower bandwidth and she would like to switch off the video track, but receive the two audio tracks (original audio plus audio annotations). She would like to do this track selection through simple changes to the URI.

4.5.6 Scenario 6: Toggle All Audio OFF

Sebo is Deaf and enjoys watching videos on the Web. Her friend sent her a link to a new music video but she doesn't want to waste time and bandwidth receiving any sounds. So when she enters the URI in her browser's address bar, she also adds an extra parameter to select the video track only.

4.5.7 Scenario 7: Toggle specific Audio tracks

Davy's girlfriend is a fan of Karaoke. She loves to be able to play back videos from the Web that have a karaoke text, and two audio tracks, one each for the music and for the singer. She practices the songs by playing back the complete video with all tracks, but uses the video in Karaoke parties with friends where she turns off the singer's audio track through a simple selection of tracks in the User Agent.

4.5.8 Scenario 8: Video aspect ratio (out-of-scope)

Silvia's television is brand new and with a display in 16:9, however she has video on her media server that are in 3:2 format. To avoid paying a premium in network fees, she would like the television to request only what can be displayed to avoid wasting bandwidth.

This particular scenario is out of scope for a media fragment URI, since it is unclear what a server should do with a request that has a different aspect ratio. It is a display issue rather than a bandwidth or clipping issue. In general, a user agent would create black borders around the video with a diverging aspect ratio. However, it is up to the user agent what to do in such a situation of diverging aspect ratio between what the server supplies and what the user agent is requested to display. For example, HTML5 has specifications for what to do in such a situation.

5 Requirements for Media Fragment URIs

This section describes the list of required media fragment addressing dimensions that have resulted from the use case analysis.

It further analyses what format requirements the media resources has to adhere to in order to allow the extraction of the data that relates to that kind of addressing.

5.1 Requirement r01: Temporal fragments

A temporal fragment of a media resource is a clipping along the time dimension from a start to an end time that are within the duration of the media resource.

Whether a media resource supports temporal fragment extraction is in the first place dependent on the coding format and more specifically how encoding parameters were set. For video coding formats, temporal fragments can be extracted if the video stream provides random access points (i.e., a point that is not dependent on previously encoded video data, typically corresponding to an intra-coded frame) on a regular basis. The same holds true for audio coding formats, i.e., the audio stream needs to be accessible at a point where the decoder can start decoding without the need of previous coded data.

5.2 Requirement r02: Spatial fragments

A spatial fragment of a media resource is a clipping of an image region. For media fragment addressing we only regard rectangular regions.

Support for extraction of spatial fragments from a media resource in the compressed domain depends on the coding format. The coding format must allow to encode spatial regions independently from each other in order to support the extraction of these regions in the compressed domain. Note that there are currently two variants: region extraction and interactive region extraction. In the first case, the regions (i.e., Regions Of Interest, ROI) are known at encoding time and coded independently from each other. In the second case, ROIs are not known at encoding time and can be chosen by a user agent. In this case, the media resource is divided in a number of tiles, each encoded independently from each other. Subsequently, the tiles covering the desired region are extracted from the media resource.

5.3 Requirement r03: Track fragments

A typical media resource consists of multiple tracks of data multiplexed together into the media resource. A media resource could for example consist of several audio, several video, and several textual annotation or metadata tracks. Their individual extraction / addressing is desirable in particular from a media adaptation point of view.

Whether the extraction of tracks from a media resource is supported or not depends on the container format of the media resource. Since a container format only defines a syntax and does not introduce any compression, it is always possible to describe the structures of a container format. Hence, if a container format allows the encapsulation of multiple tracks, then it is possible to describe the tracks in terms of byte ranges. Examples of such container formats are Ogg RFC 3533 and MP4. Note that it is possible that the tracks are multiplexed, implying that a description of one track consists of a list of byte ranges. Also note that the extraction of tracks (and fragments in general) from container formats often introduces the necessity of syntax element modifications in the headers.

5.4 Requirement r04: Named fragments

A named fragment of a media resource is a media fragment - either a track, a time section, or a spatial region - that has been given a name through some sort of annotation mechanism. Through this name, the media fragment can be addressed in a more human-readable form.

No coding format provides support for named fragments, since naming is not part of the encoding/decoding process. Hence, we have to consider container formats for this feature. In general, if a container format allows the insertion of metadata describing the named fragments, then the container format supports named fragments, if the fragment class is also supported. For example, you can include a CMML CMML or TimedText description in an MP4 or Ogg RFC 3533 container and interpret this description to extract temporal fragments based on a name given to them in the description.

5.5 Fitness Conditions on Media Containers/Resources

There is a large number of media codecs and encapsulation formats that we need to take into account as potential media resources on the Web. This section analyses the general conditions for media formats that make them fit for supporting the different types of fragment URIs.

The extraction of media fragments from a media resource is possible when these media fragments are extractable in the compressed domain. In other words, media fragments need to be expressable in terms of byte ranges referring to the parent resource. Dependent on the fragment axis, there are different requirements regarding the extraction of media fragments (i.e., obtaining media fragments in terms of byte ranges):

Not all media formats fullfil all these requirements. Hence, we distinguish the following categories:

Those media types that are capable of doing what server-side media fragments require are of interest to us. For those that aren't, the fall-back mechanisms apply (e.g., full download and then offsetting). Appendix B Evaluation of fitness per media formats lists a large number of typical formats and determines which we see fit or currently unfit for supporting the different types of media fragment URIs.

Editorial note: Silvia 

We ask for further input into the table in the attachment, in particular where there are question marks.

A References

[CMML]
Continuous Media Markup Language (CMML), Version 2.1. IETF Internet-Draft, March 2006.
[HTML 4.0]
D. Ragett, A. Le Hors and I. Jacobs. HTML Fragment identifiers. W3C Recommendation, December 1999. Available at http://www.w3.org/TR/REC-html40/intro/intro.html#fragment-uri.
[ISO Base Media File Format]
Information technology - Coding of audio-visual objects - Part 12: ISO base media file format. April 2009.
[Media Fragments URI 1.0]
Media Fragments URI 1.0. W3C Working Draft, October 2009.
[MPEG-7]
Information Technology - Multimedia Content Description Interface (MPEG-7). Standard No. ISO/IEC 15938:2001, International Organization for Standardization(ISO), 2001.
[MPEG-21]
Information Technology - Multimedia Framework (MPEG-21). Standard No. ISO/IEC 21000:2002, International Organization for Standardization(ISO), 2002. Available at http://www.chiariglione.org/mpeg/working_documents/mpeg-21/fid/fid-is.zip.
[Ontology for Media Resource 1.0]
W. Lee, T. Bürger, F. Sasaki, V. Malaisé, F. Stegmaier and Joakim Söderberg. Ontology for Media Resource 1.0. W3C Working Draft, June 2009. Available at http://www.w3.org/TR/mediaont-10.
[RFC 2119]
S. Bradner. Key Words for use in RFCs to Indicate Requirement Levels. IETF RFC 2119, March 1997. Available at http://www.ietf.org/rfc/rfc2119.txt.
[RFC 3533]
S. Pfeiffer. The Ogg Encapsulation Format Version 0. IETF RFC 3533, May 2003. Available at http://www.ietf.org/rfc/rfc3533.txt.
[RFC 3986]
T. Berners-Lee, R. Fielding and L. Masinter. Uniform Resource Identifier (URI): Generic Syntax. IETF RFC 3986, January 2005. Available at http://www.ietf.org/rfc/rfc3986.txt.
[RFC 5147]
E. Wilde and M. Duerst. URI Fragment Identifiers for the text/plain Media Type. IETF RFC 5147, April 2008. Available at http://tools.ietf.org/html/rfc5147.
[RFC 5234]
D. Crocker. Augmented BNF for Syntax Specifications: ABNF, IETF RFC 5234, January 2008. Available at http://tools.ietf.org/html/rfc5234.
[ROE]
Rich Open multitrack media Exposition (ROE). Xiph.org Wiki, April 2009.
[Skeleton]
Ogg Skeleton. Xiph .org Wiki, April 2009.
[SMPTE]
SMPTE RP 136 Time and Control Codes for 24, 25 or 30 Frame-Per-Second Motion-Picture Systems
[SVG]
J. Ferraiolo. SVG Fragment identifiers. W3C Recommendation, September 2001. Available at http://www.w3.org/TR/2001/REC-SVG-20010904/linking#FragmentIdentifiersSVG.
[Temporal URI]
S. Pfeiffer, C. Parker and A. Pang. Specifying time intervals in URI queries and fragments of time-based Web resources. Internet Draft, March 2005. Available at http://annodex.net/TR/draft-pfeiffer-temporal-fragments-03.html. IETF Internet-Draft, March 2005.
[XPointer Framework]
P. Grosso, E. Maler, J. Marsh and N. Walsh. XPointer Framework. W3C Recommendation, March 2003. Available at http://www.w3.org/TR/xptr-framework/.

B Evaluation of fitness per media formats

In order to get a view on which media formats belong to which fitness category, an overview is provided for key media formats. In the following tables, the 'X' symbol indicates that the media format does not support a particular fragment axis. The tables are separated by video/audio/image codecs and container formats.

Video CodecTrackTemporalSpatialNameRemark
H.261n/afitunfitn/a
MPEG-1 Videon/afitunfitn/a
H.262/MPEG-2 Videon/afitunfitn/a
H.263n/afitunfitn/a
MPEG-4 Visualn/afitunfitn/a
H.264/MPEG-4 AVCn/afitfit (in theory)n/aWith Flexible Macroblock Ordening (FMO), it is possible to encode spatial regions independently from each other ...
AVSn/afitunfitn/a
Motion JPEGn/afitunfitn/a
Motion JPEG2000n/afitfitn/a
VC-1n/afitunfitn/a
Diracn/afitunfitn/aWhen Dirac is stored in the Ogg RFC 3533 container using Skeleton Skeleton, ROE ROE and CMML CMML, track, temporal and named fragments are supported.
Theoran/afitunfitn/aWhen Theora is stored in the Ogg RFC 3533 container using Skeleton Skeleton, ROE ROE and CMML CMML, track, temporal and named fragments are supported.
WebMn/afitunfitn/a
RealVideon/afitunfitn/a
DVn/afitunfitn/a
Betacamn/afitunfitn/a
OMSn/afitunfitn/a
SNOWn/afitunfitn/a

Audio CodecTrackTemporalSpatialNameRemark
MPEG-1 Audion/afitn/an/a
AACn/afitn/an/a
Vorbisn/afitn/an/aWhen Vorbis is stored in the Ogg RFC 3533 container using Skeleton Skeleton, ROE ROE and CMML CMML, track, temporal and named fragments are supported.
FLACn/afitn/an/aWhen FLAC is stored in the Ogg RFC 3533 container using Skeleton Skeleton, ROE ROE and CMML CMML, track, temporal and named fragments are supported.
Speexn/afitn/an/aWhen Speex is stored in the Ogg RFC 3533 container using Skeleton Skeleton, ROE ROE and CMML CMML, track, temporal and named fragments are supported.
AC-3/Dolby Digitaln/afitn/an/a
TTAn/afitn/an/a
WMAn/afitn/an/a
MLPn/afitn/an/a

Image CodecTrackTemporalSpatialNameRemark
JPEGn/an/aunfitn/a
JPEG2000n/an/afitn/a
JPEG LSn/an/aunfitn/a
HD Photon/an/afitn/a
GIFn/an/aunfitn/a
PNGn/an/aunfitn/a

Container FormatsTrackTemporalSpatialNameRemark
MOVfitn/an/afit QTText provides named chapters
MP4fitn/an/afit MPEG-4 TimedText provides named sections
3GPfitn/an/afit 3GPP TimedText provides named sections
MPEG-21 FFfitn/an/afit MPEG-21 Digital Item Declaration provides named sections
OGGfit (1)fitn/afit (2)(1) Using ROE ROE and Skeleton Skeleton, track selection is possible; (2) Using ROE, CMML CMML and Skeleton, named addressing of temporal and track fragments is possible
Matroskafitn/an/afit
MXFfitn/an/afit
ASFfitn/an/afitMarker objects provide named anchor points
AVIfitn/an/aunfit
FLVfitn/an/afit cue points provide named anchor points
RMFFfitn/an/a?
WAVXn/an/aX
AIFFXn/an/aX
XMF?n/an/a?
AUXn/an/aX
TIFFfitn/an/afitCan store multiple images (i.e., tracks) in one file, possibility to insert "private tags" (i.e., proprietary information)

C Technologies Survey

C.1 Existing URI fragment schemes

Some existing URI schemes define semantics for fragment identifiers. In this section, we list these URI schemes and provide examples of their fragment identifiers.

C.1.1 General specification of URI fragments

  • URI FragmentRFC 3986
    http://www.w3.org/2008/WebVideo/Fragments/wiki/Main_Page#Preparation_of_Working_Draft
    cited from RFC3986: "The fragment identifier component of a URI allows indirect identification of a secondary resource by reference to a primary resource and additional identifying information. The identified secondary resource may be some portion or subset of the primary resource, some view on representations of the primary resource, or some other resource defined or described by those representations. A fragment identifier component is indicated by the presence of a number sign ("#") character and terminated by the end of the URI."

C.1.2 Fragment specifications not for audio/video

  • HTML named anchorsHTML 4.0
    http://www.w3.org/2008/WebVideo/Fragments/wiki/Main_Page#Preparation_of_Working_Draft
    refers to a specific named anchor within the resource http://www.w3.org/2008/WebVideo/Fragments/wiki/Main_Page
  • XPointer named elementsXPointer Framework
    http://www.w3schools.com/xlink/dogbreeds.xml#xpointer(id("Rottweiler"))
    refers to the element with id equal to 'Rottweiler' in the target XML document http://www.w3schools.com/xlink/dogbreeds.xml
  • text (plain)RFC 5147
    http://example.com/text.txt#line=10,20
    identifies lines 11 to 20 of the text.txt MIME entity
  • SVGSVG
                      http://upload.wikimedia.org/wikipedia/commons/d/d2/Yalta_summit_1945_with_Churchill%2C_Roosevelt%2C_Stalin.jpg#svgView(14.64,15.73,146.98,147.48)
    specifies the region to be viewed of the SVG image http://upload.wikimedia.org/wikipedia/commons/d/d2/Yalta_summit_1945_with_Churchill%2C_Roosevelt%2C_Stalin.jpg

C.1.3 Fragment specifications for audio/video

  • Temporal URI/Ogg technologiesTemporal URI
    http://example.com/video.ogv#t=12.3/21.16
    specifies a temporal fragment of the OGG Theora video http://example.com/video.ogv starting at 12.3 s and and ending at 21.16 s
  • MPEG-21MPEG-21
    http://www.example.com/myfile.mp4#mp(/~time('npt','10','30'))
    specifies a temporal fragment of the MP4 resource http://www.example.com/myfile.mp4 starting at 10 s and and ending at 30 s

C.2 Existing applications using proprietary temporal media fragment URI schemes

In this section, we list a number of proprietary URI schemes which are able to identify media fragments. Note that all of these schemes only provide support for addressing temporal media fragments.

  • Google Video (announcement) 
    http://video.google.com/videoplay?docid=3047771997186190855&ei=MCH-SNfJD5HS2gKirMD2Dg&q=%22that%27s+a+tremendous+gift%22#50m16s
    Syntax: #50m16s
  • YouTube (announcement) 
    http://www.youtube.com/watch?v=1bibCui3lFM#t=1m45s
    Syntax: #t=1m45s

    YouTube also does click-throughs on their embedded videos to time offsets (see announcement), and returns to time offsets when people browse away from a video they half watched (see http://youtube-global.blogspot.com/2009/09/release-notes-91709.html).

  • Archive.org (uses the temporalURI specification Temporal URI) 
    http://www.archive.org/download/to-SF/toSF_512kb.mp4?t=74.5
    Syntax: ?t=74.5
  • Videosurf (announcement) 
    http://www.videosurf.com/video/michael-jordan-1989-playoffs-gm-5-vs-cavs-the-shot-904591?t=140&e=184
    Syntax: ?t=140&e=184 (with t=start, e=end)

C.3 Media fragment specification approaches

Media fragment ApproachTemporalSpatialTrackName
URI based
SVGNoYesNoNo
Temporal URI/Ogg technologiesYesNoYesYes
MPEG-21YesYesYesYes
Non-URI-based
SMILYesYesNo?No?
MPEG-7YesYesYesYes
SVGNoYesNo?
TV-AnytimeYesNoNoYes
ImageMapsNoYesNo?

C.3.1 URI based

C.3.1.1 SVG
C.3.1.1.1 Spatial

Possible via SVG 1.1 Fragment Identifiers. Only rectangular spatial regions are supported:

C.3.1.2 Temporal URI/Ogg technologies
C.3.1.2.1 Temporal

A Temporal URI Temporal URI is being used to play back temporal fragments in Annodex. The clip's begin and end are specified directly in the URI. When using "#" the URI fragment identfier, it is expected that the media fragment is played after downloading the complete resource, while using "?" URI query parameters, it is expected that the media fragment is extracted on the server and downloaded as a new resource to the client. Linking to such a resource looks as follows: 

                <a href="http://example.com/video.ogv#t=12.3/21.16" />
                <a href="http://example.com/video.ogv?t=12.3/21.16" />

It it possible to use different temporal schemes, which give frame-accurate clipping when used correctly:

                <a href="http://example.com/video.ogv?t=npt:12.3/21.16" />
                <a href="http://example.com/video.ogv?t=smpte-25:00:12:33:06/00:21:16:00" />
                <a href="http://example.com/audio.ogv?t=clock:20021107T173045.25Z" />
C.3.1.2.2 Track

Tracks are an orthogonal concept to time-aligned annotations. Therefore, Xiph/Annodex have invented another way of describing/annotating these. It's only new (since January 2008) and is called: ROE (for Rich Open multitrack media Encapsulation) ROE. With ROE you would describe the composition of your media resource on the server. This file can also be downloaded to a client to find out about the "capabilities" of the file. It is however mainly used for authoring-on-the-fly. Depending on what a client requires, the ROE file can be used to find the different tracks and multiplex them together. Here is an example file: 

                <ROE>
                <head>
                <link rel="alternate" type="text/html" href="http://example.com/complete_video.html" />
                </head>
                <body>
                <track id="v" provides="video">
                <seq>
                <mediaSource id="v0" src="http://example.com/video.ogv" content-type="video/ogg" />
                <mediaSource id="v1" src="http://example.com/theora.ogv?track=v1" content-type="video/theora" />
                </seq>
                </track>
                <track id="a" provides="audio">
                <mediaSource id="a1" src="http://example.com/theora.ogv?track=a1" content-type="audio/vorbis" />
                </track>
                <track id="c1" provides="caption">
                <mediaSource src="http://example.com/cmml1.cmml" content-type="text/cmml" />
                </track>
                <track id="c2" provides="ticker">
                <mediaSource src="http://example.com/cmml2.cmml" content-type="text/cmml" />
                </track>
                </body>
                </ROE>

This has not completely been worked through and implemented, but Metavid is using ROE as an export format to describe the different resources available as subpart to one media resource. Note that ROE is also used to create an Ogg Skeleton Skeleton in a final multiplexed file. Thus, the information inherent in ROE goes into the file (at least in theory) and can be used to extract tracks in a URI: 

<video src="http://example.com/video.ogv?track=a/v/c1"/>
C.3.1.2.3 Named

To include outgoing hyperlinks into video, you have to define the time-aligned markup of your video (or audio) stream. For this purpose, Annodex uses CMML CMML. Here is an example CMML file that can be used to include out-going hyperlinks next to or into Ogg RFC 3533 streams. ("next to" means here that the CMML file is kept separate of the Ogg file, but that the client-side player knows to synchronise the two, "into" means that CMML is multiplexed as a timed text codec into the Ogg physical bitstream creating only one file that has to be exchanged). The following defines a CMML clip that has an outgoing hyperlink (this is a partial document extracted from a CMML file): 

                <clip id="tic1" start="npt:12.3" end="npt:21.16" title="Introduction">
                <a href="http://example.com/fish.ogv?t=5" >Watch another fish video.</a>
                <meta name="author" content="Frank"/>
                <img src="fish.png"/>
                <body>This is the introduction to the film Joe made about fish.</body>
                </clip>

Note how there is also the possibility of naming a thumbnail, providing metadata, and giving a full description of the clip in the body tag. Interestingly, you can also address into temporal fragments of a CMML CMML file, since it is a representation of a time-continuous data resource: 

<a href="http://example.com/sample.cmml?t=npt:4" />

With CMML and ROE you can address into named temporal regions of a CMML file itself: 

<a href="http://example.com/sample.cmml?id="tic1" />
C.3.1.3 MPEG-21

Four different schemes are specified in MPEG-21 Part 17 MPEG-21 to address parts of media resources: ffp(), offset(), mp(), and mask():

  • ffp() is applicable for file formats conforming to the ISO Base Media File Format (aka MPEG-4 part 12 or ISO/IEC 14496-12) and is able to identifiy tracks via track_ID located in the iloc and tkhd box respectively

  • offset() is applicable to any digital resource and identifies a range of bytes in a data stream (similar functionality as the HTTP byte range mechanism).

  • mp() is applicable for media resources whose Internet media type (or MIME type) is equal to audio/mpeg, video/mpeg, video/mp4, audio/mp4, or application/mp4 and provides two complementary mechanisms for identifying fragments in a multimedia resource via:

    • a set of so-called dimensions (i.e., temporal, spatial or spatiotemporal) which are independent of the coding/container format: for the temporal dimension, the following time schemes are supported: NPT, SMPTE, MPEG-7, and UTC.

    • a hierarchical logical model of the resource. Such a logical model is dependent on the underlying container format (e.g., audio CD contains a list of tracks). The structures defined in these logical models are accessed with a syntax based on XPath.

  • mask() is applicable for media resources whose Internet media type (or MIME type) is equal to video/mp4 or video/mpeg and addresses a binary mask defined in a resource (binary masks can be achieved through MPEG-4 shape coding). Note that this mask is meant to be applied to a video resource and that the video resource may itself be the resource that contains the mask.

Note that hierarchical combinations of addressing schemes are also possible. The '*' operator is used for this purpose. When two consecutive pointer parts are separated by the '*' operator, the fragments located by the first pointer part (to the left of the '*' operator) are used as a context for evaluating the second pointer part (to the right of the '*' operator).

C.3.1.3.1 Temporal

Supported by the mp() scheme:

C.3.1.3.2 Spatial

Supported by the mp() scheme:

C.3.1.3.3 Track

Supported by the mp() and ffp() schemes:

C.3.1.3.4 Named

Supported by the ffp() and mask() schemes:

C.3.2 Non-URI-based

C.3.2.1 SMIL
C.3.2.1.1 Temporal

Playing temporal fragments out-of-context

SMIL allows you to play only a fragment of the video by using the clipBegin and clipEnd atributes. How this is implemented, though, is out of scope for the SMIL spec (and for http-based URLs it may well mean that implementations get the whole media item and cut it up locally): 

<video xml:id="toc1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" clipBegin="12.3s" clipEnd="21.16s" />

It is possible to use different time schemes, which give frame-accurate clipping when used correctly:

                <video xml:id="toc2" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" clipBegin="npt=12.3s" clipEnd="npt=21.16s" />
                <video xml:id="toc3" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" clipBegin="smpte=00:00:12:09" clipEnd="smpte=00:00:21:05" />

Adding metadata to such a fragment is supported since SMIL 3.0: 

                <video xml:id="toc4" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" clipBegin="12.3s" clipEnd="21.16s">
                <metadata>
                <rdf:.... xmlns:rdf="....">
                ....
                </rdf:...>
                </metadata>
                </video>

Referring to temporal fragments in-context

The following piece of code will play back the whole video, and during the interesting section of the video allow clicking on it to follow a link:

                <video xml:id="tic1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" >
                <area begin="12.3s" end="21.16s" href="http://www.example.com" />
                </video>

It is also possible to have a link to the relevant section of the video. Suppose the following SMIL code is located in http://www.example.com/smilpresentation:

                <video xml:id="tic2" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" >
                <area xml:id="tic2area" begin="12.3s" end="21.16s"/>
                </video>

Now, we can link to the media fragment using the following URI:

http://www.example.com/smilpresentation#tic2area

Jumping to #tic2area will start the video at the beginning of the interesting section. The presentation will not stop at the end, however, it will continue.

C.3.2.1.2 Spatial

Playing spatial fragments out-of-context

SMIL 3.0 allows playing back only a specific rectangle of the media. The following construct will play back the center quarter of the video: 

<video xml:id="soc1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" panZoom="25%, 25%, 50%, 50%"/>

Assuming the source video is 640x480, the following line plays back the same: 

<video xml:id="soc2" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" panZoom="160, 120, 320, 240" />

This construct can be combined with the temporal clipping.

It is possible to change the panZoom rectangle over time. The following code fragment will show the full video for 10 seconds, then zoom in on the center quarter over 5 seconds, then show that for the rest of the duration. The video may be scaled up or centered, or something else, depending on SMIL layout, but this is out of scope for the purpose of this investigation. 

                <video xml:id="soc3" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" panZoom="0, 0, 640, 480" />
                <animate begin="10s" dur="5s" fill="freeze" attributeName="panZoom" to="160, 120, 320, 240 />
                </video>

Referring to spatial fragments in-context

The following bit of code will enable the top-right quarter of the video to be clicked to follow a link. Note the difference in the way the rectangle is specified (left, top, right, bottom) when compared to panZoom (left, top, width, height). This is an unfortunate side-effect of this attribute being compatible with HTML and panZoom being compatible with SVG. 

                <video xml:id="tic1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" >
                <area shape="rect" coords="50%, 0%, 100%, 50%" href="http://www.example.com" />
                </video>

Other shapes are possible, as in HTML and CSS. The spatial and temporal constructs can be combined. The spatial coordinates can be animated, as for panZoom.

C.3.2.1.3 Track

SMIL has no way to selectively enable or disable tracks in the video. It only provides a general parameter mechanism which could conceivaby be used to comminucate this information to a renderer, but this would make the document non-portable. Moreover, no such implementations are known. 

                <video xml:id="st1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" >
                <param name="jacks-remove-track" value="audio" />
                </video>
C.3.2.1.4 Named

SMIL has no way to show named fragments in the base material out-of-context. It has no support for referring to named fragments in-context either, but it does have support for referring to "media markers" (named points in time in the media) if the underlying media formats supports them. Yet, no such implementations are known: 

                <video xml:id="nf1" src="http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4" >
                <area begin="nf1.marker(jack-frag-begin)" end="nf1.marker(jack-frag-end)" href="http://www.example.com" />
                </video>
C.3.2.2 MPEG-7
C.3.2.2.1 Temporal
Editorial note: Raphaël 
For all dimensions covered by MPEG-7 the use of indirection should not forgotten. http://www.example.com/mpeg7file.mp7#speaker refers to the "speaker" xml element of this resource. The UA needs to parse this element in order to actually point to this fragment.

A video is divided into VideoSegments that can be described by a timestamp. MediaTimes are described using a MediaTimePoint and MediaDuration, which are the starting time and shot duration respectively. The MediaTimePoint is defined as follows: YYYY-MM-DDThh:mm:ss:nnnFNNN (Y: year, M: month, D: day, T: a separation sign between date and time, h: hours, m: minutes, s: seconds, F: separation sign between n and N, n: number of fractions, N: number of fractions in a second). The MediaDuration is defined as follows: PnDTnHnMnSnNnF with nD number of days, nH number of hours, nM number of minutes, nS number of seconds, nN number of fractions and nF fractions per second. The temporal fragments can also be defined in Time Units or relative compared to a defined time. This MPEG-7 example describes a 'shot1' starting at 6sec 2002/2500 sec and lasts for 9sec 13389/25000 sec.

                <VideoSegment id=”video” >
                <MediaLocator>
                <MediaUri>http://www.w3.org/2008/WebVideo/Fragments/media/fragf2f.mp4</MediaUri>
                </MediaLocator>
                <TemporalDecomposition>
                <VideoSegment id=”shot1”>
                <TextAnnotation>
                <KeywordAnnotation>
                <Keyword>…</Keyword>
                </KeywordAnnotation>
                </TextAnnotation>
                <MediaTime>
                <MediaTimePoint>T00:00:06:2002F25000</MediaTimePoint>
                <MediaDuration>PT9S13389N25000F</MediaDuration>
                </MediaTime>
                </VideoSegment>
                </TemporalDecomposition>
                </VideoSegment>
C.3.2.2.2 Spatial

Selecting a spatial fragment of the video is also possible, using a SpatialDecomposition-element. This MPEG-7 example describes a spatial (polygonal) mask called "speaker" which is given by the coordinates of the polygon: (40, 300), (40,210), ..., (320,300).

                <VideoSegment>
                <SpatialDecomposition>
                <StillRegion id = “speaker”>
                <TextAnnotation>
                <FreeTextAnnotation> Journalist</FreeTextAnnotation>
                </TextAnnotation>
                <Mask xsi:type="SpatialMaskType">
                <SubRegion>
                <Poly>
                <Coords> 40 300, 40 210, ..., 320 300</Coords>
                </Poly>
                <SubRegion>
                </Mask>
                </StillRegion>
                </SpatialDecomposition>
                </VideoSegment>

The spatial video fragment can be combined with temporal information thus creating a SpatialTemporalDecomposition-element. 

                <VideoSegment>
                <SpatialTemporalDecomposition>
                <MovingRegion>
                <SpatioTemporalLocator>
                <MediaTime>
                <MediaTimePoint>T00:00:06:2002F25000</MediaTimePoint>
                <MediaDuration>PT9S13389N25000F</MediaDuration>
                </MediaTime>
                </SpatioTemporalLocator>
                <SpatioTemporalMask>
                <Mask xsi:type="SpatialMaskType">
                <SubRegion>
                <Poly>
                <Coords> 40 300, 105 210, ..., 320 240</Coords>
                </Poly>
                <SubRegion>
                </Mask>
                </SpatioTemporalMask>
                </MovingRegion>
                </ SpatialTemporalDecomposition >
                </VideoSegment>

A region of an image can also be described in MPEG-7

                <Image id="image_yalta">           <!-- whole image -->
                <MediaLocator>
                <MediaUri>http://upload.wikimedia.org/wikipedia/commons/1/1b/Yalta_Conference.jpg</MediaUri>
                </MediaLocator>
                [...]
                <SpatialDecomposition>
                <StillRegion id="SR1">          <!-- still region -->
                <SpatialMask>
                <SubRegion>
                <Box>14.64 15.73 161.62 163.21</Box>
                <SubRegion>
                </SpatialMask>
                </StillRegion>
                </SpatialDecomposition>
                </Image>
C.3.2.2.3 Track

Tracks can be described using the MediaSourceDecompositionType: 

                <Mpeg7>
                <Description xsi:type="ContentEntityType">
                <MultimediaContent xsi:type="MultimediaType">
                <Multimedia>
                <MediaSourceDecomposition gap="false" overlap="false">
                <Segment xsi:type="VideoSegmentType">
                <TextAnnotation>
                <FreeTextAnnotation>video</FreeTextAnnotation>
                </TextAnnotation>
                <MediaTime>
                <MediaTimePoint>T00:00:00</MediaTimePoint>
                <MediaDuration>PT0M15S</MediaDuration>
                </MediaTime>
                </Segment>
                <Segment xsi:type="AudioSegmentType">
                <TextAnnotation>
                <FreeTextAnnotation>audio</FreeTextAnnotation>
                </TextAnnotation>
                </Segment>
                </MediaSourceDecomposition>
                </Multimedia>
                </MultimediaContent>
                </Description>
                </Mpeg7>
C.3.2.2.4 Named

Media fragments can be identified by their id.

                <StillRegion id="speaker">
                ...
                </StillRegion>

The description of this media fragment can then be retrieved using the following URI:

http://www.example.com/mpeg7file.mp7#speaker
C.3.2.3 SVG
C.3.2.3.1 Temporal

SVG relies either on SMIL or HTML5 as a foreign object to introduce temporal media fragmentation. It has no temporal fragmentation of its own. One can add a video to a scene (as can be seen in example 2). Although it is possible to add a foreign object within SVG wherein HTML5 video elements can be added. This is (at the moment) not a solution for temporal segmentation as HTML does not support it either. 

                <foreignObject>
                <div xmlns="http://www.w3.org/1999/xhtml">
                <video src="myvideo.ogg"/>
                </div>
                </foreignObject>

Here's an example of a video that starts at second 5 and has a duration of 20 seconds:

                <svg xmlns="http://www.w3.org/2000/svg" version="1.2" xmlns:xlink="http://www.w3.org/1999/xlink" width="320" height="240" viewBox="0 0 320 240">
                <desc>SVG 1.2 video example</desc>
                <g>
                <video xlink:href="test.avi" volume=".8" type="video/x-msvideo" width="320" height="240" x="50" y="50" begin=”5s” dur=”20.0s” repeatCount="indefinite"/>
                </g>
                </svg>
C.3.2.3.2 Spatial

XML snippet specifying a region of an image within SVG: 

                <svg xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
                <g id="layer1">
                <"image" id="image_yalta" x="-0.34" y="0.20" width="400" height="167"
                xlink:href="http://upload.wikimedia.org/wikipedia/commons/1/1b/Yalta_Conference.jpg"/>
                <"rect" id="SR1" x="14.64" y="15.73" width="146.98" height="147.48"
                style="opacity:1;stroke:#ff0000;stroke-opacity:1"/>
                </g>
                </svg>
C.3.2.4 TV-Anytime
C.3.2.4.1 Temporal

Within TV-Anytime, programmes can be divided in segments. Segmentation refers to the ability to define, access, and manipulate temporal intervals (i.e. segments) within an AV stream. The following excerpt of a TV-Anytime description illustrates the use of segments:

                <TVAMain>
                <ProgramDescription>
                <ProgramInformationTable>
                ...
                </ProgramInformationTable>
                <ProgramLocationTable>
                ...
                </ProgramLocationTable>
                <SegmentInformationTable>
                <SegmentList>
                <SegmentInformation segmentId="segment_2">
                ...
                <SegmentLocator>
                <MediaRelTimePoint>T00:00:06:2002F25000</MediaRelTimePoint>
                <MediaDuration>PT9S13389N25000F</MediaDuration>
                </SegmentLocator>
                ...
                </SegmentInformation>
                </SegmentList>
                </SegmentInformationTable>
                </ProgramDescription>
                </TVAMain>

SegmentLocator locates the segment within a programme (instance) in terms of start time and duration (optional). If the duration is not specified, the segment ends at the end of the programme. Note that the types of MediaRelTimePoint and MediaDuration correspond to the MPEG-7 types MediaRelTimePointType and MediaDurationType respectively.

C.3.2.4.2 Named

Supported by the segmentId attribute of the SegmentInformationType. The description of the media fragment can be retrieved as follows: 

http://www.example.com/tv_anytime_description.tva#xpointer(//*[@segmentId="segment_2"])
C.3.2.5 ImageMaps
C.3.2.5.1 Spatial

Client-side image maps: The MAP element specifies a client-side image map. An image map is associated with an element via the element's usemap attribute. The MAP element content model includes then either AREA elements or A elements for specifying the geometric regions and the link associated with them. Possible shapes are: rectangle (rect), circle (circle) or arbitrary polygon (poly) 

                <img src="image.gif" usemap="#my_map"/>
                <map name="my_map">
                <a href="guide.html" shape="rect" coords="0,0,118,28">Access Guide</a> |
                <a href="shortcut.html" shape="rect" coords="118,0,184,28">Go</A> |
                <a href="search.html" shape="circle" coords="184,200,60">Search</A> |
                <a href="top10.html" shape="poly" coords="276,0,276,28,100,200,50,50,276,0">Top Ten</A>
                </map>
              
                <map name="my_map">
                <area href="guide.html" alt="Access Guide" shape="rect" coords="0,0,118,28">
                <area href="search.html" alt="Search" shape="rect" coords="184,0,276,28">
                <area href="shortcut.html" alt="Go" shape="circle"coords="184,200,60">
                <area href="top10.html" alt="Top Ten" shape="poly" coords="276,0,276,28,100,200,50,50,276,0">
                </map>

Server-side image maps: When the user activates the link by clicking on the image, the screen coordinates are sent directly to the server where the document resides. Screen coordinates are expressed as screen pixel values relative to the image. The user agent derives a new URI from the URI specified by the href attribute of the A element, by appending ? followed by the x and y coordinates, separated by a comma. For instance, if the user clicks at the location x=10, y=27 then the derived URI is: http://www.example.com/images?10,27 

                <a href="http://www.example.com/images" >
                <img src="image.gif" ismap alt="target"/>
                </a>
C.3.2.6 HTML 5
Editorial note: Silvia 
Currently, HTML5 relies on the abilities of the used media format for providing media fragment addressing. In future, HTML5 is planning to adopt the fragment URI specifications of this document for providing fragment addressing. Input from the WHAT and HTML working groups is requested.

D Acknowledgements (Non-Normative)

This document is the work of the W3C Media Fragments Working Group. Members of the Working Group are (at the time of writing, and in alphabetical order): Eric Carlson (Apple, Inc.), Michael Hausenblas (DERI Galway at the National University of Ireland, Galway, Ireland), Philip Jägenstedt (Opera Software), Jack Jansen (CWI), Yves Lafon (W3C), Wonsuk Lee (Electronics and Telecommunications Research Institute), Erik Mannens (IBBT), Thierry Michel (W3C), Guillaume (Jean-Louis) Olivrin (Meraka Institute), Soohong Daniel Park (Samsung Electronics Co., Ltd.), Conrad Parker (W3C Invited Experts), Silvia Pfeiffer (W3C Invited Experts), David Singer (Apple, Inc.), Raphaël Troncy (EURECOM), Davy Van Deursen (IBBT)

The people who have contributed to discussions on public-media-fragment@w3.org are also gratefully acknowledged. In particular: Olivier Aubert, Werner Bailer, Pierre-Antoine Champin, Cyril Concolato, Franck Denoual, Martin J. Dürst, Jean Pierre Evain, Ken Harrenstien, Kilroy Hughes, Philip Jägenstedt, Ryo Kawaguchi, Véronique Malaisé, Henrik Nordstrom, Yannick Prié, Yves Raimond, Julian Reschke, Geoffrey Sneddon, Felix Sasaki, Philip Taylor, Christian Timmerer, Jorrit Vermeiren and Munjo Yu.

E Change Log (Non-Normative)

@@This paragraph will be replaced by the change log. DO NOT TOUCH@@