We would especially like to thank Rob Martell (Digital Renaissance) for his review and contributions to this specification.
This document is a submission to the World Wide Web Consortium (see Submission Request, W3C Staff Comment). It is intended for review and comment by W3C members.
This document is a NOTE made available by the W3 Consortium for discussion only. This indicates no endorsement of its content, nor that the Consortium has, is, or will be allocating any resources to the issues addressed by the NOTE.
This document presents Timed Interactive Multimedia Extensions for HTML (HTML+TIME). This is a proposal for adding timing and synchronization support to HTML. HTML+TIME builds upon the SMIL recommendation to extend SMIL concepts into HTML and web browsers. The current version is a result of collaboration and review among Microsoft, Macromedia, Compaq/Digital and Digital Renaissance. It is currently only a proposal and subject to change. It assumed that the reader is familiar with the ideas expressed in the W3C Recommendation: SMIL [SMIL].
The W3C has recently approved SMIL as a recommendation. SMIL introduces many valuable ideas, but has some limitations. In particular, SMIL is a data interchange format for media authoring tools and players - it does not include a means to apply the ideas to HTML and web browsers. This document describes a means of extending SMIL functionality into HTML and Web browsers. The proposal includes timing and interactivity extensions for HTML, as well as the addition of several new tags to support specific features described in the SMIL 1.0 spec. HTML+TIME also adds some extensions to the timing and synchronization model, appropriate to the Web browser domain.
HTML+TIME is not intended to supplant SMIL. The SMIL 1.0 standard supports many applications that need to describe multimedia presentations, independent of HTML and traditional web browsers. HTML+TIME is closely aligned with SMIL to make it simple for authors and tools to use both specifications. HTML+TIME also introduces a number of extensions to SMIL that are required for a reasonable level of flexibility and control. These extensions could easily be worked into the SMIL specification as well (indeed, some of the ideas have been discussed by the SYMM WG in the context of SMIL).
The layout capabilities described in the SMIL specification are subsumed by the CSS functionality standard in current browsers. Several other minor features are also standard in HTML, and are not duplicated here (but are documented in Appendix B).
Finally, an Object Model is described for HTML+TIME. The SMIL 1.0 specification did not include this, but given the tradition of HTML and the DOM, we feel this is a critical aspect of the specification.
A set of extensions are described to add additional timing, interaction and media delivery capabilities to HTML. These are modeled closely along the lines of SMIL, and attempt to reuse terminology wherever feasible. The timing and interaction support augment current script support for timers and DHTML.
Using the timing extensions, any HTML element can be set to appear at a given time, to last for a specified duration, and to repeat (i.e. loop). Simple timing is supported with a very simple syntax, but more complex timing constructs can also be described. Interactive timing is supported. The first section of this document describes how the timing support is designed, and how script writers use the timing extensions.
In order to easily integrate time-based media (movies, audio and animation content), a set of new media tags are introduced (again, based upon the SMIL 1.0 specification), and the associated integration with the timing model is documented.
Additional tags are described to support fine-grained control of synchronization and media-loading behavior. Also, the notion of temporal hyperlinks presented in SMIL is generalized to apply to HTML in general.
SMIL introduces some very powerful elements that support conditional delivery of content, specifically to support differing client platform multimedia capabilities and preference settings. These are included with minor changes in this proposal as well.
Finally, the object model for the time support is documented. Included is a discussion of support for media extensions and for extension behaviors that will take advantage of the timing support.
With the advent of CSS, the DOM and dynamic properties, it is possible for HTML to be a much more powerful medium. Designers can now begin to think of the web page not just as a static page of information, but as a dynamic, interactive presentation. Various tools exist to author animation based upon custom runtimes, and at least one (Macromedia Dreamweaver) supports animation based upon script and timers. Finally, the W3C Recommendation: SMIL [SMIL] specifies a means of describing media-rich timelines in a separate XML-based file.
None of these solutions provide a simple, standard means for HTML authors to easily add timing and interaction relationships to arbitrary HTML elements, and to coordinate these with time-based media. HTML+TIME will fill this need. It defines a simple and powerful standard for time containment within the document.
Timing support is based upon a simple model of adding attributes to HTML elements. HTML elements can be set to have a begin time and a duration. Additionally, elements can be made to repeat. We describe this as "decorating" the HTML with additional attributes. The important point is that HTML authors need not learn an entirely new syntax or document structure to add timing to pages. They simply add attributes to the elements that they want to be dynamic.
For more complex scenarios, authors can group timing into timelines. These timelines can then be controlled and timed as well. The structure is in some ways analogous to the DOM structure (cf. the <div> tag in particular), in that it defines a local time region. Two means are provided of defining local timelines: a new tag, and an attribute that can be applied to HTML container elements.
Note that the timing support augments the behavior of the elements only with respect to the time during which the element is active or visible. The rules for applying the timing are fairly simple:
For time-based media, a media player is controlled to start and stop the media when appropriate.
See the discussion of current Syntax Issues.
All HTML elements that are legal within the BODY and that represent content or style, can support timing. Appendix A presents the list of HTML elements, and the classification of elements for the purposes of timing support.
For all HTML elements that support timing, the following simple timing attributes are supported:
The current element will begin when the referenced event is raised (plus any begin
value). If the referenced event is never raised, the current element may never be
active/displayed. If a negative begin (delay) value is used with this attribute, the
element will start when the event is raised, but will start the local timeline at an
offset from 0. See also the section on negative offsets in Usage
notes below.
If the named event is "none", this element will simply wait to be turned on
(e.g. by script).
No more than one of beginWith, beginAfter or beginEvent should be specified.
Legal values include:
If the value of the "skip-content" attribute is "true", and one of
the cases above apply, the content of the element is ignored. If the value is
"false", the content of the element is processed.
The default value for "skip-content" is "true".
Reviewers - is this really necessary in the HTML/CSS context?
Clock values have the following syntax:
Signed-Clock-value ::= ("+" | "-" )? Clock-value ; default is "+"
Clock-value ::= HMS-value | Timecount-value
HMS-value ::= (Hours ":")? Minutes ":" Seconds ("." Fraction)?
Timecount-value ::= Timecount ("." Fraction)?
("h" |"min" |"s" |"ms")? ; default is "s"
Hours ::= DIGIT+
Minutes ::= 2DIGIT ; range from 00 to 59
Seconds ::= 2DIGIT ; range from 00 to 59
Fraction ::= DIGIT+
Timecount ::= DIGIT+
2DIGIT ::= DIGIT DIGIT
DIGIT ::= [0-9]
The minutes and seconds fields in an HMS-value are constrained to the range 00 to 59; leading zeros must be specified for terms between ":" and "." delimiters. Hours can be any integer value, and need not have leading zeros. The fractional seconds in both HMS-values and Seconds-values can have arbitrary precision, but nothing greater than millisecond accuracy is guaranteed.
The following are examples of legal clock values:
- Clock values:
02:30:03 = 2 hours, 30 minutes and 3 seconds
2:33 = 2 minutes and 33 seconds- Timecount values:
3h = 3 hours
45min = 45 minutes
30s = 30 seconds
5ms = 5 milliseconds- Signed clock values:
+2:10 = plus 2 minutes and 10 seconds
+300ms = plus 300 milliseconds
-10.3 = minus 10.3 seconds (10,300 milliseconds)
We can add attributes to any object in HTML to add timing. By default, all timed elements are relative to a document root context, and so exist in a single time scope for the page. I.e. no nested timing is required/defined in the examples cited. Advanced timing support allows for more powerful constructs when the authors needs them. Nevertheless, scripters just beginning to use TIME do not have to understand anything about a timing structure or hierarchy to do simple things. Also, all the offsets are in the same timespace (in the simple, default case), making it easy to align elements in time that are laid out all over the page.
Example: Making paragraphs appear over time:
...
<p t:begin="1">
This is a paragraph of text that appears after one second
</p>
<p t:begin="2">
This is a paragraph of text that appears after two seconds
</p>
<p t:begin="3">
This is a paragraph of text that appears after three seconds
</p>
...
In order to support greater flexibility in terms of relative timing, we make use of the duration and time base parameters.
Note to reviewers:
Many multimedia runtimes support complete hierarchical relative timing, tied to a scene
graph or some equivalent. This makes sense, as the containment/lexical hierarchy is
explicit with a scene graph. However, most HTML authors do not work with a user
model of HTML that includes a containment/lexical hierarchy based upon the actual DOM.
Tying time containment to DOM containment would be confusing at best. As
such, we will support default timing as described above, plus explicit time containers and
referential time bases (i.e. timing relative to another named element). HTML authors
should easily accept referential timing, as IDs are used very commonly in scripting.
Example: Defining relative timing among elements:
...
<p id="P1" t:dur="5">
This is some text that appears immediately and remains
visible for 5 seconds.
</p>
<img src="image.gif" t:beginWith="P1" t:begin="0.5">
<!-- This is an image that appears just after the first
paragraph appears (i.e. at 0.5 seconds) and remains
visible indefinitely. Note the support for fractional seconds.
-->
<p t:beginAfter="P1" t:begin="-0.5">
This is some text that appears just before the first
paragraph goes away (i.e. at 4.5 seconds) and remains
visible indefinitely. Note the support for negative offsets.
</p>
...
Users can make any given element or timeline play repeatedly (i.e. loop), using the repeat attributes. Authors can specify either a number of times to play the simple duration, or a total time for which to repeat the element timeline.
...
<t:audio t:src="intro.mid"
t:repeat="indefinite">
<t:animation id="anim1" t:src="intro.x"
t:repeatDur="2min">
<t:animation t:src="http:wakeup.x" t:beginAfter="anim1"
t:repeat="indefinite">
...
The audio is set to repeat indefinitely. The intro animation will repeat for 2 minutes, and then stop. The second animation will then begin, and will repeat indefinitely. Note that the repeat controls can also be combined with the sequence element and timelines, described below.
The timeAction attribute provides a flexible means of defining what it means for an element to be active and not on a timeline. For the purposes of control in HTML+TIME, all HTML elements are grouped into categories (see also Appendix A). By default, all elements categorized as content will be controlled with the visibility property. That is, before the element is active on the timeline (before its defined begin time), the element visibility will be set to "hidden". While the element is active on the timeline (from the defined begin until the defined end), the visibility property will be set to "visible". Again, after the end time, the visibility property will be set to "hidden". All style elements will be controlled by removing the effect of the element intrinsic behavior. It would be nice if all style elements supported an "on" property to control easily and in a well-document manner.
Examples of default timeAction usage:
...
<span t:begin="10" t:dur="20">
This is some text that appears after ten seconds and remains
visible for 20 seconds.
</span>
...
<b t:begin="1" t:dur="10">
This is some text will appear normally at first, then
be displayed bold for 10 seconds, and then revert to normal
display again.
</b>
...
<v:oval t:begin="2" t:dur="10" t:timeAction="visibility">
...
The above example shows the default behavior for timeActions. The span will be hidden when inactive. The bold element will be visible, but not bold, when inactive. The VML extension element for an oval will be hidden, as it supports a visibility property.
Example using display value:
...
<span t:begin="10" t:dur="20" t:timeAction="display">
This is some text that appears after ten seconds, and remains
visible for 20 seconds. When it becomes visible and again when
it is hidden, the document will reflow.
</span>
...
The display value is useful when the author wants the document to reflow over time. This is useful, e.g. for image sequences, where only the active image should affect the layout of the document.
Example using style value:
...
<span style="text-decoration:line-through; color:red"
t:begin="10" t:dur="20" t:timeAction="style">
This is some text will appear normally at first, then
be displayed in red strikethrough for 10 seconds,
and then revert to normal display again.
</span>
...
The style value makes it easier to control a complex set of styles over time. Any style control that can be defined using the inline style attribute can be animated over time using this timeAction setting.
Example using onOff value:
... <v:oval t:begin="2" t:end="10" t:timeAction="visibility"> <v:fill t:begin="5" t:end="10" t:timeAction="onOff"> ... </v:oval> ...
This example shows the use of the onOff value to control the intrinsic behavior of an extension style - in this case a fill element for an oval in VML. The oval will appear unfilled for 3 seconds and then the fill will be applied until the oval is hidden at 10 seconds.
The simple time attributes provide a very easy to use mechanism to add simple timing to a page. A good deal of animation is supported just via this simple syntax. Nevertheless, there will be cases in which an author wants to build up more complex timing structures, and to easily manipulate them. HTML+TIME provides a new par attribute to structure timed elements. This introduces a local, nested timeline that can be manipulated independent of the document (or parent) timing. The naming comes from SMIL, and is short for "parallel". There may be a potential conflict with the notion of "paragraphs". If need be, the token can be renamed "parallel" or perhaps "timeline".
An additional <par> tag could also be introduced, but this is syntactic sugar for <span t:par='true'>. Reviewers?
An example use might be to set up a block of paragraphs with declared timing, which the author wishes to manipulate as an independent segment of animation (i.e. a relative timeline) within the document. The par attribute defines a relative timeline which can be manipulated as a unit, moved in time, looped, cut and pasted, etc.
Timeline Attribute Syntax
An alternate syntax may used, which is equivalent in terms of the time behavior. This uses a new element to define a local timeline. Reviewers: is this necessary? It more closely follows SMIL syntax, but does not fit into HTML as cleanly.
Timeline Element Attributes
Timeline containers can be timed in the same manner as any other element. An offset value and/or a time base will offset the entire local context, and shift the time of everything within the timeline scope (except elements timed to events out of the timeline scope).
Examples
<span t:par="true" id="TL1" t:dur="10"> <!-- This begins right away, and lasts for 10 seconds --> <p> This is some text that appears immediately </p> <p t:begin="2"> This is some text that appears after two seconds </p> <p t:begin="3"> This is some text that appears after three seconds </p> </span><div t:par="true" t:beginAfter="TL1" t:begin="0.2"> <!-- This begins slightly after the first chunk is done, at 10.2 seconds --> <p> This is some much more exciting text that appears right away on the new timeline, which does not begin until the first big timeline is done. It should be 10.2 seconds into the page display before you see this... </p> <p t:begin="1"> This is some more exciting text that appears one second into the second timeline, which should be 11.2 seconds after the page Hides up. Just imagine doing a slide show with this stuff... </p> </div>
The case sometimes arises that authors want to have a series of (e.g.) images appear. This can be accomplished with the time attributes described above, but a very simple declarative syntax is indicated to support this specific case. The <t:seq> tag is provided for this purpose. Note that this element can be used for other cases as well as the simple sequence, but this is not recommended. A sequence is not a good general purpose means of declaring timing structure when the document is being edited - changing from a sequence declared with the <t:seq> tag to another timing relationship requires much more work than changing timing attributes associated with individual elements.
Note to reviewers: Should this instead be presented as an attribute to containers, a la "timeline"? If the goal is to support novices, an explicit element may be easier for them to use.
Sequence Element Syntax
Sequence Element Attributes
The sequence element can be timed in the same manner as any other element. An offset value and/or a time base will offset the entire local context, and shift the time of everything within the local timeline scope.
Contained (child) tag Attributes
Children of a sequence element can take most of the time attributes, excepting the time base. Reviewers: it appears that it is legal in SMIL to allow children of a seq element to repeat, and potentially to have indefinite duration. How much flexibility should we allow? E.g. should we allow endEvent specification? In addition, it may make sense for the default syncBehavior for all sequence children to be locked, so that sequences hold together.
Example: A slide show of images:
...
<div width="200" height="200">
<t:seq t:repeatDur="indefinite">
<!-- This will sequence the three images, repeating indefinitely.
Any timebase parameters will be ignored, as the time base is
implicit via the sequence block. Offsets are legal.
Durations are recommended, as the default is to remain
visible indefinitely, which means that nothing after that
will ever show up.
-->
<img src="image1.gif" t:dur="2" t:timeAction="display">
<img src="image2.gif" t:dur="2" t:timeAction="display">
<img src="image3.gif" t:dur="2" t:timeAction="display">
</t:seq>
</div>
...
Note that this ignores all aspects of layout. It is up to the HTML author to describe the desired layout (e.g. laid out left to right, stacked with absolute positions, etc.). With the timeAction attribute set to "display", the document will reflow over time.
Example: A sequence of styles:
...
<p>
<t:seq t:repeatDur="indefinite">
<!-- This will sequence the three styles, repeating indefinitely. -->
<FONT color="red" t:dur="2">
<FONT color="green" t:dur="2">
<FONT color="blue" t:dur="2">
Here is some text that will get a really gaudy color treatment.
</FONT>
</FONT>
</FONT>
</t:seq>
</p>
...
Multimedia without interaction is just a movie. It must be possible for the author to describe interactive responses to user actions, and to define timing variants that support interaction. In this timing model, interactive timing is just a variant in which the begin time is indeterminate. An element (or an entire timeline container) that should begin in response to some user input is simply defined with a beginEvent. When the element is not tied to a specific event (e.g. a particular button click or a stream trigger), but rather will be started by script on the page, the element can be defined with '...beginEvent="none"...' . Such an element can be started from script using a simple, familiar syntax.
Exposed action methods
In order to support interactive control of a timed element, the following methods are exposed to script:
Example script syntax
Authors can simply use the familiar script events (methods) like onclick(), onmouseover(), onmouseout(), etc. to define actions on timed HTML elements. The script method implementation simply references the timed element by id, and then calls one of the action methods exposed by the element. Note that several possible script solutions are described, just for documentation.
Note that the final image is set to begin when the slideshow is complete. It is possible to set up a timeline of actions that chain off an interactive begin. The timing for dependent elements is computed when the head of the timing dependency chain is turned on with a trigger.
<div height=200 width=300>
<t:seq id="SLIDESHOW" t:beginEvent="none">
<!-- This begins when a trigger is sent.
This will sequence the three images.
If the user clicks on an image before the assigned
duration, it will advance to the next image.
-->
<img src="image1.gif" t:dur="5" t:timeAction="display"
onclick="this.endElement()">
<img src="image2.gif" t:dur="5" t:timeAction="display"
onclick="this.endElement()">
<!-- This uses endEvent syntax. Note that 'dur'
will override if there is no click by 5 seconds -->
<img src="image3.gif" t:dur="5" t:timeAction="display"
t:endEvent="onclick">
</t:seq>
<img src="showOver.gif" t:beginAfter="SLIDESHOW"
t:timeAction="display">
<p align=center onclick="SLIDESHOW.beginElement()">
Click here to begin the slideshow.
</p>
<p align=center>
If it advances too slowly for you, just click on an image
to advance it interactively.
</p>
</div>
SMIL introduces the notion of temporal hyperlinks. Rather than create a new element to handle this, we add capabilities to the timing model to support the same functionality. The implementation must catch navigation events (i.e. docReady events as well as changes to the hash property), and then advance the root timeline to the start time of the hash element. If the hash element itself is not timed, the element parents are traversed up to the document body to find a timed element. If the hash element is not contained in any timed block, then the document timeline begins normally. If the hash element is set to begin interactively (with beginEvent="none"), the element is turned on as though it had been triggered and the document timeline plays normally.
Timed Hyperlink control Attribute Syntax
In some instances, authors will want to preclude jumping into the middle of a timeline. One example would be an advertisement before a presentation; the author may not want the end-user to be able to skip the ad. A new attribute is supported on the body tag to control this behavior, allowing the author to enable or disable the support:
This nice thing about this approach is that it requires no change to the link tag, and can cleanly work even if the link is coming from a page that has no timing defined.
The timing syntax primarily addresses relationships among elements on a page. However, there also a need to define, and to be able to control the start of overall document time. In the simple case, it will be acceptable to start document time when the document is loaded. However, in particular for long HTML documents it may be unacceptable to defer the document timeline until the document has completely loaded. Rendering of the first screenful is performed as soon as possible in most browsers, and authors will require that time can be started to run animations, etc. near the top of the page.
The model presented here provides simple controls for authors to control the start of document time. By default, document time begins when the document is fully loaded. This covers many cases, and simplifies the model for novice authors. Additional settings cause document time to begin either immediately (as soon as possible), or when the document is complete (the document and all associated media and objects have been loaded). Finally, there is an advanced option for authors to specify the point in the document at which time should begin.
Need some examples of usage, and in particular some warnings and examples of trouble they can cause themselves by starting time before document.onLoad.
Add attribute to doc root (on body tag) to specify alternatives: immediate (default), onDocLoad, onDocComplete, onStartTag.
A new attribute is supported to specify the rule for when the document root timeline starts. This attribute is only legal on the body element.
Note that if document time begins before the document has fully loaded, the author must define all timing relationships such that the timing relationships are legal when document time begins, and as the rest of the document is parsed. This means that all timing references to other timed elements (e.g. using beginWith and beginAfter) must refer to elements that are defined earlier in the document. I.e. authors may not use forward references if document time begins before the document is loaded.
A new (XML) tag is supported to control when document time starts.
Time Scope
For time attributes that reference another timed element (e.g. beginWith, beginAfter), the referenced element must be timed (i.e. it must specify one of the TIME attributes or be a TIME element) and it must be in the current timing scope. That is, the referenced element must be within the HTML subtree defined by the closest parent time container of the current tag, and must have the same parent time container (i.e. it must be a a time-sibling).
In the absence of timeline attributed containers (or timeline tags), this will be the document root and all references will be in the same scope. If a timed element using a reference is within the scope of a timeline container, the scope is the local timeline block. It is illegal to reference any timed element outside of this scope. This constraint is imposed to preclude ambiguous and potentially confusing time dependency graphs.
Negative Offsets
The model described by HTML+TIME explicitly allows negative offsets. The common use of these is in something like a sequence, where one object should appear just before another completes. In this common case, the final computed begin time is still positive. Nevertheless, there are situations in which a negative computed begin time can obtain; this is not considered illegal.
When the computed start time for an element is negative relative to the timeline container, the element is started with the parent (it can never appear or have influence before the time container does). However, the sync relationship of the local timeline to the parent is offset: the local timeline for the element is defined to begin before it actually appears, and so it effectively begins somewhere in the middle of its timeline. This can be useful in situations where an element is set to repeat, and the author wants the first repeat iteration to begin in the middle (repeating motion-paths, scrolling, etc. are sometimes authored this way).
Invalid timing definition
It is possible to describe timing relationships between or among elements that are invalid. Typically, the timing is invalid because it creates circular time-dependency references. For example, if two elements are defined to begin with one another (using either beginWith or beginEvent syntax), this is invalid. Invalid timing can result through chained combinations of begin and end timing specification. Any combination that produces a circular reference is illegal.
When an invalid timing specification is detected by the implementation, an error will be generated, and one or all of the elements involved will revert to default timing.
If both a duration and any end value (including clip-end) are specified for any element, the effective duration will be the minimum of the specified duration attribute and the computed duration for the specified end attribute.
In order to coordinate HTML and time-based media elements on a common timeline, HTML+TIME introduces new tags to easily integrate time-based media. The new media tags will simplify the declaration of time-based media elements over the traditional methods of declaring various plug-ins or other embedded player objects. It will also manage the exchange of simple timing and control information between HTML+TIME timing implementation and the media players.
The tags are defined as XML extensions in the new TIME namespace. They will be ignored by down-level browsers.
Note that the definition of time-based media is not restricted to simple media like audio and video. Support is also intended for animation media, including extension players for existing popular animation formats. HTML+TIME would not render these, but would simply coordinate the associated players in the HTML time context.
SMIL introduced a set of new tags for a variety of media types. Currently, the individual tags have no semantic significance over the catch-all <ref> tag - the semantics are really tied to the MIME type from the server or specified as an attribute. However, the use of individual media type tags allows for future extensions such as specific attributes appropriate to individual media types (e.g. audio level). We would prefer to use "media" in place of "ref", but we can adopt the SMIL naming if desired.
The media tags take a src attribute to specify the source-media URL, and an optional specifier for the MIME/media type (the MIME type provided by the server is used by default). The implementation will associate the type with an appropriate player, and manage the instantiation of a player. Implementations may handle this e.g. by injecting HTML for an EMBED tag, or it may be handled more deeply.
There is also a player attribute that supports a reference to EMBED or OBJECT element from the media tags. This allows authors to use the traditional means of declaring the player, in particular to support all the specific attributes and controls of the respective players. The referenced element must support the media player interface described in the object model below. This also supports integration of third-party media players with the HTML+TIME model.
Note that the declaration of individual media elements with associated time syntax does not preclude the implementation from associating a group of media elements with a single player instance. It may be desirable from a performance standpoint to combine the media elements into a single grouped-element (e.g. creating a temporary playlist for the associated player). However, this can have other drawbacks, including a loss of interactive control over individual elements. The noCombine attribute provides author control over this.
Media players must support a basic set of controls to integrate with HTML+TIME. The implementation or the media tags must present an interface to the media players (e.g. with a wrapper). The details of the control are provided in the object model discussion below.
Note that if no dur attribute is specified, the media wrapper will set the duration property of the element (it is legal to set the duration to indefinite). This will have the side-effect that end-time-dependents (other elements defined with beginAfter referencing this node) will have a defined begin time.
Need description of integration of stream-based events with TIME model. Should require minimal Object Model support, as events fit in like all other events. Need to discuss recommended support. Reviewers?
Stream-based events should be raised by the player object, and can be referenced as "object_id.event_name". When the player object is implicit (e.g. using a video tag), the associated media element should raise the events. This allows script and HTML+TIME event specifications to respond to all media and server generated events.
The tags described here parallel the tags defined in SMIL 1.0 [SMIL]. The only difference is the media tag which replaces the generic ref tag in SMIL 1.0, and the omission of the SMIL text tag, which is subsumed by simple HTML tags for text. Reviewers: what was the intended use of ref, and the naming used? Unlike SMIL, there is no need for the abstract region reference. There is a need for reference to a player object, but this is supported as an attribute on all types.
The media element takes the base HTML attributes appropriate to any div tag. The following are interpreted by the wrapper:
Clip-time-value ::= [Metric "="]( Clock-val | Index-val | Smpte-val
| timeID-val )
Metric ::= Smpte-type | "clock" | "index"
Smpte-type ::= "smpte" | "smpte-30-drop" | "smpte-25"
Smpte-val ::= Hours ":" Minutes ":" Seconds
[ ":" Frames [ "." Subframes ]]
Hours ::= DIGIT+
Minutes ::= 2DIGIT ; range from 00 to 59
Seconds ::= 2DIGIT ; range from 00 to 59
Fraction ::= DIGIT+
Frames ::= 2DIGIT
Subframes ::= 2DIGIT
Index-val ::= DIGIT+
timeID-val ::= [legal HTML id]
2DIGIT ::= DIGIT DIGIT
DIGIT ::= [0-9]
The value of this attribute consists of an optional metric specifier (defaults to "clock"), followed by a time value whose syntax and semantics depend on the metric specifier. The following formats are allowed:
The time value has the format hours:minutes:seconds:frames.subframes. If the frame
value is zero, it may be omitted. Subframes are measured in one-hundredth of a frame.
Unlike the strictest SMPTE format, TIME will allow for more than two digits in the hours
field. This is specifically to accommodate long format material.
Examples:
clip-begin="smpte=10:12:33:20"
clip-begin="smpte=102:12:33"
clip-begin="123.45"
clip-begin="12:05:35.3" clip-begin="142"clip-begin="Bill talks about NT"
clip-end="Bill talks about simplicity"SMIL references a textstream media element, but does not in any way define or describe it. Simple cases could be handled with HTML using timing markup. Nevertheless, having a separate media type has advantages when streaming a large amount of content - e.g. captions to a lengthy audio presentation.
The specification of a "textstream" must be formalized before it can be widely supported by media players.
To integrate timeline media and animation into the page, authors must have control over the synchronization behavior of the page. In addition, authors need to be able to define how time behaves relative to the initial loading or cueing of media. HTML+TIME unifies support for asynchronous media loading and dynamic resynchronization of players into a simple concept of sync rules and scope. HTML+TIME defines additional attributes for media elements, as well as a general mechanism for managing dynamic synchronization of timeline elements.
This level of support requires a means to specify which media elements must be ready (i.e. loaded or cued) in order to play a portion of an animation timeline. This allows the author to control when the page or any local timeline (e.g. a div) starts, relative to the media that is required. The author can force the page to wait for all media to be ready. Alternatively, the author can specify that a timeline (e.g. the main page timeline) can begin when certain specific media elements are ready, but before all media has been prepared. The author can define and control the end-user experience.
Dynamic synchronization support provides an author the means to define which elements must remain in tight synchronization, and which elements (or local timelines) can slip if the players cannot keep up (e.g. due to network congestion). This provides a balance between the requirements of coordinating an animation and the realities of network media delivery.
The author's model will be very simple: they can describe the synchronization behavior of each media element and of each time container. By definition, synchronization includes the startup sync relationship; allowing an element to slip sync provides control over media loading. By describing the sync behavior of time containers, authors can control the scope of a synchronization context. When the timing model must handle resync events (i.e. when a media player falls out of sync while playing), the sync rules of the time containers define how far-reaching the resync handling will be.
In addition, authors can describe how to fill in for elements that are not ready to play at the time they were originally authored to.
For a time container, syncBehavior="locked"
means that the local timeline must remain locked to the parent timeline. If elements
within the time container are defined with syncBehavior="canSlip",
the time container setting does not overrule the contained element setting. This only
defines the sync relationship of the time container to the parent.
Note that the most common approach to resolving sync problems will be to pause the parent
timeline. If however the parent is also defined with locked sync, the resync must be
propagated up the time tree until a parent is reached that has slip sync defined, or until
the document body is reached. In the case of a fully locked timing definition, the
entire page timeline will be paused if any element falls out of sync and raises a resync
event.
Note that the behavior is only used when a direct exit-time dependent is not ready to play. If a dependent of a dependent is not ready, or if a dependent is defined relative to the start of the current element, no fill behavior is used.
Note that when the element has multiple time dependents, the fill behavior will be used if any one of the dependents cannot begin on-time (on-sync). There is no way to define the fill behavior for individual time-dependents. An example of a potential problem is an image that shows for a few seconds and is followed by a video and some audio. Assume the image is set to fill with the intention of covering for the video until it is ready. If both video and audio are defined to begin when the image ends (beginAfter="image_id"), and if the audio is not ready to start, the image will continue to show until the audio is ready. In this case, the author should define the audio as starting with the video, and make only the video directly dependent on the image.
Default settings Attribute syntax:
Example use cases:
Need examples of use of fill behavior.
By default, everything will have loose sync. This lets authors ignore the issues with maintaining sync among all the elements. This allows all the players the most leeway as well.
Example: syncing audio and video together, independent of the rest of the page:
...
<span t:par="true" t:syncBehavior="canSlip">
<t:media src="somevideo.avi" t:syncBehavior="locked">
<t:media src="someaudio.wav" t:syncBehavior="locked">
</span>
...
The sync of the video and audio is locked, meaning that the local timeline cannot start until the media for both elements is ready to roll. Also, if either player has problems during playback, the parent container must maintain sync.
The timeline sync is defined to slip, which means that the rest of the document timeline will not be held up when the video and audio media is loading/cueing. It also means that any resync event required by one of the video or audio players losing sync, will force the timeline to resolve the sync, but not to propagate the resync event to the parent timeline (which may be the document root timeline).
Example: streaming video relative to an animated page, and holding sync loosely:
...
<body t:syncBehaviorDefault="locked" t:syncToleranceDefault="2.0">
...
<t:media src="somevideo.asf" t:begin="5">
...
The video is set to begin playing 5 seconds after the document timeline begins, and to hold sync +/- 2 seconds to the document timeline. This allows other animations on the page to stay in sync with the video. If the video falls out of sync by more than 2 seconds, the time implementation (at the document level) must resync the video (e.g. by pausing the root timeline or the video timeline until they are more closely aligned).
If an element is defined to start interactively (e.g. specifying beginEvent), the syncBehavior has a slightly modified interpretation. In this case, there is no original sync relationship defined between the element and the time container. As such, when the element is started (e.g. via script on a button), the sync relationship will be propagated to the time dependents, but no resync event is propagated to the time container. However, once the element has been started and a sync relationship has been established, the syncBehavior of the element will determine how the object will maintain sync. Thus an element can be defined to start interactively (with indeterminate sync) but to maintain sync once started. This is probably not a common authoring scenario.
It is assumed that media players and extensions provide at the basic level of control described below as the minimum for integration.
An emerging application of HTML combines the web browser with television, either as a traditional broadcast or in digital forms like DVD (e.g. see the ATVEF spec [ATVEF]. HTML+TIME is an ideal tool for these applications, providing a means of controlling time and synchronization in a web page that accompanies the television content. Several tools are provided for integrating HTML+TIME with applications like ATVEF:
SMIL introduces the notion of conditional attributes, and the switch construct built upon the conditional attributes. These are valuable constructs for many applications, and are included for support with HTML in web browsers.
The syntax described below is taken largely from the SMIL specification. Minor extensions to SMIL are described to generalize the support for HTML and the browser environment.
Test attributes provide a means of enabling or disabling an element based upon some predefined system parameters. If the expression testing a particular parameter evaluates true, the element is rendered normally. If the expression evaluates false, then the element is ignored (e.g. removed from the DOM tree), and will not be rendered.
In the context of SMIL, the use cases describe conditional delivery of various content forms, based upon built-in conditionals related to network bandwidth, screen size and depth, system language and various other user preferences. In the context of a some dedicated media players, the user preferences are associated with the SMIL renderer. Certain attributes can be directly mapped to system settings or reasonable defaults on most platforms (e.g. language). Associating some user preferences (e.g. for typical speed of the network connection) is less direct for browsers.
The recommended solution is to support typical defaults for all the parameters, and allow specific preferences in the environment. The preferences should be made available in the DOM (e.g. as attributes of the window or document). Browser installers could set these values, and/or a simple form page or equivalent would allow the user to set the test-attribute values.
The attributes supported by SMIL include (taken from the SMIL spec, with the descriptions it provides, and additional notes in italics):
Evaluates to "true" if one of the languages indicated by user preferences exactly equals one of the languages given in the value of this parameter, or if one of the languages indicated by
user preferences exactly equals a prefix of one of the languages given in the value of this parameter such that the first tag character following the prefix is "-".
Evaluates to "false" otherwise.
Note: This use of a prefix matching rule does not imply that language tags are assigned to languages in such a way that it is always true that if a user understands a language with a certain tag, then this user will also understand all languages with tags for which this tag is a prefix.
The prefix rule simply allows the use of prefix tags if this is the case.
Implementation note: When making the choice of linguistic preference available to the user, implementers should take into account the fact that users are not familiar with the details of language matching as described above, and should provide appropriate guidance. As an example, users may assume that on selecting "en-gb", they will be served any kind of English document if British English is not available. The user interface for setting user preferences should guide the user to add "en" to get the best matching behavior.
Multiple languages MAY be listed for content that is intended for multiple audiences. For example, a rendition of the "Treaty of Waitangi", presented simultaneously in the original Maori and English versions, would call for:
<t:audio t:src="foo.rm" t:system-language="mi, en"/>
However, just because multiple languages are present within the object on which the system-language test attribute is placed, this does not mean that it is intended for multiple linguistic audiences. An example would be a beginner's language primer, such as "A First Lesson in Latin," which is clearly intended to be used by an English-literate audience. In this case, the system-language test attribute should only include "en".
Authoring note: Authors should realize that if several alternative language objects are enclosed in a "switch", and none of them matches, this may lead to situations such as a video being shown without any audio track. It is thus recommended to include a "catch-all" choice at the end of such a switch which is acceptable in all cases.
The switch element allows an author to specify a set of alternative elements from which only one acceptable element should be chosen. An element is acceptable if the element is an HTML element, if any associated media-type can be decoded , and all of the test-attributes of the element evaluate to "true".
An element is selected as follows: the parser evaluates the elements in the order in which they occur in the switch element. The first acceptable element is selected at the exclusion of all other elements within the switch.
Thus, authors should order the alternatives from the most desirable to the least desirable. Furthermore, authors should place a relatively fail-safe alternative as the last item in the <switch> so that at least one item within the switch is chosen (unless this is explicitly not desired). Implementations should NOT arbitrarily pick an object within a <switch> when test-attributes for all fail.
Note that http URIs provide for content-negotiation, which may be an alternative to using the "switch" element in some cases.
Element Content
The switch element should be able to contain any HTML content.
Examples
These examples are taken directly from the SMIL 1.0 specification.
1) Choosing between content with different bitrate
In a common scenario, implementations may wish to allow for selection via a "system-bitrate" parameter on elements. The media player evaluates each of the "choices" (elements within the switch) one at a time, looking for an acceptable bitrate given the known characteristics of the link between the media player and media server.
...
<t:switch>
<img src="img_hires.gif" t:system-bitrate="40000">
<img src="img_midres.gif" t:system-bitrate="24000">
<img src="img_lowres.gif" t:system-bitrate="10000">
</t:switch>
...
2) Choosing between audio resources with different bitrate
The elements within the switch may be any combination of elements. For instance, one could merely be specifying an alternate audio track:
... <t:switch> <t:audio src="joe-audio-better-quality" t:system-bitrate="16000" /> <t:audio src="joe-audio" t:system-bitrate="8000" /> </t:switch> ...
3) Choosing between audio resources in different languages
In the following example, an audio resource is available both in French and in English. Based on the user's preferred language, the player can choose one of these audio resources.
... <t:switch> <t:audio src="joe-audio-french.au" t:system-language="fr"/> <t:audio src="joe-audio-english.au" t:system-language="en"/> </t:switch> ...
4) Choosing between content written for different screens
In the following example, the presentation contains alternative parts designed for screens with different resolutions and bit-depths. Depending on the particular characteristics of the screen, the player can choose one of the alternatives.
...
<t:switch>
<div t:system-screen-size="1280X1024" t:system-screen-depth="16">
...
</div>
<div t:system-screen-size="640X480" t:system-screen-depth="32">
...
</div>
<div t:system-screen-size="640X480" t:system-screen-depth="16">
...
</div>
</t:switch>
...
This is a list of miscellaneous SMIL elements which may need to be supported. The relevant portions of the SMIL spec are included here.
Note: for a list of SMIL elements that will not be supported, and for specific differences between SMIL and TIME extensions, see Appendix B.
The "meta" element can be used to define properties of a document (e.g., author, expiration date, a list of key words, etc.) and assign values to those properties. Each "meta" element specifies a single property/value pair.
Element Attributes
The "meta" element can have the following attributes:
If the value of the "skip-content" attribute is "true", and one of
the cases above apply, the content of the element is ignored. If the value is
"false", the content of the element is processed.
The default value for "skip-content" is "true".
The list of properties is open-ended. This specification defines the following properties:
Element Content
"meta" is an empty element.
There was a general issue related to the parameter specification syntax in the HTML+TIME specification. This has been largely resolved, but the alternatives are described here for context.
The document currently presents one syntax to illustrate the model and provide simple examples in HTML. As the chosen syntax is based upon a draft specification, it may change. Nevertheless, the syntax changes, if any, will not materially affect the model of time containment in HTML documents.
In addition, this document describes the use of embedded XML elements in the HTML. This has been referred to as "XML sprinkles". While this is not part of a current standard, it has been discussed in a related note [XMLinHTML].
This version of the document presents a model of syntax that defines parameters using XML Namespace qualified attributes. The new XML Namespace proposal [XMLNS] allows for extension attributes that are qualified with a namespace id. This provides the cleanest syntax, and is used with this document. Nevertheless, at this point the XML Namespace proposal is only a draft.
It might be argued that simple html "expando" attributes would be easiest for authors to use. Such a syntax imposes minimal changes to the HTML that they author today. However, expandos are problematic in that they violate the HTML DTD (despite the fact that both major browsers parse them without problems).
An alternative syntax was proposed that moves the parameters to a STYLE string. These STYLE-expandos are HTML DTD compliant, and CSS specifies that unknown attributes be ignored, making this a reasonable syntax. This was rejected as less elegant than the XMLNS expando syntax.
E.g. where this document describes parameters as XMLNS expandos:
<p t:begin="1" t:dur="3">Some text...</p>
the syntax with STYLE-expandos would look like:
<p style="begin:1;dur:3">Some text...</p>
and the syntax with simple expando attributes would look like:
<p begin="1" dur="3">Some text...</p>
It should be possible to support SMIL 1.0 compliant documents using HTML+TIME. The layout mechanisms of SMIL can be translated to CSS2, and the timing constructs translate directly. A relatively simple extension could support such a translation mechanism. This could even be placed within an alternate clause in a switch statement, if the browser supports HTML+TIME extensions.
Because of the asynchronous nature of loading the SMIL file, there may be some issues related to synchronization with the rest of the document. In all likelihood, the start of the SMIL timeline would normally be deferred relative to the rest of the document, but this can be easily controlled with the synchronization control facilities in HTML+TIME.
Layout within the SMIL document will be achieved using standard CSS functionality. The SMIL syntax describes a subset of CSS, and so no extensions should be required. See also the related W3C note [SMIL-CSS]. The output HTML would likely be wrapped in a div (i.e. the SMIL:import tag should essentially subclass div). The SMIL specification provides for a declaration of the dimensions of the presentation; these will define the default dimensions of the wrapping div. How should the declared dimensions interact with the use of the screen-size test-attribute?
Finally, note that any SMIL renderer that supports the defined interface for media players can be hosted as an object (or embed) on the page. In this way, a pure SMIL 1.0 presentation can be placed in the web browser context for timing and synchronization with an HTML page.
Note that the translation is only for rendering, and not for editing. As such, there is no requirement for retaining the full fidelity of SMIL information, as long as the media is correctly rendered. Certain SMIL attributes have no presentation function and these need not be preserved in the translation.
Need to formally document the exposed methods, the events and the way the timing model will handle resync events. This is still rough, and needs further discussion and review. This is incomplete, but gives an idea of where we are going.
- beginElement()
- Turns on the behavior, starting the local timeline. Fires an onBegin event and propagates time dependencies.
- endElement()
- Turns off the behavior, stopping the local timeline. Fires an onEnd event. If the element is stopping prematurely (i.e. before the defined end time) this also raises an onResync event and propagates time dependencies. Note that turnOff is different from pause(). Unlike pause, turnOff is used to advance to the end of the local timeline, as though the element had played to the end.
- cue()
- Returns a boolean to indicate whether the element is ready to display (e.g. img media is ready). Time containers return an aggregate of all descendents. Todo: need to include notes on interaction of sync rules and cue state.
- pause()
- Pauses the local timeline, and raises an onResync event.
- run()
- Resumes a paused timeline. If the element was not paused, this has no effect.
- localToGlobalTime( int ltime, string elementid )
- Convert from a local time to a global time in the referenced time space. Returns an integer indicating the count of milliseconds that represents the local time ltime in the time space of the ancestor timeline specified by elementid. If elementid is null, this converts from the local timeline to the document global timeline. This is useful both for authoring applications as well as aligning or otherwise relating elements on disparate timelines. Note that ltime need not be within the constraints of the local duration. The conversion is always done relative to the simple local duration; to convert a simple local time in iteration n of a repeating timeline, the caller must account for the offset of n-1 simple durations.
The following properties will be supported. Some platforms may fire property change events, as an alternative means of wiring time-based script or other functions to the local timeline. The list below does not indicate legal elements for the different properties.
The properties for defining the basic timing are read-only. The model supports run-time modification only through the methods defined and the currTime property.