M. Integrating SMIL Timing into Other XML-Based Languages

Previous version:
http://www.w3.org/AudioVideo/Group/Integrate/WD-Time-Integrate-19990718 (W3C members only)
Erik Hodge (ehodge@real.com) (RealNetworks)
Warner ten Kate (tenkate@natlab.research.philips.com) (Philips Electronics)
Jacco van Ossebruggen (Jacco.van.Ossenbruggen@cwi.nl) (CWI)
Patrick Schmitz (pschmitz@microsoft.com) (Microsoft)
Ted Wugofski (Ted.Wugofski@OTMP.com) (Gateway)

Table of Contents


This segment of the working draft specifies an architecture for applying timing information to XML documents. It specifies the syntax and semantics of the constructs that provide timing information. This approach builds on SMIL by preserving SMIL's timing model and maintaining the semantics of SMIL constructs.

This part of the working draft does not attempt to describe the exact syntax required to apply timing to XML documents as multiple options are still under consideration by the W3C SYMM Working Group. There are examples containing several possible syntaxes throughout this segment of the working draft, but these are for illustration purposes only and are likely to change.

1. Introduction

Currently there exists no standardized method for adding timing to elements in any arbitrary XML document. This segment of the working draft defines the mechanisms for doing so.

1.1. Background

Prior to SMIL 1.0 becoming a W3C recommendation, a significant number of W3C members expressed interest in integrating SMIL timing functionality with XHTML and other XML-based languages.

SMIL 1.0 describes timing relationships between objects, including complete XML documents. SMIL 1.0 can not control the timing of individual elements contained within these documents, e.g., the display of a single XHTML heading before the bulk body text appears, or the sequential display of the items in a list. When using SMIL 1.0 for this, a content author is forced to contain each temporal element set in a separate document, leading to very small documents in some cases. For example, consider the split up of text that must occur when creating closed captioning from a subtitle track using SMIL 1.0.

The SMIL 1.0 architecture assumes that SMIL documents will be played by a SMIL-based presentation environment. It does not treat the case where timing is an auxiliary component, and the presentation environment is defined by another language, like XHTML, a vector-graphics language, or any user-defined XML-based language and stylesheet.

This segment of the working draft specifies how SMIL timing can be used in other XML languages, providing a solution to the above cases. The work is driven by the following goals:

1.2. Use cases

The following cases require the application of timing. These use cases are not listed in any particular order:

  1. Change media presentation over time.
    Various media objects contained in or referenced in an XML-based document are made to appear and disappear over time. Note: the media can be any element that models content: a video, a paragraph, a database record, etc. An example is a series of images and captions timed to appear as a slide show.

  2. Schedule the presentation of elements of a Web page.
    E.g., an HTML[*] page could be made to change over time by controlling the display of its elements.

    [*] Note: This assumes that the HTML document is a valid XML document.

  3. Add timing to an immutable document.
    Without modifying the original content document due to copyright and/or other issues, and without being able to assume IDs are used throughout that document, apply an external timing document to that content document. In some cases, timing will be applied externally to elements based on the names of their XML mark-up tags, while in other cases timing will be applied externally to elements of certain classes or to individual elements based on their unique IDs. For example, The Daisy Consortium's "talking book" applications use HTML documents containing the text of a book whose pages are marked with <SPAN> elements containing unique IDs. An external timing document could then be used to apply unique timing to each of these <SPAN> elements.

  4. Add timing to links.
    Links could be made to be active only during certain periods.
    Note: this can already be done within a SMIL 1.0 document.

  5. Change the appearance of graphical objects over time.
    For example, add timing to elements of a graphical display so that individual graphical elements appear, disappear, and move in front of and behind each other over time.

  6. Change the style, as opposed to the visibility, of textual data over time.
    For example, make something appear red for 5 seconds and then yellow for the remainder of its duration.

1.3. Assumptions

  1. The XML language to which the timing is applied can be of any type. The language can be:
  2. The XML language can cooperate with a stylesheet. The style language used is assumed, but not restricted, to be CSS.
1.3.1 Assumptions that may need further refinement
  1. The XML language can cooperate with the content document's Document Object Model (DOM).
  2. If the full document is exposable to the content document's DOM, that DOM models the data along the tree as spanned in the body.

1.4. Requirements

  1. Should follow the SMIL time model as it evolves.
  2. Should be compatible/interoperable with SMIL.
  3. Should be possible to apply the timing model to any XML application.
  4. Should enable authoring across documents, e.g., temporal specification may be separated from the content document.
  5. Should enable timing of styles as specified in the stylesheet accompanying the XML document.
  6. Should cooperate with events as specified by the content document's DOM.
  7. Should cooperate with dynamic changes invoked through the content document's DOM. For example, if a media element's begin time is based on the end time of another media element that has ended early, the former should begin right away rather than wait until its originally-scheduled begin time is reached.
  8. Should enable construction of temporal templates, such that timing styles can be developed and taken as an authoring basis for further refinement. The precedence rules are the same as for CSS.

2. Framework

This section outlines the conceptual approach to adding timing to XML applications. The Specification section specifies the constructs used. There are three proposed methods of adding this timing:

  1. Through In-line Timing. In-line timing is simply the addition of timing syntax into a content document to schedule the presentation of its objects.
  2. Through Cascading Style Sheet (CSS) Timing. CSS Timing treats timing as style attributes and allows the application of these timing style attributes to elements of the content of a document in the same way that CSS level-1 and level-2 currently allow other styles (e.g., color, spacing) to be applied to the content. CSS timing may be added to the content document or may be contained in an external document that is referenced by the content document.
  3. Through Timesheets. Timesheets are a new concept under development that apply timing to elements in the content document. The order of the items in a timesheet determines the order of presentation of the referenced content elements. Unlike CSS Timing, a Timesheet separates timing from the content document's structure. A timesheet may be added to the content document, may be contained in an external document that is referenced by the content document, or may be a document that references the content document external to itself.

How to ensure that in-line timing cooperates uniformly with CSS Timing or Timesheets is still under consideration.

In cases where SMIL timing is placed within an XML document, a hybrid DTD may be needed containing the DTD for the SMIL Timing and Synchronization module as well as the DTD for the XML language in which the original content document was written.

2.1. Framework: In-line Timing

Reminder: the various syntaxes specified in this segment of the working draft are likely to change prior to the finalization of the working draft.

In some cases in-line timing will make authoring easier, especially in cases where the author wants the timing to flow with the structure of the content. In other cases, CSS Timing or Timesheets may be needed.

The semantics of in-line timing are the same as that of SMIL 1.0 timing, but the syntax is different. SYMM is currently considering two ways to add in-line timing to XML content.

  1. The first is to add <par>...</par> and <seq>...</seq> elements to create time blocks that apply timing to all child elements. For instance, an author could place a seq element as a parent of a list of items and consequently make those list items display one after the other.
  2. The second way is to add timing attributes within the existing XML mark-up elements. This has the advantage of not requiring the use of an additional element that might make it harder to manage the layout and other behavior of the document.
    An element would be made to act as a parent par or seq (or other time container types under consideration, e.g., <excl>), along with optional SMIL timing attributes like duration, begin time (relative to that of any parent element), and end time, to name a few. In order to declare that an element should act as a time container, a new attribute is needed, possibly named "timeLine" or "timeContainer". This attribute is only legal within grouping elements in XML documents, and specifically cannot be applied to any of the time container elements including par, seq and excl.
    Children of an element with this attribute have the same semantics as children of the respective time container elements.  
    Possible syntaxes might be timeLine="t", or timeContainer="t", where "t" is par, seq, or some other time container under consideration.
    Legal values may include:
    Defines a parallel timeline with the same timing and synchronization semantics as a par element.
    Defines a sequence timeline with the same timing and synchronization semantics as a seq element.
    Defines an exclusive or choice timeline with the same timing and synchronization semantics as an excl element.
    Default value.  Defines the current element to not have timeline container behavior (i.e. to behave as a simple time leaf).

    For example, to add timing to an XHTML <DIV> element so that it acts as a "par" SMIL time container and has a duration of display of 10 seconds, the syntax might be:
    <DIV timeLine="par" dur="10s">.

    Another attribute under consideration is the action associated with the timing. This attribute would allow the author to specify how the element's timing should be applied, e.g., to the display of its content or to style attributes like the color of its content. One syntax currently under consideration is "timeAction".  In SMIL 1.0, the begin, end, duration, and other times specified in elements are always used to place the element on its parent element's timeline. A new attribute must be created to allow alternate application of the specified time values, e.g., the begin time could be applied to a style like the color of an element without affecting the true begin time of the element.
    The syntax for making the contents of a paragraph be red starting at 5 seconds relative to the parent's timeline might look like the following. Note that the begin time of the element is not specified and is assumed to be the begin time of its parent (or the end time of its sibling if its parent is a <seq> time container):
    <P begin="5s" style="color: red" timeAction="style">

    Open issue: using in-line timing, how do you set, say, the begin time of an element as well as the begin time of its style?

Here is an example of in-line timing being used to schedule the application of color style attributes as specified in the document's style sheet: Consider the playback of a music album where the audio track plays in concert with a list of the songs. Timing is added to the list so that the song that is currently playing is colored differently from the others. "timeAction" in this example means that the style of the class "playing" should be applied (only) to the text during the duration specified. Note that, in this example, "song 1", "song 2", and "song 3" all appear throughout the entire presentation; it is only their color that has been modified over time using (in-line) timing:

     body { color: black; }
    .playing { color: red; }
  <audio ...>
  <p dur="227s" timeAction="class:playing"> song 1 </p>
  <p begin="228s" dur="210s" timeAction="class:playing"> song 2 </p>
  <p begin="439s" dur="317s" timeAction="class:playing"> song 3 </p>

2.2. Framework: Cascading Style Sheet Timing

Reminder: the various syntaxes specified in this segment of the working draft are likely to change prior to the finalization of the working draft.

CSS Timing is the use of SMIL timing within a style sheet, where timing is a style attribute, just like, for example, color and font-weight in CSS, that is applied to elements in the content document. The resultant timing structure is based on and depends on the structure of the content document. In some cases, in-line timing may be inefficient, difficult, or impossible to add particular timing. In these cases, either CSS Timing or Timesheets may be needed. Some possible cases where CSS Timing will provide a better solution than in-line timing are:

The same two attributes mentioned in the In-Line Timing Framework section, above, will be needed. The first (possibly "timeContainer" or "timeline") is needed to be able to declare that an element should act as a time container. The second (possibly "timeAction") is needed to be able to specify how the timing should be applied, e.g., to the visibility of the object(s) or alternatively to a style applied to the object(s).

How to ensure that CSS timing and in-line timing cooperate uniformly is still under consideration.

Here is a simple example containing one possible syntax for integrating timing using CSS. In this example, the list will play in sequence as dictated by the style sheet in the HEAD section of the document. Note: the style sheet, like any CSS, could alternatively exist as a separate document.

      UL { timeLine: seq; }
      LI { font-weight: bold; duration: 5s; }
      <LI>This list item will appear at 0 seconds
	  and last until 5 seconds.
      <LI>This list item will appear after the prior
	  one ends and last until 10 seconds.

2.3. Framework: Timesheets

Timesheets refer to both the conceptual model along which timing, including the structure of the timing, is integrated into an XML document, as well as one possible syntax implementation. This approach provides a solution where time can be brought to any XML document regardless of its syntax and semantics.

A Timesheet uses SMIL timing within a separate document or separate section of the content document and imposes that timing onto elements within the content document. The resultant timing structure is not necessarily related to the structure of the content document. Some possible cases where a Timesheet will provide a better solution than in-line timing are a superset of such CSS Timing cases (which are included in the list below):

2.3.1. Three document sections

Timesheets assume an XML document conceptually composed of three presentation related sections:

  1. the content part.
  2. the formatting part.
  3. the timing part.
The 3 sections of an XML document

The first section, content, relates to the particular XML document. It conforms to a DTD written for an XML language. The content part describes the media and its structure.

The second section, formatting, provides control of the properties of the elements in the content section. It conforms to a style language, which, for the purpose of this discussion, we assume to be CSS. The style section describes the style and (spatial) layout of presenting the content. "Formatting" might include matters like routing of audio signals to loudspeakers.

The third section, timing, provides control of the temporal relations between the elements in the content section. It conforms to SMIL's timing model. The time section describes the time at which content is presented as well as the time at which style is applied. The time section contains the information to prepare a presentation schedule.

Sections two and three provide presentation information to the content: the stylesheet on style and positional layout, the timesheet on temporal layout. The stylesheet and timesheet may influence each other, but there should be no circular dependencies.

The idea is that each section operates independent from and compliant with the others.

2.3.2. Principles
  1. The temporal structure is not necessarily implied by the content structure. Here is an example.
  2. A timesheet may not be sufficient to build a time graph to provide a timing structure. A timesheet can consist of independent rules (time relations), which, together with the content, build the timing graph. For example, a selector in a timesheet may apply to multiple items in the content.
  3. Unspecified timing may be left to the implementation to fill in. For example, items in a list can be declared to appear sequentially, while the temporal relations between lists and other content remain unspecified. When the author does not supply these, the template is still to be obeyed.
  4. A timesheet may over-specify time relations. Unused rules are ignored. Conflicting time relations which concern the same element are either resolved using the timesheet cascading rules (to be specified, e.g. in-line overrides a template) or are an error (also to be made explicit). For example, when the timesheet declares sequential presentation of list items, while there are none of them in the document, the rule is simply ignored. Another example is where two rules select list items specifying different durations, e.g., all list item elements have a duration of 5 seconds except the first in each list has a duration of 8 seconds.

Here is a simple example where a timesheet exists, but in-line timing is also specified and overrides the timing imposed by the timesheet:

This example has a timesheet that specifies that each "li" element will have a begin time of 10 seconds and a duration of 15 seconds. However, the in-line timing in the second "li" element has presidence over the timesheet and thus the second line item ends up having a begin time of 0 seconds and a duration of 5 seconds.  Note: this example could have been done just as easily using CSS Timing; the added power of Timesheets will be made clearer in the next example.

  li { begin=10s dur=15s }
  <li>This first line will begin at 10 sec and run for 15 sec.</li>
  <li begin="0s" dur="5s">This second line's timing is dictated
     by the in-line timing which overrides the timesheet timing
	 for each child "<li>" element.  It will thus
	 begin at 0 seconds and last 5 seconds.</li>

Following is an example showing some HTML extended with timing via a Timesheet. As with the CSS example, the Timesheet could just as well have been contained in a separate document and applied externally. CSS selector syntax [CSS-selectors] has been used. The use of CSS selectors here should not be confused with CSS Timing, proposed in the prior section of this segment of the working draft.
The expected presentation of this would be to have the two Headings appear together followed by the first list item in each list, namely Point A1 and Point B1, appearing at 3 seconds followed thereafter by the second list item in each list, namely Points A2 and B2, appearing at 6 seconds. All items would disappear at 10 seconds which is the duration of the outer <par>.

        <par dur="10">
            h1 {}
          <par begin="3">
            <!-- Selects the first LI in each list:	-->
            OL > LI:first-child { }
          <par begin="6">
            <!-- Selects the second LI in each list:	-->
            OL > LI:first-child + LI { }
      <h1>Heading A</h1>
        <li id="PA1">Point A1</li>
        <li id="PA2">Point A2</li>
      <h1>Heading B</h1>
        <li id="PB1">Point B1</li>
        <li id="PB2">Point B2</li>

Note: the property fields {.} could contain duration and syncarc relations if the author wished to add more complex timing.

Here is another example as mentioned in Use Case 2C. Assume a human body display language. In this example different parts appear and disappear in different combinations at different times regardless of the content structuring, i.e., regardless of the order of the data in the document body. The document DTD uses the human structure: human = { face, torso, 2 arms, 2 legs }. A leg has a thigh, knee, calf and foot. Etc. The document merely describes the structure of the human form. Here is an example of such a document:

         <face id="face" ...>
            <eye id="leftEye" color="green" .../>
            <eye id="rightEye" color="blue" .../>
         <arm id="leftArm" ...>
            <hand id="leftHand" .../>
         <leg id="leftLeg" ...>
            <thigh id="leftThigh" .../>
            <knee id="leftKnee" .../>
            <calf id="leftCalf" .../>
            <foot id="leftFoot" .../>

Both of the following examples are possible by applying a different timesheet in each case to the same XML document. For these examples, we use the XML "human" document, above. Note: these examples demonstrate the timesheet's ability to allow a content element to be displayed as if its parent were but with the parent not displayed, in other words the child element is displayed in the same place, spatially, as if the parent was displayed.  "These examples presume that the XML language allows a content element to be displayed as if the full document was, but with some parents not displayed. In other words the child element is displayed in the same place, spatially, as if the entire document was displayed. Not all XML languages support this."

3. Specification

Once SYMM has settled on the approach to integrating timing into XML-based documents, this section will precisely define the syntax and semantics of each. To reiterate: the exact syntax and the respective semantics are still being debated. The sample syntax in this part of the working draft currently serve as only a hint as to what is being considered as well as to what issues are in the process of being resolved.

3.1. Specification: In-line Timing

In-line timing syntax has not been specified, but several possibilities are under consideration. The In-line Timing Framework section contains an example using SMIL timing.

3.2. Specification: CSS Timing

CSS timing syntax has not been specified, but several possibilities are under consideration.

The exact specification of CSS Timing selectors is still being considered. Selector algebra will most likely be that defined by CSS2 [CSS Selectors].

The CSS Timing Framework section contains an example using SMIL timing.

3.3. Specification: Timesheets

Timesheet syntax has not been specified, but several possibilities are under consideration. The Timesheets Framework section contains several examples (1, 2) using SMIL timing.

3.3.1. Structure copying

The structure of the body may be used to impose temporal semantics, where a time property is assigned to an element. It is important to realize that time relations are imposed between the elements selected. For instance, when selecting a <ol> in a <seq> relation, it means that the ordered list is going to be displayed after or before some other element. It does not mean that the list items contained by the ordered list are to be presented in a sequence.

In order to provide a syntax for denoting temporal relations in line with the body structure, a new type of selectors is added to those already available from CSS.

CSS has the notion of class selectors. These selectors imply that the rule (time relation) they are part of should be applied for each element in the body that is a member of that class.

Timesheets add a new type of class selectors, henceforth to be called structure selectors. These selectors imply that the time relation they are part of applies to the result of expanding the structure selector into id selectors of all elements in the body that are members of that structure class. The id selectors have to appear in the order in which the elements lexically appear in the body. In this way, by selecting the class of descendants, the structure of the body section can be copied into the time section, such that the copied structure receives the temporal semantics required.

3.3.2. Structure ownership

Another form of using the structure in the XML body is called ownership. Ownership dictates whether a temporal relationship imposed on an element applies to all of its descendants or only on the element itself. Ownership applies for example in the sequenced <ol> case when child <li> element(s) contain further markup. By specifying that ownership is on, the children of <li> element(s) will also take on the same temporal relationship as their parents.

3.3.3. Timesheet selectors

As discussed earlier, in timesheets there are two ways to expand class selectors:

  1. Class selector. The timesheet's rule applies per member in the class. This is the traditional CSS meaning; the timesheet's rule is repeated per element in that class.
  2. Structure selector. The timesheet's rule applies to the set elements resulting from expanding the class selector into all its elements. For example, a structure selector is used to create a <seq> of <li> without identifying all these <li> individually.

The exact specification of timesheet selectors is still being considered. Selector algebra will most likely be that defined by CSS2 [CSS Selectors] with some additional algebra defined as necessary.

3.3.4. Timing style

In addition to selecting elements, style rules should be selectable. This enables changing style properties over time, just as we saw in the In-Line Timing color style example.

3.4. Cascading Rules

In the case where in-line timing and another method are active simultaneously, in-line timing always takes precedence if a conflict arises. This enables the creation of CSS Timing or Timesheets to be used as templates whose rules can be easily modified locally by in-line constructs.

4. DTD

This section provides the formal specification which has not yet been specified.


Extensible Markup Language (XML) 1.0,
W3C Recommendation,
Tim Bray, Jean Paoli, C. M. Sperberg-McQueen,
10 Feb 1998.
Available at <http://www.w3.org/TR/1998/REC-xml-19980210>.
[CSS-level 1]
CSS Level-1,
Cascading Style Sheets, level 1,
W3C Recommendation,
Håkon Wium Lie, Bert Bos,
17 Dec 1996, revised 11 Jan 1999.
Available at <http://www.w3.org/TR/REC-CSS1/>.
[CSS-level 2]
CSS Level-2,
Cascading Style Sheets, level 2,
W3C Recommendation,
Bert Bos, Håkon Wium Lie, Chris Lilley, Ian Jacobs,
12 May, 1998.
Available at <http://www.w3.org/TR/REC-CSS2/>.
CSS selector syntax,
Cascading Style Sheets, level 2, Specification, chapter 5,
W3C Recommendation,
Bert Bos, Håkon Wium Lie, Chris Lilley, Ian Jacobs,
12 May, 1998.
Available at <http://www.w3.org/TR/REC-CSS2/selector.html#q1>.
Timed Interactive Multimedia Extensions for HTML (HTML+TIME).
Submitted to the World Wide Web Consortium. (See Submission Request.)
Patrick Schmitz, Jin Yu, Peter Santangeli,
18 September, 1998.
Available at <http://www.w3.org/TR/NOTE-HTMLplusTIME>.

Appendix. Examples

  1. Consider an XML-based image-list language. Each document contains a list of references to JPEG images. Timing of the images relative to one another is done in line. Here is an example of such a document, where each image in the list exists on the timeline for its specified duration and is then replaced, both spatially as well as on the timeline, by the next image. The final image will be active on the timeline for only 8 of its 10-second duration because the parent is explicitly specified to end at that time. Note: the presentation of the elements is implied in this example.
    <imagelist timeLine="seq" end="28s">
      <image dur="5s" src="image1.jpg" />
      <image dur="3s" src="image2.jpg" />
      <image dur="12s" src="image3.jpg" />
      <image dur="10s" src="image4.jpg" />
  2. This example uses CSS Timing to cause an otherwise static HTML list to grow over time, where each list item shows up below the prior item, 10 seconds after the prior item began its display. Because the UL becomes a "par" time container, its list items do not disappear until the UL's end time is reached.
    /* style sheet document "growlist.css": */
    .seqtimecontainer { timeLine: seq; dur: 30s}
    LI { dur: 10s; }
    <!-- HTML document (which happens to be well-formed XML): -->
       <LINK rel="stylesheet" type="text/css" href="growlist.css" />>
        <UL class="seqtimecontainer">
    	  <LI>This is item 1.  It appears from 0 to 30 seconds.
    	  <LI>This is item 2.  It appears from 10 to 30 seconds.
    	  <LI>This is item 3.  It appears from 20 to 30 seconds.
  3. Consider a document written in some graphics language where three big squares are layed out inside a rectangle, and each square contains a smaller square. We should be able to create a timesheet that can schedule the appearance of each square at different times from the others. Note: the presentation of the elements is implied in this example.
    <rectangle id="window" geometry="..." fill="...">
      <square id="b1" ... >
        <square id="s1" ... / >
      <square id="b2" ... >
        <square id="s2" ... / >
      <square id="b3" ... >
        <square id="s3" ... / >

    In order to time the presentation of the elements so that the big squares pop up one after the other, followed by the simultaneous appearance of the small ones, the timesheet might look like this:

          #b1 { dur: 2s }
          #b2 { dur: 2s; begin: 2s; }
          #b3 { dur: 2s; begin: 4s; }
          #s1 { }
          #s2 { }
          #s3 { }

    Note: the outer "window" rectangle has not been given any explicit timing. for this example, we assume that the lack of timing implies a begin time of zero and an infinite duration if the element does not have an implicit duration.