W3C W3C Member Submission

Timesheets: XML Timing Language

W3C Member Submission 15 April 2005

This version:
http://www.w3.org/Submission/2005/SUBM-xml-timing-20050415/
Latest version:
http://www.w3.org/Submission/xml-timing/
Editors:
Teppo Jalava, Helsinki University of Technology <tjjalava@cc.hut.fi>
Mikko Honkala, Helsinki University of Technology <mikko.honkala@hut.fi>
Mikko Pohja, Helsinki University of Technology <mikko.pohja@hut.fi>
Petri Vuorimaa, Helsinki University of Technology <petri.vuorimaa@hut.fi>

Abstract

This document specifies a new language for introducing temporal control to XML documents. The XML Timing language (Timesheets) is included as a distinctive section in a document and it assigns the timing information to the elements in the other sections of the document, in the same way the Cascading Style Sheets assign spatial properties to the elements.

Compared to SMIL, the XML Timing language can be seen as a different approach to the timing of XML documents. It reuses a subset of timing primitives from the Timing and Synchronization module of SMIL 2.0, but separates content, styling and timing for easier authoring and maintenance of multimedia documents.

Status of this Document

By publishing this document, W3C acknowledges that Helsinki Univerisity of Technology have made a formal submission to W3C for discussion. Publication of this document by W3C indicates no endorsement of its content by W3C, nor that W3C has, is, or will be allocating any resources to the issues addressed by it. This document is not the product of a chartered W3C group, but is published as potential input to the W3C Process. Publication of acknowledged Member Submissions at the W3C site is one of the benefits of W3C Membership. Please consult the requirements associated with Member Submissions of section 3.3 of the W3C Patent Policy. Please consult the complete list of acknowledged W3C Member Submissions.

Table of Contents

1 Introduction
2 Overview
3 Definitions
    3.1 Timesheet element
    3.2 Media element
4 Attributes
    4.1 The begin attribute
    4.2 The dur attribute
    4.3 Syntax of the time related attributes
    4.4 The repeat attribute
5 Elements
    5.1 The timesheet element
    5.2 The par element
    5.3 The seq element
    5.4 The excl element
    5.5 The item element
6 Semantics of the timing model
7 Events
    7.1 Internal events
    7.2 User events
    7.3 Event semantics
    7.4 Additional events
8 Annotated examples
    8.1 Simple without events
    8.2 With internal events
    8.3 With user events
9 Integration with CSS layout
10 Implementations
11 Future work
12 References
13 Acknowledgements

Appendix


1 Introduction

This document defines the features and syntax for XML Timing language. The objective of the language is to define temporal styling for XML documents. Synchronized Multimedia Integration Language (SMIL) is widely used for authoring multimedia presentations. However, SMIL has some limitations when used as a common multimedia declaration language for all XML documents [Timesheets]. For instance, if a document consists of several XML languages, they cannot be properly combined with SMIL. In addition, it has predefined layout model. Also, the desire of keeping content and presentational aspects of a document separate does not realize when using SMIL. The original design principle of XML is fulfilled if content and spatial and temporal styling of a document are separated to three functional sections.

2 Overview

Timesheets can be seen as a temporal counterpart for Cascading Style Sheets (CSS). Whereas CSS defines the spatial layout of the document and formatting of the elements, Timesheets specify what elements are active at a certain moment. And as with CSS, Timesheets can be reused in multiple documents, which can provide a common temporal layout for multimedia presentations with different contents but identical storylines. The document can be shown in a user agent even if the Timesheets are not supported, since the contents and the layout are still governed by the document itself. Of course, the temporal aspect of the document is then lost, since all the elements are active all the time.

A simple example of the Timesheets is shown in section 8.1 Simple without events. It shows a general XHTML document, which represents a single slide from a slideshow. When the document is viewed, first only the title "Introduction" and the page number are shown. Then, after two seconds, the text of the first bullet is shown, followed by other bullet in two seconds intervals. The placement and styling of the elements are specified in the CSS, while the Timesheet only specify the visibility of the elements at a given moment of time.

3 Definitions

3.1 Timesheet element

An XML document may contain one or more Timesheets. The elements under these Timesheets are referred to as Timesheet elements. A timesheet element may be a time container, which has other timesheet elements as its children, or an item element, which references to one or more Media elements (cf. Section 3.2 Media element). Timesheet elements are said to be active when the they have a resolved activation time, otherwise they are inactive.

3.2 Media element

Media element is an element that takes part in the layout of the XML document. They usually represent visual or audible content that is shown when the document is presented. Also, elements that control the structure of the document (div element in XHTML, for example) are media elements. Media element is said to be visible when the contents of the element is either visible or audible.

Media elements can be divided into two separate groups according to the type of content they represent.

Static elements

Static media elements represent content that has no implicit duration. These elements include the document structure elements and elements defining textual content and images.

Dynamic elements

Dynamic media elements represent content that may have an implicitely defined duration. This type of content include video and audio. In some cases, the duration of dynamic media may also be indefinite (streaming content, for example).

From the timesheet's point of view, the static and dynamic elements are treated differently. For the static elements, the author has to explicitely define the duration, otherwise the element will be shown an indefinitely long time. The duration of the dynamic elements is defined, so the author doesn't need to explicitely define it for them. Dynamic elements stop and become inactive when their content stops.

4 Attributes

This section defines the common attributes that are used by the elements in the timing language. The syntax of the attributes follows the SMIL 2.0 Timing and Synchronization Module specification [SMIL 2.0 Timing], althought the semantics of the attributes may differ (cf. Section 6 Semantics of the timing model).

Time related attributes

4.1 The begin attribute

The begin defines the start time of the element, relative to the scheduled activation of the element. Under a parallel time container the time is relative to the beginning of the parent container. Under sequential and exclusive containers the time is relative to the time when the parent container activated the element. The attribute is not mandatory, omitting it equals the attribute value "0s".

4.2 The dur attribute

The dur defines the duration of the element. The duration is relative to the start time of the element. If this attribute is omitted, the duration depends on the type of the element. For time containers, this means that the duration is unspecified and the container will stop after all it's children have stopped. For the item element, the interpretation of an unspecified duration depends on the type of the media element the item is referencing to. For static elements the duration is indefinite and the element will stay active unless the parent element inactivates it. For dynamic content the duration will be the duration of the content itself. An item referencing to multiple media elements has a duration of the longest lasting dynamic element, or indefinite, if any of the referenced elements are static. The duration is always overridden with the duration of the parent container.

Note:

SMIL has an additional attribute end to specify the conditions to end an element. The dur attribute in Timesheets contains the functionality of SMIL's end. The dur can be described by the word until, meaning that the element is active until the condition specified by the attribute becomes true. For example, "..until 6 seconds have passed", or "..until a certain link is clicked by the viewer".

4.3 Syntax of the time related attributes

The begin and dur attributes specify points on the time-line when some action should happen, either activating or inactivating an element and it's children. These points can be specified as an offset from some other point or an offset from a certain document event, or as a combination of both. The values can also be set to "indefinite". The syntax for the different cases are explained below.

Time offset

Denoted with a numerical value followed by the unit of time, which can be either s for seconds or ms for milliseconds. The numerical value must be positive.

Event offset

An offset from an event from the DOM that can, for example, be triggered by user action or some other timesheet element. The syntax for an event offset is built from the event's target id, event's type and optionally the time offset from the catching of the event. The target and the type of the event are separated by a dot (.) and the time offset is added with a plus (+) sign. For example, to wait for a DOMActivate event of an element that has an id link1 and specifying that the action is committed two seconds after the event, the value of the attribute would be "link1.DOMActivate+2s". The time offset part of the syntax is optional, omitting it is equal to setting the offset to zero seconds. To specify a non-existing target is an error.

Multiple values

The values are separated with comma (,). There can be multiple event offset values, but only one time offset value. If multiple time offset values are specified, only the first is taken into account while the rest are ignored.

Other attributes

4.4 The repeat attribute

The repeat defines the repeat count for the element. When the element stops and the current repeat count is positive, the element is activated again and actions of the element are repeated. The repeat attribute only has meaning with time containers and items that reference to dynamic content, since repeating static content has no effect. The dur attribute specifies the total duration of the element, repetitions included. This means that, for example, if the duration is set to one and half times the original duration of the element and the repeat value is more than one, the element is shown one and a half times, since the dur attribute overrides the actual duration of the element.

The repeat attribute can also have value "indefinite", which means that the element will be repeated until it is stopped by the time container or the value of dur attribute.

5 Elements

The language specifies a parent timesheet element, three types of time containers (par, seq and excl) and an item for connecting the timesheet elements to the host media elements.

5.1 The timesheet element

The timesheet element is the parent element for all the other elements of the timing language. The children of timesheet are scheduled similarily to par element. The common attributes, defined in the attributes section have no meaning when used with the timesheet element. Instead, two optional attributes are defined for the element.

src

The value of src tells the location of an external timesheet, which will be appended to the end of the local timesheet. With this attribute a common timesheet can be reused in multiple documents without the need to rewrite it every time.

media

The media attribute is used for selecting the most suitable timesheet for current media device. It works similarily to the @media rule in the CSS stylesheets [CSS 2.1].

5.2 The par element

Parallel time container. All the children of the par are activated simultaneously and the duration of the container, if not specified with the dur attribute will be the same as with the child that has the longest duration of all the children of the container. This element is similar to the par element specified in the SMIL 2.0 Timing and Synchronization Module [SMIL 2.0 Timing].

5.3 The seq element

Sequential time controller. The children of seq are displayed sequentially so, that next child is activated as soon as the previous child finishes. The first child is activated when the parent container starts. Only one child of seq can be shown simultaneously. Total duration of the seq is either specified by the dur attribute, or if it is omitted, by the sum of durations of all the children of the element. If a child of the container has an indefinite duration, the following children will never be activated and the duration of the container will be indefinite. This element is similar to the seq element specified in the SMIL 2.0 Timing and Synchronization Module [SMIL 2.0 Timing].

5.4 The excl element

The exclusive time container can have only one child active at a certain moment, similarly to the sequential time container. The excl doesn't provide any other synchronization or scheduling to its children, it only assures that no multiple children are active simultaneously. When a child starts according to its begin attribute, the container deactivates any other child that may be active.

Unlike the other time containers, excl has implicitely indefinite duration, since there is no defined situation when the container should end. The par ends when all the children has stopped, and the seq when the last child finishes. Neither approach is applicable to the excl, since the temporal order of the children is unknown.

The excl is a special case of the seq where the container's duration is set to indefinite and all the children have indefinite or interactive begin times. It is also a simplified version of the excl element defined in the SMIL 2.0 Timing and Synchronization Module [SMIL 2.0 Timing].

5.5 The item element

The item element implements the actual connection between the timesheet and the document. It can reference a media element or a set of elements by the element's tag, id or class; the syntax and processing is similar to the CSS selector syntax. The item element has three attributes, in addition to the common attributes. The select attribute gives the references to the media elements, the prefetch attribute controls the loading of the referenced elements and the class can be used to change the properties of the referenced elements.

The select attribute

The select attribute links the timesheet to the document. The attribute follows the same syntax and processing as the CSS selectors, so that the elements can be referenced by their name, id or class, or a more complex combination of the selectors. If the attribute targets multiple elements in the document, item controls all of them. The attribute value can also specify a comma separated list of selectors.

The prefetch attribute

The prefetch controls the loading of the resources needed by the elements referenced by the item element. Some resources, like audio, video and big images, can be quite big in size and fetching them on the fly while the timesheet is already running could skew the timing of the whole timesheet. With the prefetch attribute the author can mark certain elements and their resources to be prefetched to the memory before the timesheet is started. This way the resources can be shown precisely at the specified moment of time. If the attribute is omitted or set to false, the resources are loaded just before the item is activated. When the attribute is set to true the resources are loaded into the memory before the timesheet and the timeline is started.

6 Semantics of the timing model

The timing semantics of the timesheets is based on the semantics of the SMIL timing model [SMIL 2.0 Timing], although it has been simplified a bit. This may cause the timing model not to be as powerful in expression of timing relations as the SMIL's, but it should be more approachable to the authors.

The base time, or the syncbase of a timesheet element is the moment when the element is activated by its parent. Activating an element doesn't necessary make the referenced media element visible. Rather, it sets the time moment "0s", to which the element's attributes are compared. Thus, it's not possible to specify a negative value to the begin attribute, since it would mean that the element was supposed to start in the past when it was still inactive. The timesheet element can activate itself, according to the events it is set to listen. The self-activation of the element bubbles upwards in the DOM so that the parent elements do the necessary actions to allow the activation of the correct child.

The syncbase of the child elements of a time container is depended on the type of the container. The children of par and excl elements have the starting time of their parent as their syncbase. The children of the seq element consider the end time of preceding child as their syncbase. This time is resolved only at the moment the preceding child ends.

Figure 1 shows a simple example of the semantics of a timesheet. A parallel time container has two children, which are item elements referencing two media elements. When the par is started, it activates both its children with the current time as their syncbase. The media elements referenced by the children are not activated yet. At time moment 1s, the media element item1 is activated, according to the begin value of the referencing timesheet element. At 2s, the item2 is activated. At 3s, the duration of item1 runs out, so it is stopped and the corresponding timesheet element deactivates itself. At 4s, the parent container stops according to its duration attribute, stopping all of its active children.

Time semantics

Figure 1: Timeline of a Timesheets

The duration of an element is primarily defined by the dur attribute. If the element is not stopped prematurely, due to an event or scheduling of its parent, the dur attribute tells the overall duration of the element. The element will not stop until this duration has passed, and will not stay active longer that this duration. The value of the dur attribute will prevail

If the duration is not set, the duration of an element depends on the type of the referenced elements, or the durations of the children. The timesheet items can reference to static and dynamic elements in the document. Static elements don't have implicit durations, and the implicit duration of an item element referencing a static element is indefinite. Dynamic elements have their own durations, which will be also used as the duration of the timesheet element. The duration of the timecontainer depends on the durations and activations of its children. The seq and par elements stay active until all of their children have stopped. The implicit duration of the excl is always indefinite.

7 Events

The begin and dur attributes can contain references to DOM events [DOM Events]. DOM events can be triggered by user interaction or by some other event in the document. Events are divided into two distinct groups, internal events and user events.

7.1 Internal events

Internal events are dispatched from within the timesheets. They can be used by other elements in the timesheet to create relations between different parts of the timeline. The events specified are begin event, which is dispatched when an element starts and end event, which is dispatched when element stops.

7.2 User events

User events are triggered by the actions that user makes. A typical example is that the user clicks on a link in the document, and thus a DOMActivate event is dispatched [DOM Events].

7.3 Event semantics

A timesheet element is set to listen to a certain event by specifying the event's target and type by either begin or dur attribute. When specified by the begin attribute, an inactive or active but not yet started element will be started when it receives the specified event. This may involve the activation and starting of the parent elements as well, which can require some special scheduling to be done on the upper level of the DOM tree. Consider Figure 2 as an example. At a certain time, the item with the id "1" is active. This means that the second child of the seq (i.e., the par element) can't be active. Next, the item number "3" gets an event, which it is set to listen to by the begin attribute. To activate itself, the element must first inform its parent that it is going to start. The parent (i.e., the par element) is also inactive, so it informs its parent of the activation. The parent seq element then stops the excl element and its children to allow the par element to start, and par element then activates its children according to the definition.

Events on a timesheet

Figure 2: the affect of event on a timesheet.

When the element is specified to stop according to an event, it doesn't cause that much processing in the timesheet. The element informs its parent that it has stopped and parent then decides what should happen next. Of course, some other element could be waiting to start according to the end event from the particular element.

7.4 Additional events

There are also four additional events that are specified for the seq element. They are select-next, select-prev, select-first and select-last. These are used to select a certain child of the seq element, which facilitates the creation of presentations where the user can have control of the progression of the timeline, for example, a slide show. It should be possible to add later more additional events to any other elements of the timesheet.

8 Annotated examples

8.1 Simple without events

This example shows a document that could be a single slide in a slide show. A new bullet is shown every two seconds.

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:time="http://www.x-smiles.org/2004/timesheet">

<head>
  <style type="text/css">
  ...
  </style>

  <time:timesheet>
    <time:par>
      <time:item select="#bullet1" begin="2s"/>
      <time:item select="#bullet2" begin="4s"/>
      <time:item select="#bullet3" begin="6s"/>
      <time:item select="#bullet4" begin="8s"/>
    </time:par>
  </time:timesheet>
</head>

<body>
  <div id="content">
    <p id="title">Introduction</p>
    <p id="bullet1">o Some intro to WLAN</p>
    <p id="bullet2">o Some intro to WLAN</p>
    <p id="bullet3">- Subbullet</p>
    <p id="bullet4">- Subbullet</p>
    <p id="pagenum">2</p>
  </div>
</body>
</html>

8.2 With internal events

Next example is a picture show with captions for each image shown simultaneously. The new image is shown every five seconds, which dispatches a begin event that is listened by the corresponding item under the exclusive time container. The exclusive time container takes care that there can be only one caption active at a certain time. If the selection of an image is changed later, there is no need to touch the selection of captions.

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:time="http://www.x-smiles.org/2004/timesheet">

<head>
  <style type="text/css">
  ...
  </style>

  <time:timesheet>
    <time:par>
      <time:seq>
        <time:item select="image1" id="img1" dur="5s"/>
        <time:item select="image2" id="img2" dur="5s"/>
        <time:item select="image3" id="img3" dur="5s"/>
	...
      </time:seq>
      <time:excl>
        <time:item select="text1" begin="img1.begin"/>
        <time:item select="text2" begin="img2.begin"/>
        <time:item select="text3" begin="img3.begin"/>
	...
      </time:excl>
    </time:par>
  </time:timesheet>
</head>

<body>
  <div>
    <img src="image1.jpg" id="image1"/>
    <img src="image2.jpg" id="image2"/>
    <img src="image3.jpg" id="image3"/>
    ...
    <p id="text1">Text 1</p>
    <p id="text2">Text 1</p>
    <p id="text3">Text 1</p>
    ...
  </div>
</body>
</html>

8.3 With user events

This is an extended version of the previous example. In this version, the user has full control of progression of the show. When document is loaded, there's a welcome page that suggest the user to click on a link to start the show. Showing the pictures is not done automatically this time, instead the user must click on links "Next" or "Previous" to navigate through the show. At any time, the user can also jump to the beginning or the end of the presentation.

The start and end pages listen to DOMActivate event, which is dispatched when a user clicks a link. The forward/backward navigation can't use the same event, the seq element listens to events select-next and select-prev instead (cf. Section 7.4 Additional events). In this example, the XForms [XForms] is used to dispatch the events.

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:time="http://www.x-smiles.org/2004/timesheet"
      xmlns:ev="http://www.w3.org/2001/xml-events"
      xmlns:xforms="http://www.w3.org/2002/xforms">

<head>
  <style type="text/css">
  ...
  </style>

  <time:timesheet>
    <time:seq>
      <time:item select="#start-div" begin="0s,home.DOMActivate"/>
      <time:par begin="show.DOMActivate">
        <time:seq id="imageshow">
          <time:item select="#image1" id="img1"/>
          <time:item select="#image2" id="img2"/>
          <time:item select="#image3" id="img3"/>
  	  ...
        </time:seq>
        <time:excl>
          <time:item select="#text1" begin="img1.begin"/>
          <time:item select="#text2" begin="img2.begin"/>
          <time:item select="#text3" begin="img3.begin"/>
	  ...
        </time:excl>
	<time:item select="#linkdiv"/>
      </time:par>
      <time:item select="#end-div" begin="end.DOMActivate"/>
    </time:seq>
  </time:timesheet>
</head>

<body>
  <div id="start-div">
    <a href="" id="show" ev:event="DOMActivate" 
    ev:defaultAction="cancel">Start show</a>
  </div/>

  <div>
    <img src="image1.jpg" id="image1"/>
    <img src="image2.jpg" id="image2"/>
    <img src="image3.jpg" id="image3"/>
    ...
    <p id="text1">Text 1</p>
    <p id="text2">Text 1</p>
    <p id="text3">Text 1</p>
    ...
    <div id="linkdiv">
      <a href="">
        Previous
        <xforms:dispatch name="select-prev" target="imageshow" 
        ev:event="DOMActivate" ev:defaultAction="cancel"/>
      </a>
      <a href="">
        Next
        <xforms:dispatch name="select-next" target="imageshow" 
        ev:event="DOMActivate" ev:defaultAction="cancel"/>
      </a>
      <a href="" id="home" ev:event="DOMActivate" 
      ev:defaultAction="cancel">Home</a>
      <a href="" id="end" ev:event="DOMActivate" 
      ev:defaultAction="cancel">End show</a>
    </div>
  </div>

  <div id="end-div">
    <h1>Thanks for watching</h1>
    <a href="" id="home" ev:event="DOMActivate" 
    ev:defaultAction="cancel">Watch again</a>
  </div/>
</body>
</html>

9 Integration with CSS layout

This section is informative

Since Timesheets only describes the temporal dimension of the document, it must be integrated with a host language's layout system. It can be integrated to CSS based layout by affecting the CSS properties of media elements. For instance, the CSS display property controls whether the element is displayed or not. The timesheets processor sets the CSS pseudo-class timed-inactive for the media element when the element should not be visible based on the timesheet. Then, with a default rule for the pseudo-class, the CSS property display is set to "none" to the non-visible media element. The pseudo-class overrides element's original display property and the element becomes invisible. When the media element should become visible, the implementation removes the pseudo-class from the element, thus restoring the original value.

The default style sheet should contain a declaration with the pseudo-class called timed-inactive that is used to control the visibility of the media elements:

:timed-inactive { display: none ! important;}

Additionally, the language is extended by defining a class attribute for the item element. It can be used to change style properties of the elements from the timesheet. For example, the author could specify the red background of an element to be changed to blue at some time. This is done by setting the value of class attribute to reference a CSS pseudo-class that is used in the style sheet of the document. The author doesn't have to add the same document element with the same content multiple times with different style properties, but instead only define a pseudo-class for each property change and control them from the timesheet.

10 Implementations

This section is informative

An implementation of Timesheets has been developed at Helsinki University of Technology, in the X-Smiles project [X-Smiles]. X-Smiles is an open source XML browser, which supports several XML specifications and mixed namespace XML documents styled by CSS.

Currently, the Timesheets implementation only works with documents that can be laid out using Cascading Style Sheets. The integration method is described in the previous section. In the future, it should also be able to control the temporal properties of documents styled with other techniques, such as SVG.

11 Future work

The Timesheets language is intentionally kept as simple and compact as possible, to allow easy experimentation of the idea itself. More work should be done to broaden the usability of the Timesheets. Especially, the ability to present animations needs to be added to the language. Other enhancements include a more controlled way to handle the prefetching of the resources. Also Timesheets should also be integrated with other document languages, like SVG.

12 References

Timesheets
Kate, W. et. al.: Timesheets - Integrating Timing in XML. WWW9 Workshop: Multimedia on Web, 2000. (See http://homepages.cwi.nl/~lynda/www9/Timesheets.www9.html.)
SMIL 2.0 Timing
The SMIL 2.0 Timing and Syncronization Module (See http://www.w3.org/TR/smil20/smil-timing.html.)
CSS 2.1
Cascading Style Sheets, level 2 revision 1 CSS 2.1 Specification, W3C Candidate Recommendation 25 February 2004 (See http://www.w3.org/TR/CSS21/.)
XML Events
XML Events, An Events Syntax for XML, W3C Recommendation 14 October 2003 (See http://www.w3.org/TR/xml-events/.)
DOM Events
Document Object Model (DOM) Level 2 Events Specification, Version 1.0, W3C Recommendation 13 November, 2000 (See http://www.w3.org/TR/2000/REC-DOM-Level-2-Events-20001113/.)
XForms
XForms - The Next Generation of Web Forms (See http://www.w3.org/MarkUp/Forms/.)
X-Smiles
X-Smiles Project (See http://www.xsmiles.org.)

13 Acknowledgements

This specification and the reference implementation was done in the GO-MM project at Helsinki University Of Technology. We would like to thank the partners (companies and Tekes) for providing the funding. Also, we would like to thank all researchers, who participated that project. This specification is based on the ideas and work done by Kari Pihkala in his doctoral thesis on SMIL, for which we would like to express our utmost gratitude.