Issues in Temporal Representation of Multimedia Documents : The Madeus Experience

[Next] [Previous] [First]

Design Goals

The goal of the Madeus application is to meet the following requirements:

• Structured Modelling and the definition of an Open Representation of Multimedia Information (Open Format)
• Merging the various dimensions of Multimedia Documents
• Incremental Manipulation of the Document through Interactive Interfaces
• Maintaining the Consistency of Multimedia Documents
• Automatic Production of Spatial and Temporal Layout
• Runtime support and remote access to multimedia streams

---

[Next] [Previous] [First]

Multimedia Composition Model

Multimedia documents are manipulated through four dimensions:

• Logical (structuring)
• Spatial (positioning)
• Hypermedia (linking)
• Temporal (synchronization)

---

[Next] [Previous] [First]

Temporal Composition Requirements

• Complete language (relational grammars)
• Incremental authoring methods (consistency, temporal layout)
• Flexible specifications (adjustable delays, ranges of durations, indeterministic behaviour)

---

[Next] [Previous] [First]

Temporal Abstractions

Abstract Representation of a multimedia object => Temporal Interval.

Multimedia objects can be classified into:

• Discrete              e.g. text, still pictures, graphics.
• Continuous        e.g. video, audio, virtual images.
• Indeterministic  e.g. queries, user interaction, computations.

Types of intervals:

• Free : [l, u]f value can be assigned freely in this domain
• Contingent : [l, u]c value known at end of its presentation

---

[Next] [Previous] [First]

Temporal Interval

Four important instants of the interval are considered :

• Mapping (allocation of media channels)
• Start (start of play time)
• Stop (end of play time)
• UnMapping (deallocation of media channels)

---

[Next] [Previous] [First]

Temporal Synchronization

Different levels of specification :

• Structure level (1 composite => 1 primitive)

  X= { A, B, C, D, ... } ; List_Before{X};

• Document level (n objects => m relations)

  X= {A, B, C, ..} ; A equals B; B before C; ....

• Object level (1 object => several sub-intervals or time slices)

  A where A1=A [0,5], A2=[5,end]

• System level (n objects => lip synchronization)

  X=Lip_Sync{A, B, Max_Skew} ;

• Protocol level (1 object => 1 protocol)

  Object Locator (X) = protocol://server/Object_Id

---

[Next] [Previous] [First]

A document Source Example

<doc> background = image1.jpg, synchronization=List_before

dynamic_style : { ( [0:3] sec , volume = [0, 50] ) }

<composite> name=scene1

<obj> name=A, url=http://opera.inrialpes.fr/toto.mpeg, loop =2

dynamic_style : { ( [0:10] sec, xpos = [100 : 300 pt ] );

( [end-5:end] sec, box_effect = explode ) }

</obj> ......

<relations>

A equals B; B before C; .....

</relations>

</composite>

.......

<composite> name=scene2

........

</composite>

.....

</doc>

---

[Next] [Previous] [First]

Example

---

[Next] [Previous] [First]

Example

---

[Next] [Previous] [First]

Example

---

[Next] [Previous] [First]

Temporal Representation Of the Document

• Symbolic Representation : based on a declarative style of temporal synchronization (objects + attributes + temporal relations)

  => limit the need of programming or scripting

• Internal representation : based on Hypergraphs ( 1 level of a hierarchy = 1 level of the logical structure )

---

[Next] [Previous] [First]

Temporal Constraints Management

Author:

• Add new media objects (Basic, Composite)
• Specify temporal constraints between media objects
• Define containers (for groups of related objects)

For each new specification:

• Represent the new specification in the constraint network
  (Introducing/modifying chains)
• Test the consistency of the introduced/modified chains
• In case of Inconsistency, restore previous consistent state

---

[Next] [Previous] [First]

An Example of Document Construction

---

[Next] [Previous] [First]

An Example of Document Construction

Nodes hold synchronization information + Spatio-temporal actions

---

[Next] [Previous] [First]

Types of Inconsistencies

• (a) : detecting cycles in the temporal graph
• (b) : searching for a solution for the constraint network described by the graph
• (c) : using flexibility of objects and delays to adjust dynamically the temporal scenario

( for more details see : http://opera.inrialpes.fr )

---

[Next] [Previous] [First]

Architecture of Madeus

A layered architecture of the Madeus application:

• An authoring tool allowing the manipulation of the document structure
  • Modification of the objects attributes
  • Insertion of new objects and relations
• A set of functions maintaining the internal representation of the document
• A Runtime Layer : Scheduler + TAC management + Navigation
• Object Layer :
  • Audio/video : Mpeg audio, AU format, Mpeg video, MJpeg,
  • Images : Png, Gif, Jpeg, ..
  • Formatted text (colors, fonts, ...)
  • Buttons, Links (internal and external)
• Stream Layer :
  • HTTP (Progressive handling of media streams)
  • Buffering (Prefetching by looking ahead in the temporal graph)

---

[Next] [Previous] [First]

Scheduling

---

[Previous] [First]

Conclusion

In our prototype we emphasize on:

• An open document format ( simple tagging and Objects + Attributes + Relations)
• Integrating heterogeneous media objects in a seamless way (text, images, audio, video, programs, user interactions ...)
• Taking full advantage of temporal flexibility:
  1. Free delays of temporal operators
  2. Using the temporal properties of objects (Shrink/Stretch)
  3. Handling of unpredictable behaviour (user interaction, )
• Powerful representation based on temporal graphs
  • Maintaining the consistency of the document
  • Temporal Formatting (static and dynamic)
  • Scheduling the presentation
• A full featured test bed prototype Madeus.

Several documents have been written with Madeus

---