Position paper for W3C Workshop on Frameworks for Semantics in Web Services

Semantic Multimedia Adaptation Services
for MPEG-21 Digital Item Adaptation

Dietmar Jannach, Klaus Leopold, Christian. Timmerer, and Hermann Hellwagner

… Chairman of Ad-Hoc Group on MPEG-21 Digital Item Adaptation

Department of Information Technology, Department of Business Informatics and Application Systems

Klagenfurt University, 9020-Austria



Background and research area

Over the last years, our research groups have been actively involved in the development of the ISO/IEC MPEG-21 standard, in particular on Part 7 entitled Digital Item Adaptation (DIA) [1][2]. Our recent work in that area ([3],[4]) aims at developing a standardized framework for Semantic Multimedia Adaptation Services. In the context of Universal Multimedia Access (UMA), next-generation multimedia servers will be capable of intelligently transforming the multimedia content according to the users’ context before sending it over the network. We see the main challenge in this area that the field is rapidly evolving, i.e, new video encoding and compression techniques are being developed, metadata annotations (e.g., MPEG-7, Dublin Core) allow for content-based adaptation, and the heterogeneity of networks and end-user devices is continuously increasing.

As a result of this development, we cannot be optimistic that a single software tool will be available in the near future which is capable of doing all of the possible and required transformations of all kind of coding formats (e.g., JPEG, JPEG2000, MPEG-1/-2/-4, MPEG-4 AVC/SVC, H.26x, etc.) for all the different types of user preferences, network conditions, or (end) device capabilities.

In [3] and [4] we therefore propose a framework for multimedia adaptation that solves such complex adaptation task by executing an appropriate sequence of much simpler, individual transformation steps on the original content. The main goal of the framework is to enable a simple integration of external software components that provide specialized multimedia transformation functionality. Given the specific preferences (e.g., terminal capabilities) of a user, the multimedia server computes and executes appropriate adaptation sequences on the given media. For the construction of these adaptation chains, we propose to use formal, semantic descriptions of the effects of applying an individual function on the content. Based on such precise semantics, these descriptions are then exploited by a knowledge-based reasoner (planner) that composes suitable adaptation chains.

In particular, we propose the usage of OWL-S as representation language for describing adaptation tool semantics in terms of inputs, outputs, preconditions, and effects. Additionally, we use the existing MPEG-7 Multimedia Description Schemes (MDS) and MPEG-21 DIA standards as the shared domain ontology, as these standards already define the set of terms and symbols that can be used in the description of the semantics of service execution. The feasibility of the technical integration of OWL-S descriptions, the standard Web Service grounding, and the existing MPEG standards has been evaluated in two MPEG Core Experiments (CEs) and documented in an informative Annex of [5].

Our expectations/interest in the workshop

We view our work as a real-world use case for the application of automatic service composition based on semantic descriptions. The technical feasibility and the low-level integration into the related MPEG standards was demonstrated in a prototype implementation of the framework.

Nonetheless, the acceptance of the approach (within MPEG) will be hampered, as long as OWL-S and SWRL and in particular the languages for describing execution semantics are not standardized.

As such, our aims of the workshop participation are to:

        get an up-to-date insight on current developments both with respect to the technical infrastructure as well as recent trends in the area which will potentially further influence the ongoing standardization efforts in the MPEG-21 community, and

        optionally report actively on the experiences we gained from our application domain, i.e., what is actually possible, where did we need workarounds, what are the limitations of the current approaches and so forth.

Technical details

We built a a aaaadfframework which is capable of computing and executing multi-step adaptation sequences based on semantic descriptions of the available transformation operations [3]. We use OWL-S for representing inputs, outputs, preconditions, and effects (IOPE) and a Prolog-based planning engine that interprets these descriptions and produces adequate adaptation plans. For interoperability reasons, we used the existing MPEG standards as the shared domain ontology, which for instance defines terms that can be used in the IOPE descriptions.

Our framework consists of two major parts, the adaptation decision-taking engine and the resource/description adaptation engine as depicted in Figure 1. The adaptation decision-taking engine is responsible for finding a suitable sequence of transformation steps – the adaptation plan – that can be applied on the multimedia content. The adaptation plan is then forwarded to the resource/description adaptation engine which performs the actual transformation steps on the multimedia resource. In parallel, also the accompanying (MPEG-7) content descriptions are transformed adequately.

The finding of a suitable transformation sequence is modeled as a typical state space planning problem, where actions are applied on an initial state to reach a goal state. In our domain, the start state corresponds to the original multimedia content which is specified by means of MPEG-7 MDS metadata descriptions. The goal state is an adapted version of the multimedia content which satisfies the users’ context (i.e., preferences, device capabilities, and/or network conditions) which are expressed using MPEG-21 DIA descriptions. Actions are the conversion programs that can be applied on the multimedia content and which are described in terms of inputs, outputs, preconditions, and effects (IOPE) by using OWL-S together with SWRL.


Figure 1: Knowledge-based multimedia adaptation framework [6].


The following simplified example shows the description of the start state and the goal state of a multimedia adaptation problem where an image has to be spatially scaled and the color has to be removed. Note that for the sake of readability we use an informal notation rather than the internal OWL-S representation.

The start state – the MPEG-7 description of the existing multimedia content – can be described as follows:

jpegImage(http://path/to /image.yuv), width(640), height(480), color(true).

The goal state – the MPEG-21 DIA description of the user context – can be described as follows:

jpegImage(file://path/to/image.yuv), horizontal(320), vertical(240), color(false)

The following example shows the description of the spatial scaling operation for images based on the IOPE approach. We omit the similar grey-scaling description for sake of brevity.

Operation: spatialScale

Input:               imageIn, oldWidth, oldHeight, newWidth, newHeight
Output:             imageOut
Preconditions:   jpegImage(imageIn), width(oldWidth), height(oldHeight)
Effects:                        jpegImage(imageOut), width(newWidth), height(newHeight),
                        horizontal(newWidth), vertical(newHeight)

The adaptation framework features standard read and write operations used to read images from input sources and write images to output sources. The computed adaptation plan of the adaptation decision taking engine may look like as follows:

1.       read(http://path/to/image.yuv, outImage1)

2.       spatialScale(outImage1, 640, 480, 320, 240, outImage2)

3.       greyscale(outImage2,outImage3)

4.       write(outImage3, file://path/to/output/image.yuv)

By using the IOPE approach for modeling the functionality of adaptation services, our engine remains flexible, since the core planner operates on arbitrary symbols, such that new adaptation services can be easily added without changing the implementation. Moreover, IOPE-style descriptions have shown to be expressive enough for a wide range of problem domains. Another interesting aspect in our problem domain is that the level of detail of the functional descriptions of available adaptation services can vary. In the example given, each action is an atomic, single-step picture transformation. This fine-granular specification is reasonable in cases when, e.g., open source transformation software can be used in the adaptation engine. In this scenario, the adaptation chain and the execution plan is composed of API calls to a local media processing library. On the other hand, as newer standards for semantic content annotation like MPEG-7 are increasingly established, specialized software companies provide advanced adaptation functionality as service for their clients. With the approach described in this paper, however, the potential distributed nature of the individual services is transparent for the adaptation engine.

Concluding remarks

The first edition of MPEG-21 DIA has been published by ISO/IEC in October 2004 which addresses many interoperability issues imposed by UMA, i.e., enabling transparent access to (distributed) advanced multimedia content by shielding the user from terminal and network installation, configuration, management, and implementation issues. In other words, it allows users to access multimedia content anywhere, anytime, and with any kind of device. The first amendment to DIA entitled Conversion and Permissions to be finally approved in July/October 2005 targets the description of devices in terms of its supported conversion operations among others. In this position paper we indicated how OWL-S/SWRL can be used for this purpose by utilizing existing MPEG standards as the shared domain ontology.

Discussions on this topic are also taking place on the mailing list mpeg21-uma@merl.com, which can be subscribed to by sending an email to christian.timmerer@itec.uni-klu.ac.at.


1.        Vetro, A. and Timmerer, C., “Digital Item Adaptation: Overview of Standardization and Research Activities”, IEEE Transactions on Multimedia, Special Issue on MPEG-21, vol. 7, no. 3, June 2005.

2.        ISO/IEC 21000-7:2004, “Information technology — Multimedia framework (MPEG-21) — Part 7: Digital Item Adaptation”, First Edition, October 2004.

3.        Jannach, D., Leopold, K., Timmerer, C., and Hellwagner H.: Toward Semantic Web Services for Multimedia Adaptation. In: X. Zhou, S. Su, M. Papazoglou, M. Orlowska, K. Jeffery (Eds.): Web Information Systems - WISE 2004. Berlin, Heidelberg, New York: Springer November 2004 (LNCS, 3306), pp. 641-652. (Extended version to appear in: Applied Intelligence, Springer, 2005, cf. [6]).

4.        Jannach, D., Leopold, K., Hellwagner, H., and Timmerer, C.: A Knowledge Based Approach for Multi-step Media Adaptation. In: Instituto Superior Técnico, Lisboa, Portugal (Eds.): Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2004). April 2004.

5.        Timmerer, C., DeMartini, T., and Barlas, C., (eds.), “Information technology — Multimedia framework (MPEG-21) — Part 7: Digital Item Adaptation, AMENDMENT 1: DIA Conversions and Permissions”, ISO/IEC 21000-7 AMD/1, Final Proposed Draft Amendment 1 (FPDAM/1), January 2005.

6.        Jannach, D., Timmerer, D., Leopold, K., and Hellwagner, H., “A Knowledge-based Framework for Multimedia Adaptation”, to appear in the International Journal of Applied Intelligence, Springer Science+Business Media B.V., 2005.


Other MPEG documents are available at http://www.chiariglione.org/mpeg/ under ‘Hot news’ or ‘Working Documents’. An introduction to MPEG-21 – especially to DIA – is also available at http://mpeg-21.itec.uni-klu.ac.at/.