Semantic Multimedia Adaptation Services
for MPEG-21 Digital Item Adaptation
Dietmar Jannach, Klaus Leopold, Christian. Timmerer‡, and Hermann Hellwagner
‡ … Chairman of Ad-Hoc Group on MPEG-21 Digital Item Adaptation
Department of Information Technology, Department of Business Informatics and Application Systems
Klagenfurt University, 9020-Austria
Background and research area
Over the last years, our research groups have been actively involved in the development of the ISO/IEC MPEG-21 standard, in particular on Part 7 entitled Digital Item Adaptation (DIA) . Our recent work in that area (,) aims at developing a standardized framework for Semantic Multimedia Adaptation Services. In the context of Universal Multimedia Access (UMA), next-generation multimedia servers will be capable of intelligently transforming the multimedia content according to the users’ context before sending it over the network. We see the main challenge in this area that the field is rapidly evolving, i.e, new video encoding and compression techniques are being developed, metadata annotations (e.g., MPEG-7, Dublin Core) allow for content-based adaptation, and the heterogeneity of networks and end-user devices is continuously increasing.
As a result of this development, we cannot be optimistic that a single software tool will be available in the near future which is capable of doing all of the possible and required transformations of all kind of coding formats (e.g., JPEG, JPEG2000, MPEG-1/-2/-4, MPEG-4 AVC/SVC, H.26x, etc.) for all the different types of user preferences, network conditions, or (end) device capabilities.
In  and  we therefore propose a framework for multimedia adaptation that solves such complex adaptation task by executing an appropriate sequence of much simpler, individual transformation steps on the original content. The main goal of the framework is to enable a simple integration of external software components that provide specialized multimedia transformation functionality. Given the specific preferences (e.g., terminal capabilities) of a user, the multimedia server computes and executes appropriate adaptation sequences on the given media. For the construction of these adaptation chains, we propose to use formal, semantic descriptions of the effects of applying an individual function on the content. Based on such precise semantics, these descriptions are then exploited by a knowledge-based reasoner (planner) that composes suitable adaptation chains.
In particular, we propose the usage of OWL-S as representation language for describing adaptation tool semantics in terms of inputs, outputs, preconditions, and effects. Additionally, we use the existing MPEG-7 Multimedia Description Schemes (MDS) and MPEG-21 DIA standards as the shared domain ontology, as these standards already define the set of terms and symbols that can be used in the description of the semantics of service execution. The feasibility of the technical integration of OWL-S descriptions, the standard Web Service grounding, and the existing MPEG standards has been evaluated in two MPEG Core Experiments (CEs) and documented in an informative Annex of .
Our expectations/interest in the workshop
We view our work as a real-world use case for the application of automatic service composition based on semantic descriptions. The technical feasibility and the low-level integration into the related MPEG standards was demonstrated in a prototype implementation of the framework.
Nonetheless, the acceptance of the approach (within MPEG) will be hampered, as long as OWL-S and SWRL and in particular the languages for describing execution semantics are not standardized.
As such, our aims of the workshop participation are to:
― get an up-to-date insight on current developments both with respect to the technical infrastructure as well as recent trends in the area which will potentially further influence the ongoing standardization efforts in the MPEG-21 community, and
― optionally report actively on the experiences we gained from our application domain, i.e., what is actually possible, where did we need workarounds, what are the limitations of the current approaches and so forth.
We built a framework which is capable of computing and executing multi-step adaptation sequences based on semantic descriptions of the available transformation operations . We use OWL-S for representing inputs, outputs, preconditions, and effects (IOPE) and a Prolog-based planning engine that interprets these descriptions and produces adequate adaptation plans. For interoperability reasons, we used the existing MPEG standards as the shared domain ontology, which for instance defines terms that can be used in the IOPE descriptions.
Our framework consists of two major parts, the adaptation decision-taking engine and the resource/description adaptation engine as depicted in Figure 1. The adaptation decision-taking engine is responsible for finding a suitable sequence of transformation steps – the adaptation plan – that can be applied on the multimedia content. The adaptation plan is then forwarded to the resource/description adaptation engine which performs the actual transformation steps on the multimedia resource. In parallel, also the accompanying (MPEG-7) content descriptions are transformed adequately.
The finding of a suitable transformation sequence is modeled as a typical state space planning problem, where actions are applied on an initial state to reach a goal state. In our domain, the start state corresponds to the original multimedia content which is specified by means of MPEG-7 MDS metadata descriptions. The goal state is an adapted version of the multimedia content which satisfies the users’ context (i.e., preferences, device capabilities, and/or network conditions) which are expressed using MPEG-21 DIA descriptions. Actions are the conversion programs that can be applied on the multimedia content and which are described in terms of inputs, outputs, preconditions, and effects (IOPE) by using OWL-S together with SWRL.
Figure 1: Knowledge-based multimedia adaptation framework .
The following simplified example shows the description of the start state and the goal state of a multimedia adaptation problem where an image has to be spatially scaled and the color has to be removed. Note that for the sake of readability we use an informal notation rather than the internal OWL-S representation.
The start state – the MPEG-7 description of the existing multimedia content – can be described as follows:
jpegImage(http://path/to /image.yuv), width(640), height(480), color(true).
The goal state – the MPEG-21 DIA description of the user context – can be described as follows:
jpegImage(file://path/to/image.yuv), horizontal(320), vertical(240), color(false)
The following example shows the description of the spatial scaling operation for images based on the IOPE approach. We omit the similar grey-scaling description for sake of brevity.
Input: imageIn, oldWidth, oldHeight, newWidth, newHeight
Preconditions: jpegImage(imageIn), width(oldWidth), height(oldHeight)
Effects: jpegImage(imageOut), width(newWidth), height(newHeight),
The adaptation framework features s
1. read(http://path/to/image.yuv, outImage1)
2. spatialScale(outImage1, 640, 480, 320, 240, outImage2)
4. write(outImage3, file://path/to/output/image.yuv)
By using the IOPE approach for modeling the functionality of adaptation
services, our engine remains flexible, since the core planner operates on
arbitrary symbols, such that new adaptation services can be easily added
without changing the implementation. Moreover, IOPE-style descriptions have
shown to be expressive enough for a wide range of problem domains. Another
interesting aspect in our problem domain is that the level of detail of the functional
descriptions of available adaptation services can vary. In the example given,
each action is an atomic, single-step picture transformation. This
fine-granular specification is reasonable in cases when, e.g., open source
transformation software can be used in the adaptation engine. In this scenario,
the adaptation chain and the execution plan is composed of API calls to a local
media processing library. On the other hand, as newer s
The first edition of MPEG-21 DIA has been published by ISO/IEC in October 2004 which addresses many interoperability issues imposed by UMA, i.e., enabling transparent access to (distributed) advanced multimedia content by shielding the user from terminal and network installation, configuration, management, and implementation issues. In other words, it allows users to access multimedia content anywhere, anytime, and with any kind of device. The first amendment to DIA entitled Conversion and Permissions to be finally approved in July/October 2005 targets the description of devices in terms of its supported conversion operations among others. In this position paper we indicated how OWL-S/SWRL can be used for this purpose by utilizing existing MPEG standards as the shared domain ontology.
Discussions on this topic are also taking place on the mailing list firstname.lastname@example.org, which can be subscribed to by sending an email to email@example.com.
Jannach, D., Leopold, K.,
Timmerer, C., and Hellwagner H.: Toward Semantic Web Services for Multimedia
Adaptation. In: X. Zhou, S. Su, M. Papazoglou, M.
Orlowska, K. Jeffery (Eds.): Web Information Systems
- WISE 2004.
Jannach, D., Leopold, K.,
Hellwagner, H., and Timmerer, C.: A Knowledge Based Approach for Multi-step
Media Adaptation. In: Instituto Superior Técnico,
5. Timmerer, C., DeMartini, T., and Barlas, C., (eds.), “Information technology — Multimedia framework (MPEG-21) — Part 7: Digital Item Adaptation, AMENDMENT 1: DIA Conversions and Permissions”, ISO/IEC 21000-7 AMD/1, Final Proposed Draft Amendment 1 (FPDAM/1), January 2005.
6. Jannach, D., Timmerer, D., Leopold, K., and Hellwagner, H., “A Knowledge-based Framework for Multimedia Adaptation”, to appear in the International Journal of Applied Intelligence, Springer Science+Business Media B.V., 2005.
Other MPEG documents are available at http://www.chiariglione.org/mpeg/ under ‘Hot news’ or ‘Working Documents’. An introduction to MPEG-21 – especially to DIA – is also available at http://mpeg-21.itec.uni-klu.ac.at/.