Juan C. Guerri, Carlos Palau, Manuel Esteve,
Department of Communications, Technical University of Valencia
Valencia, Spain
e-mail: jcguerri@dcom.upv.es, cpalau@dcom.upv.es, mesteve@dcom.upv.es,
1. INTRODUCTION
The main causes which are nowadays increasing the utilisation of multimedia applications in different scenarios and environments (from marketing and publicity systems to modern medical applications), are, mainly, the advances and the efforts that are being done in tasks like the formal specification of temporal relationships [5],[6],[9](interval-based specification, axes-based specification, hierarchical, reference points, timed petri nets, events or scripts), storage systems, scheduling and retrieval of information, compression systems (JPEG, MHEG,...), transport protocols for real time systems (XTP, RTP, MTP,...), high speed communication and switching systems and mainly in the integration systems and representation of the information, i.e. WWW, and the synchronisation techniques.
Among the different definitions and classifications [1], [9], that have been done over the multimedia applications, in all the literature dedicated to this topic, a multimedia system could be defined [9] as, "a system or application that supports the integrated processing of several and different kinds of information which, at least one time-dependent". Steinmetz and Balkowski, also introduce a classification of the multimedia systems using the following criteria: number of media, different kinds of media supported, and integration degree of the different kinds of media. An alternative classification, which we propose in this paper, considers also the QOS factor as an added criteria. Until now multimedia have been often defined as non-critical real time systems, however, considering the projection, they are experimenting, this point should be reconsidered, and include the multimedia applications as critical and non-critical real time applications. Consider a multimedia system like a critical or a non-critical real-time systems includes a large amount of decisions and considerations which should be taken into account, mainly when the synchronisation protocol is implemented and tested. In the critical real-time systems a high accuracy is needed, because a fault in the synchronisation process is fatal for the results and the conclusions obtained from the system. On the other hand, the human perception can not be used as a criteria in order to evaluate a synchronisation protocol, and establish the correct work of it.
From the previous considerations, can be deduced that the World Wide Web is the ideal environment for a multimedia system, in order to integrate several kinds of media, time-independent (text and images ) and time-dependant ( video and audio).
Nevertheless, the existence of a distributed system which is based in the Internet, which allows that the data stored in the Webs can be located anywhere in the World, is an advantage when the different data, which are going to be represented, have no temporal relation between them but is a problem when there is a necessity of synchronisation between the time-dependent and time-independent medias, following the temporal relationships specified by the authors of Web-based multimedia. The delay introduced by the network and the jitter affect enormously the accuracy obtained by the synchronisation protocol [2], [3].
The aim of this paper is to describe the basic ideas of a new synchronisation protocol, which could be used in the Internet by the WWW, not only with non-critical real-time multimedia applications, but with critical real-time applications. Also some results obtained, from the evaluation of the protocol, and an example of the real utilisation of the protocol are presented.
2. SYNCHRONISATION PROTOCOLS
In [6] is introduced an excellent classification of the synchronisation protocols, which are divided in three main groups: hardware, software and hybrid hardware/software. With regard to the software algorithms, that are usually used in the real-time systems and more concretely in multimedia systems, a second division is done: those which are based in a common time reference global to all the playback terminals or to a subset of them, and those which use the feedback messages from the playback terminals and without a global time reference.
In this paper an hybrid solution is proposed, which tries to improve the accuracy of the feedback protocols, and to avoid the complexity of the synchronisation protocols based in a global time in relation to the process of synchronising the multimedia flows using time-stamps. The following figure shows the new classification, in which the hybrid global/feedback protocols appear as an additional item in the group of software protocols.
With respect the work done about global time synchronisation protocols, which main goal is to synchronise the local clocks of different processes, the performance of several protocols has been evaluated. Two of them can be outlined, NTP (Network Time Protocol)[8],[10] and the synchronisation protocol using waves of messages used as a service of XTP (eXpress Transport Protocol)[4]. NTP is being widely used in the Internet, and there exists a large bibliography about it, which includes its specification, description and performance.
The second protocol has been formally described using systems of timed communicating state machines, and also its integration as a service available at the transport interface of XTP [4].
The feedback synchronisation protocols, which main goal is, analogously to the global time protocols, to keep the temporal relationships between the different flows of media but without making use of a global clock. Then in order to execute the algorithm, they base their decisions in the knowledge of some features of the network like the delay distribution, so they can calculate the playback instant of the multimedia object.
To implement the Feedback-Global protocol, NTP protocol has been chosen as the global time synchronisation protocol because of the wide use in the Internet, and a modified protocol based in an adaptive synchronisation [3] protocol for the feedback protocol.
The combination of both protocols, far from introducing complexity, provides an improvement in the synchronisation and in the performance of the multimedia applications which use it, mainly because i) the global time protocol chosen provides an accuracy in the order of milliseconds, what is enough for our purpose. This affirmation, has always validity when we talk about non-critical real-time applications. Nevertheless, in the Feedback-Global protocol this is valid for critical and for non-critical real-time applications, because over this protocol exists a simple feedback protocol. ii) In another hand, if a global time protocol which synchronises the local clocks is not used, an inaccuracy of the feedback protocol is introduced, which increases as the network jitter increases too. The utilisation of a simple global time synchronisation protocol eliminates this mistake, as will be proved later in the results section, keeping its good performance even with high delays and jitters.
3. FEEDBACK-GLOBAL PROTOCOL
The following ideas summarise the aims that we pretend to obtain with this protocol :
· Increase the accuracy in the synchronisation of multimedia objetc flows in the playback instant obtained by the feedback protocols.
· Keep the simplicity in the implementation of the feedback protocols, which allows their integration in a WWW environment.
· Use the accuracy of the global time synchronisation protocols , without introducing a time-stamp mechanism in order to keep the playback synchronised.
· Use NTP protocol as the global time synchronisation because its standardisation in the Internet.
The execution of the protocol can be divided in two phases :
1. In the first phase a common time is obtained between the Web server and the WWW clients in charge of the playback of the multimedia information. In order to achieve it, NTP or SNTP is used, which could be used as a background process in each WWW client.
2. In the second phase, the modified feedback protocol, is the one in charge of synchronising the multimedia flows, relying in the existence of a global time protocol, in order to cause simple "skip" and "pause" actions in the clients.
The following figure describes graphically the functioning of the protocol :
4. PERFORMANCE
In the following paragraphs are shown some results from measurements of the utilisation of the Feedback-Global algorithm, in order to achieve synchronisation between an audio and a video flow in an environment similar to the previous figure. The particular scenarios have been obtained by emulating different conditions of delay and jitter in order to obtain some preliminary results and valid conclusions. A playback video rate of 50 frames per second has been supposed, and a normal delay distribution, with various means {45,87.5,120}ms., and a maximum and minimum delays of [40,50] ;[75,100] ;[90,120] ms. respectively, and the maximum allowed asynchrony between data flows in the time of playback is 80 ms. Measurements have been done with a transmission rate of 20 ms in the transmitter and a consume rate of 20 1 ms. in the multimedia clients. In each experiment 10000 frames have been transmitted.
The resynchronisation method is based in obliging the slave data flow to make little "skips" or "pauses". The kind of data that is considered master flow, which does not suffer from skips or pauses, and the one slave flow, depends mainly on the kind of information with which the application is dealing. For example, if the audio features are examined , can be observed that this flow is very sensible to jitter and then it can not suffer from random delays between consecutive LDUs [1], because in other case the quality decreases, and it is highly perceptive. With respect to video, the elimination of a short number of frames, or the existence of little random delays between consecutive frames has little impact in human perception. Following this considerations is logic to think that in an application formed by a video and an audio flow, audio is chosen as master and video as slave. In order to study the influence of asynchrony in human perception and the election of the most critical data flows, a deep analysis is done in [9] and [11].
The following tables present some of the results obtained with the Feedback-Global protocol compared with the Feedback protocol implementation[3], when a conservative policy an different window sizes are used.
Normal [40 ms, 50 ms] Conservative Policy
W=0 W=1 W=2
Feedback Feedback Feedback Feedback Feedback Feedback
-Global -Global -Global
Number 0 0 0 0 0 0
Asynchrony
Average 20 48 20 43 40 46
absolute
asynchrony
(ms)
Media units 2 0 0 0 0 0
Skipped
Media units 2 0 1 0 1 0
Paused
Master 35 35 42 42 63 63
Feedbacks
Slave 5 5 6 6 9 9
Feedbacks
Normal [75 ms, 100 ms] Conservative Policy
W=0 W=1 W=2
Feedback Feedback Feedback Feedback Feedback Feedback
-Global -Global -Global
Number 1* 5 1* 3 1* 9
Asynchrony
Average 20 88 26 80 51 93
absolute
asynchrony
(ms)
Media units 3 0 0 5 0 0
Skipped
Media units 4 0 5 1 3 1
Paused
Master 45 45 54 54 81 81
Feedbacks
Slave 5 5 6 6 9 9
Feedbacks
Normal [90 ms, 150 ms] Conservative Policy
W=0 W=1 W=2
Feedback Feedback Feedback Feedback Feedback Feedback
-Global -Global -Global
Number 1* 5 1* 7 1* 10
Asynchrony
Average 28 132 34 134 40 126
absolute
asynchrony
(ms)
Media units 0 0 0 0 0 0
Skipped
Media units 7 0 6 0 5 0
Paused
Master 70 70 98 98 140 140
Feedbacks
Slave 5 5 7 7 10 10
Feedbacks
Conclusions that can be extracted from the measurements are :
· The Feedback-Global protocol, always keeps a minimum number of asynchronies.
· In the Feedback-Global protocol the synchronisation between data flows is kept in the margin [-80,80] ms., that can be considered "in-sync". Also the mean asynchrony is lower than the maximum allowed.
· The Feedback-Global protocol has a large accuracy. This implies that the clients resynchronise more times (more "skips" and "pauses"), debt to the protocol is more sensitive to differences between the audio and video flow. The main cause to this increase in the accuracy is debt to the utilisation of NTP as a protocol which offers a global time reference, and the utilisation of this time by the modified feedback protocol.
· The percentage of messages exchanged by the synchronisation protocol is small when it is compared with the total number of transmitted audio and video frames, with a value of 0.35% in the best case and 1.4% in the worse.
· The Feedback-Global protocol, has a similar behaviour although there is an increase of network delay and jitter. Initially, this characteristic converts it in a useful protocol for WANs and networks with the characteristics of Internet.
5. APPLICATION AND CONCLUSIONS
Nowadays work related with this synchronisation protocol are centred in the utilisation of the Feedback-Global in a medical multimedia application and evaluate its performance. The goal of the project is to synchronise different flows of data, generated by several devices like an electromyograph, a video camera and a lumbar monitor. These data have a temporal relationship in the moment in which they are generated, but as they are measured asynchrously and no time information is added to them, it is not possible to keep any kind of synchrony between the flows when they are retrieved for their playback. This implies that no medical conclusion, of high importance, can be extracted from them. The existence of a synchronisation between the flows of data allows the utilisation of the application in order to study the correlation between determined movements and the muscular activity, to detect the causes of a locomotive deficiency analysing the movements done by the patient, or also design training equipment for sport people emphasising some movements. The following image shows the application :
Besides the main aim previously exposed, there are some additional objectives. One of them is obtein a real testbed in order to evaluate the implementation and performance of new synchronisation protocols, and once the application has been tested, future work will be centred in the utilisation of this multimedia application in the WWW environment, doing the access via Internet.
6. REFERENCES
[1] R.Steinmetz, "Synchronization Properties in Multimedia Systems," IEEE
Journal on Selected Areas in Communications, Apr. 1990.
[2] T.D.C.Little, A.Ghafoor, "Multimedia Synchronization Protocols for
Broadband Integrated Services," IEEE Journal on Selected Areas in
Communications, Dec. 1991.
[3] S.Ramanathan, V.Rangan., "Adaptative Feedback Techniques for
Synchronized Multimedia Retrieval over Integrated Networks", IEEE/ACM
Transactions on Networking, Apr. 1993.
[4] M.Esteve, C.Palau, and J.C.Guerri, "A Synchronization Service
Framework for XTP," Proc. 20th Annual Conference on Local Computer
Networks, Minneapolis, Oct. 1995.
[5] T.D.C.Little, A.Ghafoor, "Interval-Based Conceptual Models for
Time-Dependent Multimedia Data," IEEE Trans. on Knowledg and Data
Engineering, Aug. 1993.
[6] Ramanathan, K.G.Shin, R,Butler, "Fault-Tolerant Clock Synchronization
in Distributed Systems", IEEE Computer, Oct. 1990.
[7] John A. Stankovic, "Misconceptions About Real-Time Computing" IEEE
Computer, Oct. 1988.
[8] D. Mills, "Internet Time Synchronization: The Network Protocol," IEEE
Trnasactions on Communications, Oct. 1991.
[9] G.Blakowski, R.Steinmetz,"A Media Synchronization Survey: Refernece
Model, Specification, and Case Studies," ," IEEE Journal on Selected
Areas in Communications, Jan. 1996.
[10] RFC 1305, "Network Time Protocol"
[11] Hern-Yow Chen, "Multysinc: A Synchronization Model for Multimedia
Systems", IEEE Journal on Selected Areas in Communications, Jan. 1996.