The digital video album

On the merging of media types in multimedia

The computer as a medium for other media

Isolated resources and distributed environments 

Hypertext and hypermedia

Media types and linearity

A digital video album 

Conclusion

References

The computer as a medium for other media

Ever since its emergence in electronic-digital form, the computer has been a meeting place of some kind. Like an agora, computer technology and environments have, in their different stages of development, constituted a place where persons, disciplines, media and information have converged and interacted. Starting out as enormous mainframe computing machines performing arithmetical tasks far beyond individual, manual achievements, the computer has developed into a small, relatively cheap but powerful tool marketed to millions of individual users.

Today, not only numbers, but words, drawings, pictures, sound, and video may be generated, processed and distributed by means of personal computers. The computer as we know it today has developed into a powerful and complex medium activating and forcing the use of multiple senses and capable of receiving, manipulating and transmitting all the traditional media types. Different media developed in distinct but overlapping and related institutions and technologies - such as painting, writing, print, photography, telephone, radio, film, television and video - now merge as digitally coded information in the hardware and software of microcomputers. Connected in global networks of end-users, the computer medium has constituted itself as a true super- or multimedium - a medium for other media.

Adding the digital machine's own capabilities of interaction and random access to vast amounts of information, this multimedia vehicle now represents the most technological advanced and complex medium in the joint history of human communication and information technology. Grounded on this infrastructure, we experience the emergence of a postprint, electronic-digital public sphere of only vaguely known structure, function and consequence.

Isolated resources and distributed environments

This relationship can be seen as parallel to the distinction between systems 'published' on physically unchangeable, isolated media, such as CD-ROMs, and, on the other hand, distributed systems based on electronic networks. This difference is important and will generate various conventions for composing multimedia messages. But although a fully developed network may have all advantages over publications of physical, unchangeable entities, the latter may still have a long life. Two thousand years of manuscript and book technology (followed by the success of video and audio cassettes) has instituted a market of producers and consumers where messages as goods are intimately related to physical entities. Despite the technical possibility of fully developed on-line networks, I believe it will take a considerable amount of time before the current mode of production will allow a radical redefinition of the individual message as a product and object. If the information commodity, in the form of electronic multimedia texts should only be available on-line, the text will loose its appearance as a concrete sensible object unambiguously pointing at its origin the author-producer. Since our western economic system seems to outlast every challenge from non-market economies, it is likely that the reification of information and knowledge existing in the age of the printed book will survive the transference to electronic, interactive media and thus for a long time favour the publication of units and genres compatible with this tradition.

The opposition between distributed and isolated multimedia messages is evident and will probably become more significant in the future. Though for the time being many of the problems we are dealing with in multimedia production, for instance the dynamic integration of media types and the establishing of meaning-effective conventions, are problems that need to be solved whether the messages are distributed over a network or published on a compact disc.

Hypertext and hypermedia

'Let me introduce the word 'hypertext'***** to mean a body of written or pictorial material interconnected in such a complex way that it could not conveniently be presented or represented on paper. It may contain summaries, or maps of its contents and their interrelations; it may contain annotations, additions and footnotes from scholars who have examined it. Let me suggest that such an object and system, properly designed and administered, could have great potential for education, increasing the student's range of choices, his sense of freedom, his motivation, and his intellectual grasp******. Such a system could grow indefinitely, gradually including more and more of the world's written knowledge. However, its internal file structure would have to be built to accept growth, change and complex informational arrangements. [...]

Films, sound recordings, and video recordings are also linear strings, basically for mechanical reasons. But these, too, can now be arranged as non-linear systems - for instance, lattices - for editing purposes, or for display with different emphasis. [...] The hyperfilm - a browsable or vari-sequenced movie - is only one of the possible hypermedia that require our attention.

*****The sense of 'hyper-' used here connotes extension and generality; cf. 'hyperspace.' The criterion for this prefix is the inability of these objects to be comprised sensibly into linear media, like the text string, or even media of somewhat higher complexity. [...]

******I will discuss this idea at length elsewhere.'(2)

One might object, then, that hypertext is basically a paper based textual form implemented in book technology through features such as footnotes, references, allusions, etc. Although hypertext frequently has been described and explained as 'the generalised footnote', the second footnote in our quote shows the limitations of textual interconnections in paper-based technology compared to the qualities of hypertext. The second footnote is a reference to something that is not there, unavailable to the reader, an elaboration which exists 'elsewhere', outside the present paper.

Nelson uses hypertext and hypermedia at distinct grammatical levels. Hypertext, hyperfile, and hyperfilm are mentioned as examples of hypermedia. Hypermedia then becomes a generic term and not the name of a kind of hypermedium itself. The 'hyper-' prefix denoting non-linear organisation of either text, pictures, sound, or video, is here thought of as a structure implemented in the separate media, not a device for linking across platforms. Later, Ted Nelson has adjusted his point of view and according to what is now common usage let 'hypermedia' be compatible with 'hypermedia document', 'hypermedia production', etc. Today, hypertext and hypermedia are being used as synonyms since 'text' is applied as a metaphor for all kinds of media types not just verbal text.

Before Nelson, the now classic paper by Bush (3), and Engelbart (4) were conceiving and implementing similar ideas - Bush in an environment of analogue mechanics before and after the 2nd World War and Engelbart as the first to radically explore the potential of the digital computer. Within the frames of Engelbart's strategic project on 'augmenting the human intellect', most of the features on any modern microcomputer or workstation were developed: text editing, windows, the mouse, hypertext, outlining, groupware and e-mail. For both Bush and Engelbart as well as the later Nelson, the integration of media types was a major concern. Engelbart, for instance, demonstrated combinations of text, graphics and video on networked workstations as early as twenty-five years ago.

During the 1980s there was much talk about the future of multimedia and hypermedia, when only the supporting technological environments finally would be in place. During this period interesting projects based on combinations of analogue and digital equipment where developed, projects that also led to the establishment of conventions (5). In the early 1990s the technology gave sufficient support for full integration of audio and video on any medium range personal computer, especially the Apple Macintosh. The status of hypermedia today is not exactly what one would have imagined five to ten years ago. The all-digital environment is here, but real attempts of innovative integration of the various media types have still not shown up. Instead, one might detect a splitting in academic, lingocentric hypertext as one extreme and image dominated computer games as the other.

Media types and linearity

This focusing on the problems of linearity gives us a perspective on the relationships between the various types of media. In the context of this essay, we may focus on four main media types: image, verbal text, video (moving images), and audio. In the perception and consumption of visual images there are no fundamental rules of linearity we must follow when exploring the image; which succession we go through to combine the elements into a whole, an image. Our gaze may travel relatively freely over the surface. Of course, there are limitations and structures which generate preferred readings of images, like foreground and background, constraints of light and shade, etc., but in general we might say that this is a succession or a linearity based on the subject's actions, a subjective linearity.

With written text it is quite different: if we are to generate the information and meaning invested in a written text at all, we have to follow the rules for reading written language. Vi must add letter to letter, words to words, sentences to sentences, etc. The rules of succession constituting writing are founded in language as an inter-subjective praxis among social human beings. We may then say that with the media type verbal text we are facing an inter-subjectively founded linearity. At the micro level this linearity is necessarily very constrained, but the bigger the chunks of texts the less constrained and the less linear, and the more they may be consumed in multi-successive sequences. We do not read all texts from beginning to end, but feel free to jump back and forth, depending on our needs and interests. Writing at the macro level then, allows for a certain degree of interactivity as in encyclopaedias and newspapers.

With video (including television and film) we face a machinery which is presupposed before the media type starts to exist. The apparatus has to display a certain rate of frames, and the spectator must follow this flow of moving images, conditioned by time. A sequence can normally not be stopped; neither is it normal to choose the speed of consumption. The playback speed should be identical to the speed of recording. We may shut the apparatus down, but then this media type disappears as well. We may say that video as a media type is conditioned by an object founded kind of linearity.

Sound as a continuum is also dependent on an external source for organisation and existence. While image, text and video are visually based information, sound is aural. Audio information does not have the same potential of being selected, with sound space being reduced to the instant now, with a before and an after. Sound is thus an extreme kind of media type, and its linearity is a temporal linearity. Space may in audio occur as parallels of accords and harmony. These four media types find their places on an axis with the still image as the extreme space based exponent and with audio on the temporal side. Text and video are mixtures in between (see figure 1). Interactivity and the potential of selection are best when there is a space dominant modus supporting the subjective kind of linearity.

Integration of media types is not new in the history of media and communication. Writing and print technology have combined images and verbal text; and the silent movie changed from one configuration of multimedia (moving images + verbal text) to another (moving images + sound) when the talking movie was introduced. But these examples of integration are fundamentally different from what we might expect of computer based, digital multimedia. In literature, text usually dominates the image. The verbal text gives the ambiguous and meaning-rich environment of images direction and precision, rules on how to understand and use them. In the silent movie era text was subsumed under the speed and rhythm of the actions depicted in the film. Also, subtitles and text on television are always presented on the premises of moving images and their temporal linearity. In digital multimedia these relationships of repression and dominance do not have to exist, since the computer has the capacity to include and maintain the founding premises of each individual media type in integrated documents. The problem, however, is then to create a dynamic integration of the different sign types and still conserve their unique individual qualities without tending to dominance and subsumption. How can video and text be combined in a smooth way without the one dominating the other; or to put it another way: how can video be given qualities originally belonging to verbal text like a larger degree of selection and interactivity; and can text benefit from the video mode of linearity? The following discussion - related to the idea of a video album concretised in the production of a specific multimedia system - is a tentative attempt to provide some information about these problems.

A digital video album

Facing the challenge of creating structure and form in a multimedia system - 'The Interactive Kon Tiki Museum' - for presenting the excavations and expeditions of the Norwegian explorer Thor Heyerdahl, the idea of a video album was central. All Heyerdahl's major expeditions have been well-documented on film, photographs and in literature. But for a system planned as a supplement in the exhibition at the Kon Tiki Museum in Oslo, it was important to make much of the video available to the visitors. But the filmed material was all in the format of feature films or television documentaries which are not suitable for the museum environment and mode of information acquisition. The video material had to be made available to the users in short but self-contained chunks, and it had to be possible for the viewer to choose between various themes to get more detailed information on topics of their interest - still in the video format. Cross-references and linking in the video media, film and television are completely absent. One way of making video interactive was by adapting the footnote mode of reference, developed in literature, and use it as a model for how to organise the many chunks of specially edited digital video clips; but at the same time the dynamics, the rhythm and tempo of the whole presentation, had to be maintained. I will discuss these solutions in relation to the three following still images (screen shots) from the system.

The image in figure 2 depicts the scene in the system which corresponds to the table of content in a book or a magazine. In a time span of a couple of minutes, this scene introduces all the main topics or sections of the system. To introduce one of the items, four actions or sequences are joined to create a continuous flow or movement: The spinning of the globe, the marking of a path or a place, the zooming of an image from the geographic point to the button, then the short movie displayed on the button, before the whole sequence is repeated with the next button. These four different actions all contain different kinds of movement but are linked in a sequence to get the impression of a moving wave transported from the globe spinning to the movie icon (micon) on the button. This wave movement helps to create the impression of continuity, despite the discontinued elements and different types of actions involved.

The image in figure 3 shows the introductory scene to the Easter Island section. The two buttons named 'The Origins' and 'Statues' are the video footnotes. In literature, the convention of footnotes - the relationship between the main text and the footnote text - is marked with identical signs. Here, a different kind of pointing or suggesting relation was chosen. In the previous example the relations between the geographical point on the globe and the corresponding button were created by the movement of a growing image, starting as a pixel on the surface of the globe and ending as the first frame in a mini movie on the button. In the Easter Island scene this convention is more formalised. On the Macintosh, an animated frame indicates that a document is being opened. Following this convention, the image itself does not move, only an animated frame representing the reference to the video footnote.

If following the link to 'Origins' the whole screen changes and a new background appears with a blank field which does not display a video but text; three texts that like ingresses introduce the buttons to additional video clips (see figure 4). Instead of a time based media type like video introducing the scene, text alone was supposed to introduce the videos at this sub-level. However, this became problematic. When the user had got used to the dynamics of the two first scenes and their video based tempo and rhythm, something felt wrong when entering this scene. Suddenly, the user was facing text (a medium type with no intrinsic tempo for information aquisition) instead of video (with its object-based linearity). The dynamics of the presentation as a whole was gone, integration had not taken place in the intended way. Video had been supplied with characteristics from text. Now this had to be done the other way round; text had to be given the time dependent qualities of video. This was done in two ways. The text was divided into three paragraphs showing up on the screen in a sequence. Also, a voice over was added, an audio track reading the text. Thus the original non-temporal text was turned into a talkie, with changes over time both in the audio and in the video track. To achieve dynamic integration and flow one had to compromise a bit, the result was that video became interactive and selective like a text, and text time based like video and sound, but with the interactivity intact.

Conclusion

References