View source for TTML/changeProposal014

<small>[[TTML/ChangeProposalIndex | < Change Proposal Index]]
</small>

= Audio Rendering - CLOSED-IMPLEMENTED =

__NOTOC__
<div id="boilerplate" style="border-left:3px solid #00CC77; padding: 0.5em 1em 0.5em 1em; background-color:#CCFFCC; color:#000; margin:auto; width: 70%; font-size:0.9em;">

* Owner: Nigel Megitt.
* Started: 14/06/13 
* Closed: 20/01/15
</div>

== Issues Addressed ==

[https://www.w3.org/AudioVideo/TT/tracker/issues/10 ISSUE-10] - Embedded Audio - Closed - [https://dvcs.w3.org/hg/ttml/rev/d16d284100b9].

== Summary ==

TTML is currently predicated on the visual representation of timed text. It may be useful additionally to present the timed text in audio, e.g. for accessibility purposes. That audio presentation may have been pre-recorded or generated by text to speech technology, in both cases using the document instance as a source. Issue-10 concerns links to external audio resources. Additionally, if the TTML is to be used as a source for generating audio tracks for "audio description" (European term) or "video description" (equivalent US term) then extra markup may be useful to guide the conversion to audio, regardless of whether that converter is a human or a machine. For example pronunciation guidance may be needed.

For both the above use cases and more general metadata capture for subtitles and captions, markup of emotion would also be a useful addition to TTML.

Pronunciation markup may be referenced as an external [http://www.w3.org/TR/pronunciation-lexicon/ PLS] document.

Emotions may be expressed using [http://www.w3.org/TR/emotionml/ EmotionML], however further work is needed to define how to extend TTML with EmotionML semantics.

Speech synthesis markup is probably not needed here - if it were needed, [http://www.w3.org/TR/speech-synthesis11/ SSML] would appear to be relevant.

== Edits to be applied ==

=== Strawman ===

==== Pronunciation lexicon ====
Add a <code>lexicon</code> element to <code>tt:head</code> using the same semantics as in [http://www.w3.org/TR/speech-synthesis11/#edef_lexicon SSML §3.1.5.1]

Not required, since the lexicon element would be in a non-TTML namespace, and it is already permitted to add foreign (non-TTML) namespace XML. TTML-only processors must ignore it, but any TTML + PLS processors can do whatever they see fit with it.

==== Emotion markup ====
;DONE
:Find a way to integrate EmotionML semantics into TTML

No edits are required: adding XML content from the EmotionML namespace into TTML is already permitted - as it's a foreign namespace TTML-only processors must ignore it, but any TTML + EmotionML processors can do whatever they see fit with it.

==== Map external audio cue fetch and playback to Javascript ====

 use the onenter associated with cues, and getCueAsAudio(), thus:

 <nowiki>
    cue.addEventHandler("enter", 

          function(sender)
          {
              var theVideo = document.getElementByName("theVideo");
              var savedVolume = theVideo.volume;
              theVideo.mute();
              // set watchdog on video in case it overruns the description duration
              var h = theVideo.addEventHandler("timeUpdate", 
                    function(video) {
                        if(video.currentTime > cue.endTime) video.pause();
                    }
              var myAudio = sender.getCueAsAudio();  // if this is too slow do outside handler
              myAudio.addEventHandler("ended", 
                    function(description) {
                        theVideo.removeEventHandler(h);
                        theVideo.Volume = savedVolume;
                        if(theVideo.paused) theVideo.play();
                    }
              myAudio.play();
          }
    } 
</nowiki>

== Edits applied ==

Audio resources added as specific type of generic data resource, in change https://dvcs.w3.org/hg/ttml/rev/d16d284100b9

== Impact ==

== References ==

;Pronunciation Lexicon Specification
: [http://www.w3.org/TR/pronunciation-lexicon/ PLS].

;Emotion Markup Language
: [http://www.w3.org/TR/emotionml/ EmotionML].

;Speech Synthesis Markup Language
:[http://www.w3.org/TR/speech-synthesis11/ SSML].