mass-moved component to LC1

The event that is returned should include information as to which track may be affected in order to provide a hint as what metadata may have been modified. At least the type of track that has been modified should be returned. Without this change there is no means for an application to be aware of added and deleted tracks in a track list. 

The bug should clarify which events are reported since it is very vague as to what may have changed: Here is a proposed list of events:
- event on video track
- event on audio track
- event on text track (subtitles) 

Note-This proposed changed is based in discussions in W3C webTV IG - media piping task force.

Ok my plan is to do the following:
* Define a TextTrackList that has a .length, a getter, and an onaddtrack handler. (The TextTrackCueList interface can be used for copy-pasta.)
* Add an onaddtrack handler to VideoTracks and AudioTracks.
* Define a AddTrackEvent interface with a single attribute 'track' of type 'object'.
* Add to the "steps to expose a media-resource-specific text track" a step to fire an onaddtrack event.
* Add to the resource fetch algorithm entries in the big <dl> that handle the discovery of new video or audio tracks and fire an onaddtrack events accordingly.

Tracks can never be removed, since if a track existed at a particular time, it'll still exist if you seek back to that time. I guess with an infinite stream where the /earliest possible position/ has gone past the point where a track was last usable one could imagine dropping the track, but I'll add events for that if it ever becomes a real problem.

Checked in as WHATWG revision r6606.
Check-in comment: Have a TextTrackList object instead of a TextTrack[] array, so we can add members to it.
http://html5.org/tools/web-apps-tracker?from=6605&to=6606

Checked in as WHATWG revision r6607.
Check-in comment: Add onaddtrack to all the TrackList interfaces. Also, make the text that introduces event handler IDL attributes more consistent across the spec.
http://html5.org/tools/web-apps-tracker?from=6606&to=6607

Checked in as WHATWG revision r6608.
Check-in comment: Define an event interface for addtrack events.
http://html5.org/tools/web-apps-tracker?from=6607&to=6608

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted
Change Description: see diffs given above and below
Rationale: Concurred with reporter's comments.

Checked in as WHATWG revision r6609.
Check-in comment: Fire addtrack events whenver a track is added to a media element.
http://html5.org/tools/web-apps-tracker?from=6608&to=6609

With regards to the proposed solution to simply indicate add tracks information on a live stream I am concerned that changes overtime may be lost. This will be a problem for an application to know what track(s) is not available anymore.

For example there may be user preferences handled in the application that go beyond a single language. My preference may be (1) spanish, (2) english, and (3) french. The application may indicate spanish when a list of audio/subtitle tracks are listed for a scheduled program but then next scheduled program the spanish audio/subtitle is removed. How does an application keep track of the removed track?

Implementation:
Here is a suggestion on the handling of the array with the available tracks if it is not already done this way. The suggestion is that the array is a static view. When an event occurs one can retrieve the latest array. This implies the array is not dynamic and will not change without explicitly requesting for the update. There are some other details with this approach but I would like to request initial response before pursuing this approach.

It's not clear what the behaviour should be when the media of a track ends. Although it may make sense that the track object is never removed from the list (for the reason you give and also to avoid modifying the indices of the other tracks), the page still needs to know that the media is no longer available so that it could switch to a different track.

For example consider a programme where alternative language audio is available some, but not all, of the time. If the alternative audio was the user's preferred audio, then the page would want to switch to that when it was available and switch back to the default audio at other times.

It's not clear if a track become unavailable and then available again whether this should result in a new track being added ? And if so, how would the page know that this was the return of the same track which was earlier removed.

A possible solution would be to add an 'available' boolean to each track with an event to indicate changes in availability.

Reopening - see comment.

Please file separate bugs for separate issues.

(In reply to comment #12)
> Please file separate bugs for separate issues.

I need a bit more info on what you consider a separate issue.

The issue of tracking when a a particular track has media data and when it does not is IMHO a separate issue than the issue of tracking whether a particular track exists and notifying the script when a new track is added to the list.

(I agree that the issue of tracking when a track has data is a real one, though I haven't yet studied what use cases actually rely on having this knowledge. For example, sign-language overlay tracks frequently are sparse relative to the primary video. I could believe that there are use cases that need the ability to determine during what time regions such a track would be available, but before determining what API to use to expose this, and how to report changes to this information, we will need to enumerate the use cases for using the information in question. This is better done in its own dedicated bug so that if we e.g. get a revert request or escalation for the original topic of this bug, we don't have to track multiple issues under the same bug number.)

Is there any change required here specific to the diff checked in above (comment 8) or are the only outstanding comments relating to the new material mentioned in comment 14 to be filed as a separate bug?

(In reply to comment #15)
> Is there any change required here specific to the diff checked in above
> (comment 8) or are the only outstanding comments relating to the new material
> mentioned in comment 14 to be filed as a separate bug?

If a live stream has, say, a spanish language track and that media stops because the live stream has moved on to some programme without spanish language available, then this could be considered as either "the spanish language track is deleted", or as "the spanish language track is still present but has no media".

Which choice you make is a design decision about which I don't have a strong opinion (though see below). The current proposal takes the second approach.

Nevertheless, the use-case was included in the original bug, so whilst this could be tracked as a separate issue it is not "new" and should be dealt with as an LC1 issue too.

I do have a mild concern about the proposed approach: we should consider what happens with extremely long-running streams (live streams) which may consist of a lot of different material spliced together, where the available tracks in each segment are completely different. The current approach implies an ever-growing track list and I wonder if we really should delete the "dead" ones in some way.

(In reply to comment #16)
> The current approach implies an
> ever-growing track list and I wonder if we really should delete the "dead" ones
> in some way.

In a typical broadcast stream, e.g., MPEG-2 TS, there are two kinds of significant changes that affect the existing track collection:

(1) the addition, change, or removal of entries from a PMT (program map table) whose content is being presented, where each entry is effectively a video, audio, or data (everything else) track, i.e., an Elementary Stream (ES);

(2) the addition, change, or removal of entries from the PAT (program association table), where each entry points at a PMT (via its PID);

Starting with case (2), removal of or change of PID for the current presenting PMT should cause a removal event for all tracks, followed by an add event for all tracks in the new PMT (if one exists). One may not exist if the current PMT is the only PMT, and it is removed from the PAT without adding a new PMT.

Then for case (1), removal of or change of an elementary stream entry for the current presenting PMT should cause a removal for the affected track, followed by an add if it is replaced by a new elementary stream PID.

If the only change in the TS is that the the content delivered in the currently presenting ESs of the currently presenting PMT changes, e.g., due to a splice but without changing the PID #s, then there would be no way to detect that change absent of some other form of metadata.

For client side code to maintain a coherent model of track collection, I don't see a reasonable way (or need) to avoid some form of remove track event. Defining HTML5 video behavior without fully plumbing the actual practice will likely prove a problem in the future if an over-simplistic approach is adopted (e.g., ignoring track removal).

I think for the MPEG-2 TS case we can safely, and should, ignore the multi-program transport stream (MPTS) case (where the program association table PAT comes into play). The HTMLMediaElement models the MPEG-2 single program TS (SPTS): a media resource (MPEG-2 program) with multiple tracks (MPEG-2 elementary streams). There is no HTML5 corollary to a MPTS. Also, the IP delivery network provides a program multiplexing scheme that would seem to obviate the need for the equivalent function of the MPTS.

With regards to the detecting tracks that are removed, it seems that the UA will already have the necessary information to do this. Consider an MPEG-2 SPTS case: the program map table (PMT) has to be accessed by the UA codec so it can locate the video/audio elementary streams to play the media resource. The PMT has the complete description of tracks (packet IDs in MPEG-2 parlance) that are available. The PMT will change when a track is added or deleted and so the UA can detect that. My understanding is the WebM and Ogg have the equivalent structure of the PMT. With regards to tracks in the program, the PMT, or equivalent, is analogous to the manifest file used with adaptive bit rate delivery methods.

Bob

(In reply to comment #17)
> (In reply to comment #16)
> > The current approach implies an
> > ever-growing track list and I wonder if we really should delete the "dead" ones
> > in some way.
> 
> In a typical broadcast stream, e.g., MPEG-2 TS, there are two kinds of
> significant changes that affect the existing track collection:
> 
> (1) the addition, change, or removal of entries from a PMT (program map table)
> whose content is being presented, where each entry is effectively a video,
> audio, or data (everything else) track, i.e., an Elementary Stream (ES);
> 
> (2) the addition, change, or removal of entries from the PAT (program
> association table), where each entry points at a PMT (via its PID);
> 
> Starting with case (2), removal of or change of PID for the current presenting
> PMT should cause a removal event for all tracks, followed by an add event for
> all tracks in the new PMT (if one exists). One may not exist if the current PMT
> is the only PMT, and it is removed from the PAT without adding a new PMT.
> 
> Then for case (1), removal of or change of an elementary stream entry for the
> current presenting PMT should cause a removal for the affected track, followed
> by an add if it is replaced by a new elementary stream PID.
> 
> If the only change in the TS is that the the content delivered in the currently
> presenting ESs of the currently presenting PMT changes, e.g., due to a splice
> but without changing the PID #s, then there would be no way to detect that
> change absent of some other form of metadata.
> 
> For client side code to maintain a coherent model of track collection, I don't
> see a reasonable way (or need) to avoid some form of remove track event.
> Defining HTML5 video behavior without fully plumbing the actual practice will
> likely prove a problem in the future if an over-simplistic approach is adopted
> (e.g., ignoring track removal).

(In reply to comment #16)
> 
> Nevertheless, the use-case was included in the original bug, so whilst this
> could be tracked as a separate issue it is not "new" and should be dealt with
> as an LC1 issue too.

Yes, if you file a bug on the topic in comment 14, please don't hesitate to mark it as an LC1 bug.


> I do have a mild concern about the proposed approach: we should consider what
> happens with extremely long-running streams (live streams) which may consist of
> a lot of different material spliced together, where the available tracks in
> each segment are completely different. The current approach implies an
> ever-growing track list and I wonder if we really should delete the "dead" ones
> in some way.

There's also an ever-growing buffer (which the user agent might eventually start discarding data from), an ever growing "current time", etc. Are we actually expecting this to be a practical problem?

I'd rather not solve issues that never become real problems. They add complexity to the implementations, the test suites, the spec, etc that is not justified.

We can always add a solution to this later if it becomes a real problem.

Is there anything that still needs to be addressed here other than the issue raise in bug 14492?

(In reply to comment #20)
> Is there anything that still needs to be addressed here other than the issue
> raise in bug 14492?

One question relating to the product allocation of the bugs and how this bug and 14492 will be resolved.

The bug 14492 is being allocated as HTML5 spec and not kept as L1 HTML5. I did not get a clarification on the reallocation. I am concerned that there is a dependency on the two bugs and if there is not a unified solution in my view this bug is not complete.

As indicated by several comments toward this bug the issue with removal of tracks has to be addressed. The consequence of separate allocation is that 14492 may affect the solution provided by this bug.

Question:

1. Will 14492 be handled within LC1 HTML5 drop?

2. If 14492 modifies the solution from 13358 will this be a problem if this bug is closed?

> 1. Will 14492 be handled within LC1 HTML5 drop?

I will handle bugs in the LC1 bucket in the same way as bugs in any other bucket.
I don't know how the chairs will handle them.


> 2. If 14492 modifies the solution from 13358 will this be a problem if this bug
> is closed?

No.


At the end of the day, the reason to separate these bugs is primarily so that if I decide to reject or delay the request to add a way to notice when tracks are dropped, the risk is minimised that any escalation will as a side-effect also remove the ability to tell when a track is added.

On response (1) I have a concern. The bug 14492 was created simply on a request to handle the removal of tracks separately. If looking at the original text of this bug it states adding and deleting tracks and several comments on this bug reflect a need for a solution for deleting.

To only address "adding" of tracks does not address the full intention of the original bug. If the 14492 is handled independently from 13358 and not within LC1 there is an incomplete solution for the webtv IG Media Piping TF requirements.

I understand this is a decision of the chairs and not yours but without a response from the chair I am only able to highlight this issue through the bug.

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted
Change Description: see diff given below
Rationale: see above.

I'm marking this FIXED for now and will address the removal issue in bug 14492, which has been marked LC1 by the chairs.