"Media elements must not stop playing just because all references to them have been removed; only once a media element is in a state where no further audio could ever be played by that element may the element be garbage collected."

Because any event handler can call play(), this would require preventing GC until no more events could possibly be fired. For example, one would have to wait until a media element with preload=auto and a suspend event handler has finished loading completely before GC. That does not seem reasonable.

Here's when Blink currently prevents GC:

1. If there is an audio track and the media element is playing.
2. If there are pending events in the media element task source.

This isn't reliable if one sets a src and assumes that loadedmetadata will fire, so I'd like to add:

3. If the delaying-the-load-event flag is true. (Blocks until loadeddata, suspend or error, bascially.)

It would be nice if the spec were a bit more specific, maybe with this as a starting point.

(In reply to Philip Jägenstedt from comment #1)
> 
> Because any event handler can call play(), this would require preventing GC
> until no more events could possibly be fired. For example, one would have to
> wait until a media element with preload=auto and a suspend event handler has
> finished loading completely before GC. That does not seem reasonable.

Why not? It's what's intended.

If you don't want to wait for that, and want to GC earlier, just suspend the download early. You're allowed to do that whenever you want.

(In reply to Ian 'Hixie' Hickson from comment #2)
> (In reply to Philip Jägenstedt from comment #1)
> > 
> > Because any event handler can call play(), this would require preventing GC
> > until no more events could possibly be fired. For example, one would have to
> > wait until a media element with preload=auto and a suspend event handler has
> > finished loading completely before GC. That does not seem reasonable.
> 
> Why not? It's what's intended.
> 
> If you don't want to wait for that, and want to GC earlier, just suspend the
> download early. You're allowed to do that whenever you want.

OK, I guess one could add some tricks like that if wasted bandwidth turns out to be a problem, which does seem a bit unlikely TBH.

Here's an ad-hoc test for GC while loading:
http://software.hixie.ch/utilities/js/live-dom-viewer/saved/3097

Firefox gets all the way to suspend, IE gets to loadstart, and WebKit/Blink don't get any events at all, which is what I wanted to fix.

Robert, do you know what the precise GC behavior of Gecko is here?

Eric, do you have any opinions on how to change this in WebKit?

I don't think the spec needs to say anything specific about GC here. We GC media elements when it's safe to do so without affecting observable behavior, and this seems to work fine in practice.

The key part of our implementation is here:
http://dxr.mozilla.org/mozilla-central/source/content/html/content/src/HTMLMediaElement.cpp#3471

Thanks Robert!

Translating that to spec terms, I think you prevent GC if any of the following are true:

1. the delaying-the-load-event flag is set
2. !paused (ended now implies pause)
3. seeking
4. autoplaying flag is set
5. networkState is NETWORK_LOADING

That seems pretty good to me, certainly better than what Blink/WebKit does.

The wording of the spec changed in html5.org/r/5975 but I never picked up on that and so Presto's rule was to prevent GC if !paused && !ended && playbackRate != 0.

IE11 apparently has yet another way of doing things, so I think specifying this could help interoperability.

I don't understand what needs to be specified here.

Implementations should behave as if there's no GC. Isn't that all that needs to be said?

Since special GC rules are needed for media elements, interoperable implementations seem more likely if the rules are made explicit.

But if everyone except me thinks that specifying this would be silly, I can accept that.

The rules _are_ explicit: "only once a media element is in a state where no further audio could ever be played by that element may the element be garbage collected".

I don't understand what is wrong with this rule. What cases should we garbage collect that are different from what that rule implies?

(In reply to Ian 'Hixie' Hickson from comment #8)
> The rules _are_ explicit: "only once a media element is in a state where no
> further audio could ever be played by that element may the element be
> garbage collected".
> 
> I don't understand what is wrong with this rule. What cases should we
> garbage collect that are different from what that rule implies?

If comment #3 was ambiguous, I'm OK trying to implement what the spec currently implies. If listing conditions that can be mapped directly to an implementation doesn't fit in editorially, fine, WONTFIX it is.

My concern with listing conditions explicitly is that it's easy to get the conditions wrong and then we end up coerced into GC-exposing behaviour.

True, that is a risk. The fact that this is non-trivial to get right is probably part of the reason why it's currently not very interoperable, but I'll try add something to web-platform-tests instead.

I'm reopening this because of Chromium issue https://code.google.com/p/chromium/issues/detail?id=412876 .

The WebGL conformance tests instantiate transient video elements, add 'playing' and 'timeupdate' event listeners, set the videos to autoplay, and play them. Then the JavaScript reference to the video element goes out of scope. They're very infrequently being GC'd before they fire the first event, causing the test to hang.

roc's statement that Firefox avoids GC'ing the video element until doing so would not be observable is spot on. This behavior needs to be defined more precisely.

(In reply to Kenneth Russell from comment #12)
> I'm reopening this because of Chromium issue
> https://code.google.com/p/chromium/issues/detail?id=412876 .
> 
> The WebGL conformance tests instantiate transient video elements, add
> 'playing' and 'timeupdate' event listeners, set the videos to autoplay, and
> play them. Then the JavaScript reference to the video element goes out of
> scope. They're very infrequently being GC'd before they fire the first
> event, causing the test to hang.
> 
> roc's statement that Firefox avoids GC'ing the video element until doing so
> would not be observable is spot on. This behavior needs to be defined more
> precisely.

I don't see why. It's clear, given the current spec, that Chrome's behavior there is a bug.

It's not clear to me. The videos in question have no audio track. According to the current spec, wouldn't it be legal to GC them at any time, even though doing so would change observable behavior?

Good point, maybe it should say "in a state where no further audio could ever be played and no further events could ever be fired"? Unless in spec terms addEventListener takes a reference until removeEventListener is called?

Hm, I didn't think about the case of a video with no audio track but with a video track. My fear with relying on just events is that in theory, events could fire for forever — consider a user agent that decides to suspend and discard buffered data whenever memory pressure increases, and then aggressively recaches when the pressure is relieved. Even if paused, such a media element would forever be firing 'progress' and 'suspend' events.

Would it be possible to explicitly call out the case of delivering network-related events while the video is paused, and say GC is allowed to occur in that case?

(In reply to Kenneth Russell from comment #14)
> It's not clear to me. The videos in question have no audio track. According
> to the current spec, wouldn't it be legal to GC them at any time, even
> though doing so would change observable behavior?

This spec --- all specs, really --- have an implicit frame condition that you can't change observable behavior unless something in the spec says you can.

For example, the DOM spec does not explicitly forbid a UA from suddenly removing all an element's attributes for no reason. Nevertheless it's clear to all of us that such behavior would be non-conformant.

Likewise, suddenly deciding not to fire any more events for a media element is a behavior that has no mandate in any spec, and is non-conformant.

(In reply to Philip Jägenstedt from comment #1)
> "Media elements must not stop playing just because all references to them
> have been removed; only once a media element is in a state where no further
> audio could ever be played by that element may the element be garbage
> collected."

I guess this could be interpreted as spec permission to stop firing events on elements where "no further audio could ever be played". But I think, given the first clause, it's less than clear.

If this is causing confusion I think the comment should just be removed, or we should clarify that the statement is explanatory rather than normative.

(In reply to Ian 'Hixie' Hickson from comment #16)
> Hm, I didn't think about the case of a video with no audio track but with a
> video track. My fear with relying on just events is that in theory, events
> could fire for forever — consider a user agent that decides to suspend and
> discard buffered data whenever memory pressure increases, and then
> aggressively recaches when the pressure is relieved. Even if paused, such a
> media element would forever be firing 'progress' and 'suspend' events.

In that case Gecko would simply GC the element, which I think is spec compliant: it's observably equivalent to the browser simply deciding not to recache for that element, and it's clear from the spec that decisions about whether and how much to buffer are up to the UA.

Yeah. It's just a question of how to phrase this, if we have to (which I agree is an open question). The current text is clearly not ideal.

So, looking at this more closely, I think actually the spec is fine.

(In reply to Philip Jägenstedt from comment #1)
> 
> Because any event handler can call play(), this would require preventing GC
> until no more events could possibly be fired. For example, one would have to
> wait until a media element with preload=auto and a suspend event handler has
> finished loading completely before GC. That does not seem reasonable.

Actually, this isn't true, since preload=auto is just a hint. The UA could decide just not to cache any more, and then it'd be good to GC, similar to what roc says in comment 20.

(In reply to Kenneth Russell from comment #14)
> It's not clear to me. The videos in question have no audio track. According
> to the current spec, wouldn't it be legal to GC them at any time, even
> though doing so would change observable behavior?

Not if they had event listeners, because the event listeners could change the media resource to something with an audio track.

(In reply to Ian 'Hixie' Hickson from comment #22)
> (In reply to Kenneth Russell from comment #14)
> > It's not clear to me. The videos in question have no audio track. According
> > to the current spec, wouldn't it be legal to GC them at any time, even
> > though doing so would change observable behavior?
> 
> Not if they had event listeners, because the event listeners could change
> the media resource to something with an audio track.

That's subtle. I think some non-normative clarifying text would be helpful to implementers.

Checked in as WHATWG revision r8806.
Check-in comment: Add more explanatory text around media element GC behaviour
https://html5.org/tools/web-apps-tracker?from=8805&to=8806

Is the above sufficient? I wasn't sure 100% how to phrase it.

Thanks, the new text looks good to me.

The text is OK but I think it's a bit misleading at the end. An element with no audio tracks needs to be kept around if it has event handlers that could fire, not really because those event handlers could load a resource with audio, but because that event handler could do *anything at all* with observable side effects, e.g. alert().

The spec doesn't require that it not be GC'ed while there could be observable side effects. Indeed, there will always be observable side-effects — GC'ing itself has one (reduced memory usage), so if we couldn't GC if there was an observable side-effect, we could never GC. That's why the spec is phrased in terms of a specific effect (audio).

The non-normative text which I added, right now, is accurate because it refers to what the normative text says matters for garbage collection.

We could change the normative text, though. Do you have a proposal for what it should say instead?

(In reply to Ian 'Hixie' Hickson from comment #28)
> The spec doesn't require that it not be GC'ed while there could be
> observable side effects. Indeed, there will always be observable
> side-effects — GC'ing itself has one (reduced memory usage), so if we
> couldn't GC if there was an observable side-effect, we could never GC.
> That's why the spec is phrased in terms of a specific effect (audio).

With some specific exceptions, we generally treat performance (space and time) as falling outside the set of observable effects that should be normatively specified.

But whether and when event handlers will be executed is something we generally *do* consider should be normatively specified. Even when the execution of event handlers is nondeterministic we still specify constraints on such execution.

> We could change the normative text, though. Do you have a proposal for what
> it should say instead?

Currently it says:
Media elements that are potentially playing while not in a Document must not play any video, but should play any audio component. Media elements must not stop playing just because all references to them have been removed; only once a media element is in a state where no further audio could ever be played by that element may the element be garbage collected.

I think we can cut this down. We don't need to say "must not play any video"; elements not in a document have no CSS boxes so there's nowhere for video to render, hence no need to specify that it does not render.

I think we should remove the final sentence; it's redundant, since an element suddenly stopping playing and never resuming even though the underlying resource is available is clearly a bug even if this paragraph is deleted. We could make the part before the semicolon a non-normative note. The part after the semicolon is misleading since it suggests that when an element can no longer play audio it can be GCed, although in fact some other conditions must also be true for it to be GCable. I'd just delete that part altogether.

If there's no strong reference to the element, what stops it from being garbage collected (thus stopping audio playback)? Everywhere else in the HTML spec we explicitly call out what the implied strong references are, so that it's well-defined when something with no explicit references can be GCed. (This matters e.g. for WebSockets, Documents, etc.)

(In reply to Ian 'Hixie' Hickson from comment #30)
> If there's no strong reference to the element, what stops it from being
> garbage collected (thus stopping audio playback)?

Because just stopping playback for no reason is a behavior that finds no justification in the spec.

> Everywhere else in the
> HTML spec we explicitly call out what the implied strong references are, so
> that it's well-defined when something with no explicit references can be
> GCed. (This matters e.g. for WebSockets, Documents, etc.)

I wasn't aware of that. Consistency might be a reason to spec it here too, I suppose. But I don't like the idea that we have to explicitly say that features must not suddenly stop working.

IMHO we have to be explicit because it's not clear to me how else we would define what observable side-effects are ok and what observable side-effects are not.

Do you think an HTML5 canvas implementation that suddenly clears the canvas for no particular reason is conformant? I don't recall any spec text saying that's not allowed.

In other words, I think observable UA behavior not explicitly permitted by some spec is invalid. I'm happy to draft a definition of "observable" if you don't think it's clear enough.

The spec defines the value of the bitmap, and everything that affects the bitmap, so no, that wouldn't be conforming.

But if you like, you can look at it the other way: if we don't define when something can be GCed, then nothing can ever be GCed.

(In reply to Ian 'Hixie' Hickson from comment #35)
> But if you like, you can look at it the other way: if we don't define when
> something can be GCed, then nothing can ever be GCed.

That's not true at all. This is extra-clear in programming language theory: some languages have formal specifications, and typically those specifications never mention GC, but implementations of those languages implement GC as a semantics-preserving optimization.

It's also true in our setting. Given
  function f() {
    var p = document.createElement('p');
  }
you can take out everything any spec says about GC, and GC will still be possible, because there is no way to distinguish an implementation that GC's the created element from an implementation that never GCs but has infinite memory. The latter conforms to the spec, of course, and therefore so does the former.

That's true for <p>, but it's not true for some other elements. For example, it's not true for <img>. And yet you can still GC <img> (you just have to make sure the network traffic survives). Same, in reverse, with the object returned document.images — you're allowed to GC that even if it has visible side-effects.

(In reply to Ian 'Hixie' Hickson from comment #37)
> That's true for <p>, but it's not true for some other elements. For example,
> it's not true for <img>. And yet you can still GC <img> (you just have to
> make sure the network traffic survives).

If you're talking about the code
  function f() {
    var img = document.createElement('img');
    img.src = "http://example.com/resource";
  }
then if the spec says nothing about GC, a conforming implementation will have to behave observably equivalent to an implementation that never GCs. I.e. it will have to ensure the HTTP fetch completes, but can collect the element after that (or even before that, as long as the HTTP fetch does complete). What's the problem here?

If you're talking about something else, please describe the issue in more detail.

> Same, in reverse, with the object
> returned document.images — you're allowed to GC that even if it has visible
> side-effects.

I have no idea what you mean here. I can't see how GC of the document.images HTMLCollection could have observable side effects.

You can set a property on the collection and later get document.images again. If it was GCed the property is not there. The spec allows GC there despite it being observable.

Where does the spec do that? And why does it do that?

Actually document.images is defined to always return the same object. But getElementsByTagName:

[[
When invoked with the same argument, the same HTMLCollection object may be returned as returned by an earlier call.
]]

https://dom.spec.whatwg.org/#concept-getelementsbytagname

As to why, implementors refused to do anything else because returning the same object every time is like leaking memory when used only once, and returning a new object every time has bad performance when used in a loop.

(In reply to Simon Pieters from comment #41)
> As to why, implementors refused to do anything else because returning the
> same object every time is like leaking memory when used only once,

That isn't necessary so. For example, consider the following code:
  function f() {
    var images = document.images;
    return images.length;
  }
Assuming this is the only getter of document.images in the program, then after running f(), the 'images' object can be GCed, even though the spec says the same object must be returned. That's because the only way to observe whether document.images is returning a new object is to retain a reference to the old object for comparison purposes, or add an expando to the old object and test for its presence on the new object, and this program doesn't do either of those things.

Having said that, the language in the spec for getElementsByTagName is OK. Note that it doesn't say anything specifically about GC, it just lets the implementation be nondeterministic about when and whether it reuses objects. Likewise, for media elements, letting the implementation nondeterministically stop loads for non-playing elements gives us useful leeway to GC media elements without violating specified behavior.

Well, we could just strip out all the stuff about strong references and GC and so on and just talk about what GC side-effects are acceptable or some such. That's a lot of work for minimal gain as far as I can tell, though, so I'm reluctant to do it at this time.

OK, but I hope I've convinced you that would be the right direction to go. I think in this bug and elsewhere we can proceed on the assumption that implementations should behave as if there is no GC.

If there's no GC, then APIs like PortCollection have no purpose. This is clearly not true; we need APIs like PortCollection to _enable_ GC. Though of course PortCollection itself exposes GC, unfortunately, not sure what to do about that... since that's the whole point, in a way...

You can have APIs that are designed to enable GC, without explicitly specifying their GC effects.

For example, an alternative way to define PortCollection would have been to remove the 'iterate' method and instead expose a 'postMessage' method that broadcasts a message to every port in the collection. Then PortCollection wouldn't need to specify anything about GC behavior, yet it would be a valuable tool that, when used properly, would allow implementations to GC dead ports without affecting observable behavior.

A similar example is JS WeakMaps. WeakMaps are carefully designed to allow key/value pairs in the map to be GCed; the critical design feature is that WeakMaps have no API letting you iterate through the key/value pairs in the map. But again, WeakMaps don't specify any GC behavior.

GC is an optimization. We should design APIs that facilitate important optimizations. We can do this without explicitly specifying the optimizations.

That wouldn't work because you couldn't transfer stuff if you did that. Also, that would still expose GC behaviour, unfortunately.

(In reply to Ian 'Hixie' Hickson from comment #47)
> That wouldn't work because you couldn't transfer stuff if you did that.

It would work, it just wouldn't be as efficient as it could be. I don't think transfers make sense for a broadcast-oriented API anyway.

> Also, that would still expose GC behaviour, unfortunately.

How?

Anyway the design of PortCollection is not the point. But I don't know how to make further progress in this conversation.

I don't understand how you could transfer, say, a port, using the API you describe.

It would expose GC behaviour because you could set up a bunch of ports that do nothing but respond to messages, then drop all references to them other than a reference to them in a PortCollection (so that they can get GCed), then keep sending messages and seeing which ones you receive.

(In reply to Ian 'Hixie' Hickson from comment #49)
> I don't understand how you could transfer, say, a port, using the API you
> describe.

I don't think it makes sense to transfer anything, including a port, in a broadcast-style API.

> It would expose GC behaviour because you could set up a bunch of ports that
> do nothing but respond to messages, then drop all references to them other
> than a reference to them in a PortCollection (so that they can get GCed),
> then keep sending messages and seeing which ones you receive.

As long as a Worker can receive messages and respond to them in some observable way, it should not be GCed. Maybe that doesn't match the spec right now, but that's the way it should work.

(In reply to Robert O'Callahan (Mozilla) from comment #50)
> (In reply to Ian 'Hixie' Hickson from comment #49)
> > I don't understand how you could transfer, say, a port, using the API you
> > describe.
> 
> I don't think it makes sense to transfer anything, including a port, in a
> broadcast-style API.

Right, that's why this isn't a broadcast-style API. It just calls you back for each port.

The use case is something like this: Suppose you had a document editor system, with a shared worker dealing with all the communication with the server, and several tabs were open on a particular document. At some point, the server contacts you to ask you if any of your readers are looking at a particular page. Using this API, you can have all the people who are looking at this document have a port in a particular PortCollection, and when you get that request you can send them all a request to find out the answer, passing them a port on which to send the reply.

If we only had a broadcast-style API, you couldn't do this. You'd have to use the main port for the doc to handle replies to this particular query, instead of being able to divide conversations into specific ports.


> > It would expose GC behaviour because you could set up a bunch of ports that
> > do nothing but respond to messages, then drop all references to them other
> > than a reference to them in a PortCollection (so that they can get GCed),
> > then keep sending messages and seeing which ones you receive.
> 
> As long as a Worker can receive messages and respond to them in some
> observable way, it should not be GCed. Maybe that doesn't match the spec
> right now, but that's the way it should work.

That requires that authors manage the lifetime of ports. The point of the PortCollection API is that it lets you have weak references to ports so that if both sides are done, the browser can GC the ports without the two sides having to explicitly confirm that that's ok.

As far as I can tell, the original issue here has been addressed by the addition of the note. If there's any additional changes you think should happen, please file new bugs. Thanks.

Yep, and this has been fixed in Blink now too. Thanks for your time!