14:52:19 RRSAgent has joined #mediawg 14:52:19 logging to https://www.w3.org/2021/06/15-mediawg-irc 14:52:21 Zakim has joined #mediawg 14:53:31 Meeting: Media WG - call on WebCodecs 14:53:43 RRSAgent, make logs public 14:54:25 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2021-06-15-Media_Working_Group_Teleconference-agenda.md#agenda 14:59:23 sandersd has joined #mediawg 15:01:38 chcunningham has joined #mediawg 15:01:42 present+ Francois_Daoust, Chris_Cunningham, Chris_Needham, Dan_Sanders, Jan-Ivar_Bruaroey, Youenn_Fablet, Zachary_Cava, Eric_Carlsson, Gary_Katevman 15:03:14 present+ Peng_Liu, Dale_Curtis 15:04:39 pengliu has joined #mediawg 15:04:47 zacharycava_ has joined #mediawg 15:05:13 zacharycava_ has joined #mediawg 15:05:13 pengliu has joined #mediawg 15:05:13 chcunningham has joined #mediawg 15:05:13 sandersd has joined #mediawg 15:05:13 jib has joined #mediawg 15:05:13 cpn has joined #mediawg 15:05:13 hober has joined #mediawg 15:05:13 QABaglan has joined #mediawg 15:05:13 Germain has joined #mediawg 15:05:13 padenot has joined #mediawg 15:05:13 BrandonCabral has joined #mediawg 15:05:13 gkatsev has joined #mediawg 15:05:13 mounir has joined #mediawg 15:05:13 sangwhan has joined #mediawg 15:05:13 heejin_ has joined #mediawg 15:05:13 gregwhitworth has joined #mediawg 15:05:53 Chair: Chris_Needham 15:06:14 scribe: tidoust 15:06:35 cpn: Follow-up discussion to last week's call. Let's continue the discussion on window/worker first. 15:06:45 https://github.com/w3c/webcodecs/issues/211 15:06:55 Topic: Issue #211 - Should WebCodecs be exposed in Window environments? 15:07:01 present+ Jer_Noble 15:07:06 Chair+ Jer_Noble 15:07:15 dalecurtis has joined #mediawg 15:07:46 cpn: I'd like to briefly recap where I think we are in terms of discussion and see if there's progress we can make during this call to find consensus. 15:08:32 ... I see both sides of the arguments. Strong arguments in either direction. I'd like to separate ImageDecoder from audio/video. Do the same concerns apply to image decoders? 15:08:48 ... Do we see that there is any potential risk to exposing image decoders on window? 15:09:11 chcunningham: From our side, there are no specific concern. 15:09:45 youenn: In general, image decoding is a one time operation. Being delayed by 100ms is probably fine. I have less concerns there. I haven't really looked at the API. 15:10:57 dalecurtis: If we have no concern for Image decoders, we should clarify what our concerns are for the rest, as image decoding is at the basis of decoding p-frames and the like. 15:11:39 ... The APIs don't necessarily belong together but if we feel there's no issue with image decoder, then I question what our issues are with the rest. 15:12:12 jib: When people talk about low-latency, if there's motion involved, would you rather have constant performance, or jank. 15:12:34 dalecurtis: First paint latency, that's low latency that I had in mind in this issue. 15:13:16 ... If we want devs to improve first paint latency, then workers can be a hindrance, time taken to set things up. 15:13:31 cpn: Setup cost would be a one time thing, right? 15:13:50 eric has joined #mediawg 15:14:11 dalecurtis: Yes. For multi-frame cases, the overhead per frame is the time to get to your worker and execute an async operation. 15:14:20 ... For single-frame cases, the setup cost is the main cost. 15:15:22 https://github.com/w3c/webcodecs/issues/211#issuecomment-860981641 15:15:49 chcunningham: The core of Apple's concern is performance and whether developers can be trusted to make the right calls for their app. 15:16:05 ... Two sub-issues: whether performance issue is real and whether developers can be trusted. 15:16:40 ... The developers that we've spoken to so far have shared an awareness of the API and inner workflow. 15:16:55 ... Many of them have done just workers, or just window, or both. 15:17:35 ... We're basing our conclusions on the experience that developers have. 15:18:02 ... We are also arguing that the performance issues have not materialized in our experience. 15:19:01 ... One final point is that there is tension on APIs that are or are not exposed to workers. There are draft specifications for things such as MediaStreamTrack, but the work is very much still at early stages 15:19:47 jib: It's not about not trusting the devs, but more about having the right defaults for the devs. 15:20:48 ... In the past where there have been better models, there has been a lot of friction towards moving to the better but slightly more complex model. The better model needs to be imposed somehow. 15:21:11 ... Every feature we had in the insecure model removes some incentive to move to the default secure model. 15:21:48 ... So, rephrasing as "better defaults". We're not sure that the cost of abandoning better defaults is worth the benefits 15:22:21 cpn: It seems to me that there are many APIs that could be used in conjunction with WebCodecs are not available on workers today. To what extent is that blocking? 15:22:48 chcunningham: There aren't applications that cannot be built today and that could be built in the future when these APIs are exposed to workers. 15:23:09 ... The problem is performance, as you'd be using postMessage throughout between main thread and worker. 15:23:32 ... The complexity of being in a worker is a technical pain that we're hearing a lot from developers. 15:24:29 cpn: This leads me to a comment from Dale. If we were to restrict the API to workers, under what conditions would we revisit that decision? What information beyond what we currently know do we need to see to inform a future decision on window? 15:25:03 youenn: If we start with only workers, it's possible that someone writes a JS shim that would allow to do things on the main thread, paying the cost of postMessage. 15:25:30 ... We can look at inefficiencies with that approach and if the cost is really high. If it is, it may delay or forbid some applications. 15:26:42 ... Currently, exposing to window does not provide new feature. Having measurements would be useful. Gathering them would greatly help to evaluate whether exposing to window is very useful or marginally useful. 15:27:17 dalecurtis: We have this origin trial, several developers speaking about their experience and performances. 15:27:28 ... You're saying that this is not enough data. 15:28:28 chcunningham: I want to also mention how effective the strategy can be. If people start using the shim instead of switching to workers, they will pay a higher cost. 15:28:56 ... Jan-Ivar knocked that down with "we can remove the library if that happens". But once a library is out there, it's out there. 15:29:47 jib: The first thing I would point out. Possible extreme outcomes: On one hand, people will not shim and abandon. On the other hand, putting the default to worker will be too hard and people will always use the shim. 15:29:56 ... I'm hoping that we can end up in the middle. 15:30:13 ... Problem is if we expose on window today, we can undo this later on. 15:30:55 ... Not exposing to window provides some incentive to move to workers right now, giving us some way to measure things properly. 15:31:24 chcunningham: There are many cases for WebCodecs that are not in the WebRTC space, e.g. video editor. 15:32:52 ... We don't know how app developers will handle this for sure. The starting point is often that you find a library and if it addresses some friction such as working around the need to use to worker, then you'll go with it regardless of whether that is a good design decision. 15:33:35 jib: The Web is full of dead libraries. As soon as browsers say "we made a mistake, we can expose to window", then libraries that use postMessage will disappear. 15:34:19 chcunningham: My fear is if we continue the position that exposing to window is problematic, then the library will continue to be used. 15:34:32 jib: If we fail to convince people, then we should expose to window. 15:34:41 youenn: I agree. 15:35:01 ... [gives some WebRTC example for comparison] 15:35:31 ... In WebRTC, I hope that that we can use the same codepath for WebRTC or WebTransport + WebCodecs. 15:35:41 ... It's difficult as we're in the design phase. 15:36:02 ... If we screw things up today, then we're effectively screwed. Hard to rollback. 15:36:57 chcunningham: I find the idea of making a totally different API for window than the one for workers totally unappealing. 15:37:15 ... 35 minutes in the call, I feel we're still looping. 15:37:27 ... I would like to encourage the chairs to escalate this to a CfC. 15:38:27 jib: In the interest of new ideas, I thought there was an issue on real-time mode. Just on top of my head, could realtime mode be only exposed in worker and non realtime mode be exposed to worker and window? 15:38:50 ... Exposing the API to workers has consensus within the group. 15:39:03 ... I'm hoping that the CfC is not about that, but focused on exposure to window. 15:40:05 cpn: I put forward a proposed resolution in the thread at the end of last discussion. Expose to window + provide developer guidance. If we move towards a vote, would we be happy for us to use that phrasing? 15:40:42 ... We could phrase it in the opposite direction. Defer exposition on window until we have clearer experience and feedback. 15:41:04 jib: I would prefer that second formulation, as it leaves the door open. 15:41:33 chcunningham: Just to clarify, I'm not asking for a realtime CfC, but one sent to the group's mailing-list. 15:42:40 jib: Given the lack of documentation, the shipping calendar in Chrome is aggressive. 15:43:39 youenn: I also prefer Jan-Ivar's formulation, which is clearer. Defer or now is clearer that we're not asking about taking a strong position on exposing to window now. 15:43:49 ... Ship it now or defer the decision to later. 15:44:06 cpn: I would like to see some criteria on what the conditions would be for a later decision. 15:45:00 jib: We could say use cases below a certain threshold of ms but I think that we will know if we've made the wrong decision in a year from now. 15:45:28 ... That's also the benefit of deferring. If there is lack of consensus, process-wise, that will create issue down the line. 15:46:05 cpn: Personally, I'm sympathetic to that point of view. Having a decision that closes the door. Something that allows us with new information to revisit this would be preferable. 15:46:19 ... Chris, would you be ok with that formulation? 15:46:26 chcunningham: Yes, as long as my vote can be "Do not defer" 15:46:40 Topic: Issue #239 - Is exposing https://w3c.github.io/webcodecs/#enumdef-hardwareacceleration a good idea 15:47:04 https://github.com/w3c/webcodecs/issues/239 15:47:14 chcunningham: Starts with Zoom and others saying "we really want to use your hardware codecs, and we really do not want to use your software codecs". 15:47:40 ... The difficulty is to expose that as the right granularity for privacy preserving reason. 15:48:01 ... [summarizes discussion on the issue] 15:48:39 ... Last comment from someone from Zoom. The takeaway is that these people have their own set of features and that they need to use hardward codecs. 15:49:18 jernoble: Chair hat off. We have decided in past discussions not to expose hardware details. 15:49:43 ... To me personnally, that does not seem to be enough of a use case to warrant exposing them today. 15:50:55 ... Hardware or power efficient: For instance, Zoom could come to us and say that the Intel H264 chipset has a bug and that they want to know precisely where it is deployed. 15:51:05 ... We do not want to expose that. 15:51:39 chcunningham: Zoom and VLC have their own implementation of codecs. There is the case of avoiding a bug as you suggest. That is not the only case. 15:52:02 ... There's also the "I know my code, I prefer to use my stuff unless you can guarantee me that you can give me hardware acceleration" 15:52:39 youenn: They probably care more about power efficiency than hardware acceleration. Low-resolution, they may prefer their implementation. For HD, they will likely prefer hardware decoding. 15:52:49 ... Media Capabilities gives you that. 15:53:25 ... To increase fingerprinting, there should be a high motivation. 15:53:47 ... In that specific case, there are heuristics that you can use and we should clarify why this is not good enough. 15:53:52 q? 15:54:45 chcunningham: My issue with that argument is that there are two pieces. The stack for WebCodecs is not necessarily the same as the stack for RTC. 15:55:30 ... If the gain is 1% of increased fingerprinting, then it's not much. 15:56:05 youenn: That should be brought to PING, who may look at it and realize that it is for a restricted set of users and push back. 15:56:22 chcunningham: PING reviewed the specification and did not raise anything on this topic. 15:56:37 ... 1% was just to counter your 99%. 15:57:08 youenn: Fingerprinters will use that API. There are good reasons to believe that this will be the main usage of WebCodecs. That is the case with WebRTC and that is very sad. 15:57:17 chcunningham: I think that this is very speculative. 15:57:32 ... The main usage of Media Capabilities is not fingerprinting for instance. 15:57:58 ... Using MediaRecorder, apps can create a hash of the hardware. 15:58:23 youenn: And that is fine. If the bits MediaRecorder create create an issue, we can address that later on and improve the output. 15:59:09 ... The specific boolean is exposing new information that is not currently exposed. Zoom would probably manage to come up with a good heuristic. 15:59:46 chcunningham: If they can find a good heuristic, then the boolean does not really increase the fingerprinting surface. 16:00:04 ... The mitigations exist. 16:00:36 youenn: If we start to do that mitigation, then many web sites will expect things to work in Safari and it won't. 16:00:51 chcunningham: This is not true. Developers will have to be prepared for the fallback. 16:01:44 youenn: The user may not be served well as the decision may then be per browser with less optimal decisions taken in some browsers. 16:02:52 ... I don't want Safari to be more privacy-preserving, I want the APIs to be privacy-preserving with Safari implementing them as planned. 16:03:21 jib: I agree with Youenn that Web specs need to increase interoperability and to do that, we need to create abstractions. 16:03:48 ... If we expose hardware, we cannot compete on privacy since privacy is no longer part of the abstraction 16:04:19 ... PING raised a similar comment on WebRTC. They may have missed that in WebCodecs, but we should probably raise that with them. 16:05:36 s/WebRTC/WebRTC Stats/ 16:05:46 ... It would be good to close the loop with PING on that. 16:06:19 https://www.w3.org/blog/2019/06/privacy-anti-patterns-in-standards/ 16:06:38 cpn: Tess dropped a link the other day by someone from PING that a new feature should not point to existing features as a justification for increasing the fingerprinting surface, as it prevents reducing the surface down the road. 16:07:24 chcunningham: If we believe that this can be solved with existing features, then we are effectively locking us in providing reliable information for Media Capabilities. 16:08:17 ACTION: chcunningham to reach out to Ping regarding hardware decoder 16:08:47 s/decoder/acceleration 16:08:48 Topic: Issue #213 - Should ImageDecoder IsTypeSupported be (a)synchronous? 16:08:51 https://github.com/w3c/webcodecs/issues/213 16:09:17 chcunningham: Media Capabilities is departure from previous APIs in that we've been doing things asynchronously. 16:10:09 ... The argument for making ImageDecoder asynchronous is that image codecs are increasingly video. We may in the future rely on hardware systems, using background process, etc. 16:10:19 ... So we propose to make the method asynchronous 16:10:46 jernoble: Chair hat off again. In Safari, all of these questions are answered in a separate process and that's fine. 16:11:02 ... Making things synchronous is not that of a big deal. 16:11:20 ... That is just one implementers experience of jumping to another process. 16:11:46 ... It reminds me of the discussion on the autoplay policy API. 16:12:09 ... I feel that it is not justifiable intrinsically that it gets done asynchronously. 16:12:22 ... A lot of things would be easier in my daily life if APIs were more synchronous. 16:12:54 ... But that is not what we should care about. Implementers are way down in the chain: end users, then dev, then... implementers. 16:13:17 ... I don't see the justification for imposing this on developers. 16:13:33 cpn: Is it the same criteria as for the Autoplay Policy API? 16:13:44 jernoble: From my point of view, that is the same situation. 16:13:52 ... Same arguments, same conclusion. 16:14:01 chcunningham: I wasn't part of the autoplay discussion. 16:14:41 ... It surprises me that Safari would want to hide an asynchronous API under the hoods from developers. Being transparent about asynchronoucity seems a good thing for developers to improve their performances. 16:14:48 jernoble: We just solve this with cachin. 16:14:54 s/cachin/caching/ 16:15:01 ... XPC != asynchronous 16:15:33 ... When I say that something happens in another process, it is not necessarily intrinsically asynchronous. 16:16:19 jernoble: I know that Youenn and I disagree here. Strictly speaking for myself, this does not meet the muster of making something asynchronous 16:17:05 ... When autoplay came up before, the async keyword was not available in all platforms that were being targeted. 16:17:22 ... Nested promises were a cost. 16:17:29 chcunningham: About caching? 16:18:11 jernoble: Talking specifically about canPlayType. We don't cache specific responses about codec strings, but caches about container types, and that maps pretty well with the API space that we're targeting here. 16:18:17 ... These can be cached. 16:18:47 ... We push the answer to that answer to every process around. We pay the cost once and it is never paid again. 16:19:08 ... That is an implementation detail that may not apply to other implementations. 16:20:05 chcunningham: The people who own the part of the chrome would not allow me to hold these responses in cache. At best, for some calls, there will be some cache miss. 16:20:27 s/chrome/chrome startup/ 16:21:04 chcunningham: If Youenn and you disagree on this point, should we let you align on a position with Youenn? 16:21:17 jernoble: I was willing to convince everyone on this call. 16:21:48 cpn: From a developers points of view, we have a number of APIs that do similar types of queries and that are all synchronous. Is consistency a good argument? 16:22:02 chcunningham: It's a mixed bag. We have both sync and async in practice. 16:22:31 cpn: Wondering about next steps. 16:22:45 chcunningham: This conversation is a little bit less stuck and less high priority. 16:23:10 ... Fine with giving Jer time to exchange with Youenn and come back to the group. 16:23:26 Topic: Next meeting 16:23:37 cpn: Do we feel we've discussed these issues enough? 16:24:10 jernoble: Would you like to see some follow-up discussion on the second issue in particular, chcunningham? 16:24:35 chcunningham: We'll go back to would-be users of this feature and perhaps call for another discussion afterwards. 16:25:42 cpn: OK, we're available to organize a meeting if and as needed. 16:26:22 RRSAgent, draft minutes 16:26:22 I have made the request to generate https://www.w3.org/2021/06/15-mediawg-minutes.html tidoust 18:24:02 Zakim has left #mediawg 18:48:59 dalecurtis has joined #mediawg 18:51:07 chcunningham has joined #mediawg