IRC log of mediawg on 2022-12-13

Timestamps are in UTC.

16:46:56 [RRSAgent]
RRSAgent has joined #mediawg
16:46:56 [RRSAgent]
logging to https://www.w3.org/2022/12/13-mediawg-irc
16:47:02 [Zakim]
Zakim has joined #mediawg
16:47:13 [cpn]
Meeting: Media WG
16:47:28 [cpn]
Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-12-13-Media_Working_Group_Teleconference-agenda.md
16:54:14 [alwu]
alwu has joined #mediawg
17:01:12 [Matt_Wolenetz]
Matt_Wolenetz has joined #mediawg
17:01:18 [Matt_Wolenetz]
present+
17:03:28 [jernoble]
jernoble has joined #mediawg
17:04:10 [youenn]
youenn has joined #mediawg
17:04:14 [Matt_Wolenetz]
Hi Jer :) - I don't see anyone else on webex yet.
17:04:23 [jernoble]
Zakim, prepare the meeting
17:04:23 [Zakim]
RRSAgent, make logs Public
17:04:24 [Zakim]
please title this meeting ("meeting: ..."), jernoble
17:04:43 [jernoble]
meeting: Media WG Teleconference - 2022-12-13
17:04:51 [Matt_Wolenetz]
likewise
17:04:52 [jernoble]
* Same
17:05:18 [tidoust]
tidoust has joined #mediawg
17:06:17 [jernoble]
present+
17:06:46 [cpn]
present+ Chris_Needham
17:06:49 [cpn]
scribe+ cpn
17:07:10 [Matt_Wolenetz]
present+
17:07:32 [tidoust]
RRSAgent, draft mintes
17:07:32 [RRSAgent]
I'm logging. I don't understand 'draft mintes', tidoust. Try /msg RRSAgent help
17:07:34 [tidoust]
RRSAgent, draft minutes
17:07:34 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust
17:08:53 [hta]
hta has joined #mediawg
17:09:08 [tidoust]
Present+ Francois_Daoust, Dale_Curtis, Alastor_Wu, Peter_Thatcher, Harald_Alverstrand, Frank_Liberato, Youenn_Fablet, Eric_Carlson
17:09:10 [tidoust]
scribe+
17:09:21 [tidoust]
Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-12-13-Media_Working_Group_Teleconference-agenda.md#agenda
17:09:44 [tidoust]
Topic: ITU-T SG16 Liaison statement on WebCodecs
17:09:46 [youenn_]
youenn_ has joined #mediawg
17:09:50 [ericc]
ericc has joined #mediawg
17:10:01 [tidoust]
cpn: We received an incoming liaison statement from ITU-T SG16.
17:10:09 [cpn]
https://github.com/w3c/media-wg/blob/main/liaisons/2022-10-28-itu-t-sg16.md <- Draft reply
17:10:14 [tidoust]
... Around WebCodecs, and also around new VVC codec.
17:10:50 [tidoust]
... I drafted a reply, which describes WebCodecs, the use cases, a few indications about our own plans such as current work on VideoFrame metadata registry.
17:11:11 [tidoust]
... I shared this. Got a thumbs up from Bernard, Jer, Paul.
17:11:30 [tidoust]
... I want to make sure that everything we write here is representative.
17:11:39 [tidoust]
... I was hoping to get this out before the Christmas break.
17:11:58 [tidoust]
... If you haven't had a chance to look at it yet, now would be a good time.
17:12:12 [tidoust]
Present+ Bernard_Aboba
17:12:52 [tidoust]
youenn: I like the fact that you state that the group would be open to add a registration provided there was support from implementors.
17:13:00 [tidoust]
... I assume that means user agent implementors?
17:13:32 [tidoust]
cpn: That's a question for the group perhaps. H.263 comes to mind for instance.
17:13:57 [tidoust]
Dale_Curtis: I don't think that we want to be gatekeepers of what the registry contains, even though there isn't support in web browsers per se.
17:14:09 [tidoust]
... We'd still want some technical constraints to be met.
17:14:28 [tidoust]
cpn: Right. That would apply to any future registration as well.
17:14:47 [tidoust]
Topic: WebKit update on Audio focus/audio session API
17:15:10 [tidoust]
Slideset: https://docs.google.com/presentation/d/1jICP-gAT5FydEEi_fUR0kXFHIYcbxlCo2J54rIxBCeg/edit
17:15:52 [cpn]
scribe+ cpn
17:16:10 [cpn]
Youenn: We received reports that audio handling on iOS isn't easy, e.g., VC applications
17:16:24 [cpn]
... The intent of the application may not match our heuristics for setting up the audio pipeline
17:16:33 [cpn]
... So a new API may be appropriate
17:16:51 [cpn]
... You might remember the Audio Focus API, initially in Media Session, then split out from that
17:17:00 [cpn]
... There's an explainer, linked from the slides
17:17:16 [cpn]
... The overall goal is to get feedback, is the scope right, next steps?
17:17:41 [cpn]
... Compared to the original Audio Focus API, we wanted to reduce scope, for the iOS platform
17:18:04 [cpn]
... We focused on the audio session category, and interruptions
17:18:20 [cpn]
... The API should support future features such as requesting or abandoning audio focus
17:18:27 [cpn]
... Handling audio providers as a group
17:18:35 [cpn]
... We wrote an explainer, and a prototype in WebKit
17:18:49 [tidoust]
[slide 3]
17:18:50 [cpn]
... Some examples: setting the audio session category, you can open the demo in iOS
17:19:08 [cpn]
... playAudio and capture functions, for microphone input
17:19:36 [tidoust]
s|Slideset: https://docs.google.com/presentation/d/1jICP-gAT5FydEEi_fUR0kXFHIYcbxlCo2J54rIxBCeg/edit|Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Dec/att-0000/AudioSession_API.pdf|
17:19:44 [cpn]
... If you call playAudio initially, then capture, it's disruptive in iOS. The reason is that when you play using Web Audio, it's ambient
17:20:03 [cpn]
... Two different audio levels when going from ambient to play & record. Something we want to avoid
17:20:15 [tidoust]
i/Youenn: We received/[Slide 2]
17:20:20 [cpn]
The setCategory function allows you to set the category to play & record, don't use ambient
17:20:28 [cpn]
s/The/... The/
17:20:42 [tidoust]
[slide 4]
17:21:09 [cpn]
Youenn: On interruption, when you're in a video call, you might receive a phone call, which is higher priority, and the website is interrupted, capture stopped, audio or video elements may be stopped
17:21:17 [cpn]
... But the website may not know that
17:21:34 [cpn]
... It's also not clear to the website whether to restart audio after the phone call
17:21:56 [cpn]
... Providing the concept of an audio session, which can go between active and interrupted, allows the website to change what is visible to the user
17:22:25 [cpn]
... On an interruption, it could show a UI, or UI to allow the user to restart capture
17:22:39 [tidoust]
[slide 5]
17:23:03 [cpn]
... We tried to keep the API small. There's an audio session state and audio session type. Then we added an AudioSession interface, which we though was clearer
17:23:29 [cpn]
... Use that to say it's ambient (mix with others), or play & record, so the UA can set the audio pipeline accordingly
17:23:55 [cpn]
... There are event handlers, no constructor. For simple use cases, a getter on navigator to get the audio session
17:24:12 [cpn]
... A default global audio session. Use this object to query or tailor it
17:24:16 [tidoust]
[slide 6]
17:24:28 [cpn]
... My main interest is not to go into specific issues. More issues are welcome
17:24:55 [cpn]
... Question: is this of interest, is it going in the right direction? Any thoughts on potential next steps?
17:25:37 [cpn]
Dale: From a Chrome point of view, Mounir and Becca worked on it. At a glance, seems reasonable. There might be worry about duplication between Media Session and Audio Session, but no specific thoughts on that
17:26:03 [cpn]
Youenn: The API shape is different, there might be only one Media Session in a page, but only one Audio Session
17:26:16 [cpn]
... The call to split the two things in the past is OK
17:26:43 [cpn]
... We decided to delay the grabbing and releasing of audio focus. There might be other things to consider, e.g., auto play
17:27:26 [jernoble]
q?
17:27:50 [cpn]
... A question I have, is it's not yet submitted in the WG. Is it already in scope?
17:28:35 [tidoust]
cpn: Looking at the charter, Audio Focus API is in the list of potential normative deliverables
17:28:53 [tidoust]
... We just need to run a call for consensus to adopt the spec to the Media WG
17:29:23 [cpn]
Sushanth: How to handle audio from multiple tabs?
17:29:39 [cpn]
Youenn: This would help with that
17:30:04 [tidoust]
present+ Sushanth_Rajasankar
17:30:04 [cpn]
Sushanth: If the audio type requested by one browser is playback, and from another is ambient, only one can exist at a time
17:30:47 [cpn]
Youenn: You'd mimic what two native applications would do. One session with playback would probably not be interrupted by another that requests ambient
17:31:05 [tidoust]
cpn: At what point would we be ready to run a call for consensus on this?
17:31:24 [tidoust]
youenn: If there's already consensus in this call, we'd be interested to run it as soon as possible.
17:31:33 [tidoust]
... No particular hurry, but the sooner the better.
17:31:48 [tidoust]
... If there's no consensus, we'd like to know what to work on.
17:32:12 [tidoust]
cpn: Just worried about support from other browser vendors.
17:32:32 [tidoust]
youenn: We talked a bit with Mozilla. I can check with them and get back to you.
17:33:03 [tidoust]
alwu: From Mozilla Firefox perspective, that's an API we'd be interested in supporting as well.
17:33:33 [tidoust]
Dale: And no reason to hold off calling for consensus while we figure things out internally.
17:34:10 [tidoust]
jernoble: In the meantime, feedback on existing issues is welcome.
17:34:34 [tidoust]
cpn: So proposed resolution is to run a CfC.
17:34:47 [tidoust]
Topic: Consistent SVC metadata between WebCodecs and Encoded Transform API
17:38:16 [tidoust]
Bernard: [going through slides]. Sequence of unsigned long dependencies. There's also some missing information.
17:38:33 [tidoust]
... We're essentially re-inventing WebCodecs in another spec, perhaps not the right way to go.
17:38:47 [tidoust]
... Two different SVC metadata dictionaries could be avoided.
17:39:21 [tidoust]
... Temporal may be shipping in Safari, but spatial is not shipping anywhere.
17:39:32 [tidoust]
Dale: I'm in favor of unifying what we can.
17:39:56 [tidoust]
Bernard: Proposal is for a few of us to get together and prepare a PR to harmonize things
17:40:04 [tidoust]
... This would at least avoid future issues.
17:40:33 [tidoust]
... We made some progress in the last couple of days, and Youenn prepared a bunch of PRs that solved a number of type mismatches.
17:41:57 [tidoust]
cpn: Is this something for the WebCodecs spec itself or the metadata registry?
17:42:10 [tidoust]
Bernard: This is for encoded metadata for which we don't have a registry.
17:43:34 [tidoust]
Topic: Media Pipeline architecture - Media WG input and WebRTC collaboration planning
17:43:58 [tidoust]
cpn: Back at TPAC, we identified several places where we may benefit from coordination between groups.
17:44:08 [tidoust]
... This is picking up on where we're at with this.
17:44:28 [tidoust]
Bernard: We created a Media Pipeline architecture repo following discussions.
17:44:50 [tidoust]
... Issues and pointers to sample code covering integration of next generation web media apis.
17:45:20 [tidoust]
... Also to go beyond just the specs we mentioned already, e.g. WebTransport which could be used to transport media.
17:45:49 [tidoust]
... From time to time, it's hard to undertand whether there are performance issues in the specs, implementations or in the code sample.
17:47:04 [tidoust]
Bernard: When I started of, I was thinking about capture with Media Capture and Streams Extensions, then encode/decode with WebCodecs (and also MSE v2 to some extent), Transport (WebTransport, WebRTC data channels in workers), and Frameworks (WHATWG streams, WASM)
17:47:35 [tidoust]
Bernard: The pipeline model is based on WHATWG Streams, through TransformStreams piped together.
17:47:54 [tidoust]
... When you're sending frames, you have a several options, e.g. reliable/unreliable, etc.
17:48:14 [tidoust]
... To stream these pipelines together, you have to use all of these APIs together. Does it all make sense?
17:48:31 [tidoust]
... I don't know that many developers who understand all of these APIs.
17:48:48 [tidoust]
... Some issues already created in the repo.
17:49:04 [tidoust]
-> https://github.com/w3c/media-pipeline-arch/ Media Pipeline architecture repo
17:49:12 [tidoust]
... A lot of the issues are focused on transport.
17:49:35 [tidoust]
... There are a few things that are worth discussing here.
17:50:16 [tidoust]
... E.g. rendering and timing. Media Capture Transform is an interesting API. Does VideoTrackGenerator have a jitter buffer? Does it not?
17:50:25 [tidoust]
... That is not particularly well defined in the spec.
17:50:48 [tidoust]
... We have two samples at the momen. One is a WebCodecs encode/decode in worker in the WebCodecs repo.
17:51:20 [tidoust]
... The second one adds WebTransport to that. This one took more work to optimize the transport. It adds serialization/deserialization.
17:51:40 [tidoust]
... We use frame/stream transport. That's not exactly RTP but it's close.
17:51:52 [tidoust]
... We're using SVC at baseline and partial reliability.
17:52:04 [tidoust]
... Overall, it's working surprisingly well.
17:52:26 [tidoust]
... I had to do a reorder buffer but still not a full jitter buffer.
17:52:43 [tidoust]
Bernard: Here are some of the things that you can play with.
17:53:24 [tidoust]
... You can play with this stuff. At the end, it generates a Frame RTT graph. That does not really give you glass to glass measurements.
17:53:39 [tidoust]
... Performances are pretty reasonable now after some work.
17:53:56 [tidoust]
... Slide shows an example with AV1 at full-HD.
17:54:46 [tidoust]
... What's interesting is that key frames can be transmitted within a single congestion window.
17:55:07 [tidoust]
... General question is what do we do with this?
17:56:06 [tidoust]
cpn: That's really great to get that practical feedback from building things.
17:57:02 [tidoust]
Bernard: Yes, we're seeing a lot of stuff. Similarly, there are a few things where I don't know enough of the internals to understand what needs to be done.
17:57:39 [tidoust]
... You have to be cautious of await calls with WHATWG Streams, since they are going to block. Debugging is also hard.
17:58:13 [tidoust]
youenn: Note you may use JS implementations or ReadableStream and WritableStream to ease debugging.
17:58:54 [tidoust]
Bernard: Good idea. You can get a dozen stages and you don't really know where things are in the different queues. It's not easy to figure out what happens. The code is fairly small though.
17:59:24 [tidoust]
cpn: Immediate next step?
18:00:02 [tidoust]
Bernard: Adding APIs in multiple groups adds question. It's worthwhile checking in on this periodically.
18:00:12 [tidoust]
... I don't want to act like I have a handle on this.
18:01:04 [tidoust]
cpn: OK, we'll talk more about how to improve that cross-group collaboration.
18:01:45 [tidoust]
cpn: Our next meeting will be on the new year. Happy Christmas and looking forward to seeing you next year!
18:02:45 [tidoust]
RRSAgent, draft minutes
18:02:46 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust
18:31:09 [tidoust]
i|Bernard: [going through slides]|Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Dec/att-0002/MEDIAWG-12-13-2022.pdf
18:32:11 [tidoust]
i|Bernard: [going through slides]|[Slide 2]
18:32:16 [tidoust]
RRSAgent, draft minutes
18:32:16 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust
18:32:49 [tidoust]
i|Bernard: [going through slides]|[Slide 3]
18:32:52 [tidoust]
i|Bernard: [going through slides]|[Slide 4]
18:33:01 [tidoust]
i|Bernard: [going through slides]|[Slide 5]
18:33:46 [tidoust]
i|Bernard: We created a Media Pipeline architecture repo|[slide 6]
18:34:09 [tidoust]
i|Bernard: When I started of|[Slide 7]
18:34:29 [tidoust]
i|Bernard: The pipeline model is based on WHATWG Streams|[Slide 8]
18:35:07 [tidoust]
i|... Some issues already created in the repo.|[Slide 9]
18:35:42 [tidoust]
i|... We have two samples at the momen|[Slide 10]
18:36:01 [tidoust]
i|Bernard: Here are some of the things|[Slide 11]
18:36:29 [tidoust]
i|... Slide shows an example with AV1 at full-HD.|[Slide 12]
18:36:34 [tidoust]
RRSAgent, draft minutes
18:36:34 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust
18:39:27 [tidoust]
RRSAgent, bye
18:39:27 [RRSAgent]
I see no action items