IRC log of mediawg on 2022-09-15
Timestamps are in UTC.
- 22:26:26 [RRSAgent]
- RRSAgent has joined #mediawg
- 22:26:26 [RRSAgent]
- logging to https://www.w3.org/2022/09/15-mediawg-irc
- 22:26:33 [Zakim]
- Zakim has joined #mediawg
- 22:26:54 [dom]
- Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Sep/att-0008/TPAC_2022_WebRTC_WG___Media_WG___MEIG_Joint_Meeting.pdf
- 22:27:00 [alwu]
- does anyone know what the passcode is for the webRTC joint meeting? thanks.
- 22:27:00 [bneedham]
- bneedham has joined #mediawg
- 22:27:15 [sandersd]
- sandersd has joined #mediawg
- 22:27:17 [cpn]
- present+ Chris_Needham, Paul_Adenot, Dominique_Hazael_Massieux, Francois_Daust, Hisayaki_Ohmata, Bernard_Aboba
- 22:27:17 [bneedham]
- present+ Bradley_Needham
- 22:27:59 [cpn]
- present+ Jake_Holland
- 22:28:00 [fluffy]
- fluffy has joined #mediawg
- 22:28:08 [fluffy]
- present+ Cullen Jennings
- 22:28:18 [cpn]
- present+ Kaz_Ashimura
- 22:28:20 [TuukkaToivonen]
- TuukkaToivonen has joined #mediawg
- 22:28:43 [cpn]
- present+ Mike_English, Alastor_Wu, Dan_Sandsers
- 22:28:49 [fluffy]
- present+ Cullen_Jennings
- 22:28:54 [cpn]
- s/Dan_Sandsers/Dan_Sanders/
- 22:29:14 [cpn]
- Present+ Tuukka_Toivonen, Pater_Thatcher
- 22:29:30 [nigel]
- nigel has joined #mediawg
- 22:29:50 [kajihata]
- kajihata has joined #mediawg
- 22:30:07 [cpn]
- Present+ Youenn_Fablet
- 22:30:16 [cpn]
- Present+ Tatsuya_Igarashi
- 22:30:36 [herre]
- herre has joined #mediawg
- 22:31:21 [cpn]
- Present+ Brian_Baldino, Daisuke_Kodakima
- 22:33:06 [dom]
- Present+ Eric_Carlson, David_Singer, Elad_Alon
- 22:33:08 [dom]
- scribe+
- 22:33:14 [tguilbert]
- tguilbert has joined #mediawg
- 22:33:19 [kajihata]
- present+ Hiroshi_Kajihata
- 22:33:21 [eugene]
- eugene has joined #mediawg
- 22:33:25 [ken]
- ken has joined #mediawg
- 22:33:25 [ohmata]
- ohmata has joined #mediawg
- 22:33:27 [tguilbert]
- present+ Thomas_Guilbert
- 22:33:42 [gendler]
- gendler has joined #mediawg
- 22:33:46 [jib]
- jib has joined #mediawg
- 22:33:47 [gendler]
- present+
- 22:33:58 [Orphis]
- Orphis has joined #mediawg
- 22:34:02 [Orphis]
- present+
- 22:34:08 [youenn_]
- youenn_ has joined #mediawg
- 22:34:10 [eugene]
- present+ Eugene Zemtsov
- 22:34:12 [herre]
- present+ Tony_Herre
- 22:34:16 [ken]
- present+
- 22:34:17 [bialpio]
- bialpio has joined #mediawg
- 22:34:17 [ohmata]
- present+ Hisayuki Ohmata
- 22:34:19 [klausw]
- klausw has joined #mediawg
- 22:34:19 [youenn_]
- present+ Youenn Fablet
- 22:34:22 [jholland]
- jholland has joined #mediawg
- 22:34:30 [TuukkaToivonen]
- present+ Tuukka Toivonen
- 22:34:31 [klausw]
- present+ klausw
- 22:34:32 [jib]
- present+ Jan-Ivar Bruaroey
- 22:34:34 [jholland]
- present+ Jake_Holland
- 22:34:38 [klausw]
- present+ Klaus Weidner
- 22:34:38 [englishm]
- present+ Mike_English
- 22:34:42 [bialpio]
- present+
- 22:34:45 [dom]
- Topic: Introduction
- 22:34:53 [dom]
- [slide 5]
- 22:34:53 [eric-carlson_]
- eric-carlson_ has joined #mediawg
- 22:34:56 [eehakkin]
- eehakkin has joined #mediawg
- 22:34:58 [dsinger]
- dsinger has joined #mediawg
- 22:35:01 [eric-carlson_]
- +present
- 22:35:06 [dom]
- Bernard: some context for this joint meeting
- 22:35:08 [eric-carlson_]
- present+
- 22:35:09 [dom]
- [slide 6]
- 22:35:14 [eeeps]
- eeeps has joined #mediawg
- 22:35:21 [dsinger]
- present+ dsinger
- 22:35:22 [riju_]
- present+ rijubrata bahumik
- 22:35:24 [dom]
- Bernard: the pandemic has brought a new wave a technology to the mass market
- 22:35:30 [eehakkin]
- present+ Eero_Häkkinen
- 22:35:34 [eeeps]
- present+ Eric Portis
- 22:35:53 [dom]
- ... podcasting (75% of China once/week), video conferencing, video streaming (vod, live streaming)
- 22:36:07 [dom]
- ... ...
- 22:36:24 [hfujisawa]
- hfujisawa has joined #mediawg
- 22:36:41 [dom]
- ... a bunch of these are being compiled in the webrtc-nv use cases and webtransport
- 22:37:00 [dom]
- ... they blur the lines between streaming and realtime
- 22:37:09 [dom]
- [slide 7]
- 22:38:01 [dom]
- [slide 8]
- 22:38:35 [dom]
- RRSAgent, draft minutes
- 22:38:35 [RRSAgent]
- I have made the request to generate https://www.w3.org/2022/09/15-mediawg-minutes.html dom
- 22:38:39 [dom]
- RRSAgent, make log public
- 22:38:42 [dom]
- [slide 9]
- 22:40:00 [igarashi_]
- igarashi_ has joined #mediawg
- 22:40:14 [igarashi_]
- present+ Tatsuya_Igarashi
- 22:40:27 [dom]
- [slide 10]
- 22:41:07 [kaz]
- kaz has joined #mediawg
- 22:41:30 [kaz]
- zakim, who is on the call?
- 22:41:30 [Zakim]
- Present: Chris_Needham, Paul_Adenot, Dominique_Hazael_Massieux, Francois_Daust, Hisayaki_Ohmata, Bernard_Aboba, Bradley_Needham, Jake_Holland, Cullen, Jennings, Kaz_Ashimura,
- 22:41:33 [Zakim]
- ... Mike_English, Alastor_Wu, Dan_Sandsers, Cullen_Jennings, Tuukka_Toivonen, Pater_Thatcher, Youenn_Fablet, Tatsuya_Igarashi, Brian_Baldino, Daisuke_Kodakima, Eric_Carlson,
- 22:41:33 [Zakim]
- ... David_Singer, Elad_Alon, Hiroshi_Kajihata, Thomas_Guilbert, gendler, Orphis, Eugene, Zemtsov, Tony_Herre, ken, Hisayuki, Ohmata, Fablet, Toivonen, klausw, Jan-Ivar, Bruaroey,
- 22:41:33 [Zakim]
- ... Weidner, bialpio, present, eric-carlson_, dsinger, rijubrata, bahumik, Eero_Häkkinen, Eric, Portis
- 22:41:52 [eugene]
- present+ Eugene_Zemtsov
- 22:42:03 [dom]
- [slide 11]
- 22:42:03 [jib]
- present+ Jan-Ivar_Bruaroey
- 22:42:54 [dom]
- Bernard: lots of SdOs involved, and even lots of different groups involved on these topics in W3C itself
- 22:43:34 [dom]
- ... as a developer, you have to combine all these APIs together to build an app
- 22:43:52 [dom]
- ..; the webcodecs sample code illustrate that as they need to use many different of APIs
- 22:43:54 [dom]
- [slide 12]
- 22:44:01 [kaz]
- s/..;/.../
- 22:44:13 [kaz]
- rrsagent, make log public
- 22:44:18 [kaz]
- rrsagent, draft minutes
- 22:44:18 [RRSAgent]
- I have made the request to generate https://www.w3.org/2022/09/15-mediawg-minutes.html kaz
- 22:45:08 [dom]
- [slide 13]
- 22:45:44 [yonet]
- yonet has joined #mediawg
- 22:45:56 [dom]
- Bernard: the webrtc worked with WHATWG to make the WHATWG streams work better for media use cases
- 22:46:23 [dom]
- ... converting video frames from WebCodecs into other formats is another need that has emerged
- 22:47:02 [dom]
- ... the support for worker thread for the pipeline is still inconsistent
- 22:47:35 [kaz]
- meeting: Web Real-Time Communications Working Group and Media Working Group and Media and Entertainment Interest Group - Joint meeting
- 22:47:39 [dom]
- ... separating worker threads for send and receive pipelines is not easy to put in place in practice
- 22:47:46 [dom]
- [slide 14]
- 22:48:32 [dom]
- Bernard: goal is to identify owners of the identified problems and next steps
- 22:48:49 [dom]
- q?
- 22:48:58 [cpn]
- q?
- 22:49:49 [dom]
- [slide 15]
- 22:50:01 [kaz]
- q+
- 22:50:15 [dom]
- q?
- 22:50:20 [dom]
- ack kaz
- 22:50:56 [dom]
- Kaz: Web of Things are thinking of using WebRTC for use in video surveillance cameras
- 22:51:10 [dom]
- ... I wonder about security considerations for these types of applications
- 22:51:38 [dom]
- Youenn: does this need DRM?
- 22:51:55 [dom]
- Kaz: not necessarily, but the cameras would still want to protect their data
- 22:52:01 [fluffy]
- q+
- 22:52:30 [dom]
- youenn: WebRTC provides hop-by-hop encryption
- 22:52:32 [Kodajima]
- Kodajima has joined #mediawg
- 22:52:44 [dom]
- ... complemented by end-to-end encryption done in the encoded content
- 22:53:01 [dom]
- ... that may be something to consider here as well
- 22:53:14 [dom]
- bernard: please file an issue for use cases
- 22:53:29 [dom]
- -> https://github.com/w3c/webrtc-nv-use-cases/issues WebRTC NV Use Cases Repo
- 22:53:31 [dom]
- q?
- 22:53:36 [dom]
- ack fluffy
- 22:53:38 [kaz]
- ack k
- 22:54:27 [dom]
- fluffy: it would be important to have support for end-to-end encryption - beyond JS-based encryption, it would be good to re-use our existing infrastructure (certificates management, webauthn) to enable E2EE
- 22:54:33 [cpn]
- q?
- 22:54:34 [dom]
- ... probably anchored into MLS
- 22:55:06 [dom]
- Bernard: would that be separate from WebRTC then?
- 22:55:10 [dom]
- Cullen: yes
- 22:55:20 [dom]
- ... we have a much better understanding of what's needed
- 22:55:47 [dom]
- ... we need to make sure our architecture exposes API in the pipeline where the browser doesn't necessarily have access to the underlying data
- 22:56:06 [dom]
- Bernard: there is a related issue in the webcodecs repo, filed in the DRM context
- 22:56:07 [kron]
- kron has joined #mediawg
- 22:56:08 [dom]
- q?
- 22:56:54 [cpn]
- scribe+ cpn
- 22:57:27 [cpn]
- Dom: Code samples is a good way to identify where the gaps are
- 22:57:50 [cpn]
- Action: Set up a place where we can discuss cross-group or cross-API architecture issues
- 22:58:31 [dom]
- Topic: Issue 90/Issue 121: Pluggable codecs
- 22:58:45 [dom]
- -> https://github.com/w3c/webrtc-encoded-transform/issues/90 Adopt feedback streams in RTCRtpScriptTransformer #90
- 22:58:54 [youenn]
- youenn has joined #mediawg
- 22:58:55 [dom]
- -> https://github.com/w3c/webrtc-encoded-transform/issues/121 Need APIs for using WebCodec with PeerConnection #121
- 22:59:08 [dom]
- [slide 16]
- 22:59:41 [dom]
- Harald: once you take a media and you transform it, it no longer has the same format, which is problematic for SDP-based negotiations
- 22:59:53 [dom]
- ... this creates issues where the pipeline is configured for the wrong media
- 23:00:13 [gkatsev]
- present+ Gary_Katsevman
- 23:00:34 [dom]
- ... one property that we need from the communications machine for webrtc is the ability to speak truth that the content being sent doesn't conform to the spec
- 23:00:58 [dom]
- ... there is also an issue around packetization/depacketization
- 23:01:37 [peter]
- peter has joined #mediawg
- 23:01:48 [dom]
- ... we should figure an API so that once you install a transformation in the pipeline how to handle the data in the communications machinery
- 23:02:05 [dom]
- ... that should operate in terms of codec description
- 23:02:28 [kaz]
- rrsagent, draft minutes
- 23:02:28 [RRSAgent]
- I have made the request to generate https://www.w3.org/2022/09/15-mediawg-minutes.html kaz
- 23:02:28 [dom]
- [slide 17]
- 23:02:53 [dom]
- Bernard: this is key to enabling to combine webcodecs & webrtc
- 23:03:01 [dom]
- ... also needed for E2EE
- 23:03:07 [dom]
- s/[slide 17]//
- 23:03:25 [dom]
- Harald: this problem has indeed arisen already in the context of E2EE
- 23:03:38 [dom]
- ... also, should this be done in terms of packets or in terms of frames
- 23:03:38 [kaz]
- s/Hisayaki/Hisayuki/
- 23:03:55 [kaz]
- chair: cpn
- 23:04:06 [cpn]
- q?
- 23:04:16 [dom]
- Bernard: in terms of next steps, the idea is to sollicitate proposals?
- 23:04:57 [dom]
- Youenn: re WebCodecs, the traditional view is that if you're using webcodecs you will use web transport or datachannel to transport the data
- 23:05:10 [dom]
- ... what's the use case for using webcodecs inside something very integrated like peerconnection
- 23:05:36 [dom]
- ... it's good to provide proposals, but I would focus on use cases and requirements
- 23:05:52 [dom]
- Bernard: the key reason is that niether datachannels nor webtransport will generate low latency
- 23:06:07 [dom]
- ... getting to 100ms can only be achieved with WebRTC
- 23:06:16 [kaz]
- present+ Harald_Alvestrand, Alex_Turner, Brandon_Jones, Brian_Baldino, Chao_He
- 23:06:17 [dom]
- Youenn: I agree with that
- 23:06:32 [dom]
- ... one of the main benefits of WebCodecs is for the use of hardware encoder/decoder
- 23:06:45 [dom]
- ... when you expose that in WebCodecs, we should expose it in PC
- 23:06:57 [kaz]
- present+ Dale_Curtis, Dan_Sanders, Eero_Haekkinen
- 23:07:02 [dom]
- ... the main benefits would be to finetune encoders/decoders behavior
- 23:07:10 [eeeps]
- eeeps has joined #mediawg
- 23:07:17 [dom]
- ... not clear what the benefits would be vs native encoder/decoder integration in PC
- 23:07:27 [kaz]
- present+ Eric_Portis
- 23:07:30 [dom]
- Harald: in a browser, the code supporting the codecs for the 2 are the same
- 23:07:38 [dom]
- ... but the capabilities in the 2 APIs aren't
- 23:07:55 [kaz]
- present+ Hiroshi_Fujisawa, Hiroshi_Kajihata
- 23:08:00 [dom]
- ... the most intensive control surface would be in WebCodecs rather than in WebRTC
- 23:08:08 [peter]
- q+
- 23:08:16 [jib]
- q+
- 23:08:16 [dom]
- ... you control the interface via WebCodecs with an input/output in RTP
- 23:08:21 [eugene]
- q+
- 23:08:23 [dom]
- ... this requires supporting the integration
- 23:08:46 [dom]
- Bernard: WebCodecs hasn't been for long, but they're already exposing a lot more capabilities than WebRTC
- 23:08:47 [fluffy]
- q+
- 23:09:13 [kaz]
- present+ Johannes_Kron, Klaus_Weidner, Daisuke_Kodajima
- 23:09:16 [dom]
- Peter: +1 to Youenn that the same codecs are exposed in both WebRTC & WebCodecs
- 23:09:36 [tovep]
- tovep has joined #mediawg
- 23:09:54 [dom]
- ... but as we add new low level controls - would we want to add them in both places or in a single place?
- 23:09:57 [bialpio]
- present+ Piotr_Bialecki
- 23:09:57 [dom]
- ack peter
- 23:10:01 [dom]
- ack jib
- 23:10:02 [kaz]
- present+ Mike_English, Riju, Sung_young_Son, Thomas_Guilbert, Tuukka_Toivonen
- 23:10:22 [kaz]
- rrsagent, draft minutes
- 23:10:22 [RRSAgent]
- I have made the request to generate https://www.w3.org/2022/09/15-mediawg-minutes.html kaz
- 23:10:27 [dom]
- jib: we have multiple APIs that are starting to overlap in usages
- 23:10:36 [tovep]
- present+ Tove_Petersson
- 23:10:43 [dom]
- ... what might help would be to ask what we prefer this or that API
- 23:10:54 [dom]
- ... I'm hoping we can avoid "this API is better than that API"
- 23:11:05 [cpn]
- q?
- 23:11:16 [dom]
- ... what makes WebCodecs better? if it's more codec support, we can add more codecs to WebRTC
- 23:11:36 [dom]
- ... or for WHATWG stream support in WebTransport, that could be exposed in data channels
- 23:11:38 [kaz]
- present+ Peter_Thatcher
- 23:12:11 [dom]
- ... it may not be necessary to break open WebRTC to benefits from the RTP transport benefits
- 23:12:17 [dom]
- ack eug
- 23:12:17 [peter]
- q+
- 23:12:27 [kaz]
- present+ Tatsuya_Igarashi
- 23:12:36 [dom]
- eugene: it's been mentioned that RTCDataChannel and WebTransport are not truly optimized for RTC
- 23:12:42 [dom]
- ... can something be done about it?
- 23:12:53 [dom]
- ... A/V is nto the only application for real-time transport
- 23:13:22 [eugene]
- q-
- 23:13:27 [dom]
- Bernard: this is under discussion in a number of IETF WGs
- 23:13:30 [kaz]
- present+ Youenn_Fablet
- 23:13:36 [dom]
- ... some things can be done but it's still a research problem
- 23:13:55 [dom]
- ... in the WebTransport WG, we disucssed interactions with the congestion control algorithms
- 23:14:06 [dom]
- s/disucss/discuss/
- 23:14:18 [dom]
- ... not sure how much momentum there would be to change congestion control for datachannels
- 23:14:25 [cpn]
- q?
- 23:14:26 [dom]
- Harald: datachannels are built on SCTP
- 23:14:35 [dom]
- ... it's now easier to play with the congestion control for that
- 23:14:53 [dom]
- ... understanding congestion control is hard, improving it even harder
- 23:15:00 [dom]
- ... unlikely to be solved in the short term, in IETF
- 23:15:03 [dom]
- ack fluffy
- 23:15:48 [dom]
- fluffy: in terms of congestion control, ReNo CC can be moved to BBRv2
- 23:16:28 [dom]
- ... in terms of driving interest on WebCodecs, the next generation of codecs driven by AI are providing substantial improvements to this space
- 23:16:57 [dom]
- ... these are proprietary codecs that some would like to deploy at scale in browsers
- 23:17:14 [dom]
- Bernard: a lot of these technologies aren't that easy to do in WebCodecs
- 23:17:18 [dom]
- ... e.G. AI integration
- 23:17:29 [cpn]
- q?
- 23:17:33 [dom]
- ... pre/post-processing can be done with webcodecs
- 23:17:38 [cpn]
- ack peter
- 23:17:58 [dom]
- peter: re congestion control, there are something we could do at the W3C level by exposing more information about what's happening in the transport so that app can adjust what they send
- 23:18:05 [dom]
- ... I've volunteered to help with that
- 23:18:44 [dom]
- ... on the codecs side, re which API to use - even if we allow "bring your own codec" and an RTP Transport, there is still a question of the jitter buffer which is uniquely exposed in WebRTC (not in WebTransport)
- 23:19:29 [dom]
- CPN: additional control surface in WebCodecs?
- 23:19:48 [dom]
- Paul: we can add knobs in WebCodecs when they apply to everything
- 23:20:04 [dom]
- ... otherwise, we add knobs in struct in the registry
- 23:20:54 [dom]
- ... e.g. tuning for drawing vs real-life for video
- 23:21:00 [dom]
- ... or packet size for audio
- 23:21:19 [dom]
- q?
- 23:21:36 [dom]
- CPN: we welcome issues for additional control surface in the webcodecs repo
- 23:21:59 [dom]
- youenn: one of the things we discussed was asking higher quality in the area of the video whtere there is e.g. a face
- 23:22:34 [dom]
- Topic: Issue 131: Packetization API
- 23:22:38 [dom]
- [slide 17]
- 23:22:53 [dom]
- -> https://github.com/w3c/webrtc-encoded-transform/issues/131 Should we expose packetization level API to RTCRtpScriptTransform? #131
- 23:23:11 [dom]
- Youenn: rtcpeerconnection used to be a blackbox with an input and output
- 23:23:23 [dom]
- ... webrtc-encoded-transform opens it up a little bit
- 23:23:27 [dom]
- ... it works for some things
- 23:23:33 [dom]
- ... but it doesn't cover all use cases
- 23:23:39 [cpn]
- q?
- 23:23:43 [dom]
- ... we tried to identify use cases where there isn't good support
- 23:24:06 [dom]
- ... e.G. when adding redundancy for audio to make it more reliable over transport
- 23:24:32 [dom]
- ... but adding redundancy expose the risk of having packets being discarded
- 23:26:00 [dom]
- ... Giving more control on the generation packets for video frame would also help in the context of encryption (SPacket)
- 23:26:54 [dom]
- ... RTP packets come with RTP headers that aren't exposed directly - it could be nice to give r/w access to apps e.g. to improve the voice-activity header
- 23:27:35 [cpn]
- q?
- 23:27:36 [dom]
- ... I plan to gather use cases and requirements since there is some interest - hope others will join me
- 23:27:49 [dom]
- cpn: where would that be done?
- 23:27:51 [yonet]
- yonet has joined #mediawg
- 23:27:59 [dom]
- youenn: issue #131 is probably the right place
- 23:28:20 [cpn]
- q?
- 23:29:12 [Bernard]
- Bernard has joined #mediawg
- 23:29:26 [dom]
- fluffy: it's hard to know how to find where the discussions happen; I'm interested
- 23:29:52 [dom]
- bernard: this relates to the challenge of keeping track of work happening across all these groups
- 23:30:40 [dom]
- fluffy: the architectural discussion is whether packetization is part of the pipeline we need to consider in our cross-group coordination
- 23:31:04 [cpn]
- q?
- 23:31:20 [englishm]
- fluffy++
- 23:31:41 [dom]
- cpn: we need to seed our x-greoup repo with a description of where we are and where we want to go
- 23:32:09 [jib]
- q+
- 23:32:12 [dom]
- bernard: another possibility would be a workshop to help with these x-groups discussions
- 23:32:16 [jib]
- q-
- 23:33:17 [dom]
- jib: I played with the API which illustrated that packetization is both codec and transport specific
- 23:33:23 [eric-carlson]
- eric-carlson has joined #mediawg
- 23:33:43 [dom]
- ... is our quesiton "where should it belong architecturally?"
- 23:33:47 [dom]
- bernard: great question
- 23:34:13 [dom]
- jake: has anyone talked with IETF on the need to surface the MTU to the encoder?
- 23:34:28 [dom]
- youenn: the packetizer has the info on MTU
- 23:34:38 [dom]
- ... the question is whether the packetizer would give the info back to the app
- 23:35:19 [cpn]
- Action: Create initial architecture description
- 23:35:26 [dom]
- Topic: Issue 99 & Issue 141: WebCodecs & WebRTC
- 23:35:31 [dom]
- [slide 18]
- 23:35:48 [dom]
- -> https://github.com/w3c/webrtc-encoded-transform/issues/99 Encoded frame IDLs share definitions with WebCodecs #99
- 23:35:52 [Larry_Zhao]
- Larry_Zhao has joined #mediawg
- 23:35:58 [dom]
- -> https://github.com/w3c/webrtc-encoded-transform/issues/141 Relationship to WebCodecs #141
- 23:36:25 [dom]
- Youenn: webrtc-encoded-transform and webcodecs share some very similar structures, but with differences incl on mutability of some of the data
- 23:36:37 [dom]
- ... it's a little bit inconvenient we have these different models
- 23:36:59 [dom]
- ... but that ship has probably sailed
- 23:37:16 [cpn]
- q?
- 23:37:20 [dom]
- ... for metadata, we're thinking of referring back to webcodecs
- 23:37:38 [dom]
- ... this got support from the webcodecs involved in the WebRTC WG meeting on Monday
- 23:38:16 [dom]
- Topic: Issue 70: WebCodecs & MediaStream transform
- 23:38:29 [dom]
- -> https://github.com/w3c/mediacapture-extensions/issues/70 Issue 70: WebCodecs & MediaStream transform
- 23:38:35 [dom]
- [slide 19]
- 23:39:03 [dom]
- Youenn: media capture transform allows to grab frames, modify them, and repackage them as a mediastreamtrack
- 23:39:06 [peter]
- q+
- 23:39:08 [dom]
- ... this is based on the VideoFrame object
- 23:39:23 [dom]
- ... cameras nowadays are capable to expose information about e.g. face positions
- 23:39:32 [dom]
- ... it's cheap to compute and could be usefully exposed to Web apps
- 23:39:49 [dom]
- ... since this is tightly synchronized data, it would be good to package it with the video frames
- 23:39:56 [nigel]
- nigel has joined #mediawg
- 23:40:13 [dom]
- ... in our dsicussion on Monday, we agreed to have a metadata dictionary in VideoFrame
- 23:40:30 [dom]
- ... WebRTC would expose its set of metadata for e.G. face detection
- 23:40:48 [dom]
- ... we discussed the possibility to have webapp specific metadata e.g. through JS objects
- 23:40:56 [dom]
- ... this could be useful in the context of AR/VR
- 23:41:24 [dom]
- ... the initial PR won't go that far, but it should open the way to that direction
- 23:41:36 [dom]
- ... the first use case will focus on browser-defined metadata
- 23:41:58 [Bernard]
- q+
- 23:42:17 [peter]
- q-
- 23:42:21 [cpn]
- q+
- 23:42:21 [dom]
- ... these would be exposed as the top-level, and there would be a .user sub-object that would need to be serializable
- 23:42:28 [nigel_]
- nigel_ has joined #mediawg
- 23:42:33 [dom]
- q+ ada
- 23:42:52 [dom]
- Bernard: this has application beyond the use cases on the slide
- 23:42:54 [bialpio]
- q+
- 23:43:10 [dom]
- ... e.g. face detection could help encoders around faces vs blurred background
- 23:43:24 [dom]
- ack Bernard
- 23:43:44 [dom]
- ... we would need to consider whether the encoder should be expected to look at that metadata
- 23:43:47 [dom]
- youenn: +1
- 23:43:55 [dom]
- ack cpn
- 23:44:06 [dom]
- cpn: how common is face detection in the camera level?
- 23:44:10 [dom]
- eric: it's very common
- 23:44:39 [dom]
- youenn: we could also expose a face detection blackbox that would transform a camera feed and annotate it with face detection metaata
- 23:44:41 [Bernard]
- q-
- 23:44:48 [dom]
- ack ada
- 23:44:58 [dom]
- Ada: (Immersive Web WG co-chair)
- 23:45:15 [dom]
- ... some of of our WG participants are very intereesting in bringing more AR feature to WebRTC
- 23:45:22 [TuukkaToivonen]
- TuukkaToivonen has joined #mediawg
- 23:45:27 [dom]
- ... they do AR by running WASM on the camera feed from WebRTC
- 23:45:55 [dom]
- ... if you were to fire up the various AR systems on the devices, you could surface more metadata e.G. the current position of the device or solid objects in the environments
- 23:46:25 [cpn]
- q?
- 23:46:26 [kaz]
- q+
- 23:46:30 [dom]
- Youenn: very important we get the AR/VR input on this work
- 23:46:53 [dom]
- ... we're also thinking of exposing requestVideoFrame to expose some of these metadata as well
- 23:47:05 [ada]
- ada has joined #mediawg
- 23:47:18 [dom]
- -> https://github.com/w3c/webcodecs/issues/189 WebCodec as pass through to application metadata #189
- 23:47:29 [dom]
- bialpio: also from the IWWG
- 23:47:53 [dom]
- ... we also have a use case to integrate WebXR animation loop with video frames used in XR session
- 23:48:10 [dom]
- ... how would we correlate a video frame from gUM with poses?
- 23:48:59 [dom]
- youenn: in WebRTC, accessing frames from the camera is via a track processor in a worker
- 23:49:13 [dom]
- ... correlating that with timestamps might work
- 23:49:15 [Bernard]
- q+
- 23:49:23 [cpn]
- ack bia
- 23:49:25 [dom]
- ... would be interesting to see if this is workable with readablestreams
- 23:49:29 [cpn]
- q+
- 23:50:05 [dom]
- bialpio: in this particular use case, WebXR would be introducing video frames, they could be added as a metatadata into video frames
- 23:50:55 [dom]
- ... some devices are doing pose prediction - this might make it tricky to sync with past video frames
- 23:51:08 [dom]
- ... the question is what we would be the best place to discuss this?
- 23:51:24 [dom]
- youenn: VideoFrame sounds like the place where this is converging
- 23:51:36 [fluffy]
- q+
- 23:51:39 [dom]
- ... so webcodecs repo in the Media WG would seem good
- 23:51:48 [cpn]
- q- cpn
- 23:51:58 [dom]
- cpn: also worth discussing in the architectural repo
- 23:52:18 [dsinger]
- dsinger has joined #mediawg
- 23:52:28 [bialpio]
- q-
- 23:52:34 [cpn]
- Action: Include image capture for AR applications and stream correlation in architecture description
- 23:52:38 [dom]
- ack kaz
- 23:52:57 [dom]
- kaz: spatial data wg also looking at sync between location and time
- 23:53:06 [cpn]
- q?
- 23:53:10 [dom]
- ... useful to consider sync with video stream as well
- 23:53:14 [dom]
- ack Bernard
- 23:53:21 [dom]
- Bernard: the synchronization issue has come up several time
- 23:53:32 [dom]
- ... the videotrackprocess gives you videoframe with timestamps
- 23:53:40 [dom]
- ... how to render that accurately in sync with audio
- 23:53:50 [dom]
- ... it's not always clear that you get the sync that you want
- 23:54:04 [dom]
- ... is that the appropriate API when dealing with all these operations
- 23:54:16 [dom]
- ... what's the right way to render sync'd audio/video?
- 23:54:28 [dom]
- ... this is probably worth some sample code to figure it out
- 23:54:28 [cpn]
- q?
- 23:54:32 [dom]
- ack fluffy
- 23:55:05 [dom]
- fluffy: one of the architectural issues is that these various systems are working at different timing with different control loops
- 23:55:24 [dom]
- ... how to synchronize them and render them correctly is an architectural issue - it needs the big picture of how it fits together
- 23:55:46 [dom]
- Bernard: half of the questions I get is in that space
- 23:56:18 [cpn]
- Action: capture synchronization issues in the architecture desctiption
- 23:56:45 [nigel]
- nigel has joined #mediawg
- 23:56:59 [dom]
- paul: this metadata seems to become a central piece
- 23:57:13 [riju_]
- Thanks Youenn for bringing this up.. Others if there's something specific to FaceDetection you want to contribute/ask here's an explainer https://github.com/riju/faceDetection/blob/main/explainer.md
- 23:57:15 [dom]
- ... it seems worth focus on this to save trouble down the line
- 23:57:48 [cpn]
- q?
- 23:58:05 [dom]
- [slide 20]
- 23:58:36 [dom]
- Topic: WebCodecs
- 23:58:41 [jholland]
- apologies, I have to leave early. Thanks for an informative session.
- 23:59:49 [kaz]
- present+ Masaki_Matsushita
- 23:59:55 [dom]
- Subtopic: w3c/webcodecs #198 Emit metadata (SPS,VUI,SEI,...) during decoding