BEGIN:VCALENDAR VERSION:2.0 PRODID:-//Sabre//Sabre VObject 4.5.8//EN CALSCALE:GREGORIAN LAST-MODIFIED:20241016T205507Z BEGIN:VTIMEZONE TZID:America/Los_Angeles X-MICROSOFT-CDO-TZID:13 BEGIN:STANDARD DTSTART:20231105T090000 TZOFFSETFROM:-0700 TZOFFSETTO:-0800 TZNAME:PST END:STANDARD BEGIN:STANDARD DTSTART:20241103T090000 TZOFFSETFROM:-0700 TZOFFSETTO:-0800 TZNAME:PST END:STANDARD BEGIN:DAYLIGHT DTSTART:20240310T100000 TZOFFSETFROM:-0800 TZOFFSETTO:-0700 TZNAME:PDT END:DAYLIGHT END:VTIMEZONE BEGIN:VEVENT UID:49386363-7a65-4f4a-9580-bff867a1c6e9 DTSTAMP:20241016T205507Z SUMMARY:Evolved Video Encoding with WebCodecs DTSTART;TZID=America/Los_Angeles:20240925T111500 DTEND;TZID=America/Los_Angeles:20240925T121500 DESCRIPTION:https://www.w3.org/events/meetings/49386363-7a65-4f4a-9580-bff8 67a1c6e9/\n\n[WebCodecs](WebCodecs) provides a low-level API to do encodin g and decoding of video with control over settings on a per-frame basis. A s a relatively young API it currently lacks some more advanced features\, such as temporal/spatial scalability that are important for real-time use cases like video conferencing.\n\nThis session is intended to discuss a nu mber of potential next steps\, to find which features are highest priority \, and what benefits or problems we face with each of those. \n\nSome of t he topics for discussion:\n\n#### Explicit reference frame control\nBy all owing the user to specify which reference buffers to reference and which t o update on a per-frame basis\, it is possible to implement a number of im portant reference structures and coding features including temporal/spatia l/quality layers\, long-term references\, low-latency 2-pass rate control\ , etc.\n\nIn short\, any of the scalability modes listed in [Scalable Vide o Coding (SVC) Extension for WebRTC](https://www.w3.org/TR/webrtc-svc/)\, any many more can be implemented with a small set of tools. If done right\ , this could even be done in a manner that is codec and implementation agn ostic.\n\nThis way of modeling an encoder does also present some issues. T he user needs to be able to determine how many reference buffers are avail able\, how many can be referenced per frame and know which references are allowed or disallowed based on various circumstances. How do we expose suc h data in a way that is both user friendly\, compatible with the current A PI\, and avoids unnecessary finger printing surfaces?\n\nThere are also tr adeoffs when it comes to integrating with existing encoder implementations \, a small subset of which may not fit well into this model.\n\n#### Spati al/Quality Scalability\n\nSpatial scalability can be achieved by changing the `encode` call to take a _sequence_ of encoding options\, instead of a single option\, per input frame. Each option would then represent a differ ent layer and would include a desired encoded resolution. With reference f rame scaling\, a user may reference a buffer containing a different resolu tion.\n\nAgain\, this comes with some challenges. Different codec types mi ght have different bounds on the scaling factors\, and even certain implem entations have limitations in this regard - if it is supported at all. Som e codecs allow only reference frame scaling within the same temporal unit\ , while other support any reference at any time. How do we handle encoders with special optimized mode such as "multi-res" or "S-mode aware" encodin g?\n\n#### Rate Control\n\nWhen dealing with layered encoding\, rate contr ol becomes much more involved. The easiest way is to just support CQP\, pu tting all of the rate control control with the user. If CBR is desired\, t he encoder needs to understand the bitrate target and expected frame rate for each spatio-temporal layer\, this means it suddenly needs to be SVC aw are even if the user is doing all of the reference frame control.\n\n#### Auxiliary\n\nThere are many other knobs that could potentially be added. S peed/Quality control\, segmentation/ROI-mapping\, etc\nWhat's on the wish- list of the community?\n\n#### Other Sessions of Interest\n\nNote that the re will also be a first-step proposal discussed at the [joint Media/WebRTC WG Meeting](https://www.w3.org/events/meetings/b2c200b8-7362-4330-b74c-7b 2efc05d87f/) on the 26th.\n\nFurther\, there is a proposed [breakout sessi on](https://github.com/w3c/tpac2024-breakouts/issues/13) on [RtpTransport] (https://github.com/w3c/webrtc-rtptransport)\, an API that allows users to send custom-encoded frames over the RTP channel of a `PeerConnection` and is intended to go hand-in-hand with WebCodecs.\n\nAgenda\n\n**Chairs:**\n Erik Språng\, Eugene Zemtsov\n\n**Description:**\n[WebCodecs](WebCodecs) provides a low-level API to do encoding and decoding of video with control over settings on a per-frame basis. As a relatively young API it currentl y lacks some more advanced features\, such as temporal/spatial scalability that are important for real-time use cases like video conferencing.\n\nTh is session is intended to discuss a number of potential next steps\, to fi nd which features are highest priority\, and what benefits or problems we face with each of those. \n\nSome of the topics for discussion:\n\n#### Ex plicit reference frame control\nBy allowing the user to specify which refe rence buffers to reference and which to update on a per-frame basis\, it i s possible to implement a number of important reference structures and cod ing features including temporal/spatial/quality layers\, long-term referen ces\, low-latency 2-pass rate control\, etc.\n\nIn short\, any of the scal ability modes listed in [Scalable Video Coding (SVC) Extension for WebRTC] (https://www.w3.org/TR/webrtc-svc/)\, any many more can be implemented wit h a small set of tools. If done right\, this could even be done in a manne r that is codec and implementation agnostic.\n\nThis way of modeling an en coder does also present some issues. The user needs to be able to determin e how many reference buffers are available\, how many can be referenced pe r frame and know which references are allowed or disallowed based on vario us circumstances. How do we expose such data in a way that is both user fr iendly\, compatible with the current API\, and avoids unnecessary finger p rinting surfaces?\n\nThere are also tradeoffs when it comes to integrating with existing encoder implementations\, a small subset of which may not f it well into this model.\n\n#### Spatial/Quality Scalability\n\nSpatial sc alability can be achieved by changing the `encode` call to take a _sequenc e_ of encoding options\, instead of a single option\, per input frame. Eac h option would then represent a different layer and would include a desire d encoded resolution. With reference frame scaling\, a user may reference a buffer containing a different resolution.\n\nAgain\, this comes with som e challenges. Different codec types might have different bounds on the sca ling factors\, and even certain implementations have limitations in this r egard - if it is supported at all. Some codecs allow only reference frame scaling within the same temporal unit\, while other support any reference at any time. How do we handle encoders with special optimized mode such as "multi-res" or "S-mode aware" encoding?\n\n#### Rate Control\n\nWhen deal ing with layered encoding\, rate control becomes much more involved. The e asiest way is to just support CQP\, putting all of the rate control contro l with the user. If CBR is desired\, the encoder needs to understand the b itrate target and expected frame rate for each spatio-temporal layer\, thi s means it suddenly needs to be SVC aware even if the user is doing all of the reference frame control.\n\n#### Auxiliary\n\nThere are many other kn obs that could potentially be added. Speed/Quality control\, segmentation/ ROI-mapping\, etc\nWhat's on the wish-list of the community?\n\n#### Other Sessions of Interest\n\nNote that there will also be a first-step proposa l discussed at the [joint Media/WebRTC WG Meeting](https://www.w3.org/even ts/meetings/b2c200b8-7362-4330-b74c-7b2efc05d87f/) on the 26th.\n\nFurther \, there is a proposed [breakout session](https://github.com/w3c/tpac2024- breakouts/issues/13) on [RtpTransport](https://github.com/w3c/webrtc-rtptr ansport)\, an API that allows users to send custom-encoded frames over the RTP channel of a `PeerConnection` and is intended to go hand-in-hand with WebCodecs.\n\n**Goal(s):**\nFind the highest priority features in the com munity\, and what aspects needs more consideration\n\n\n**Agenda:**\nThe a genda is to discuss the proposal to add reference frame control to WebCode cs\, and gather feedback and comments on the path forward. The session con sist of a few parts:\n\n* General goals\n* Our initial proposal\, a minimu m viable useful implementation of reference control\n* How to query what t he encoders are capable of\n* Spatial Scalability (SVC\, Simulcast)\n* Rat e Control \n* Miscellaneous\n\nSee also [WebCodecs spec](https://www.w3.or g/TR/webcodecs/) and [github issue](https://github.com/w3c/webcodecs/issue s/285) for the reference control.\n\n**Materials:**\n- [slides](https://ww w.w3.org/2024/Talks/TPAC/breakouts/evolved-webcodecs.pdf)\n- [minutes](htt ps://www.w3.org/2024/09/25-evolved-webcodecs-minutes.html)\n- [Session pro posal on GitHub](https://github.com/w3c/tpac2024-breakouts/issues/71)\n\n* *Track(s):**\n- Real-time Web STATUS:CONFIRMED CREATED:20240916T215856Z LAST-MODIFIED:20241016T205507Z SEQUENCE:1 ORGANIZER;CN=W3C Calendar;PARTSTAT=ACCEPTED;ROLE=NON-PARTICIPANT:mailto:nor eply@w3.org LOCATION:-1 Lower Level - Catalina 2 CATEGORIES:TPAC 2024,Breakout Sessions END:VEVENT END:VCALENDAR