15:00:26 RRSAgent has joined #me 15:00:26 logging to https://www.w3.org/2021/12/07-me-irc 15:00:37 meeting: Media and Entertainment IG 15:01:20 JohnRiv has joined #me 15:01:32 present+ Kaz_Ashimura, Cris_Needham, Xabier_Rodriguez_Calvar, Yuhao, Takio_Yamaok, Joey_Parrish 15:01:51 present+ John_Riviello 15:01:52 igarashi_ has joined #me 15:02:06 +present igarashi 15:02:20 present+ Tatsuya_Igarashi, Will_Law 15:02:42 present+ Jake_Holland, Francois_Daoust 15:02:50 present+ MFX_Worldwide 15:02:56 zakim, who is on the call? 15:02:56 Present: Kaz_Ashimura, Cris_Needham, Xabier_Rodriguez_Calvar, Yuhao, Takio_Yamaok, Joey_Parrish, John_Riviello, Tatsuya_Igarashi, Will_Law, Jake_Holland, Francois_Daoust, 15:02:59 ... MFX_Worldwide 15:03:11 present- Cris_Needham 15:03:17 present+ Chris_Needham 15:03:26 calvaris has joined #me 15:04:13 present- Yuhao 15:04:16 present+ Yuhao_Fu 15:04:17 Louay has joined #me 15:04:26 join+ Louay_Bassbouss 15:04:35 present+ Louay_Bassbouss 15:04:51 present+ Fuqiao_Xue 15:05:33 present+ Todd_Smith 15:06:06 hi 15:06:12 scribenick: jholland 15:06:56 Topic: TPAC 2021 outcomes 15:07:12 chris: tpac 2021 had 2 meetings: first on app performance on tv devices 15:07:49 ...should we be looking at dev technologies, canvas, etc. there was a specific idea about control of cache usage because of limited space on devices 15:07:57 ...plus some more 15:08:13 ...2nd meeting about timing accuracy, latency, etc on iot devices 15:08:41 ...please state your interest and volunteer to lead a discussion if you're interested, we can follow up on these 15:08:41 Agenda: https://lists.w3.org/Archives/Public/public-web-and-tv/2021Dec/0000.html 15:09:12 ...discussion between this group and ???, would like to see a collaboration between our groups 15:09:25 ...nothing specific at this stage, but more to come on that front 15:09:33 s/???/MiniApps/ 15:09:44 ...will: who is interested in cmaf over webtransport? 15:10:16 rrsagent, make log public 15:10:20 rrsagent, draft minutes 15:10:20 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html kaz 15:10:24 chris: nhk expressed interest, had a specific question, follow up offline 15:10:32 s/nhk/NHK/ 15:11:00 chris: great media production workshop at smtpe, hoping to bring some of the stuff they raised to the wg 15:11:04 Chair: Chris_Needham, Tatsuya_Igarashi 15:11:28 ...participant survey to find out who's interested and willing to contribute, it needs active participation to move forward 15:11:52 present-Takio_Yamaok 15:11:59 present+ Takio_Yamaoka 15:12:06 ...report from workshop is forthcoming. there were 25 talks or so, transcripts and presentations at link on slide 2 15:12:25 s/present-Takio_Yamaok/present- Takio_Yamaok/ 15:12:50 present+ MFX_Worldwide 15:13:26 yuhao: at bytedance web team, working on 2 players 15:13:49 ...2 proposals to present today 15:14:16 ...1st proposal: Video.requestVideoLoadCallback 15:15:04 ...easy way to get request and response info from video element, similar to video.requestVideoFrameCallback, won't video from MSE or VideoStream 15:15:19 rrsagent, draft minutes 15:15:19 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html kaz 15:16:04 ...use case 1: report http errors with supplemental info to user, instead of just "not supported". today there's no way to distinguish what error happened 15:16:10 scribe+ cpn 15:16:36 present+ Kazuhiro_Hoya 15:17:53 ...use case 2: user watching live stream with hls can get info from cdn response headers for troubleshooting, but video tag cannot. 15:19:00 ...use case 3: timing for segments req/response not available with video tag as opposed to hls player, cannot troubleshoot bad user experience because info is hidden 15:19:18 video.requestVideoLoadCallback() Explainer -> https://github.com/leonardoFu/video-rvlc/blob/main/explainer.md 15:19:23 ...explainer: https://github.com/leonardoFu/video-rvlc/blob/main/explainer.md 15:20:15 ...get callbacks when a request starts, when response happens, so timing is visible, headers are visible. cancelable 15:20:44 ...makes response info available just like fetch api, content-length, response code 15:21:29 jholland: Does the cancel call cancel the request or just the callback? 15:21:49 yuhao: yes 15:22:05 jake: cancel is about the callback, not the request, right? yuhao: yes 15:22:16 q+ Will 15:22:18 ack w 15:22:27 will: if you want this info, why not use mse and fetch? you then have all this info 15:23:19 yuhao: yes, that's what we're doing. we can get all this info, it works well. but for mp4 we don't have a good stable library to play. most web devs prefer to use video tag instead of player 15:23:57 ...i use mse players exactly for this, but many colleagues who want to play mp4 just want to use the video tag 15:24:35 will: if you made a library for this, that would work, right? I say this because adding apis is difficult, and it seems faster to just make a library maybe 15:25:33 will: to 2nd point: I work at a cdn, so I understand the troubleshooting point. you shouldn't be using ip address. apple is hiding this info, and many people will have the same ip 15:26:09 ...the way forward is to use standard media client data to append info to the request such as session id, that will go into logs and will be handled consistently by the cdn, and will go into their logs 15:26:57 yuhao: that makes sense, yes. fetch api to load media streams gets session info in the headers, but video tag we don't get to put stuff in headers so we can't add the session id the same way. also a similar problem there. 15:27:12 ...request callback gives access to insert headers in the request 15:27:44 will: query args are pretty easy and can go into the url today. yes, custom headers are harder, but query args is a reasonable hook. 15:28:33 chris: video with source url case does seem to have a gap--media element makes requests and there's lower visibility with less info about why a failure happened on a failure 15:28:52 ...is there a parallel with other elements elsewhere on the web platform? 15:29:18 yuhao: recently heard a method to get request/response info: worker can intercept video request, but I haven't tried it yet. 15:29:29 ...i might be able to try a demo to see if that's useful 15:30:40 chris: interesting. anyway, for hls case you'd get this info on every request? 15:31:16 yuhao: yes, and we can use regex to see which requests are what. resources with manifest file vs. segments is generally easily distinguished 15:31:50 chris: clarification: so the callback would return the manifest as well? or just metadata about the response success/failure? 15:32:34 yuhao: data would be read-only at most to prevent plugins from editing manifest. we just need metadata for what we're doing. 15:33:10 chris: joey, has this come up in your discussions about building media players? 15:33:44 joey: all we're building is ??-based. we haven't had requests to get response metadata from the hls playbacks, not sure if serviceworker would work 15:34:00 ...possible to bypass all that with a player and mse 15:34:25 ...you don't have an option on ios/iphones, so i can see how this would bbe useful 15:34:41 ...seems preferable to just have mediasource everywhere, but we don't. 15:35:11 chris: yes, for file playback where not using hls/dash this makes sense. have you considered raising a bug or github issue against the html spec? 15:35:33 chris: that seems the most likely place to raise this 15:36:14 joey: yes, you could argue if there's something similar for images or other parts of the web, maybe this applies to something lower-level applicable to other parts of the web 15:36:54 ...maybe serviceworker is the place for that. could potentially raise bugs with the browser vendors. if i were doing this i'd experiment with service workers to find out where there's gaps 15:37:13 chris: yes, makes sense. big question is whether service worker can do this 15:37:27 joey: yes, and next big question is whether hls on ios can do this too 15:38:05 will: original question is: given mse is too complex to get this out, maybe serviceworker doesn't solve this? 15:38:23 joey: yes, but maybe we can do that with a library instead of extending web 15:38:50 ...2nd thing was cmcd injection into these requests. if the serviceworker can do that too, then maybe we don't need to add an api to the browser 15:39:53 yuhao: like will said, most web developers uses video element to load the resource directly, not just for us but everywhere. serviceworker might be a good place to solve this, makes it easier to understand for web devs and easier to use 15:40:40 ...when i just want to play back video, if it's not a standard library it's hard to know what to use 15:41:04 ...adds complexity to the website. have to worry about page size and such 15:41:15 ...but we can make it best practice for our devs if this works. 15:42:07 chris: ok, some good feedback, maybe we should capture this in meig repo to track results of this investigation. once we know what serviceworker can do we can use that as guidance on next steps 15:42:12 [I guess one difficulty with Service Worker is that you won't have access to the DOM, so no easy way to associate the request with a specific video element. This may not be fully needed though] 15:42:23 ...track with a ticket in the repo 15:42:46 yuhao: 2nd proposal: getting sei event from video element when frame is parsed in the video 15:42:50 video SEI event Explainer -> https://github.com/leonardoFu/video-sei-event/blob/main/explainer.md 15:43:05 i/2nd proposal/Topic: Video SEI events/ 15:43:24 ...sei event contains info about the video: timestamp to synchronize iwth video frame 15:43:50 ...available with mse or mediastream. conversation with ??? said they're considering taking out sei info from webcodecs 15:44:14 SEI w/ webcodecs discussions -> https://github.com/w3c/webcodecs/issues/198 15:44:23 ...had the question for me if we don't need to decode and just use the demuxer to decode? 15:44:26 s/conversation with ???/conversation with xfq/ 15:45:06 ...video annotations ("bullets") for text & emoji, subtitles. useful to avoid covering people with the bullets 15:45:31 ...ai stuff can place the text overlay to avoid people by body shape, including with subtitles 15:45:31 Masking in Bullet Chatting -> https://w3c.github.io/danmaku/usecase.html#masking 15:45:51 ...the reason this works is with sei. that gets put into video stream 15:46:18 ...so we need the same mechanism here 15:47:03 ...2nd use case is quiz overlays on the video, part of interactive live events. question panel can pop up where users can click 15:47:20 ...needs to sync with time-limited question asked on the video live stream. sei provides the sync info here 15:47:44 ...3rd use is to capture the time delay. 15:47:59 ...allows measurements about user experience and performance 15:48:13 ...proposal adds sei event for video element 15:48:35 ...gives the payload data for the sei info, and the timestamp (which frame it's in) 15:48:47 ...use the timestamp to start a timeout for various overlays 15:49:07 ...timestamp tells when to render the sei info 15:49:26 ...needed for synchronization. for now we use media source, we can parse the sei stream 15:49:41 ...but we still need the timestamp there to map from video timestamps to the current system time 15:49:48 [In the body masking scenario, frame accuracy seems needed. The main way to achieve that today is through WebCodecs, so even though the app does not need to access decoded frames in theory, in practice it probably does to control the time at which the frame and the masked overlay get rendered] 15:50:14 ...so we can synchronize properly. helps user experience. when using webcodecs, this would help to avoid having to remux 15:50:49 ...so sei info is about how to synchronize the supplemental info with the video element 15:51:06 chris: this fits with another area of work in the interest group around looking at dash emsg boxes 15:51:07 Karen has joined #ME 15:51:23 ...sei is in the video bitstream, not in the media container, right? 15:51:28 yuhao: yes. 15:51:49 ...in china we're still using rtmp because flash was popular before and cdn still has rtmp 15:52:16 ...for web live streams it's easy to move from rtmp to flv streams. 15:53:00 chris: in the other work, we were exposing this in text tracks. those can have metadata and the application can look at events that are scheduled 15:53:44 ...the difference here is you have an event list triggered when the frame is shown? or when it's parsed? it seems like this has a timing requirement 15:54:11 ...so this needs to be linked to the frame so that overlays can avoid collision? or is the timing not as strict so it can have some delay? 15:54:54 yuhao: renderer itself can cause some time delay. that's why i'm looking to get info about when it's parsed, so we know when to render the effect 15:55:40 ...so parsed not rendered. first version if the api asked for it when rendered. that's also good 15:56:13 chris: so the data extracted includes timestamp, so you can know when to apply the rendering 15:56:29 yuhao: yes, timestamp is based on video's current time, not video file's time. 15:57:36 ...synchronizing across multiple broadcast also. timestamps need to change for merging these kinds of events 15:57:47 ...so it might not increase smoothly and might come back to 0 15:58:18 ...complex to deal with sei timestamp, as opposed to current time 15:58:19 q? 15:59:06 will: i support this. we went down this path for dash some. much more efficient while parsing in the internal browser just informs the app. this is possible already in javascript by parsing it 15:59:22 [Other possibilities to expose SEI metadata would be to expose in requestVideoFrameCallback() and/or WebCodecs] 15:59:30 ...but much more efficient to return this from the parser that's already rendering and parsing the video 16:00:11 chris: we'll invite to discuss this use case in media timed events. will follow up separately, this should be included in that work. 16:01:27 cpn: we'll be in touch 16:01:36 ... we have GitHub issues, etc. 16:01:44 ... will share the information with you 16:01:56 ... any other comments? 16:02:02 (none) 16:02:10 cpn: thanks a lot for participating 16:02:22 ... good use cases, well presented and described 16:02:41 ... we're adjourned 16:02:49 [adjourned] 16:02:59 rrsagent, make log public 16:04:02 rrsagent, draft minutes 16:04:02 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html kaz 16:04:39 i/Other possibilities/ I need to run to another meeting, but I would love to have SEI metadata somehow. Could be in a callback, could be in a texttrack. I agree with Will. I would not want to do this in a library./ 16:04:57 rrsagent, draft minutes 16:04:57 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html cpn 16:05:33 i/we'll be/scribe+ kaz/ 16:05:36 rrsagent, draft minutes 16:05:36 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html cpn 16:05:50 xfq_ has joined #me 16:06:36 s//Joey:/ 16:06:53 rrsagent, draft minutes 16:06:53 I have made the request to generate https://www.w3.org/2021/12/07-me-minutes.html cpn 17:05:06 myles has joined #me 17:16:13 Karen has joined #ME 18:01:16 Zakim has left #me 19:36:45 Karen has joined #ME 22:06:04 Karen has joined #ME 23:26:15 Karen has joined #ME