W3C

– DRAFT –
360 video

Attendees

Present
@shaoboyan_, cabanier, cpn, cwilso, mounir, stepsteg
Regrets
-
Chair
Samira
Scribe
cpn

Meeting minutes

samira: content protection issues for DRM
… chicken and egg problem, is it worth their while for content and service providers?
… there's an opportunity to solve this, increase engagemnt in VR devices
… why do we need WebGL?
… 360 cameras have several lenses, frames from these are stitched together
… [world map example]
… or 6 different cameras pointing in different directions, give a cube map
… WebXR needed to render the pixels on the device
… where should the standard live?
… could be on the video element, add an isSpherical attribute
… include all the spherical information in the media container
… immersive web group has made progress on device API

<yanghau> where can I get the link of slides?

samira: can use this to render video and subtitles
… spherical audio is another consideration
… subtitles are an important part of this, to make 360 video accessible
… would benefit all users
… subtitles should always be visible, regardless of where the user is looking
… what requirements do we have for making 360 video easily available on websites?

Jer: previous efforts on standardising a file format for 360 video had problems - bandwidth where only a proportion is used at any time
… is there an HLS like codec, with streams at varying resolutions at different positions, switch up and down to higher resolution streams
… it's a known problem from previous efforts
… will be difficult to have enough varying resolutions in a single file

samira: resolution is an issue, 8K is avalilable, but most cameras do 4K

andreas: we filed an issue with the immersive web CG on captioning, history of discussion
… where to standardise? what layer to tackle, and if we know what we want, which is the right group to follow up?
… response from immersive web was that 360 video isn't a primitive, so not right time
… there's more to do on this. captions is one part, there's a lot of content out there now, but without captions. this is unfortunate
… does the 360 problem need to be discussed in a certain environment, but where to do that?

samira: want discussion with other browser implementers

jer: interesting authoring problem, use webvtt to avoid burning in captions. but how to do the rendering and keep visible?

mark_foltz: is the focus on the container format for the spherical video, or is it about compositing with other content in WebGL?
… metadata so you know the point of reference, to be able to render faithfully

horiuchi: is real time communication in scope?

samira: yes
… these are some initial thoughts, want a larger discussion

andreas: would be good to hear from the immersive web community

brandon: bandwidth is a concern, this happens over the wire, and also decoding a large amount of data through the GPU, bus bandwidth
… you can't do the same kinds of high resolutions that native apps can do because there's a copy of the data to the textrure into webGL
… having a more native support can help
… WebXR would like to have a dedicated layer type that can accept a video
… This would give more control, but allow DRM compatible rendering

Philip_Lamb: In Firefox we have support in a medial element, allows it to be projected. the missing piece is projection metadata
… be it rectangular or polar. if the video is delivered as 2d, we need a common language to express projections

<Zakim> plamb_mozilla, you wanted to comment

Philip_Lamb: projections may be appropriate for different kinds of content, transformations
… we've seen with mobile devices the bus bandwidth issue
… regarding captions, there'll be a subtitle track placed 6 times around the user, baked in, which is undesirable

samira: the metadata might be the least friction path, needed anyway

mounir: how many libraries are doing 360 video?

samira: three.js has it, also react 360. can't give numbers on how popular
… for simple sites that just want to play video, it might not be worth it given the effort

mounir: what's the main benefit, e.g, copying of data?

samira: performance, also DRM content, ease of integrating into the website

cpn: i recall previous discussion in immersive web, want to get feedback before standardising higher level

alex_turner: what's the delta between existing general purpose API and getting to native support, build a high level component to narrow the gap. want to identify what that gap is to help move the conversation forwards

brandon: another concrete need is that on many VR devices, if you hand the video off to the OS more directly, there are reprojection methods that can be used to render, rather than render to a WebGL buffer. it boils down to performance and security that can be improved

alex_turner: also native playback controls, sometimes a pro, sometimes a con

brandon: it seems nobody wants the standard controls

mounir: pretty much all websites don't use the standard controls, but we'd have to create them and it's a lot of work
… i'd want to see numbers to show that it's worth adding

jer: it requires a lot of work, the easiest thing would be to pull in three.js
… might be able to implement the controls in JS, similar to what we currently do
… would need new APIs on the video element to control projection

brandon: an approach to that is to add primitives to WebXR, which we're already doing, it's on our roadmap
… then, more responsibility would be on the author

jer: can I use WebXR?

brandon: yes, we got feedback that don't want to do the rendering pipeline twice, we get inline rendering

alex_turner: on native platforms for a skybox, want an equirect video, then there are issues of defining projections

jer: we could punt the video element discussion to webxr

andreas: i'd like to share some results from a European project
… project on immersive accessibility for 360 video
… they built a player with closed captions, signing, audio description support
… the demos give a feel for the direction, it works on XR devices
… the player uses WebXR with a threejs based library to translate it to WebGL, then burn in the access content
… [demo of player controls]
… you can customise the access services, e.g, font size of subtitles
… for 360 there's a feature requirement for captions to indicate the location of the speaker, show an arrow

<samira> https://‌1drv.ms/‌p/‌s!AnTGccCUFUkIgawGY1QzITAtM0tGCw?e=JSbURO

andreas: have done user testing to get feedback. two kinds of subtitles: position in 360 sphere attached to the current speaker, or in a fixed position on screen that follows your view

andreas: we use IMSC for caption authoring, it's rendered on a canvas using threejs
… it's not a standard way to do it, the experience in XR devices is worse than in a regular browser
… there's no place to render the captions with XR devices
… on XR devices, you look down to open the menu
… we started the discussion two years. should we try to pursue standardisation?

<Zakim> plamb_mozilla, you wanted to comment on novel advantages

mounir: we could add use the generic cue proposal

andreas: yes, this would enable the browser to handle the rendering

brandon: thank you for the demo. there are two problems. one is a question of how is it best to present the subtitles?
… then, how do we standardise the metadata to allow this to be done
… it seems that a lot of progress has been made on the first question, but is it not a solved problem?

andreas: there's not a standard way of doing it. IMSC and WebVTT are made for a 2d plane, but for this kind of content, it's unknown
… if you want to decouple authoring and rendering, you need a standard

brandon: I agree. there are some questions to be answered about how to render in a headset, so it's difficult to standardise what the metadata would be until we have a good handle on appropriate ways to show this
… this would indicate to me that the appropriate place to continue this to a common solution would be in a library, if they can show how they do it
… assuming this is showing how to do it, i'd suggest a library start to push for their approach for delivering the data they need

andreas: i'm concerned about time passing, as we have inaccessible content out there, so it's already a problem. want to see a short/medium term solution
… we raised issues in the immersive web CG, there's an open source library doing this
… we raised this in WICG
… not so much activity from implementers, so want that before going to standardisation

alex_turner: there can be different parts that can be standardised independently
… could start from video element solution an work back from there
… or from where we are and work forwards
… eg, create an authoring standard that is consumed by a library

brandon: what can an implementer bring to the table, that's been missing so far?

andreas: something like a TextTrack for HTML video, the browser is responsible for overlaying the subtitles
… could be similar to the DOM overlay solution

brandon: i agree. all this is happening in the browsers via developer supplied libraries
… there's a couple of things to do before we'd implement. we'd want to see library support that proves out the approach

philip: what's the overlap between this and TTWG?

andreas: there's a request to produce a TTML version this year that supports this
… but we're missing expertise in the group, need input from the immersive web community
… we'd deal with standardisation in TTWG. my idea was to go to the WICG to bring different expertise together

<atai1> WICG Proposal on 360 subtitles: https://‌discourse.wicg.io/‌t/‌proposal-subtitles-for-360-videos/‌3856

alex_turner: getting to a common understanding of the problem space and all the steps involved, and where there could be standards, could help unblock

<atai1> Requirements for 360 subtitles: https://‌github.com/‌immersive-web/‌proposals/‌issues/‌40

alex_turner: don't want to wait for one big thing to happen before being able to start
… let's use this issue for the launching point?

<atai1> Demo for 360 Player with subtitles: http://‌imac.i2cat.net/‌player/

cpn: another CG has been proposed

andreas: chistopher patnoe from Google has proposed a CG

cpn: next steps?

alex_turner: post thoughts on the issue, issue 55 in the proposals repo

andreas: and create a repo in WICG

[adjourned]

Minutes manually created (not a transcript), formatted by Bert Bos's scribe.perl version 86 (Fri Aug 23 14:14:18 2019 UTC), a reimplementation of David Booth's scribe.perl. See history.