W3C

- DRAFT -

Web & VR TPAC breakout

21 Sep 2016

Agenda

See also: IRC log

Attendees

Present
Jonathan_Jeon(ETRI), Charles_LaPierre, Sangwhan_Moon, Louay, Bassbouss, Anssi_Kostiainen, Jungbin_Kim(Letsee)
Regrets
Chair
SV_MEETING_CHAIR
Scribe
dom

Contents


<scribe> ScribeNick: dom

Anssi: Chrome has an experimental implementation
... Edge is working on one
... Samsung and Firefox have implementations as well
... Ningxin (Intel) will show a demo of hand tracking from the Web
... Frank from MS will report on their implementation work
... Then we will have Louay discussing how to make 360° video first class citizen on the Web
... we'll go through these talks and then open the discussion
... any suggestion already for topics worth discussing

Changes to the WebVR spec

Frank: I'll summarize the changes to the WebVR spec brought in September
... We've looking at how to apply the WebVR spec to Hololens
... and worked with Mozilla and Google on adapting the API to more devices
... we've suggested adding VRDisplay.getFrameData()

Anssi: for context, WebVR is a low-level API

<scribe> ... completed by higher level frameworks such as a-frame to make it easy for Web dev to create content

Frank: the good outcome of the changes to the WebVR spec is that it makes it easier to adopt to more devices in these frameworks
... other changes to the spec include deprecation of some methods
... happy to get questions and feedback on the WebVR spec

Anssi: I'll give you a quick overview of what a high-level VR framework looks like
... [showing a-frame.io and its usage of web components for declarative VR scenes]

Hand tracking use case for VR

Ningxin: [showing leapmotion usage in WebVR]
... showing the user's hands in the VR view is very natural
... and enables to use gestures as a mode of interaction in immersieve experiences
... MS Hololens has also gestures added to their mixed-reality experience
... Intel project alloy has also looking at this space
... the device brings depth sensing
... Intel's crosswalk web runtime engine has an experimental handtracking Web API
... it exposes a gesture API to Web content
... [demo with a depth camera]
... the depth information facilitates the hand recognition & gesture detection
... [depth sensing capture demo, with hand detection highlighted]
... VR Head mounted displays come with cameras that can be used for environment detection
... the 3D detection enables precise mapping of the hand position

360° video

[Louay presenting]

Louay: from Fraunhofer Fokus
... we are also working on the Web VR topic, focusing more on streaming and 360° video in the browser and TV sets
... in this slide I show the different ways of streaming and processing 360 videos
... each coming with their advantages and challenges
... in the 1st case, the server streams the whole 360° content to the browser and leaves it to the browser to render it correctly
... the processing component enables to change the current view port
... the advantage of that approach is that there is low motion latency when moving the view port
... the second option is to processing on the server side, and streams the result of that processing to the client
... this removes any requirement from the client
... this is particularly relevant to TV sets - it enables to play 360 video in HBTV terminal
... where the motion is done via a remote control
... the problems here that network introduces latency
... and it also requires a lot of server processing since each client needs its own dedicated processing
... the 3rd option works the same way as adaptive streaming does
... with prepared pre-processed 360° videos on the server
... you do all the pre-processing and push the resulting files on a static Web server and use MSE to combine the segments on the client
... there can be different algorithms to prepare the content
... we need intelligent caching since we have an additional dimension (time & space)
... [showing a demo of the 3rd option]
... WebVR expects all the rendering to be done on the client, which isn't quite the case here
... this solution introduces delay which might be problematic for some use cases
... it's probably more useful e.g. in second screen cases than HMD, but maybe with intelligent buffering it could be used in the WebVR context

Anssi: I think you need a lower latency for HMD

Louay: you could play around the quality / latency trade off

Zoltan: what's the latency figure for client side processing?

Louay: the processing on the client side requires streaming the whole video
... in smartphones and laptops that would work, but probably not for Chromecast or TV sets
... finishing here with some open questions
... a browser that supports natively 360 video would need to expose events and information for the viewport
... likewise, MSE likely needs to adapt to 360 videos

Frank: do you support other projections schemes?

Louay: @@@
... 6 viewports ought to be sufficient, but in practice we do more of those to have a smoother rendering
... this of course requires a lot of storage capacity

Anssi: this illustrates how to bring VR as a progressive enhancement to the Web

Open discussion

Anssi: WebVR is an important piece of the overall Web & VR space, but it's just one piece
... there are many other issues that needs solving
... and that's what the Web & VR workshop will be looking at
... [showing the list VR workshop topics]

-> https://www.w3.org/2016/06/vr-workshop/ W3C Workshop on Web & Virtual Reality, October 2016, 19-20, San Jose, CA, USA

Anssi: topics include adapting to new input methods, accessibility support
... formats and codecs, declarative content
... 360 videos
... VR as progressive enhancement; in particular, making sure that Web developers can re-use their skills - content is king
... there are intersections between AR & VR that the workshop will consider, but the main focus of the event is VR

DKA: one of the issues we are running into in Samsung when it comes to showing VR content is performance
... will that be covered?

Anssi: sure, that sounds like something we would gladly accept as a topic (and a pull request on the workshop page)

DKA: we're just starting to think at the issue
... there is a question of how to tackle it at the platform level
... and then how to make it so that developers create performance applications
... I think a key is that in VR, performance is a health issue

Anssi: Frank, how has that been approached in your implementation?

Frank: from our perspective, it's very doable

Cyril: working on an open source project named GPAC doing video streaming
... there are 2 different topics: interfaces to control the display and interfaces
... and then one on video processing
... I wonder where W3C stands on these fields
... Note that other SDOs are working on VR

<cyril> https://www.surveymonkey.com/r/MPEG-Virtual-Reality

Cyril: MPEG is working on video & audio codecs on VR and has started a survey
... trying to identify the need in terms of compression, streaming for VR
... the results will be presented at the MPEG meeting next month

anssi: on where W3C fits, that's part of the question we want to discuss at the VR workshop
... we will hear from instance from Khronos with Neil who will be at the workshop

cyril: on media vs api, one thing that I see as needed is accessing the metadata that will end up in media VR files

Mahesh_Samsung: what metadata are you thinking of more specifically?

cyril: e.g. data captured by sensors during the capture
... the project being used

Mahesh: for instance we want metadata on 180 vs 360° video type
... we're interested to hear about more specific needs in this space

Dave_Singer: I also go to MPEG & 3GPP meetings, and I encourage them to scope more specifically what they are doing in this space
... and obviously would encourage W3C to do the same
... MPEG will likely be looking at compression and streaming
... 3GPP at the impact on network
... and W3C should certainly look at the API that can feed the network needs

Charles: first, it's critical to have accessibility baked in from the very beginning
... 2nd, the hand recognition demo was excellent, and it would provide a great way to do sign language interpretation

Doug: do you have concrete ideas on the scene graphs? for declarative scenes

Dom: a-frame.io is a good example of what's possible, and how to use web components for experimentation in this space

Doug: there has been some talks in SVG to make the SVG group a "graphics" WG (with canvas & others)

Anssi: would be good to have someone from the SVG WG at the workshop

Doug: I'll find out
... (but SVG is not appropriate for that kind of stuff in general)

https://github.com/w3c/vr-workshop

-> https://github.com/w3c/vr-workshop GH repo of the VR workshop page

Anssi: Thanks for joining us! Hopefully this got you a bit excited about the topic

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/09/21 10:05:51 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/Lua (?)/Louay/
Succeeded: s/(shucks)//
Succeeded: s/open source project/open source project named GPAC/
Succeeded: s/@@@/Mahesh/
Found ScribeNick: dom
Inferring Scribes: dom
Present: Jonathan_Jeon(ETRI) Charles_LaPierre Sangwhan_Moon Louay Bassbouss Anssi_Kostiainen Jungbin_Kim(Letsee)
Agenda: https://www.w3.org/wiki/TPAC2016/SessionIdeas#Web_.26_Virtual_Reality

WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Got date from IRC log name: 21 Sep 2016
Guessing minutes URL: http://www.w3.org/2016/09/21-webvr-minutes.html
People with action items: 

[End of scribe.perl diagnostic output]