Immersive Web WG/CG TPAC 2023 Day3

15 September 2023


ada, alcooper, bialpio, bkardell_, Brandel, cabanier, CharlesL, etienne, jamesn, Laszlo_Gombos, Leonard, tidoust, vicki, yonet
ada, cabanier, vicki, yonet

Meeting minutes

<atsushi> TPAC 2023 Day2 minutes


ada: this might be a long one. we have a repo in the CG for spatial favicons, not a lot has happened.
… maybe the favicon for the web app manifest, maybe for the page
… might not totally be within our charter but let's talk about it

ada: seems like a very cool thing. could user agents punch out to an SVG to a blobby thing to make a 3d favicon look good?
… any thoughts? or should we archive this?

cabanier: magic leap implement this a few years ago, made orig proposal
… that the time there were a lot of questions
… icons are underspecified

they are not fetched by the page, by the browser
… I'm not sure that matters

as long as we don't say, let's support more mime types

I think there's a nice fallback mechanism that would be nice to see

ada: I think it makes sense. I want to just drop in my Cistine chapel.gltf

cabanier: it would be great if there were an organization or association to define that

ada: maybe the Alliance for Open USD
… something with a very limited feature set, very small
… I'd love to see what people do with a restricted medium like this

Brandel: AOUSD initial agreement is to go through current material and turn it into a standard
… and next, to find out where other relevant standards are necessary
… web is next
… one benefit of NURBS is that they can be used to represent low-res things

ada: it would be fun to see the standards evolve for small model formats

cabanier: would it make sense to reach out to the working group to see

ada: that would be a good idea, this kind of thing takes a long time
… work out the WHATWG for page favicons, or Web App WG for web app manifest
… wasn't there something like, "its not up to us to decide what formats to do". wasn't there a list somewhere?
… this was pre-model

cabanier: if image supports this, maybe support for model too

ada: browsers that don't support it could just ignore it
… if someone were to implement it...

cabanier: ideally there would be a document somewhere to specify

ada: esp now that SVG favicons are supported

ada: do people still feel positive? we'd probably be the lynchpin
… should we do a CG repor to establish what it means to do a small one?
… we can do it in the spatial favicons repo
… establish some base guidelines for what a small model should be
… guidelines for both gltf and usd
… this is approx what we think it should do. mention NURBS, mention low-poly, etc

<cabanier> proposal for size constraints from magic leap: https://ml1-developer.magicleap.com/en-us/learn/guides/portal-icon-guidelines

ada: I don't think the spatial favicons effort will ever leave the CG, more like "this is a things we think" vs a standard
… +1 if you think this is a good idea

<yonet> 0

<etienne> +1 for "small models"

… 0 for neutral, -1 if against

<bialpio> +0.25

<gombos> +1

<cabanier> +1

cabanier: I'm a little worried bc I don't know who is making the right decision, there could be arguing g
… maybe we'll end up just doing our own thing

ada: the history of favicon is pretty loose, I think someone from Netscape was just like "here's a thing I added" one evening
… this might be all of the discussion we need around this
… let's pick up another 15 min issue

<ada> immersive-web/administrivia#201


yonet: is there anything else we should add/ask from our community? there's a Help Wanted tag on this
… one is types

cabanier: I think editors have not really gone through this, some issues are very old
… maybe we should start by going through this

<yonet> https://github.com/immersive-web/webxr/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22

ada: it would be nice if /help added the Help Wanted label
… should I remove the HW thing from email? do people actually read it?

cabanier: maybe assign it to someone and they get an email every day

yonet: if we have anything to ask we could promote it a little

ada: I guess now we are getting closer to Rec
… maybe it could be useful for like, "Hey, we want feedback for this" if someone wants to leave comments, it would be welcome on these issues
… action: add a new label for Needs Feedback, put that in the email instead, and add a new command for people to request feedback
… I don't think anyone reads the mailing list. or the emails I send out! :p
… I guess people do turn up for the F2F
… I think this will be a good action for us to do (yonet agrees)
… BTW if anyone in this room wants navigation to work, feel free to drop in and work on it!

break time! \o/

ada: thank you for joining us
… we'd like to expose more a11y through webxr
… for new to webxr, it's an API that sits on top of xr
… some devices are attached to a computer but most frequently, the display is the computer
… such as the vision pro and quest
… or augmented reality on smart phone
… the issue we have with adding a11y to WebXr is that we are on top of WebGL which has no a11y features
… we have some api's that can increase semantics
… the pixels on the screen are rendered by the user
… we need help with some of these issues
… the other item we are working on, unrelated to WebXR is a new API called the <model> element
… it's a new html element that can display 3D model
… they are no immersive but can be integrated in the page

jamesn: is this what we discussed at the last TPAC?

ada: yes

<yonet> Accessibility considerations: https://github.com/immersive-web/webxr/blob/main/accessibility-considerations-explainer.md

ada: it would be a good idea for the developer to expose what is on the screen at the time
… so we can report back to the user what we are looking at
… the naive thing would be to label everything on the screen but it doesn't give you the context
… the developer can confer that information on what they're rendering on the screen
… if there was a way to properly get that information out
… this api should use the proper tools for the job

vicki: are there other elements where you're drawing with webgl, what do the other a11y tools look like

jamesn: canvas had an experiment with html overlays
… so rather than trying every line being available, you could do potentially do something
… so there's no single description for an entire screen

<bkardell_> https://www.bbc.co.uk/accessibility/forproducts/xr/presentations/

bkardell_: could someone break down if there is research or experiments in this area?
… there are numerous other things that have been explored

ada: I wish that the people from xr access where here

yonet: one thing that they did, it was to say what the objects are around you
… that is one of the challenges but they didn't have solutions for that

CharlesL: xr access has been doing a lot of research at cornell

<CharlesL> https://xraccess.org/workstreams/prototype-for-the-people/

CharlesL: they have prototype
… they want people to experiment
… and need help. They broke it down for the different types of disability
… the developer knows the scene such as a meeting room
… the user could have preferences on how the room is laid out
… depending on the level they are querying, the metadata can be there

cabanier: I know that at TPAC Japan an example from 2D canvas that was only implemented in firefox for an a11y tree that when you draw pixels you also render a semntic tree.

cabanier: screenreaders would need to be existed with it to work and shouldn't be too bad for developers. It wouldn't be hard for developers to render it. But it would fall to the developers.

tidoust: I want to find solutions to expose the a11y information
… so we need to teach developers on how to expose this
… if there's information in USD
… how can this format be extended to give accessible information
… that can easily be targeted at developers

yonet: I'm concerned that if you have a model, there are still a lot of other things around you

ada: we can do both
… webxr has a long time problem that needs to be solved
… for both webxr and model
… adding a11y information to the models is important

ada: as cabanier said, having a render buffer where you render colors referencing a tree
… this could work really well. You can probably do it within a single render
… it could be a good approach to see if it makes sense

jamesn: I want to jump back to canvas
… if we do something with a dom renderer, having something that is overlaid
… rather than something separately

ada: unfortunately there are issues here because you can't show html in webxr
… we've been trying to fix this over the year but extremely hard
… dom content is typically rectangular which doesn't match webgl
… you can have more advantage if you have a pixel map

bkardell_: I mentioned the research

<bkardell_> https://abilitynet.org.uk/news-blogs/apple-vision-pro-has-vision-disabled-inclusion

bkardell_: apple launched the vision pro with a lot of a11y
… it would be great if people could share your a11y research
… we need to gather more information and do studies
… hard to come up with answers in this room

yonet: the meta research team wanted to present but didn't get permission on time

CharlesL: can we have different rendering on top would be nice for color blind people
… the mention of eye tracking, my eyes shake so I'm worried about this technology

yonet: for hololens we weren't reactive to the eye movements. The system itself would also train on you

CharlesL: I'm using different headsets to try this out

cabanier: The tree itself would be written out into the DOM, but the colours would map to the elements in this tree.

<cabanier> bkardell_: I assume that this works on simple canvases

yonet: it would be good if the same view was used for everyone

<cabanier> ... does this work on more complex content

cabanier: the colour index buffer is just used for mapping pixels to elements rather than being visible

the canvas dom overlay technique, it works for a certain size but does it actually work for things are infinitely big e.g. google maps? Or an open world game with a massive amount of data.

cabanier: as you walk through the universe you would update the DOM

bkardell_: does this work today

cabanier: it was removed, it was liked but it was removed

ada: you would only apply colors to the leaf nodes?

CharlesL: I was trying to figure out, is there an aria type solution?

ada: is there a way to output arbitrary information for a screen reader
… today, it has to be a dom tree and the user has to traverse it

CharlesL: my thinking we only have screen readers for headsets
… do we need screen readers for headset?

ada: the headset can be a dumb screen that is attached to a computer
… it's up to the browser itself

bkardell_: are you sure? This is needed for all the apps
… for browsers, you have an OS level a11y tree
… with the exception of vision pro, they're all android based so could use virtual touch

yonet: it would be great what was done for canvas
… and would like to know the challenges in the XR space
… how can we work with the canvas group

jamesn: the APA group is not necessarily the right group
… we know a11y api very well
… it's unclear if the people in the APA group have experience with accessibility APIs, those with the relevant API experience are more likely to be in ARIA
… apart from vision pro
… I'd love to be involved in this conversation

<CharlesL> https://www.w3.org/TR/xaur/

semantic labels

cabanier: we added support for semantic labelling in webxr with gives you information about the room you are in, when the user sets up their device they can annotate their environment floors, tables, walls, paintings, this information is giving to WebXR and used to prevent users from running into objects. It seems useful even at a system level. This is useful for everybody. There are

only a limited set of labels right now. E.g. No ottoman label.

<yonet> Semantic labels: immersive-web/semantic-labels

CharlesL: is this done through a W3C registry?

cabanier: right now it's in a repo people can contribute to

CharlesL: intnl, the hardware can do it

cabanier: the hardware can do it

<jcraig> https://www.aswf.io/blog/community-support-needed-for-new-accessibility-initiative/

james craig: Apple has been working with the Academy Software Foundation (ASWF) to improve a11y. They are going to be working with the USD format to... [connection lost]

<yonet> immersive-web/webxr-accessibility

<jcraig> https://wiki.aswf.io/plugins/servlet/mobile?contentId=74940495#content/view/74940495

<jcraig> Quoting from last link…

<jcraig> USD Accessibility needs :

<jcraig> - Method to label an object (container or leaf node)

<jcraig> - Method to label a time range (possibly use a caption format? like VTT metadata)

<jcraig> - If this format can include video and/or audio, how is loc/lang handled? We may need something similar same for captions and audio descriptions.


cabanier: this issue is something we have noticed that start of in VR that then progress into AR, so for a big portion of AR you are doing all the AR things, cameras, depth, reprojection which is a lot of wasted power.

cabanier: I Don't think it is something we the browser can figure out

ada: It might be more useful to change session time within the session
… if you can pre ask the features. It might be request a change for XR session. I think it would be a more useful way , like going from inline to iimmersive modes

rik: you are proposing while in session, a new request for a different session?

ada: or devs can request for more than one immersive sesion and we give the new xr session.
… I don't think it would break any current usage
… it would be something like xrsession.transition or something

rik: I think it would be awkward

ada: if you put it as an optionall feature

rik: you can't make it an optional feature

ada: if we request as a feature, it would be like showing another room when you want to switch the mode.

Rik: I would prefer if it was in the render state.

ada: when you are in immersive-vr you are not going to test for hit test or anything

rik: if you know what the room looks like, you could do hit testing on the room. You want to keep the environment if you want to keep the AR mode
… creating a new session will have discontinuity

ada: I think you are right, especially on handheld, in that situation what you want is still be in the handheld mode but switch from ar to vr

rik: do you think a flag?

ada: I don't think every platform can do it instantanously, so it should be a promise
… if it is system app start taking over doing ar stuff and you couldn't get access again for immersive ar session

rik: yes. I think I can see it both ways.

ada: for some things like change the mode from transparency

rik: Hololens and a device like that would fail the promise.

ada: you might also want to fire it as an optional feature, that way the device can get ready for the session change

bialpio: vr to ar is tricky. I don't know if we want to allow that. How do you display permission priompt

ada: you can still have ar session. You call the session immediately

bialpo: we don't need that feature now right.

ada: For the escape room case, for the situation where you fail the camera. More work need to be done. You say this is reserved

bialpo: for the vr, the device can render over everything. After the ar mode, it will render everything over again. I think this is over optimization

ada: you are not going to get the camera feed instantly
… we will need a transition state

bialpo: I agree if update render state might not know if the camera is rendered or not, for those cases we might need a promise
… I think we will know if we are on additive devices or not. This case is not enough for the feature as a promise.
… for hit test we don't require ar session. you could expose hit test. the only thing is that we don't have a way of hit testing in vr in virtual objects

alcooper: is there anythign in OpenXR that has an example for how they have done this and whether they actually can stop doing work even if we ask it not to.

alcooper: I think as long as the session is going the camera will still be going but that could just be for our implementation,

alcooper: I don't see going from immersive-ar to immersive-vr beign something feasible since they are essentially two seperate runtimes.

cabanier: on OpenXR there is a system ready call and do passthrough, and a pass through layer which can be removed from the lift of layers.

cabanier: to bialpio since it could be a hint for additive devices they can just fulfill the promise immediately.

bialpio: I think this makes sense, I wanted to add some details about ARCore in which case i think we can immediately switch at least in one direction or maybe both for the camera availability.

alcooper: Yeah we should be able to do that but we may at least save a little optimisation although it may do it anyway.

bialpio: yeah it might be a very small performance improvement for android

bialpio: there might be some features that the site may want access to which we may be able to disable but I don't think we would be able to disable these on the fly in ARCore they need to be set on session creation which add latency to the camera such as image tracking which add a couple of ms of delay. Which is noticable, we probaby wouldn't be able to turn these off.

cabanier: i agree that depth sensing or image tracking are turned on then it won't do anything

alcooper: one thought i had regarding passthrough, would it make sense to have pass through be a request in immersive-vr too so pass through is just a feature that can be turned on or off

cabanier: I don't think the developer thinks of their app as vr.

<yonet> alcooper: to rik's point, maybe it is weird to have these ar vs vr with the pass through

bialpio: i think this is starting to blur the lines between AR and VR I don't think many of the feature sassociated with AR are blocked in VR sessions and it's worth while considering AR sessions as more sensitive in terms of privacy so perhaps some features should be restricted to AR only.

cabanier: one more reason they should start from AR then disable, is preerving the mode type in android since we don't want people starting in cardboard mode then going AR

bialpio: also it would need to solve getting the permission prompt

cabanier: it would also cause the session to grid to a halt for a few seconds

<Leonard> [AFK, BRB]

alcooper: if it is a hint we need to have someway to show it can fail e.g. if you are doing raw camera and ask us to disable your camera that should probably fail

<ada> bialpio: re raw camera access we would need to start handing off camera frames whilst the user cannot see what is being visible by the camera.

ada: if you request a raw camera access, what is stopping the dev to render a big square to hide the environment

<ada> alcooper: for what it's worth you could do the same thing with getUserMedia.


ada: I've chat with hdr experts. I've been trying to work out what we need to expose to the user
… there are some xr devices, like vision pro, meta quest pro, have a display that is hdr
… currently all xr sessions are srgb
… what was it that you explained about quest pro? do you just stretch out the color space?

rik: most websites do it in srgb but spec says it is linear
… I think this is how webgl works too.
… we were creating linear colorspace. some browsers implementation colors are wrong

ada: what Rik told me gl layers can be floats

rik: spec does not allow floats but can be updated
… webgl can render to floating point rgb buffer for hdr, that is totally possible
… if not, we need to request the color spaces

ada: I was thinking having a prefered color spaces. list of supported ones
… there needs to be a way to say what color space you are using, the color space you are rendering needs to be supported by the frameworks. You need to be able to tell which one you are doing.
… for headsets like vision pro, head room might change
… we provide the user list of color space, and the developer can tell us which color space they are rendering. There might be some reason they might do headspace
… I think we will have to draft something

Leonard: I think this may be resolved, but is this about colourspaces or HDR?

Brandel: This is more sRGB vs P# which is also HDR

Leonard: These calculations are done in Linear colourspace and then would need to go through tonemapping to go the displays.

Leonard: You Emmet Liash from Google is an expert in this field.

<Leonard> References: P3 - https://en.wikipedia.org/wiki/DCI-P3; sRGB - https://en.wikipedia.org/wiki/SRGB

<eeeps> The HDR on the Web breakout session (https://www.w3.org/events/meetings/009a5b81-0459-4ae4-9b33-f88dd9a9d89f/) was good and a few of the issues discussed there seem relevant

<Leonard> Different color spaces have a major impact on ensuring consistency of display results between different devices

cabanier: @ada re the headroom we need to expose, WebXR is based on a canvas the developer instantiates before the sessio nstarts

ada: the canvas context needs to know the details of the devie rather the computer so the information eeds to be present on the XR session

cabanier: the canvas already needs to have a way to retarget itself to what ever the display is what ever they do with that we can treat the canvas context the same on the headset.

<bialpio> Related, 2d canvas has a way to specify color space: https://chromestatus.com/feature/5807007661555712

need to get the list of supported depth formats for the system compositor so that the developer can choose the correct.

rik: I am not sure if we support floating points

ada: it seems like we are going to be blocked. on the hdr meeting they showed us all required. 3 thinks were blockers

Leonard: are we going to discuss this on our regular meetings? I want to make sure the experts will join

ada: I will bring the topic back, otherwise the issue is going to get worse with more devices

Leonard: the color space is the issue, different than hdr

ada: I guess only hdr is dependent on buffers

rik: I don't think we have a problem currently about the color space

ada: we should at least expose prefer color spaces

rik: you mean, prefered texture formats

ada: color space

Leonard: is this the final output or the colour space physically based rendering is done in

Brandel: it's the tonemapping, the final output

ada: I was pretty sure threejs is using linear these days and then convert to srgb at the end

bialpio: opengl assumes srgb. You all the math in linear space, and then turn it into srgb. We might be confused about some of this.
… webgl and opengl will always assume linear

ada: does correct colors won't work in gl

rik: yes

rik: we have to do texture formats

ada: down the road, we will have to display everything to the user.

<Leonard> Do you really want to commit to WebGL as it is [slowly] dropping out in favour of WebGPU?

ada: if you have 3 headsets connected to a computer they each will have a different profile

<Leonard> TY all - great meeting. I need to leave now.

<yonet> Thanks Leonard

Automated testing of WebXR sessions with WebDriver

ada: there's a w3c group wrt the webdriver
… it would be great if webxr could use webdriver
… I think it would be useful for developer building complext website
… but also in depth testing for webxr
… I was wondering if people think our group should do
… and then submit it to the group that manager webdriver

alcooper: we have the webxr test api
… which chrome uses to drive tests
… I believe Apple contributed to that as well

<bialpio> WebXR Test API: https://immersive-web.github.io/webxr-test-api/

alcooper: maybe more could be done there

ada: that's great.
… maybe that could be the basis for this type of work
… that sounds great
… should the group be involved in this

bialpio: do we know what is expected here?
… we already have the web platform tests
… it mostly mocks some data
… depending on what is needed, it's sufficient
… in chromium, you can't take advantage of this outside of web platform tests
… we don't initialize the components all the time. It only exists in our tests

ada: what I'd like to see happen, is that I can put my hmd in the computer and then run my tests
… and then dump it to a file, generate screenshots, diff them , etc
… so if I make a change, everything still works as expected
… it's not just mocking, I want to actually drive the hardware

bialpio: you can send triangles through the mock api
… we have an implementation that drives this
… you can't control ref space, input sources
… so you could write a script that moves you around
… related to plugging in the headset, ARCore allows us to record an AR session
… this is how we do testing internally
… check how the api is working
… If you go to session recording, it's very backend specific
… we don't something special, just feed the precorded session to the application
… the test API might be sufficient

ada: is this API defined?

bialpio: yes, Alex and Manish are the editors
… for all the APIs we ship, we've been updating the data
… we have a javascript implementation that implements the runtime

alcooper: the goal of the test API is to serve as a fake device
… if you want to inspect bits of state
… it drives and controls the state
… you need a special build of chrome. We only do it as a polyfill
… some of our tests, do rely on generic webdriver functionality

ada: it sounds that we want to build on top of this
… we don't want to have users wear the headset while the tests run

alcooper: we don't ship any of that

bialpio: just to clarify, it's not really a polyfill
… because the native code still runs
… we talk to a javascript backend. Just not arcore
… I'm looking at the spec and it looks like we need to relax it if we want it to run on shipped builds

ada: if we go down the selenium group, we want to say that it references this api
… but that it can ship in browser

alcooper: I believe Manish implemented this in servo
… I think it's fair to discuss a separate interface
… that can control a generic device

ada: a developer could use the test api withouth selenium without us needing to make changes there
… I'm glad we had this conversation

Minutes manually created (not a transcript), formatted by scribe.perl version 221 (Fri Jul 21 14:01:30 2023 UTC).


Succeeded: s/tit/it

Succeeded: s/topic: Agenda bashing//

Succeeded: s|TPAC 2023 Day2 minutes -> https://www.w3.org/2023/09/12-immersive-web-minutes.html||

Succeeded: i|topic: https://github.com/immersive-web/spatial-favicons/issues/7|TPAC 2023 Day2 minutes -> https://www.w3.org/2023/09/12-immersive-web-minutes.html

Succeeded: s/ the aria group is not necessarily the right group/ the APA group is not necessarily the right group/

Succeeded: s/it's unclear if they have accessibility APIs/it's unclear if the people in the APA group have experience with accessibility APIs, those with the relevant API experience are more likely to be in ARIA/

Succeeded: i/ada: you would only apply colors to the leaf/scribe+ cabanier/

Succeeded: s/Apple have been /Apple has been /

Succeeded: s/??? foundation/Academy Software Foundation (ASWF)/

Succeeded: s/something/the same thing

Succeeded: i/Leonard: I think this may be resolved, but is/scribe+ ada/

Maybe present: bialpo, rik

All speakers: ada, alcooper, bialpio, bialpo, bkardell_, Brandel, cabanier, CharlesL, jamesn, Leonard, rik, tidoust, vicki, yonet

Active on IRC: ada, alcooper, atsushi, bialpio, bkardell_, Brandel, cabanier, CharlesL, eeeps, etienne, gombos, jamesn, jcraig, Leonard, tidoust, vicki, yonet