Meeting minutes
<atsushi> TPAC 2023 Day2 minutes
immersive-web/spatial-favicons#7
ada: this might be a long one. we have a repo in the CG for spatial favicons, not a lot has happened.
… maybe the favicon for the web app manifest, maybe for the page
… might not totally be within our charter but let's talk about it
ada: seems like a very cool thing. could user agents punch out to an SVG to a blobby thing to make a 3d favicon look good?
… any thoughts? or should we archive this?
cabanier: magic leap implement this a few years ago, made orig proposal
… that the time there were a lot of questions
… icons are underspecified
they are not fetched by the page, by the browser
… I'm not sure that matters
as long as we don't say, let's support more mime types
I think there's a nice fallback mechanism that would be nice to see
ada: I think it makes sense. I want to just drop in my Cistine chapel.gltf
cabanier: it would be great if there were an organization or association to define that
ada: maybe the Alliance for Open USD
… something with a very limited feature set, very small
… I'd love to see what people do with a restricted medium like this
Brandel: AOUSD initial agreement is to go through current material and turn it into a standard
… and next, to find out where other relevant standards are necessary
… web is next
… one benefit of NURBS is that they can be used to represent low-res things
ada: it would be fun to see the standards evolve for small model formats
cabanier: would it make sense to reach out to the working group to see
ada: that would be a good idea, this kind of thing takes a long time
… work out the WHATWG for page favicons, or Web App WG for web app manifest
… wasn't there something like, "its not up to us to decide what formats to do". wasn't there a list somewhere?
… this was pre-model
cabanier: if image supports this, maybe support for model too
ada: browsers that don't support it could just ignore it
… if someone were to implement it...
cabanier: ideally there would be a document somewhere to specify
ada: esp now that SVG favicons are supported
ada: do people still feel positive? we'd probably be the lynchpin
… should we do a CG repor to establish what it means to do a small one?
… we can do it in the spatial favicons repo
… establish some base guidelines for what a small model should be
… guidelines for both gltf and usd
… this is approx what we think it should do. mention NURBS, mention low-poly, etc
<cabanier> proposal for size constraints from magic leap: https://
ada: I don't think the spatial favicons effort will ever leave the CG, more like "this is a things we think" vs a standard
… +1 if you think this is a good idea
<yonet> 0
<etienne> +1 for "small models"
+1
… 0 for neutral, -1 if against
<bialpio> +0.25
<gombos> +1
<cabanier> +1
cabanier: I'm a little worried bc I don't know who is making the right decision, there could be arguing g
… maybe we'll end up just doing our own thing
ada: the history of favicon is pretty loose, I think someone from Netscape was just like "here's a thing I added" one evening
… this might be all of the discussion we need around this
… let's pick up another 15 min issue
immersive-web/administrivia#201
yonet: is there anything else we should add/ask from our community? there's a Help Wanted tag on this
… one is types
cabanier: I think editors have not really gone through this, some issues are very old
… maybe we should start by going through this
<yonet> https://
ada: it would be nice if /help added the Help Wanted label
… should I remove the HW thing from email? do people actually read it?
cabanier: maybe assign it to someone and they get an email every day
yonet: if we have anything to ask we could promote it a little
ada: I guess now we are getting closer to Rec
… maybe it could be useful for like, "Hey, we want feedback for this" if someone wants to leave comments, it would be welcome on these issues
… action: add a new label for Needs Feedback, put that in the email instead, and add a new command for people to request feedback
… I don't think anyone reads the mailing list. or the emails I send out! :p
… I guess people do turn up for the F2F
… I think this will be a good action for us to do (yonet agrees)
… BTW if anyone in this room wants navigation to work, feel free to drop in and work on it!
break time! \o/
ada: thank you for joining us
… we'd like to expose more a11y through webxr
… for new to webxr, it's an API that sits on top of xr
… some devices are attached to a computer but most frequently, the display is the computer
… such as the vision pro and quest
… or augmented reality on smart phone
… the issue we have with adding a11y to WebXr is that we are on top of WebGL which has no a11y features
… we have some api's that can increase semantics
… the pixels on the screen are rendered by the user
… we need help with some of these issues
… the other item we are working on, unrelated to WebXR is a new API called the <model> element
… it's a new html element that can display 3D model
… they are no immersive but can be integrated in the page
jamesn: is this what we discussed at the last TPAC?
ada: yes
<yonet> Accessibility considerations: https://
ada: it would be a good idea for the developer to expose what is on the screen at the time
… so we can report back to the user what we are looking at
… the naive thing would be to label everything on the screen but it doesn't give you the context
… the developer can confer that information on what they're rendering on the screen
… if there was a way to properly get that information out
… this api should use the proper tools for the job
vicki: are there other elements where you're drawing with webgl, what do the other a11y tools look like
jamesn: canvas had an experiment with html overlays
… so rather than trying every line being available, you could do potentially do something
… so there's no single description for an entire screen
<bkardell_> https://
bkardell_: could someone break down if there is research or experiments in this area?
… there are numerous other things that have been explored
ada: I wish that the people from xr access where here
yonet: one thing that they did, it was to say what the objects are around you
… that is one of the challenges but they didn't have solutions for that
CharlesL: xr access has been doing a lot of research at cornell
<CharlesL> https://
CharlesL: they have prototype
… they want people to experiment
… and need help. They broke it down for the different types of disability
… the developer knows the scene such as a meeting room
… the user could have preferences on how the room is laid out
… depending on the level they are querying, the metadata can be there
cabanier: I know that at TPAC Japan an example from 2D canvas that was only implemented in firefox for an a11y tree that when you draw pixels you also render a semntic tree.
cabanier: screenreaders would need to be existed with it to work and shouldn't be too bad for developers. It wouldn't be hard for developers to render it. But it would fall to the developers.
tidoust: I want to find solutions to expose the a11y information
… so we need to teach developers on how to expose this
… if there's information in USD
… how can this format be extended to give accessible information
… that can easily be targeted at developers
yonet: I'm concerned that if you have a model, there are still a lot of other things around you
ada: we can do both
… webxr has a long time problem that needs to be solved
… for both webxr and model
… adding a11y information to the models is important
ada: as cabanier said, having a render buffer where you render colors referencing a tree
… this could work really well. You can probably do it within a single render
… it could be a good approach to see if it makes sense
jamesn: I want to jump back to canvas
… if we do something with a dom renderer, having something that is overlaid
… rather than something separately
ada: unfortunately there are issues here because you can't show html in webxr
… we've been trying to fix this over the year but extremely hard
… dom content is typically rectangular which doesn't match webgl
… you can have more advantage if you have a pixel map
bkardell_: I mentioned the research
<bkardell_> https://
bkardell_: apple launched the vision pro with a lot of a11y
… it would be great if people could share your a11y research
… we need to gather more information and do studies
… hard to come up with answers in this room
yonet: the meta research team wanted to present but didn't get permission on time
CharlesL: can we have different rendering on top would be nice for color blind people
… the mention of eye tracking, my eyes shake so I'm worried about this technology
yonet: for hololens we weren't reactive to the eye movements. The system itself would also train on you
CharlesL: I'm using different headsets to try this out
cabanier: The tree itself would be written out into the DOM, but the colours would map to the elements in this tree.
<cabanier> bkardell_: I assume that this works on simple canvases
yonet: it would be good if the same view was used for everyone
<cabanier> ... does this work on more complex content
cabanier: the colour index buffer is just used for mapping pixels to elements rather than being visible
the canvas dom overlay technique, it works for a certain size but does it actually work for things are infinitely big e.g. google maps? Or an open world game with a massive amount of data.
cabanier: as you walk through the universe you would update the DOM
bkardell_: does this work today
cabanier: it was removed, it was liked but it was removed
ada: you would only apply colors to the leaf nodes?
CharlesL: I was trying to figure out, is there an aria type solution?
ada: is there a way to output arbitrary information for a screen reader
… today, it has to be a dom tree and the user has to traverse it
CharlesL: my thinking we only have screen readers for headsets
… do we need screen readers for headset?
ada: the headset can be a dumb screen that is attached to a computer
… it's up to the browser itself
bkardell_: are you sure? This is needed for all the apps
… for browsers, you have an OS level a11y tree
… with the exception of vision pro, they're all android based so could use virtual touch
yonet: it would be great what was done for canvas
… and would like to know the challenges in the XR space
… how can we work with the canvas group
jamesn: the APA group is not necessarily the right group
… we know a11y api very well
… it's unclear if the people in the APA group have experience with accessibility APIs, those with the relevant API experience are more likely to be in ARIA
… apart from vision pro
… I'd love to be involved in this conversation
<CharlesL> https://
semantic labels
cabanier: we added support for semantic labelling in webxr with gives you information about the room you are in, when the user sets up their device they can annotate their environment floors, tables, walls, paintings, this information is giving to WebXR and used to prevent users from running into objects. It seems useful even at a system level. This is useful for everybody. There are
only a limited set of labels right now. E.g. No ottoman label.
<yonet> Semantic labels: immersive-web/
CharlesL: is this done through a W3C registry?
cabanier: right now it's in a repo people can contribute to
CharlesL: intnl, the hardware can do it
cabanier: the hardware can do it
<jcraig> https://
james craig: Apple has been working with the Academy Software Foundation (ASWF) to improve a11y. They are going to be working with the USD format to... [connection lost]
<yonet> immersive-web/
<jcraig> https://
<jcraig> Quoting from last link…
<jcraig> USD Accessibility needs :
<jcraig> - Method to label an object (container or leaf node)
<jcraig> - Method to label a time range (possibly use a caption format? like VTT metadata)
<jcraig> - If this format can include video and/or audio, how is loc/lang handled? We may need something similar same for captions and audio descriptions.
immersive-web/webxr-ar-module#90
cabanier: this issue is something we have noticed that start of in VR that then progress into AR, so for a big portion of AR you are doing all the AR things, cameras, depth, reprojection which is a lot of wasted power.
cabanier: I Don't think it is something we the browser can figure out
ada: It might be more useful to change session time within the session
… if you can pre ask the features. It might be request a change for XR session. I think it would be a more useful way , like going from inline to iimmersive modes
rik: you are proposing while in session, a new request for a different session?
ada: or devs can request for more than one immersive sesion and we give the new xr session.
… I don't think it would break any current usage
… it would be something like xrsession.transition or something
rik: I think it would be awkward
ada: if you put it as an optionall feature
rik: you can't make it an optional feature
ada: if we request as a feature, it would be like showing another room when you want to switch the mode.
Rik: I would prefer if it was in the render state.
ada: when you are in immersive-vr you are not going to test for hit test or anything
rik: if you know what the room looks like, you could do hit testing on the room. You want to keep the environment if you want to keep the AR mode
… creating a new session will have discontinuity
ada: I think you are right, especially on handheld, in that situation what you want is still be in the handheld mode but switch from ar to vr
rik: do you think a flag?
ada: I don't think every platform can do it instantanously, so it should be a promise
… if it is system app start taking over doing ar stuff and you couldn't get access again for immersive ar session
rik: yes. I think I can see it both ways.
ada: for some things like change the mode from transparency
rik: Hololens and a device like that would fail the promise.
ada: you might also want to fire it as an optional feature, that way the device can get ready for the session change
bialpio: vr to ar is tricky. I don't know if we want to allow that. How do you display permission priompt
ada: you can still have ar session. You call the session immediately
bialpo: we don't need that feature now right.
ada: For the escape room case, for the situation where you fail the camera. More work need to be done. You say this is reserved
bialpo: for the vr, the device can render over everything. After the ar mode, it will render everything over again. I think this is over optimization
ada: you are not going to get the camera feed instantly
… we will need a transition state
bialpo: I agree if update render state might not know if the camera is rendered or not, for those cases we might need a promise
… I think we will know if we are on additive devices or not. This case is not enough for the feature as a promise.
… for hit test we don't require ar session. you could expose hit test. the only thing is that we don't have a way of hit testing in vr in virtual objects
alcooper: is there anythign in OpenXR that has an example for how they have done this and whether they actually can stop doing work even if we ask it not to.
alcooper: I think as long as the session is going the camera will still be going but that could just be for our implementation,
alcooper: I don't see going from immersive-ar to immersive-vr beign something feasible since they are essentially two seperate runtimes.
cabanier: on OpenXR there is a system ready call and do passthrough, and a pass through layer which can be removed from the lift of layers.
cabanier: to bialpio since it could be a hint for additive devices they can just fulfill the promise immediately.
bialpio: I think this makes sense, I wanted to add some details about ARCore in which case i think we can immediately switch at least in one direction or maybe both for the camera availability.
alcooper: Yeah we should be able to do that but we may at least save a little optimisation although it may do it anyway.
bialpio: yeah it might be a very small performance improvement for android
bialpio: there might be some features that the site may want access to which we may be able to disable but I don't think we would be able to disable these on the fly in ARCore they need to be set on session creation which add latency to the camera such as image tracking which add a couple of ms of delay. Which is noticable, we probaby wouldn't be able to turn these off.
cabanier: i agree that depth sensing or image tracking are turned on then it won't do anything
alcooper: one thought i had regarding passthrough, would it make sense to have pass through be a request in immersive-vr too so pass through is just a feature that can be turned on or off
cabanier: I don't think the developer thinks of their app as vr.
<yonet> alcooper: to rik's point, maybe it is weird to have these ar vs vr with the pass through
bialpio: i think this is starting to blur the lines between AR and VR I don't think many of the feature sassociated with AR are blocked in VR sessions and it's worth while considering AR sessions as more sensitive in terms of privacy so perhaps some features should be restricted to AR only.
cabanier: one more reason they should start from AR then disable, is preerving the mode type in android since we don't want people starting in cardboard mode then going AR
bialpio: also it would need to solve getting the permission prompt
cabanier: it would also cause the session to grid to a halt for a few seconds
<Leonard> [AFK, BRB]
alcooper: if it is a hint we need to have someway to show it can fail e.g. if you are doing raw camera and ask us to disable your camera that should probably fail
<ada> bialpio: re raw camera access we would need to start handing off camera frames whilst the user cannot see what is being visible by the camera.
ada: if you request a raw camera access, what is stopping the dev to render a big square to hide the environment
<ada> alcooper: for what it's worth you could do the same thing with getUserMedia.
immersive-web/webxr#1345
ada: I've chat with hdr experts. I've been trying to work out what we need to expose to the user
… there are some xr devices, like vision pro, meta quest pro, have a display that is hdr
… currently all xr sessions are srgb
… what was it that you explained about quest pro? do you just stretch out the color space?
rik: most websites do it in srgb but spec says it is linear
… I think this is how webgl works too.
… we were creating linear colorspace. some browsers implementation colors are wrong
ada: what Rik told me gl layers can be floats
rik: spec does not allow floats but can be updated
… webgl can render to floating point rgb buffer for hdr, that is totally possible
… if not, we need to request the color spaces
ada: I was thinking having a prefered color spaces. list of supported ones
… there needs to be a way to say what color space you are using, the color space you are rendering needs to be supported by the frameworks. You need to be able to tell which one you are doing.
… for headsets like vision pro, head room might change
… we provide the user list of color space, and the developer can tell us which color space they are rendering. There might be some reason they might do headspace
… I think we will have to draft something
Leonard: I think this may be resolved, but is this about colourspaces or HDR?
Brandel: This is more sRGB vs P# which is also HDR
Leonard: These calculations are done in Linear colourspace and then would need to go through tonemapping to go the displays.
Leonard: You Emmet Liash from Google is an expert in this field.
<Leonard> References: P3 - https://
<eeeps> The HDR on the Web breakout session (https://
<Leonard> Different color spaces have a major impact on ensuring consistency of display results between different devices
cabanier: @ada re the headroom we need to expose, WebXR is based on a canvas the developer instantiates before the sessio nstarts
ada: the canvas context needs to know the details of the devie rather the computer so the information eeds to be present on the XR session
cabanier: the canvas already needs to have a way to retarget itself to what ever the display is what ever they do with that we can treat the canvas context the same on the headset.
<bialpio> Related, 2d canvas has a way to specify color space: https://
need to get the list of supported depth formats for the system compositor so that the developer can choose the correct.
rik: I am not sure if we support floating points
ada: it seems like we are going to be blocked. on the hdr meeting they showed us all required. 3 thinks were blockers
Leonard: are we going to discuss this on our regular meetings? I want to make sure the experts will join
ada: I will bring the topic back, otherwise the issue is going to get worse with more devices
Leonard: the color space is the issue, different than hdr
ada: I guess only hdr is dependent on buffers
rik: I don't think we have a problem currently about the color space
ada: we should at least expose prefer color spaces
rik: you mean, prefered texture formats
ada: color space
Leonard: is this the final output or the colour space physically based rendering is done in
Brandel: it's the tonemapping, the final output
ada: I was pretty sure threejs is using linear these days and then convert to srgb at the end
bialpio: opengl assumes srgb. You all the math in linear space, and then turn it into srgb. We might be confused about some of this.
… webgl and opengl will always assume linear
ada: does correct colors won't work in gl
rik: yes
rik: we have to do texture formats
ada: down the road, we will have to display everything to the user.
<Leonard> Do you really want to commit to WebGL as it is [slowly] dropping out in favour of WebGPU?
ada: if you have 3 headsets connected to a computer they each will have a different profile
<Leonard> TY all - great meeting. I need to leave now.
<yonet> Thanks Leonard
Automated testing of WebXR sessions with WebDriver
ada: there's a w3c group wrt the webdriver
… it would be great if webxr could use webdriver
… I think it would be useful for developer building complext website
… but also in depth testing for webxr
… I was wondering if people think our group should do
… and then submit it to the group that manager webdriver
alcooper: we have the webxr test api
… which chrome uses to drive tests
… I believe Apple contributed to that as well
<bialpio> WebXR Test API: https://
alcooper: maybe more could be done there
ada: that's great.
… maybe that could be the basis for this type of work
… that sounds great
… should the group be involved in this
bialpio: do we know what is expected here?
… we already have the web platform tests
… it mostly mocks some data
… depending on what is needed, it's sufficient
… in chromium, you can't take advantage of this outside of web platform tests
… we don't initialize the components all the time. It only exists in our tests
ada: what I'd like to see happen, is that I can put my hmd in the computer and then run my tests
… and then dump it to a file, generate screenshots, diff them , etc
… so if I make a change, everything still works as expected
… it's not just mocking, I want to actually drive the hardware
bialpio: you can send triangles through the mock api
… we have an implementation that drives this
… you can't control ref space, input sources
… so you could write a script that moves you around
… related to plugging in the headset, ARCore allows us to record an AR session
… this is how we do testing internally
… check how the api is working
… If you go to session recording, it's very backend specific
… we don't something special, just feed the precorded session to the application
… the test API might be sufficient
ada: is this API defined?
bialpio: yes, Alex and Manish are the editors
… for all the APIs we ship, we've been updating the data
… we have a javascript implementation that implements the runtime
alcooper: the goal of the test API is to serve as a fake device
… if you want to inspect bits of state
… it drives and controls the state
… you need a special build of chrome. We only do it as a polyfill
… some of our tests, do rely on generic webdriver functionality
ada: it sounds that we want to build on top of this
… we don't want to have users wear the headset while the tests run
alcooper: we don't ship any of that
bialpio: just to clarify, it's not really a polyfill
… because the native code still runs
… we talk to a javascript backend. Just not arcore
… I'm looking at the spec and it looks like we need to relax it if we want it to run on shipped builds
ada: if we go down the selenium group, we want to say that it references this api
… but that it can ship in browser
alcooper: I believe Manish implemented this in servo
… I think it's fair to discuss a separate interface
… that can control a generic device
ada: a developer could use the test api withouth selenium without us needing to make changes there
… I'm glad we had this conversation