W3C

- DRAFT -

Invisible DOM breakout session

24 Oct 2018

Attendees

Present
Regrets
Chair
SV_MEETING_CHAIR
Scribe
heycam

Contents


<scribe> ScribeNick: heycam

domenic: there are two concepts
... this is a WIP, we realized that these are related
... we'll show you each of them, tentative APIs, not baked, then connections between them
... here are 2 approaches to simliar problems

rakina: first is the background
... there's been an increasing amount of virtual content on the web
... hte content is captured in some non-DOM data structures
... some portion is converted into DOM nodes and inserted
... this is done for some reasons, one of the biggest is saving the cost of rendering contents that aren't needed yet
... ideally any site with substantial DOM should be virtualizing
... but many things don't work
... they rely on content being in the DOM
... e.g. find in page, a11y, building a tab nav, indexability
... we're proposing a searchable invisible dom
... a new way to expose the DOM to the browser that is searchable, but invis
... we want need to pay the cost of rendering
... current API shape: an attribute

<div invisible>

scribe: if you make an element invisible, you're making its descendants invisible too
... and an event
... when things from the brwoser that might signal that an invisible element needs to be visible occurs, e.g. find in page happens
... there will be an event called "activateinvisible", sent to the ancestors have the invisible attribute
... because to make an element visible, you need tis ancestor chain to be visible
... the default action of the event will be to remove the invisible attribute

[shows example]

scribe: for items that should be visible, you add them as normal elements, for those not visible initialy\ly, you async add them with invisible
... then when the activateinvisible event happens, you get the element that needs to be activated, e.g. by looking at the position
... what about display:none?
... the biggest different is that find-in-page, anchor links, tab navigation, indexability won't work
... here, style calculation is actually done, since you need to see if it's display:none
... also thinking about a stronger version "static"
... the point is not doing additional things, e.g. custom elements, script not run, style sheets not loaded, etc.
... invisible="static"
... we have a lot of work to do, hammer out the design, reln with display locking, details on static mode

domenic: we do want to give people an overview of display locking

rniwa: for the activateinvisible, you said that when the browser finds the contents it will fire
... what about scrolling?
... fire them when you reach the point? or predictively/preemptively fire?

domenic: the current thinking is that you wouldn't fire on scrolling because there's already mechanisms to tell you've scrolled past the visible stuff
... it's cases where you don't know that, e.g. tab nav
... tab nav would just skip over the display:none things
... scrolling is easier to deal with with today's technology
... that's our current thinking
... avoid having to think about size/position of these things

JakeA: [?]

TabAtkins: templates are a much wider field
... don't want to spam them in

JakeA: being dead in a wider sense is good, it's a static thing
... real problem is that it's hard to toggle on/off

TabAtkins: also problems with templates being in arbitrary spots in the DOM

JakeA: no, you can

<JakeA> heycam: sorry, my question was, why not <tempate searchable>

JakeA: API wise it's all very early and changing
... when we have two divs, ctrl+f hello world can match, or that doesn't work?

rakina: for find in page styling is still done

JakeA: you would lay out?

rakina: you just need the style values

chris: to match the text you need to know the style
... but in order to show a highlight you need layout etc.
... that's what the event is for
... script has the opportunity to instantiate the element

JakeA: so on Ctrl+F you would layout everything?

domenic: lazily
... the number of results will grow over a few seconds, it's async

mstange: what invisible elements activate the event on?

rakina: there is a concept of "active match" for find in page
... it's just one element at a time

chris: all ancestors in the chain that have the invisible bit

JakeA: it could return a range

mstange: also on prev siblings? they might be important for layout

domenic: yes, that's what the example was about
... let's say find in page takes you way down in the middle of the list, so now the default action would be to show that one element
... the top of the list plus one element

mstange: the default action would do that?

domenic: yes, it would remove that one's invisible

mstange: how does it know it's in the range of the page if the previous siblings could affect it?

domenic: jsut talking about the default action
... this example shows, before you do the scrolling, go through and hide things at the top, show the things right around me

mstange: so let's say you don't handle this event
... the browser finds an element that has matching text
... and then it will unhide all the elements in the ancestor chain?
... so if I have a long virtaul list, with diff sized rows, if I match text in the 10th row of 20, it will unhide the 10th row, but because the prev siblings don't exist in the DOM, it won't be in the right spot

domenic: yes
... it's very debatable how useful the default behavior is

chris: let's do display locking
... I work in rendering on chrome
... display locking is something that vlad and I have been working on
... prototpying, thinking through, writiing explainers for
... to solve a similar problem
... display locking is a way of preventing rendering of a subtree of the DOM, when you know that that rendering might take a long time
... so e.g. right now if you modify DOM, then as soon as the script yields, the next render frame will cause the whole pipeline to occur
... first of all, that rendering might take as long as it takes
... depends on the size of the DOM / mutations
... second is that it happens all at once, and it's not scheduled
... for the whole time it's running, ebcause rendering and JS are on the same thread, you can't respond to user input

<bz> +q

chris: so display locking is a way to say I'm going to make an expensive DOM mutation
... I want to do it in the background, in parallely with work unrelated to that subtree
... e.g. oyu have a page with a tab UI, when you want to render the tab with new content, you might lock the root of the tab, update its DOM with a seq of JS tasks
... and because it's locked, the rendering will not occur until you unlock
... the script itself can schedule updates to the DOM
... if using a framework like virtual DOM, and might have significant computation cost, those things can be chunked as well
... can update the DOM in parts
... don't have to worry about partial renders/flickering
... when you're done, the lock will commit, in the background
... there is not a single task running on the main thread event loop
... so the event loop is responsive to user input
... also allows you to update different parts of the screen at different rates
... because while you're doing that long update to the tab, you might be typing into an input box, or doing something else, in a different part of the DOM
... there's a related project from React, trying to fix similar things -- React Fibre / React Async
... they've demod this concept of speed of UI reaction
... if you're typing into a search-as-you-type search box, dynamically interpreting the text, and trying to render autocompletes and search contents
... the complexity of that th ing might be dependent on anything
... very important when typing into the input box, that input is responsive
... an API like this will allow you to do those renders to guarantee input responsiveness, but have a best effort to render
... I'll show you the API

[shows example]

scribe: in this mode there's an element not in the tree
... you want to insert it into the tree in an async manner
... make an element, call acquireDisplayLock on it, pass it a callback and in there you should modify the DOM, then insert it into the document
... the return value of acquireDisplayLock is a Promise
... for when the callback is finished and when the cooperative commit is done, but not yet displayed on the screen

[shows second example]

scribe: the callback gets a context param
... with that you can schedule another callback
... like setTimeout but associated with the lock
... so you call context.schedule(fn), when all the callbacks are done, the lock commits and the promise resolves
... so we designed this pattern so it's difficult for devs to forget to unlock
... another thing to point out, while the rendering is locked, the state of the DOM is out of sync with what the user can see
... as a result we can't process input events

Rossen: from the PoV from the visual layer you're freezing the state
... until you commit
... what about other projections of the DOM? a11y? what are the implications?

chris: how will an a11y UI respond ...

Rossen: are these changes observable by a11y, throughout the display lock?

chris: I hadn't thought that through, I think they should not be visible

Rossen: my hope would be that if we're not doing anything visual, then a11y would match

alice: in the case of invisible, part of what to achieve here is that if you have virtualized content, you need a way to tell a11y content about it
... to generate a list of headers, go to next

domenic: the way I think about that is the sighted user sees only what's in the viewport, the AT will be aware of a list of all headings in the document
... it's not quite 1:!

bz: the assumption that the DOM and layout are in the same thread
... does this proposal still make sense when that stops being true?

chris: I believe so
... it's an async API that just waits for an update to occur
... which may or may not occur on the same thread
... if rendering were entirely done OMT, and did not block script from running again, that would solve some problem of the jank
... it wouldn't solve 1) a way for the dev to express the relative reln of two simulatenous updates, which should complete first, what should be display visually in the meantime
... that's the main thing

bz: for the actual behavior. what does the UA cache in terms of display?
... if I have a browser window, not full screeen, the page locks the DOM, the user resizes the window
... what does the user observer happening?

chris: in this example here, we have these different modes for these things
... in the case here we're locking the element before placing it in the DOM, it's newly in the DOM, so there's no visual

bz: if you lock the root of the doc?

chris: in that case, our proposal is that the browser caches the display list, or pixel representation, of the content
... if a resize occurs then there's a guttering effect

domenic: but also there was a display:none option>

<JakeA> +q

chris: if you lock an elemetn in the tree, the element isn't locked itself, it's the subtree

<AutomatedTester> RRSAgent: make minutes

chris: you can conceptually resize the element. but the subtree content would have a gutter displayed

bz: presumably this would need to apply to isolation primitive roots?

chris: only makes sense when you ahve containment isolatuion
... we're planning to require content:contents or stronger
... (style, layout, paint)
... it's a stacking contet, a containing block for fix/abspos descendants
... ebcause it's contain:paint it also has a clip

futhark: what abuot style elements inheriting down into the locked subtree?

chris: that still happens

domenic: but the pixels don't update

futhark: if you're constantly chanign things in the outer part...

chris: then it will delay the commit
... there are ways you can screw yourself up
... and cause the background commit to restart of go longer
... also the subtree you've locked, there's nothing stopping the dev mutating it outside of the lock
... or calling some style/layout inducing method
... so we plan to just have the existing behavior, it forces layout/style
... this whole API is designed to fit into the ways that at least chromium/safari work
... I'm hoping/assuming that others behave similarly
... and we don't have to introduce a whoel new set of async APIs
... or disallow the sync ones

jib: is display locking about optimization only?

chris: it's about perf yes
... in terms of schedulability and responsiveness
... it's not intended to increase parallelism per se
... but I do believe that when browsers have bg rendering techniques they maybe can use it more

jib: this would make the app get involved with optimizing rendering in a way
... does this mean the UA is barred from rendering

chris: yes
... the script is allowed to cooperatively apply its mutations to the dom

jib: for reasons other than perf? e.g. rendering elements you don't want to be shown? or just to make things faster

hris: to make things faster and keep the benefits of atomic display

domenic: browsers are barred from rendering display:none...
... Chris is talking about leaving the old pixels there
... there's a version, maybe the dev does ahead of time, you make it display:none, out a progress spinner on it, then lock it, make it display:block, fill it, then release and render
... from that perspective it doesn't seem that different

jib: can you lock things that are visible?

chris: input events are ignored
... if you wish to get those events anyway, put the handler on the ancestor element

JakeA: it's as if they're pointer-events:none

chris: some other options we're exploring, but we're leaning towards this one
... other option would be to queue up events
... which is what janking browsers do now
... that's perhaps a nice feature for some use cases
... downside is that you click but then something moves out from under you

iank: also can get double event dispatch
... e.g. a keyboard event handler
... it would fire twice

chris: or maybe twice and an event fires twice

rniwa: what happens if the user is in the middle of typing chinese in an input element that's display locked
... capture the content that's in the IME?

chris: yes

rniwa: and if they keep typing it's ignored?
... including caret?

chris: yes

rniwa: caret is blinking...

JakeA: it would stop blinking

chris: a real app should provide an affordance

??: you wouldn't lock the input element right?

chris: I think it's just a bad UX practice
... it would be a corner case that would be ugly and unexpected

rniwa: let's say you're in the middle of selecting text somewhere
... if you lock in the middle, you would freeze in that state?

chris: yes

JakeA: the technique of keeping the lock open async, you call a method to say I want to keep this open?

chris: you get the first callback with a context object
... you can call context.schedule with another callback

JakeA: that seems close to IndexedDB transacation model developers don't like
... if I'm streaming contnet from the network, and I'm generating table rows from that, and I want to display a row at a time -- would that use case be supported?

chris: if you need to wait for an external event the suspend/resume methods would be better

JakeA: I suggest looking at web locks

chris: yesterday mstange you said the case of locking the thing in the dom is more expensive
... and causes the UX fails that rniwa mentioned
... so maybe the first API should not have that feature?
... that's what you mentioned?

mstange: having an API that only supports locking an element not in the DOM would not have all the problems that rniwa mentioned
... would be easier to implement, since sometimes you don't have a display list that you can cache
... e.g. chrome doesn't have display lists for elements far away from the current viewport

annevk: why do you need to lock something in the tree? why is it expensive?

Domenic: you want to lock things in the DOM that are invisible

mstange: atm if you insert a fragment into the DOM, the browser will do the layout in one step
... might jank
... with this API you can say that it doesn't need to be done sync
... even with an impl that does layout OMT, you can say this one is less important

annevk: more like a flag that ...

chris: in a way the API is expressing lesser importance
... or parallelism and degrees of importance

mstange: as a library you create fragment, lock, insert into dom, callback fires, remove the old placeholder content, them the browser paints

chris: another use case for display locking is speculative layout
... you can lock youre content you're putting in the dom, wait for it to paint, when the promise resolves you could remove it from the dom but actually look at offsetTop
... since it's already laid out, it should be free
... to avoid off screen hacks

rniwa: why is it free?

chris: you're doing it in the promise resolution callback, so layout has been completed

Domenic: the promise fulfills when layout is complete

rniwa: I see

chris: it's not guaranteed since other script could dirty something in between
... this gets back to some details on when the resolution callback happens
... back to inviisble DOM, someone observed this lets you indicate lesser importance content
... you could imagine a declarative version of this
... maybe an async attribute on some element
... to indicate you prefer perf of the rest of the page
... then it's similar to an invisible="" element
... but instead of events, you mark some things as async, then perhaps the browser could determine when it's far off screen it should render it, then when it comes close or you do find in page, it starts to render it

Domenic: I think that's an advanced version that would be cool, the browser does everything right wtih the attribute
... other direction is what if you have an imperative API for the searchable/visible case
... the make it searchable/visible is locking
... stop doing main thread work for this
... the use cases we care about, we could display:none it instead of freeze pixels
... making it visible again, this version is better since it smears out rendering/layout over multiple frames
... so there's a lot of connection at the conceptual level

chris: thats' what the searchable option [in the example], if you lock it and invidcate it's searchable, it would be visible to UA features but don't want to render

rniwa: how scalable is this solution, when oyu have a million entries/
... I work on perf, in some issues it's not tractable to make DOM nodes for everything on the page
... 200,000 data points, creating an element for each is not ok

chris: at some point the DOM will get so big that just having that in memory isn't acceptbale
... virtualized content with so much data is not addressed by this proposal.
... but say you had a scroller for instagram posts, seems reasonable to have at most 10,000 of them...?
... my point is that because each of those things would have containment isolation, we coul make them as invisible, the layout cost when you mark something as visible again is at most proportional to 10K elements
... the containment isolation is important, you can leave them uninvisible

mstange: if you have a virtual list, avoiding layout / painting is not all, it's also the DOM
... with react, it really has to iterate over all DOM node in the element, to check if it's going to match the new DOM nodes
... so it does have some complexity proportional to the number of nodes

chris: the react virtual scroller...?

Domenic: it doesn't have that problem

jib: would any of this affect audio or video?

Domenic: the static variant we mentioned would block loads and presumably auto playing, and I think both variants would block video for being invisible

jib: are you missing the video?

Domenic: I don't know
... the static version the video doesn't get download

mstange: could we have an API that doen't even need the DOM nodes for the find in page case?

Domenic: rakina explored this
... but there are many other cases like find

RRSAgent: make minutes

RRSAgent: make minutes public

RRSAgent: make logs public

RRSAgent: make minutes

<Domenic> Thank you heycam for minuting!!

RRSAgent: make minutes

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes manually created (not a transcript), formatted by David Booth's scribe.perl version 1.154 (CVS log)
$Date: 2018/10/24 12:04:34 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.154  of Date: 2018/09/25 16:35:56  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)


WARNING: No "Present: ... " found!
You can indicate people for the Present list like this:
        <dbooth> Present: dbooth jonathan mary
        <dbooth> Present+ amy
        <amy> Present+

Found ScribeNick: heycam
Inferring Scribes: heycam

WARNING: No "Topic:" lines found.


WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth


WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: No "Topic: ..." lines found!  
Resulting HTML may have an empty (invalid) <ol>...</ol>.

Explanation: "Topic: ..." lines are used to indicate the start of 
new discussion topics or agenda items, such as:
<dbooth> Topic: Review of Amy's report


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)


[End of scribe.perl diagnostic output]