<scribe> ScribeNick: heycam
domenic: there are two
concepts
... this is a WIP, we realized that these are related
... we'll show you each of them, tentative APIs, not baked,
then connections between them
... here are 2 approaches to simliar problems
rakina: first is the
background
... there's been an increasing amount of virtual content on the
web
... hte content is captured in some non-DOM data
structures
... some portion is converted into DOM nodes and inserted
... this is done for some reasons, one of the biggest is saving
the cost of rendering contents that aren't needed yet
... ideally any site with substantial DOM should be
virtualizing
... but many things don't work
... they rely on content being in the DOM
... e.g. find in page, a11y, building a tab nav,
indexability
... we're proposing a searchable invisible dom
... a new way to expose the DOM to the browser that is
searchable, but invis
... we want need to pay the cost of rendering
... current API shape: an attribute
<div invisible>
scribe: if you make an element
invisible, you're making its descendants invisible too
... and an event
... when things from the brwoser that might signal that an
invisible element needs to be visible occurs, e.g. find in page
happens
... there will be an event called "activateinvisible", sent to
the ancestors have the invisible attribute
... because to make an element visible, you need tis ancestor
chain to be visible
... the default action of the event will be to remove the
invisible attribute
[shows example]
scribe: for items that should be
visible, you add them as normal elements, for those not visible
initialy\ly, you async add them with invisible
... then when the activateinvisible event happens, you get the
element that needs to be activated, e.g. by looking at the
position
... what about display:none?
... the biggest different is that find-in-page, anchor links,
tab navigation, indexability won't work
... here, style calculation is actually done, since you need to
see if it's display:none
... also thinking about a stronger version "static"
... the point is not doing additional things, e.g. custom
elements, script not run, style sheets not loaded, etc.
... invisible="static"
... we have a lot of work to do, hammer out the design, reln
with display locking, details on static mode
domenic: we do want to give people an overview of display locking
rniwa: for the activateinvisible,
you said that when the browser finds the contents it will
fire
... what about scrolling?
... fire them when you reach the point? or
predictively/preemptively fire?
domenic: the current thinking is
that you wouldn't fire on scrolling because there's already
mechanisms to tell you've scrolled past the visible stuff
... it's cases where you don't know that, e.g. tab nav
... tab nav would just skip over the display:none things
... scrolling is easier to deal with with today's
technology
... that's our current thinking
... avoid having to think about size/position of these
things
JakeA: [?]
TabAtkins: templates are a much
wider field
... don't want to spam them in
JakeA: being dead in a wider
sense is good, it's a static thing
... real problem is that it's hard to toggle on/off
TabAtkins: also problems with templates being in arbitrary spots in the DOM
JakeA: no, you can
<JakeA> heycam: sorry, my question was, why not <tempate searchable>
JakeA: API wise it's all very
early and changing
... when we have two divs, ctrl+f hello world can match, or
that doesn't work?
rakina: for find in page styling is still done
JakeA: you would lay out?
rakina: you just need the style values
chris: to match the text you need
to know the style
... but in order to show a highlight you need layout etc.
... that's what the event is for
... script has the opportunity to instantiate the element
JakeA: so on Ctrl+F you would layout everything?
domenic: lazily
... the number of results will grow over a few seconds, it's
async
mstange: what invisible elements activate the event on?
rakina: there is a concept of
"active match" for find in page
... it's just one element at a time
chris: all ancestors in the chain that have the invisible bit
JakeA: it could return a range
mstange: also on prev siblings? they might be important for layout
domenic: yes, that's what the
example was about
... let's say find in page takes you way down in the middle of
the list, so now the default action would be to show that one
element
... the top of the list plus one element
mstange: the default action would do that?
domenic: yes, it would remove that one's invisible
mstange: how does it know it's in the range of the page if the previous siblings could affect it?
domenic: jsut talking about the
default action
... this example shows, before you do the scrolling, go through
and hide things at the top, show the things right around me
mstange: so let's say you don't
handle this event
... the browser finds an element that has matching text
... and then it will unhide all the elements in the ancestor
chain?
... so if I have a long virtaul list, with diff sized rows, if
I match text in the 10th row of 20, it will unhide the 10th
row, but because the prev siblings don't exist in the DOM, it
won't be in the right spot
domenic: yes
... it's very debatable how useful the default behavior is
chris: let's do display
locking
... I work in rendering on chrome
... display locking is something that vlad and I have been
working on
... prototpying, thinking through, writiing explainers
for
... to solve a similar problem
... display locking is a way of preventing rendering of a
subtree of the DOM, when you know that that rendering might
take a long time
... so e.g. right now if you modify DOM, then as soon as the
script yields, the next render frame will cause the whole
pipeline to occur
... first of all, that rendering might take as long as it
takes
... depends on the size of the DOM / mutations
... second is that it happens all at once, and it's not
scheduled
... for the whole time it's running, ebcause rendering and JS
are on the same thread, you can't respond to user input
<bz> +q
chris: so display locking is a
way to say I'm going to make an expensive DOM mutation
... I want to do it in the background, in parallely with work
unrelated to that subtree
... e.g. oyu have a page with a tab UI, when you want to render
the tab with new content, you might lock the root of the tab,
update its DOM with a seq of JS tasks
... and because it's locked, the rendering will not occur until
you unlock
... the script itself can schedule updates to the DOM
... if using a framework like virtual DOM, and might have
significant computation cost, those things can be chunked as
well
... can update the DOM in parts
... don't have to worry about partial renders/flickering
... when you're done, the lock will commit, in the
background
... there is not a single task running on the main thread event
loop
... so the event loop is responsive to user input
... also allows you to update different parts of the screen at
different rates
... because while you're doing that long update to the tab, you
might be typing into an input box, or doing something else, in
a different part of the DOM
... there's a related project from React, trying to fix similar
things -- React Fibre / React Async
... they've demod this concept of speed of UI reaction
... if you're typing into a search-as-you-type search box,
dynamically interpreting the text, and trying to render
autocompletes and search contents
... the complexity of that th ing might be dependent on
anything
... very important when typing into the input box, that input
is responsive
... an API like this will allow you to do those renders to
guarantee input responsiveness, but have a best effort to
render
... I'll show you the API
[shows example]
scribe: in this mode there's an
element not in the tree
... you want to insert it into the tree in an async
manner
... make an element, call acquireDisplayLock on it, pass it a
callback and in there you should modify the DOM, then insert it
into the document
... the return value of acquireDisplayLock is a Promise
... for when the callback is finished and when the cooperative
commit is done, but not yet displayed on the screen
[shows second example]
scribe: the callback gets a
context param
... with that you can schedule another callback
... like setTimeout but associated with the lock
... so you call context.schedule(fn), when all the callbacks
are done, the lock commits and the promise resolves
... so we designed this pattern so it's difficult for devs to
forget to unlock
... another thing to point out, while the rendering is locked,
the state of the DOM is out of sync with what the user can
see
... as a result we can't process input events
Rossen: from the PoV from the
visual layer you're freezing the state
... until you commit
... what about other projections of the DOM? a11y? what are the
implications?
chris: how will an a11y UI respond ...
Rossen: are these changes observable by a11y, throughout the display lock?
chris: I hadn't thought that through, I think they should not be visible
Rossen: my hope would be that if we're not doing anything visual, then a11y would match
alice: in the case of invisible,
part of what to achieve here is that if you have virtualized
content, you need a way to tell a11y content about it
... to generate a list of headers, go to next
domenic: the way I think about
that is the sighted user sees only what's in the viewport, the
AT will be aware of a list of all headings in the
document
... it's not quite 1:!
bz: the assumption that the DOM
and layout are in the same thread
... does this proposal still make sense when that stops being
true?
chris: I believe so
... it's an async API that just waits for an update to
occur
... which may or may not occur on the same thread
... if rendering were entirely done OMT, and did not block
script from running again, that would solve some problem of the
jank
... it wouldn't solve 1) a way for the dev to express the
relative reln of two simulatenous updates, which should
complete first, what should be display visually in the
meantime
... that's the main thing
bz: for the actual behavior. what
does the UA cache in terms of display?
... if I have a browser window, not full screeen, the page
locks the DOM, the user resizes the window
... what does the user observer happening?
chris: in this example here, we
have these different modes for these things
... in the case here we're locking the element before placing
it in the DOM, it's newly in the DOM, so there's no visual
bz: if you lock the root of the doc?
chris: in that case, our proposal
is that the browser caches the display list, or pixel
representation, of the content
... if a resize occurs then there's a guttering effect
domenic: but also there was a display:none option>
<JakeA> +q
chris: if you lock an elemetn in the tree, the element isn't locked itself, it's the subtree
<AutomatedTester> RRSAgent: make minutes
chris: you can conceptually resize the element. but the subtree content would have a gutter displayed
bz: presumably this would need to apply to isolation primitive roots?
chris: only makes sense when you
ahve containment isolatuion
... we're planning to require content:contents or
stronger
... (style, layout, paint)
... it's a stacking contet, a containing block for fix/abspos
descendants
... ebcause it's contain:paint it also has a clip
futhark: what abuot style elements inheriting down into the locked subtree?
chris: that still happens
domenic: but the pixels don't update
futhark: if you're constantly chanign things in the outer part...
chris: then it will delay the
commit
... there are ways you can screw yourself up
... and cause the background commit to restart of go
longer
... also the subtree you've locked, there's nothing stopping
the dev mutating it outside of the lock
... or calling some style/layout inducing method
... so we plan to just have the existing behavior, it forces
layout/style
... this whole API is designed to fit into the ways that at
least chromium/safari work
... I'm hoping/assuming that others behave similarly
... and we don't have to introduce a whoel new set of async
APIs
... or disallow the sync ones
jib: is display locking about optimization only?
chris: it's about perf yes
... in terms of schedulability and responsiveness
... it's not intended to increase parallelism per se
... but I do believe that when browsers have bg rendering
techniques they maybe can use it more
jib: this would make the app get
involved with optimizing rendering in a way
... does this mean the UA is barred from rendering
chris: yes
... the script is allowed to cooperatively apply its mutations
to the dom
jib: for reasons other than perf? e.g. rendering elements you don't want to be shown? or just to make things faster
hris: to make things faster and keep the benefits of atomic display
domenic: browsers are barred from
rendering display:none...
... Chris is talking about leaving the old pixels there
... there's a version, maybe the dev does ahead of time, you
make it display:none, out a progress spinner on it, then lock
it, make it display:block, fill it, then release and
render
... from that perspective it doesn't seem that different
jib: can you lock things that are visible?
chris: input events are
ignored
... if you wish to get those events anyway, put the handler on
the ancestor element
JakeA: it's as if they're pointer-events:none
chris: some other options we're
exploring, but we're leaning towards this one
... other option would be to queue up events
... which is what janking browsers do now
... that's perhaps a nice feature for some use cases
... downside is that you click but then something moves out
from under you
iank: also can get double event
dispatch
... e.g. a keyboard event handler
... it would fire twice
chris: or maybe twice and an event fires twice
rniwa: what happens if the user
is in the middle of typing chinese in an input element that's
display locked
... capture the content that's in the IME?
chris: yes
rniwa: and if they keep typing
it's ignored?
... including caret?
chris: yes
rniwa: caret is blinking...
JakeA: it would stop blinking
chris: a real app should provide an affordance
??: you wouldn't lock the input element right?
chris: I think it's just a bad UX
practice
... it would be a corner case that would be ugly and
unexpected
rniwa: let's say you're in the
middle of selecting text somewhere
... if you lock in the middle, you would freeze in that
state?
chris: yes
JakeA: the technique of keeping the lock open async, you call a method to say I want to keep this open?
chris: you get the first callback
with a context object
... you can call context.schedule with another callback
JakeA: that seems close to
IndexedDB transacation model developers don't like
... if I'm streaming contnet from the network, and I'm
generating table rows from that, and I want to display a row at
a time -- would that use case be supported?
chris: if you need to wait for an external event the suspend/resume methods would be better
JakeA: I suggest looking at web locks
chris: yesterday mstange you said
the case of locking the thing in the dom is more
expensive
... and causes the UX fails that rniwa mentioned
... so maybe the first API should not have that feature?
... that's what you mentioned?
mstange: having an API that only
supports locking an element not in the DOM would not have all
the problems that rniwa mentioned
... would be easier to implement, since sometimes you don't
have a display list that you can cache
... e.g. chrome doesn't have display lists for elements far
away from the current viewport
annevk: why do you need to lock something in the tree? why is it expensive?
Domenic: you want to lock things in the DOM that are invisible
mstange: atm if you insert a
fragment into the DOM, the browser will do the layout in one
step
... might jank
... with this API you can say that it doesn't need to be done
sync
... even with an impl that does layout OMT, you can say this
one is less important
annevk: more like a flag that ...
chris: in a way the API is
expressing lesser importance
... or parallelism and degrees of importance
mstange: as a library you create fragment, lock, insert into dom, callback fires, remove the old placeholder content, them the browser paints
chris: another use case for
display locking is speculative layout
... you can lock youre content you're putting in the dom, wait
for it to paint, when the promise resolves you could remove it
from the dom but actually look at offsetTop
... since it's already laid out, it should be free
... to avoid off screen hacks
rniwa: why is it free?
chris: you're doing it in the promise resolution callback, so layout has been completed
Domenic: the promise fulfills when layout is complete
rniwa: I see
chris: it's not guaranteed since
other script could dirty something in between
... this gets back to some details on when the resolution
callback happens
... back to inviisble DOM, someone observed this lets you
indicate lesser importance content
... you could imagine a declarative version of this
... maybe an async attribute on some element
... to indicate you prefer perf of the rest of the page
... then it's similar to an invisible="" element
... but instead of events, you mark some things as async, then
perhaps the browser could determine when it's far off screen it
should render it, then when it comes close or you do find in
page, it starts to render it
Domenic: I think that's an
advanced version that would be cool, the browser does
everything right wtih the attribute
... other direction is what if you have an imperative API for
the searchable/visible case
... the make it searchable/visible is locking
... stop doing main thread work for this
... the use cases we care about, we could display:none it
instead of freeze pixels
... making it visible again, this version is better since it
smears out rendering/layout over multiple frames
... so there's a lot of connection at the conceptual level
chris: thats' what the searchable option [in the example], if you lock it and invidcate it's searchable, it would be visible to UA features but don't want to render
rniwa: how scalable is this
solution, when oyu have a million entries/
... I work on perf, in some issues it's not tractable to make
DOM nodes for everything on the page
... 200,000 data points, creating an element for each is not
ok
chris: at some point the DOM will
get so big that just having that in memory isn't
acceptbale
... virtualized content with so much data is not addressed by
this proposal.
... but say you had a scroller for instagram posts, seems
reasonable to have at most 10,000 of them...?
... my point is that because each of those things would have
containment isolation, we coul make them as invisible, the
layout cost when you mark something as visible again is at most
proportional to 10K elements
... the containment isolation is important, you can leave them
uninvisible
mstange: if you have a virtual
list, avoiding layout / painting is not all, it's also the
DOM
... with react, it really has to iterate over all DOM node in
the element, to check if it's going to match the new DOM
nodes
... so it does have some complexity proportional to the number
of nodes
chris: the react virtual scroller...?
Domenic: it doesn't have that problem
jib: would any of this affect audio or video?
Domenic: the static variant we mentioned would block loads and presumably auto playing, and I think both variants would block video for being invisible
jib: are you missing the video?
Domenic: I don't know
... the static version the video doesn't get download
mstange: could we have an API that doen't even need the DOM nodes for the find in page case?
Domenic: rakina explored
this
... but there are many other cases like find
RRSAgent: make minutes
RRSAgent: make minutes public
RRSAgent: make logs public
RRSAgent: make minutes
<Domenic> Thank you heycam for minuting!!
RRSAgent: make minutes
This is scribe.perl Revision: 1.154 of Date: 2018/09/25 16:35:56 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) WARNING: No "Present: ... " found! You can indicate people for the Present list like this: <dbooth> Present: dbooth jonathan mary <dbooth> Present+ amy <amy> Present+ Found ScribeNick: heycam Inferring Scribes: heycam WARNING: No "Topic:" lines found. WARNING: No meeting chair found! You should specify the meeting chair like this: <dbooth> Chair: dbooth WARNING: No date found! Assuming today. (Hint: Specify the W3C IRC log URL, and the date will be determined from that.) Or specify the date like this: <dbooth> Date: 12 Sep 2002 People with action items: WARNING: No "Topic: ..." lines found! Resulting HTML may have an empty (invalid) <ol>...</ol>. Explanation: "Topic: ..." lines are used to indicate the start of new discussion topics or agenda items, such as: <dbooth> Topic: Review of Amy's report WARNING: IRC log location not specified! (You can ignore this warning if you do not want the generated minutes to contain a link to the original IRC log.)[End of scribe.perl diagnostic output]