TPAC WebPlat WG Day 2 (Editing TF)

23 Oct 2018



garykac, Olli_Pettay, marcosc, AvK, BoCupp, Yves, chaals, sangwhan, Yoichi, Osato, Marijn, Kruisselbrink, Piotrek, Koszulinski, johanneswilm, Addison, YoshifumeInoue
chaals, marcosc, garykac



JW: The editing TF has been in place for a few years, trying to tackle contentEditable. We have made an inputEvent specification that is implmeneted by two browsers, and is hopefully in the final stages of getting through the process.
... So now we are looking at other problems we have with text editing on the web.

Clipboard events

GK: Clipboard… been around for a long time and made little progress.
... thre is the sync event-based API, and a new async promise=based one giving more access.
... the old one based on iffy things like execCommand can be sort of hacked but isn't going forward.
... So we need to get a replacement in place.
... Also there is a huge desire for putting iumages on the clipboard, and you can't do that. Big sticking points are long-term sdhape of the API because it needs to handfle more than images, and there are security concerns with images.
... THere is a more general security concern on writing into the clipboard, and a privacy concern on reading from it.
... You can't sanitise images in the main thread so it has to be asynch. Hence the new asynch apporach.

<garykac> https://docs.google.com/document/d/1sMcajoYAQ_M_VOIn1RLokG1sVB7skjoeviqt9aZOcDk/edit

GK: A few months ago I tried to write up the open issues.

<garykac> This document is a proposal

GK: Whenever images come up you get the other issues, and you can't solve issues by itself. So what shape can get us there?
... at a high level: images. Delayed generation of content...
... an app that can write 5 or 6 different expensive formats. You want to put a placeholder that will call for the information if you need it.
... The platforms support delayed content, so the browser should build on top of that.
... Custom data types, which is potentially a security issue. If you want to support a type on Windows, you allocate an unrecoverable resource, so there is a DoS vector for supporting lots of them.
... Chrome and Safari pickle them so you can't find out exactly what is there.
... And then there is the question of baseline data type support, and how to put data on the clpboard.
... we had relied on dataTransfer, but it's not a great fit. That overlaps with Drag and Drop and we would like to keep this separate from that.
... Goal for today - feedback on whether the proposed shape is reasonable. There are lots of details to shake down...
... and to find out if anyone is working on implementation so we can make sure they are in the dicussion.

OP: Chrome has asynch implmentation for the text part?

GK: Right. Because text is easier. we have permissions for read/write, and the ability to read and write plain text in the clipboard.
... everything else goes through the read API - how do you represent multiple items in multiple formats, even though most clipboards only handle a single item (but many have multi-format)
... Big problem with images, is to deal with security we have to transcode/sanitise it. e.g. we decode a PNG and make a new version of it with the browser's PNG code. But if you carefully craft every pixel, you don't want to lose the work, metadata, etc.
... Suggestion is to allow unsanitised images if we had a permission. E.g. if you are a PWA you can be more trusted than a random website, and that might allow e.g. image read/write directly, or custom MIME types.

JW: Looks good. Wonder if it can solve a different problem. I need to serialise text to the clipboard and I don't know where it will be pasted, maybe gdocs, a blog platform, … I know how those things will clean the data. So they all have the same MIME type but I want to custom-serialise depending on where it is going to be pasted.
... When the user copies, I recreate the thing they copy and put that into the clpboard. I can put whatever HTML I want, but until I know where it will be pasted I need to make complex HTML that I hope will come out right when pasted into the destination.
... so can I have several versions with the same MIME type on the clipboard?

GK: How would wordpress get the right thing for them, instead of teh Gdocs version?

JW: I don't know, but some kind of tagging mechanism

GK: Don't think we can fix the problem of target platforms manipulating what they get.

AvK: Seems better for target to standardise on sanitising.

JW: So take footnotes. Some things have them, some don't…

GK: For things that big, you want separate MIME types. We should document how the data gets cleaned up before it gets on the clipboard.

JW: So I could use a custom MIME type?

GK: Yep - and between websites on one browser you can do that already

Winston: Safari doesn't allow pasting custom MIME types across origins, to prevent leaking sensitive details.
... if you paste on a website you could be pasting additional sensitive data

AvK: If other browsers don't do that is it helpful?

Winston: we can revisit.

AvK: What about using streams for data?

GK: Hadn't thought of it...

SM: Thought about it. Question is whether platform APIs support it.

GK: Wanted to keep the API succinct but Apple wanted a data-type and explicity denote that it is delayed. If we have that we have a dictionary and we can add streaming later.
... instead of sending a function, that implies delayed data.
... Streaming thing sounds interesting - will look.

AvK: Write seems to be synchronous now. How does that work with delayed data.

GK: You can pass a promise as data.

AvK: Not in the spec...

GK: Oh, yeah. data can be data, JS function or a promise, as written right now.
... I haven't updated the spec, so the scribble proposal is what to look at now before I update the spec text.
... I didn't write down IDL, don't want to suggest this is a done deal. Chrome has only implemented readTextand writeText. None of this has been implemented yet..

OP: We also have code for readText and writeText, not shipping.

??: Can we do delayed clipboard? Windows requires synch message. If we delay clipboard we need to hold in the browser process.

scribe: if you have large clipboard data you should handle onload…

GK: e.g. Photoshop puts data and says "do you want it all"? That would have to be supported in UA.
... We want to make something that can support this, even if we don't actually build all the support now. So e.g. level 1 i images and custom types, and we might do delayed data later.
... so long as our proposal actually supports doing that I think we are OK. Is there a need for custom types now?
... There is a difference in trust between a browser tab and an installed PWA.
... if there are permissions associated with that, we could give more permission to a PWA. But then how much support do we give inside a tab?

AvK: Is it true that I have more trust in something I install?

GK: It's the same with native apps

CMN: Sure but we don't describe that system, and I don't think there really is a distinction people understand.

AvK: Can we block on dodgy systems?

GK: The idea is to have more permissions to control more dangerous features… so you only get it in some place where there is more trust.
... user agent makes the choice on how to decide if you are in a trusted environment.

AvK: You need to consider the UX too?

GK: Sure.

AvK: hmm. 3 prompts for a clipboard could be a problem.

GK: You can easily make a terrible permissions UI… and we should address that.

AvK: We should not try to make it someone else's problem.

GK: Like there is a manifest of permissions. So when you only do permissions when something is required, you end up with repetitive requests. Grouping them together reduces that problem.

[general discussion that we are getting into permission fatigue and security problems]

GK: If you can auto-download an image anyway, maybe we're worrying about something that is already possible.
... if that lets us remove transcoding we solve a lot of issues.
... I am hopeful that it will play out that way.
... There is still custom types consuming resources - if we throttle it, maybe.

Wins: if we pickle it we have one type. Maybe we should agree on a pickling format, which would solve this...

GK: Sort of. Native apps would need to reach into that….
... if we make it too easy to get it, why not just directly write the data?

AvK: Makes it an explicit opt-in for native apps, which seems good.

OP: Moz has a pickling format for custom data types?

GK: We can't write lots of custom types. Because it allows a DoS attack.

AvK: E.g. we allow PNG< and an app puts in bytes as PNG, the user pastes that data into an app and the bytes bite the user.
... so instead of writing PNG, we write app/ourformat, and in there we know how to read out the data an find a PNG in it.
... So if we had a standard picklign format like webClip, people would have decoders that are built to decode securely because content comes from the Web.

BoCupp: This seems mixing issues. Who is going to use a secure decoder somewhere, but not in other places?
... so if the browser doesn't take responsibility for images, would you still pickle for custom mime types?

GK: Yes.
... should you be able to download an image payload?

W: We have stuff in Safari that slow this down to see if the user is sure.

SM: Does that show up for images?

W: nope.

GK: We also have video/audio.

AvK: Would we only allow images that browsers can decode, or allow bytes with the MIME type?
... in your model, you check if the browser can decode it. If you have arbitrary bytes, then you have a different concern about what is in it.

SM: Treatng image types as image types seems fine.

GK: This isn't hypothetical, the vector has been used.

SM: Don't know if pickling will go well if you cannot paste into a native app.

GK: Pickling solves a problem of regitering arbitrary mime types allowing DoS.

JW: Is pickling stopping people read the data?

GK: Nope. It means we only need to register one MIME type to handle all kinds of data.

BC: WHy not just limit the number of MIME types you can register?

[question is how many types does it take to create a problem?]

Avk: Websites will target native types and drop bad code into them...

BC: I don't think native apps assume that they have trusted content from the clipboard, because it is already a shared resource.

Avk: Like the idea of pickled format, because native apps ahve to opt in and avoid trust.

Wins: agree with AvK. Have seen apps who think they are the only ones writing their format, and look, problems.

Avk: If we can protect the OS, we should, and not just blame them for getting it wrong.

BC: To back up ti the point about delayed data blocking the UI - you are advertising a lot of stuff, but if nobody asks for the whole collection, you save doing the work. Unblocking the UI isn't the value, it is saving the up-front work of building the copy...

GK: Yeah, that's the intent.

AvK: Wonder if there are concerns on fingerprinting…

GK: Next steps: standard pickling? sanititation, what do we do?

BC: Want sanitisation out. Pickling limits compatibility of the API, so you have to go back and rebuild everything before you can work with the web.

Wins: We know interop comes at a price of apps being able to get into a bad state.

BC: You can't write text without permissions anyway. I am not sure we need to have the restrictions, when you already have someone granting permission to use the clipboard.

Avk: but the user cannot reason about the security of native apps.

BC: I claim that already happens with script today.

JW: The security concerns for images, woud they apply for rich text? Does the pickling apply to that?

GK: The question is what should we need to pickle?

BC: no, not rich text formats either. I think we should be relying on the permissions API

Avk: I think we would object a lot.

JW: If I close the browser, and I have delayed data on clipboard, what happens?

GK: you get a notice saying "before I go, do you want me to paste the data into the clpboard?"

BC: So do native apps run through all 24 formats when you shut down?
... or just the core format

GK: What I ahve seen so far is all or nothing

[scribing lacuna]

scribe: you can also use a promise.

JW: Say I have a very unusual high-cost format for a few users? That's a cost.

GK: I find it hard to be concerned about that

BC: We have metrics to check how long things take to close down.
... how do we know if the time taken is reasonable for a hard job, or just something going wrong.
... think we need to think through the UI

GK: But I think that is browser not spec.
... will be following up with this through bugs on the clipboard spec. So if you know someone interested, please direct them to the bug tracker for that.
... I know MS has been asking for this, so that e.g. Office365 can cut/copy/paste in Chrome… and similar things that people expect to be easy.
... To get a sense on where are we on sanitisation I will follow up, but assume that there is likely a desire to samitise images.

Avk: If we pickled them as well?

GK: That is the alternative path to sanitise.
... Note that we also need to think about a lot more formats - video, audio, …
... Also, giant pickles always are not necessarily good. So we figure out whether to put up unsanitised, or we pickle some things, and sanitise sometimes, or blend these things.
... if we don't have to sanitise we can do a lot more nicely. If we can be comfortable rate-limiting requests for custom mime types that might make life better too.

<smaug> https://www.w3schools.com/tags/tryit.asp?filename=tryhtml5_a_download

Input events i18n

i18n issues on input events

<marcosc> scribe: marcosc


"Character is not defined"

ap: character can have different meaning depending on context... it means one thing in unicode, it means something else to other people. Without being clear, the spec can get into trouble.
... so, you have things like "insert transpose"

jw: The reason we don't need to be more specific is because we are not defining what a user sees as a character. We are saying that the user gave this command to transpose this browser... and the user agent may do its own thing.

chaals: so, if you have a set of "characters" and you want to shift characters.

ap: if you are not specific, "character" could mean a bunch of different things. And browsers, and other specs, are very specific about.

<chaals> MC: As a browser vendor, internally we have to know what it is we are trying to swap, so we need a definition.

jw: we are not trying to specify what the browser does at that particular point. We've only looked at one browser. So, JS could intervene, and change various things.

ap: I understand what you are saying - but if you are not clear what you mean, it could lead to interoperability issues. Otherwise, you could get issues with unicode characters that are composed as multiple code points, so if it's unclear, then you could end up with the wrong "character" representations when transposition happens.

jw: if you have a transpose functionality in the browser, then an event should fire a "insert transpose" event.

pc: which browsers have this insert transpose events?

??: Chrome and Safari

pc: with the event, if there is no clear guidance to developers for when the author prevents the default, then it's unclear what they should do.

chaals: so, maybe you can rely on the browser's defintion on what it is transposing - so you can rely on what the browser is doing, a developer will have some idea what they are going to get with regards to a "character" (and what a character is).

jw: the event would work on the target range. But we would need to be sure that it's a particular thing, like a "graphine" (?)

ap: yes, and but my suggestion is to be clear what these are going to be "e.g., graphine clusters".

jw: so would be good to add a note to the spec?

<addison> https://w3c.github.io/bp-i18n-specdev/#characters


"'deleteWordForwards' and 'deleteWordBackwards' not well defined"

ap: again, the same issue... "word" is not defined.

jw: yes, sometimes a JavaScript library will have a better idea of what a word might be. Which is why one might want to intercept that at times.

ap: ECMA is working on a word delimiter. We have discussed with them why it's hard.


"Code points and graphemes in backward deletion"

ap: there has been discussion since this that you really mean "code point" some of the time. Challenge being four things forward selection, backwards selection, forward/backward deletion. So dealing with that would be not to select, for instance, half a character. The question starts to become, are there cases where there are code points where you don't want to delete by themselves... like with emojis, which have complext corner cases.

YI: as unicode doesn't seems keen to define this, so it's kinda up to the browser vendors here to define how it works here.

ap: we talked to the unicode folks about this

[ cross talk]

jw: there is an explanatory note about how to deal with forward delete, etc. but it's not supposed to say exactly how UAs should behave.

YI: we need some kind of specification in that defines some kind of browser behavior.

chaals: that feels like a separate task. What would be great would be something like having a list of characters and how to deal with delete things when such sequence is encountered

ap: I think it makes sense to describe that somewhere, but not here in the spec. The i18n WG may document this.


How to carry directional information with a string and handle insertion

ap: strings have, for instance, strong directional information. It would be great if directional information from the context was carried along. However, depending on where it gets pasted, the it may behave differently of what's expected.

YI: we might need to consider the bidi boundary

jw: if we want to standardize how it behaves, what do we need to do?

ap: the unicode algorithm defines this

A diagram gets drawn input carets ... discussion about how input carets work with regards to unicode bidi algorithm

scribe: right to left... left to right... carets...

jw: so, this is up to the user agent until it hits javascript. Then after javascript does the insertion, it's unclear what js is supposed to do.

ap: it's actually quite well understood by user agents, because of unicode. So, when you paste into left to right/right to left, the user agent knows how to deal with it, even tho it can be confusing.

pc: when copy/paste happens, you will get a bunch of base direction information. But this will may be different with just plain text.

YI: it can change line by line...
... we should nonetheless define the behavior for UAs regarding the events
... I don't know how well all this is defined already

chaals: yes, this stuff can be quite weird
... more drawings on whiteboard ...
... more talk of left to right/right to left...

ap: ... so, when you paste stuff in, you can end up with a bunch of additional control characters
... bidi characters don't behave like other characters when carets move over them (they are skipped)

jw: we could add more explanatory notes, but it feels like this should be defined elsewhere.

cp: there are platform implications (e.g., paste) - but it's like you are putting a different layer thing on top, like "transponse"
... you are also producing a set of ranges, which would be equivalent to what the browsers would have done.
... ... so are we going to equivalent between the js implementation and the browser dom implementation

jw: so, in content editable, this also comes up... like when you delete across nodes, how to merge multiple nodes (e.g., two paragraphs)
... when we file an issue on you, we also track in our own spec. And our own issue we have marked it to be closed as being satisfied with the response

MC: johanneswilm, could you please go through each of the issues and just make a note about what was decided.

<chaals> scribe: chaals

Content-editable disabled

JW: We should have a list of the things you want to disable, so they don't appear in context menus etc. I assmebled a list of what JS editors want in the list and we have it.
... so instead of a space-separated attribute, should we make it an API, on the document or on an individual contentEditable element.
... I don't mind having it on an element, but on a whole document it makes no sense because you may have differently configured editors running on different parts of the document.

Winston: did a Proof of concept for the API…

JW: Piotr and I looked at it and it seems to make sense

PK: It solves a bunch of issues for us. Regarding the API, we aren't that fussed about whether it is a method or attribute.
... but if we have a list of format options and disable them one by one, we get stuk if a borwser introduces a new one not on our list.

JW: Seems editors want a whitelist of what is enabled, and browsers want to have a blacklist of what to disable, so they can ship a new feature and have it work normally...
... this exists on various platforms - you see it clearly on the touchbar on Mac.

Winston does a demo.

Winston: There is a lot of compat risk with editors, because yu see a thing that gets broken. We want to disable the controls.
... we might want to make justify more granular, to have right/center/left …
... we would make the same thig apply to context menu

MC: That list could change what is there. So if you take things away new random things appear.

Winston: They would appear, but if implementors want to turn them off they can. The model we have would allow for finer granularity as well as broader categories like "justification", "font-variants" ...

CMN: Doing this, if you introduce a new formatting command we hope that it comes with a label we can use in here, and where it fits in broader catgegories...

BC: In favour of future-prrofing - start with nothing and list the things that you do support.

GK: problem is the OS adds a new thing and you can't inherit it.

JW: Editors don't just hold teh HTML, they have a further step in the middle. If you add a thing they don't know,, you get potential breakage.
... so they check for other changes, and revert them if they don't understand.

Winston: using beforeInput imagine JS using beforeInput to understand how to handle things.

BC: This is about cleaning up the browser UI so it doesn't have broken options.

Winston: we want to allow browsers to ship new UI by default.
... so we can expose a thing the page already knows how to expose.

JW: The issue comes when there is a new feature that wasn't known when you built the editor.
... But we would like something even if it isn't as much as we would like.

CMN: Propose we go forward with a blacklist of atomic things, but look in the future to add some classes like fontVariant, Justification, and ask that as people introduce new things in the future they fit them into classes if they make sense

JW: Alternative, disable all, and then allow enabling one by one.

YI: Or provide a list of commands browsers have, so you can dynamically query it and work out what to do.

CMN: Nice. But again, it will mean in implementation new browser features are typically disabled.

YI: There is a list of commands ion MacOS.

PK: Problem is that the list can change…
... if there is a new button we cannot handle, we generally throw away the effect of it.

YI: Is this nestable, or can we restrict it to the root of where it appears. Also, Shadow COM…?

CMN: Nestable is a nice feature, but hard to think about.

YI: Shadow DOM is meant to be isolated, so it gets hairy.

RESOLUTION: we are not going to allow nesting.
... We want an API, so it can tell you what commands are available, on an editable element

MC: There is a fingerprinting vector...

OP: But we can add features that don't actually work in the list.

DesignMode …

JW: now we have an API instead of an attribute, we can deal with designMode on...

<garykac> scribe: garykac



"copy/cut and text-transform"

bo: likes idea of keeping innerText aligned with the clipboard

innerText is a decent text representation of the DOM

JW: JS editors said that they preferred the html as it was originally, with no CSS transforms.

eg: if there is CSS that changes text to be all upper case, then they don't want that applied - they want the original html

moving the styles to be inline is OK, just don't change the raw data

bo: text-transform can be moved inline - that seems fine

innerHtml is defined in the HTML spec

<BoCupp> https://html.spec.whatwg.org/multipage/dom.html#the-innertext-idl-attribute

<BoCupp> Step 4 says to account for text-transform when collecting text

Want to spec how HTML is serialized to be placed on the clipboard

Need new tracking issue for that.


<BoCupp> Relevent issue https://github.com/w3c/csswg-drafts/issues/627

Next issue : https://github.com/w3c/editing/issues/176

<PiotrekKoszulinski> https://docs.google.com/document/d/1PKRVAXtqR2vkkI3qE5P7UH24Cdo03ebPUqCWlxwJmkY/edit?usp=sharing

<smaug> https://github.com/whatwg/html/pull/2742

<smaug> https://github.com/whatwg/html/issues/2730

<smaug> https://developer.mozilla.org/en-US/docs/Web/HTML/Element/menuitem

<PiotrekKoszulinski> Demo: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/menu

<PiotrekKoszulinski> Create a ticket with a content of this presentation that browsers should help in maximising the available space.

<PiotrekKoszulinski> Create a ticket for retrieving/providing geometry of editing UI

github issue for "maximizing the available space": https://github.com/w3c/editing/issues/182

Please add more info/context to it.

github for "retrieving/providing geometry of editing UI" : https://github.com/w3c/editing/issues/183

<PiotrekKoszulinski> https://docs.google.com/document/d/1yCGVFRUy-ouMpWvLyBwMANsr5h7xKmP08I9ZIDv8HIE/edit?usp=sharing

Range plus selection decorations

<marcosc> https://github.com/w3c/webplatformwg/issues/120

Summary of Action Items

Summary of Resolutions

  1. we are not going to allow nesting.
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2018/11/02 16:21:53 $