Virtual Keyboard Control
Facilitator: Bo Cupp
Speakers: Anupam Snigdha
Discuss the Virtual Keyboard API and Virtual Keyboard PolicyMinutes (including discussions that were not audio-recorded)
Transcript
Okay, hi everyone.
My name is Bo, and I am a engineering manager on the Microsoft Edge Team, working on editing and put in DOM, and with me I've got, Hi everyone, I'm Anupam, and I'm also already put in DOM Team Microsoft.
Okay, let's say our objective for today is to try to make some progress on issues that we haven't closed yet related to controlling the VirtualKeyboard.
We're also happy to identify any new issues if folks, spot them as we give a presentation to sort of warm everybody up.
We're also looking for collaborators if somebody wants to help us turn our explainers into a spec and especially if you have experience doing that sort of thing, we're happy to collaborate with you.
And so the way that we thought we'd go about it is doing a presentation and demo, and then answering questions.
We think the presentation takes somewhere between 10 and 15 minutes, and then we'll get into issues discussion.
So in terms of the scope, there's really two parts to the proposal.
I'll start off with the first part.
The first part is Adapting the Layout of a webpage to changes in visibility for the Virtual Keyboard.
And so here's an illustration that is a mock-up of a foldable device that has two screens, the black rectangles represent the left and a right screen of this device, and we use this dual-screen device because it makes it real easy to see the problems.
On such a device if we don't adopt a proposal like we've put forth here then we expect something like the following to happen.
And I'll do this back and forth a few times just so you can see it.
When the keyboard is coming up, what we're showing here is a reduction in the height of the Visual Viewport, and a sliding up of the Layout Viewport.
And so the layout of the page is unchanged it's just shifted and you can see that there's some space on the left screen that we've added to the bottom.
And so this is so that we have room to shift the Layout Viewport up so that all of the content which is rendered within it is above the virtualKeyboard.
A couple of things are undesirable about this, obviously on the left screen we are wasting some space.
And the second thing that may be is a little less obvious is that as the user pans around because we now have a Visual Viewport that is smaller than our Layout Viewport, the user can take their finger and pan this content around so that they can see portions of the Layout Viewport that are not currently visible and that gets a little bit awkward whenever you have subscrollers which it looks like we have in this mock-up app.
Because sometimes your finger will pan and move the subscroller while other times it will move the contents to the Layout Viewport itself.
So we think that we can do better than this experience and it requires some cooperation from the author what we would like to have is this, and I'll just do that a few times.
In here what you see is that the Visual Viewport and the Layout Viewport are remaining unchanged and that some portion of the webpage is being relocated in response to the virtualkeyboard showing.
(keyboard clacking) So here's the before and after we think the after looks better than the before, (keyboard clacking) and the way in which the author would cooperate with us in order to achieve the desired result is in the current proposal, we have an opt-in mechanism accessible from script on a new virtualKeyboard interface which we have hang off of the navigator object.
So the author says overlaysContent true and it's named as such because now when set to true when the virtualKeyboard appears if the author changes nothing about their page, the virtualKeyboard will be overlaying and obscuring some of the content of the page.
In order to adapt to that we have some CSS environment variables, so here we're showing keyboard-inset-height is adjusting the bottom of that fixed position action-bar that you saw moving up and down.
And because I've got a dual-screen device, you'll see the media queries just above it.
There were some questions about, well how do you know what screen the keyboard is showing on?
And when it's appropriate to be adapting your layout to it in foldable scenarios.
So here we're borrowing apart of a separate proposal it's not part of the virtualKeyboard control but we're borrowing CSS environment variables using them in media quiries to detect what mode we're in on the foldable and whether or not the position of the keyboard is on the right or if you had a desire to adjust it on the left and so forth, you would add additional media in quiries like you see there.
Okay so that is the mechanism by which the author can adjust the page, later in some code you'll see that we have a JavaScript version of the events as well, instead of just CSS environment variables.
And the on last point I wanna make about adapting layout is that while I showed some demos on the dual-screen device we think it's equally applicable to single-screen devices as well (keyboard clacking) so that when the keyboard comes up this was an application that was sized maybe using viewport units to fit your viewport, and when the keyboard comes up it's introduced some slop on the page.
So that the user can now paint it around and like I said earlier with subscrollers, it's not necessarily a desirable experience.
And so if you were to do something like this, instead, you can see that the offer relocates a portion of their page, and they could additionally, that looks like a subscroller region on the left that might have a list of messages.
You could add some space to the bottom of that scroller to ensure that the user can swipe until all of the messages were up above the keyboard.
That increases the layout stability and we think makes for a more usable application.
So that's basically the pitch for part one.
(keyboard clacking) For part two, I'm gonna turn it over to Anupam here, I'll just escape out of the presentation for a second and he's going to demo some stuff about having the author control when the virtualKeyboard is hidden or shown.
(chair clonks) So in this demo we have a new attribute on the element called virtualkeyboardpolicy.
And virtualkeyboardpolicy has like two values, auto and manual.
Virtualkeyboardpolicy, if it's set to auto, it behaves just like the default editable element.
So if user taps on the virtualkeyboardpolicy order the keyboard comes up automatically.
And if user taps on the readonly element, then keyboard dismisses automatically.
So where there was not have to handle anything when the focus is on virtualkeyboardpolicy auto with respect to the visibility of the keyboard.
But for virtualkeyboardpolicy goes to manual, this provides more control over the visibility of the keyboard.
So if virtualkeyboardpolicy manual is set to focus, it doesn't change the state of the keyboard.
So is it now the keyboard is hidden, if I set focused on the element with virtualkeyboardpolicy's manual it stays in.
So it doesn't change the state of the keyboard but if it's up then I switch focus to a manual policy element, it stays up.
So virtualkeywordpolicy doesn't change the state of the virtualkeyboard and using the virtualkeyboardpolicy's manual and this behavior, user can use API's that we have defined on virtualKeyboard to explicitly control the visibility of the keyboard.
So they can call show and hide to show the virtualKeyboard and hide the virtualKeyboard when focus is on the element with virtualkeyboardpolicy as manual.
So I want to contrast the virtualkeyboardpolicy manual with the inputmode properlly, so there's been some confusion regarding inputmode=none and the behavior of the inputmode, sorry.
So when inputmode=none is focused at least on Windows, Firefox and Chromium.
The behavior is pretty similar to what we say for, virtualkeywordpolicy's manual.
So if I set focus on the, inputmode=none as you can see in the keyboard state in change and if the keyboard is already up, it stays up.
But inputmode=none is not interoperable and it has a different behavior in software iOS.
So if the editable element has the focus and inputmode=none is specified then the keyboard hides.
To show the keyboard you have to flip it to some other value, something other than none.
But inputmode really is an attribute that controls this shape of the virtualKeyboard on Windows.
So when inputmode=numeric is specified numeric layout for the virtualKeyboard is shown.
So I'll show you an example where I use inputmode=numeric and the virtualkeyboardpolicy manual to show you how the inputmode attribute works in connection with virtualkeyboardpolicy manual.
So if I switch focus to virtualkeyboardpolicy manual I type element with inputmode numeric, and I pull up the keyboard manually so using the icon on the shell you see the virtualkeyboard layout is still numeric.
But the user, or the other author has explicit control over the visibility of the keyboard.
So they can call virtualkeyboard at hide, right now, to hide the keyboard and they can call it virtualkeyboard darkshow to show the keyboard.
I'll show you in another demo how this thing works, the along with the work all lay content.
So in this demo we have a keyboard icon over here that we used to call, that we use to explicitly show and hide keyboard, and there's some content with palatable element and it has virtualkeyboardpolicy attribute as manual.
If you set focus over here, you see the guide is rendered but the keyboard didn't come up.
And if you just select some content over here I also use grouper, yes.
Select contents and tap on the keyboard icon so the keyboard comes up automatically because the author calls a bunch of keyboard or show API that triggers the show.
And you also saw that the selected content supposition just above the keyboard so it can be showed into view and the scroll by height also changes.
Like if I dismiss the keyboard by pressing the keyboard icon again, you'll see that the scroll by height changes too.
So this is a much better user experience I'll show you in the code, like how we are, (mouse clicks) It's up in the head for the style thing.
And go all the way down to the bottom, I think is our script.
(mouse clicks) So here I've said, oops.
(both laughs) I will set, overlayContent's flag to true.
And when author sets overlayContentflag=true it ignores the default resizing of the Visual Viewport.
So the visual view, when the keyboard shows up or gets hidden, the Visual Viewport and the Layout Viewport in recent change.
So then, so the, well, author Ben gonna listen for geometrychange event which is the JavaScript event that Bo talked about and this geometrychange event can be used to listen for visibility changes.
Like if the keyboard is shown the boundingRect property on the virtualkeyboard will have a height which is going to be zero.
And it will have a, sorry, it will have a live down Rect which gets updated when the virtualKeyboard is shown right here.
And this double keyboard is the keyboard icon Rect, that going to show the height keyboard.
It basically checks for respond in their property and if it's greater than zero that means the keyboard is shown.
The article's height to hide the keyboard and if it's out then we show the keyboard.
We also use CSS environment variables to position the content above the keyboard.
If the content is uploaded by the keyboard or you want to positions some portions of the webpage so you can use this environment variable of the keyboard and set height to, like I said, the position of the page template rows which are basically the content that you're seeing on the the height of element And then we use the geometrychange event and to also scroll, set the selected content into view, so that it doesn't get activated by the virtualKeyboard.
So that's pretty it for the demo and we should ask these two questions I'll hand it over to Bo.
Move the chairs back (chairs clonks) All right, so what's the, I'll pop back to the presentation for a second.
So what we showed was the hide and show show API to control the Virtual Keyboard Visibility along with the Virtual Keyboard policy attribute.
And opt-in mechanism so that you don't change the layout for Visual Viewport when the keyboard is shown or hidden and new events and environment variables to adapt your layout.
Before we get into any issues discussions anybody just have any questions about what we were showing?
