Mobile Accessibility Task Force Teleconference -- 14 Nov 2019

Topic Custom interactions

Jake: we need to have a definition of what custom interactions are – that is difficult
... last week I had an idea – turning around what if we can define the common interactions and everything else is a custom interaction
... the next thought is what exactly are we trying to solve, because we are trying to solve two issues
... if we take the example of the email item or option and you swipe it to the left and halfway through the swipe it gives you two options – archive, something else
... basically it's a single extended swipe, but suddenly you've got two interactive elements if you only swipe half but with a full swipe it gets deleted. So it's not only about a swipe it's about seeing elements or activating elements
... so that's about combining interactions – extended interactions
... another thing is if there's something somewhere it would be great to have an indicator or help and instruction – an indicator like an arrow
... so we have an indicator, we have multiple actions below another interactions, and we might have a gesture or a keystroke or maybe three keystrokes to press where you can create a new email or another post, but it's all combining stuff

Detlev: What you mean by combining stuff

Jake: combining gestures and you can have something being excavated by adjuster that is not known to people

Detlev: is it covering both standard gestures or interactions and the typically different interactions that you That when you turn on the screen reader?
... typical for androids Talkback's context menu

Jake: for now, no, you can create your own combined gesture – just mentioning different scenarios
... There's one more thing – pointer gestures. When we talked about Poyner gestures was a moment when we talked about starting point and end point and a point in between. And so in our conversation about custom interactions, what is a custom interaction – how can we ever define every customer interaction etc.
... There was Also a question why is it a custom interaction when you have aria etc.
... turning around, pin down what is a standard interaction. This might be a lot easier to define. And say everything else is a custom interaction
... so, for instance if we define a swipe left is a standard interaction and you build something with a swipe left there's only one action when you swipe left, you're always fine. When there's another interaction like using an arrow key or tab to your shift tab or enter space , those are all standard interactions.
... whatever wages you define, aria they all make use of standard interactions
... once you say several keys etc. you are out of standard interactions. The moment you do something that is not in the list of Standard interactions, you have to define it
... Swipe all the way across and one thing that standard, but the moment you provide choices halfway it's not standard and you need to define
... so that's the starting point. Let's make sure we define our standard interactions, and if we agree on those, everything else, or every combination of two standard interactions automatically becomes a custom interaction

Detlev: I'm not sure whether dividing line is between standard gestures and everything else – if you have things like documented gestures like tap with four fingers of the top of the screen to move the focus to the top, those are documented and standard for screen reader users. Do you think we can bracket out the particular gestures that you get with screenreader?

Jake: let's take a concrete example – you are already diving into nonstandard

Detlev: trying to see whether this is whether or not assistive technology is turned on

Jake: you started already like the 4 finger stop on the top of the screen that's documented for android, so you're already okay. I'm not sure about assistive technology if we talk about using WCAG for hardware or software, the assistive technology itself – if it's not part of the list we define this is just a first try to get those standard gestures out there. If you do a six finger swipe to the middle and up and you documented you are fine because it's do[CUT]
... as long as it's not on the list we define automatically becomes a custom gesture

<JakeAbma> https://docs.google.com/document/d/1ouVFq4w-i0rchNHtTAG_JoRwHfYm9mN2MkxFBct1JSI/edit

Kim: The key thing is the concept is we come up with a basic list for gestures and a basic list for keyboard, everything else is custom gesture. The standard ones are easy to define, and then everything else is custom – that's the idea

Jake: keyboard – The keys that are pretty well defined as standard. Everything else, including, for instance open application and press and for something new. It's not on this list, please Document it

Detlev: it might be over the top to go through the entire list of things that are defined as standard to find something that is not standard
... the testing aspect is difficult because you can't tell what weird actions there might be that you have to document

Detlevv: also gray area of pass-through gestures – work on the standard PC but not on a Mobile keyboard

Detlev: just not easy to pin down those things that are generally accepted standard gestures – you may not have Some keys on keyboards

Jake: if you provide an example works on PC keyboard but not on mobile – Is that relevant – if it doesn't exist is not part of the success criteria

Detlev: hard to find what isn't defined – in those cases where it's not documented how we going to detect that

Jake: that's not related to the solution because from the other side if you don't have your standard list you have the same problem
... this concept is a standard list is a pretty clear concept and most of the time people know – you don't have to test those. Look, this is the standard list, but if you do another combination please provide instructions
... you have a question like one finger, two finger, three finger and maybe there's some part only operate standard when you turn on the screen that's good we can leave it out of the list, and become even more concise. So we have less in the list – the standard list. I had the biggest problem defining what is motivation actuation so I just put something in there to Start
... keyboard is well-defined
... you can also tell people do we need to provide for the majority of interactions or interaction elements – this one is not about official indication of standard gesture, so people say well you see a list but there's no arrow On there and we don't want it from aesthetic perspective. You're fine with the success criteria, but the moment it becomes a custom interaction you need to provide that or instruction or tooltip or whatever.
... But for all the other cases Which are the majority, you don't have to provide anything because this is only about the custom ones

Detlev: I'm trying to think of examples that you would want to use. Say you have a select – it behaves like a standard select so you basically press arrow down and it will open up. You can use the arrow keys to go through the options. Now all that is a standard, Right?
... so how do we implement the option that in that context you do something to delete. I think the list of commands has to be seen in the context of expected behavior. You have dependencies between those things in your sequences of actions which may be expected or unexpected. So I think it can quickly get quite complex in terms of trying to pin down what standard behaviors and what is – just a warning

Jake: combo box, press the arrow down down down, standard, fine. But someone presses delete and it deletes an option will normally in a web application you can never delete an option – is that a standard interaction?
... is that a standard interaction?

Kim: do you have an example of that interaction?
... can you think of an example that illustrates your point – there may be a standard context but unusual interaction? Would help to have a concrete example of that to advance the conversation

Jake: if we see any nonstandard OS browser or technology interaction by definition – arrow up and then delete on an action, that's not standard browser behavior, and it is documented
... I'm trying to look for if it's already covered or we need some extra wording. If we specifically say including gestures, keyboard interactions blah blah blah, that is the part we define as the standard interactions.

Detlev: I think it's a good approach, I'm trying to see if it holds up – I'm basically sympathetic to it. Make sure this is not getting into the area where people say this is holding back interaction because it's hard to define a standard list

Jake: we've had the same discussions – we took a separate set we defined ourselves. That would be also a list which may be changed in time. On the other side wouldn't it be great to say the whole world has a set of standard finger gestures, keyboard keys, and the more difficult ones I hope you can help with, and just say this is the standard set and we ask for feedback and maybe two or deleted Or two are added because someone has a brilliant idea.
... I didn't put definitions in there because it takes time, but we can provide some definitions in the standard list.

Marc: I like the idea of being able to provide instructions. I think of this one is one that should be fairly easy to get through because it has the get out of jail free – just by having a help section that covers all this. As I read through this I see at a minimum I can have a help section on my site that covers any custom gestures that we have
... we don't know what people are going to be coming up with next month or next year, but I'm just trying to think how much we need to define – were giving so many options, and help on the site is the easiest way out, then we offer much better practices. The instructions come up as your doing the gestures, all those.

Jake: the big difference between the previous version and this version is the most difficult thing is – and I totally agree with that, we can define a custom gesture, so let's turn around, let's define standard gestures, and everything else is a custom gesture
... most people the Example is when we talk about standard gestures with fingers is left right up down. The moment you say go left first and then up you are combining to standards and that becomes a custom gesture that you need to define
... in short, instead of trying to define what custom is, we define what standard is. I think we can pretty well define it in a clear set and then say everything else is custom.

Detlev: a small thing I see is an issue is the standard gestures, now talk about finger but other success criteria we have Extended to pointer

Marc: I almost struggle with anything more than one finger being standard

Jake: the concept of turning around is not trying to define custom, just defining standard and leaving everything else up to
... for example top widgets, area widgets, is it custom or standard. Basically when you define Standard keystrokes, you're fully covered for every area widget because they all work the same right now. So the question is not anymore if any area widget implemented custom, no because you can use them with all Standard keystrokes

Detlev: I think the gesture part might be simplified to just one pointer and not have two finger and three finger
... any standards with two fingers

Jennifer: I like the idea but I'm wondering if it's easy to maintain?

Jake: keyboard standards have been stable for like 30 years

Jennifer: what about mobile web?

Jake: we need to have consensus and more people can think of what is the standard, but swipes, maybe a small set will be added extra like two finger or drag or turn a dial, and of course everyone is invited to come up with more complicate adjusters but when they are there it is a custom direction so you need to – that's exactly what you need them to be defined
... anybody can do anything – swipe left instead of up-and-down, but at that moment it is custom so provide instructions on how to use it
... how many standard gestures with your finger will be invented – I don't think we have many more options than going around Uptown

Jennifer: I think one more – press and hold

Jake: those are pretty standard, I have the first set and we just need to refine it. I've been thinking about it now for three days and I'm comfortable with the standard stuff

Detlev: you scroll a page and at some point something happens for example the header disappears or you have a little thing popping up which takes a fixed position at the edge or something.
... also need to make it show for blind users. Do you need instructions for that? How do you draw the line? When you would you need a custom instruction?

Jake: you do something and it will reveal something, or it will activate something. But on the side will also show you a pop-up. The moment it is not actionable
... interesting case – Instead of scrolling you swipe halfway to the left and it looks like you are swiping off the screen and there will be a functionality activated, but at the same time there's a balloon popping up mentioning something to you. Status message success criteria kicks in. But your point is those are two things that one so it's not standard. Because if it was only the one action it would be fine. What would be the difference if you have mo[CUT]

Interaction provided to you

Jake: it's still in standard interaction but nothing will get Activated – there's something different if you swipe to the left something appears and it gets activated but also something else gets activated

Detlev: I see the point as soon as new actions are revealed to you but nothing happens it's all right
... you have this example with mail halfway – file options, if you swipe halfway the nothing happens yet, so you can think if you want to do it or not. My example is scroll up and down the page and at some point something appears at the side. It's partly dependent on your scroll status but also dependent on the gesture and it may be triggered by that gesture but whether it's part of that gesture or some dynamic change that's triggered at some point in time.
... my point is it might be difficult to define exactly what's part of the interaction and what isn't to Draw the line of what is standard and what isn't.

Jake: it's just all that's standard or available through standard interactions. So it's all functionality that uses custom interaction or through custom interaction, for example when it's reviewed
... I see a small loophole over here but that's more the wording

Detlev: so the example I mentioned you scroll a page and something pops up at a certain point, it reveals something, but it doesn't trigger something you don't execute a function so you just reveal options, so deleting something by scrolling that would be custom interaction, but if you just review something by scrolling that would be all right – would that be the right distinction?

Jake: yes, because that would be a standard interaction
... one thing wrong with the normative Text, all functionality that's available by custom interactions – that shouldn't be there, should be that's usable by, that's activated by a custom interaction
... or is available through custom interaction, that's okay. But the next part says instructions that say how to activate functionality, that's not okay. Sounds like the hidden button you have to say, But it's not about that is purely about custom interactions. That can be fixed.
... when you do a swipe to the left, like with the email example and there are five actions underneath the swipe left, that is not a problem. If you scroll up or down and something reveals that is not a problem. That is a standard interaction and it doesn't have something activating.
... the moment you scroll up in your page get deleted and something else will be shown, then there's a really weird pattern custom interaction.

Detlev: so the critical point about the swipe all the way and delete would be the delete part and not the reveal part

Jake: it's one of the other, delete is not a problem, reveals not a problem, the moment have to swipe reveals in all the way swipe deletes, then it's a custom interaction
... is it a swipe or is it a drag and we want to make a difference?
... because if you swipe left you always archived with, but if you drag halfway you need to Hold it

Detlev: I think this Needs working through a number of examples to see if the normative text holds up. But I think the general approach is sound.

Jake: I think if we agree with the standard list for gestures pointers Keyboard and then eventually something with motion actuation
... the two other issues are solved, but the one issue is how do you discover what's accustomed gesture if you don't know what's there – testing
... then you have to do weird things to find if something is hidden or not

Marc: my pitch is to really simplify it – keep it to one Finger, most basic of standard gestures

Jake: that would be great – please adjust the list. Also the keyboard list – that's a standard list
... for the mouse click, double-click, right-click, scroll if scroll wheel, left right up down – I think that's the same as the pointer
... so that would also be a very simple list

Detlev: I would call it single pointer for consistency and consistency with the pointer gesture and consistency with the pointer gesture success criteria
... and then add the mouse will is obviously only for mouse that doesn't really matter

Jake: so I don't think it will become a list with lots of standard gestures, it will be like 5678 maybe nine and that's it
... Five, six, seven, eight, maybe nine and that's it
... I think the standard list will not change – same with the keyboard
... add comments for standard list, and custom interactions examples

I agree with Mark that it would be useful to keep the list simple – easier to agree on a list that way as well

reiterating – feel free to add comments to the list, especially on what should be standard and any examples you may find

<scribe> ACTION: Jake to continue with the define a standard list and everything else is a custom interaction concept for this SC

<trackbot> Created ACTION-80 - Continue with the define a standard list and everything else is a custom interaction concept for this sc [on Jake Abma - due 2019-11-21].

- DRAFT -

Mobile Accessibility Task Force Teleconference

14 Nov 2019

Attendees

Contents

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output