Testing Protocol and Methodology

<michael_fairchild> https://github.com/w3c/aria-at/issues/5#issuecomment-510122453

<michael_fairchild> scribe: MF

<michael_fairchild> Matt: Attached is my findings for the menubar example, looking at row 10.

<michael_fairchild> Matt: Columns 1, 2, and 3 form a sentence.

<michael_fairchild> Matt: In row 10, I added notes that describe the "incorrect support" result

<michael_fairchild> Matt: However, there there slightly different expectations based on the keys that were pressed. In notes, I have two different expectations outlined

<michael_fairchild> Matt: Test, activity, SR mode, Tested element, tested attributes, assertion will be the same for all screen readers

<michael_fairchild> Importance might depend on the screen reader, but it should be the same for every screen reader.

<michael_fairchild> Matt: Test methods, result, and notes would change for each screen reader

<michael_fairchild> Matt: I haven't considered braille here. We might need a new set of assertions for braille.

<michael_fairchild> Yohta: JF had the idea of hiding some of the columns which might not be helpful for some of the columns.

<michael_fairchild> Matt: element, importance, and attributes might not be necessary for testers

<michael_fairchild> Yohta: I like the idea of hiding information for testers

<michael_fairchild> JF: I'm finding this conversation helpful. One thing that is important for us to think about - we need to use language that will be easy for testers to understand. Avoiding domain language is important (such as 'assertion', etc)

<michael_fairchild> JF: I've seen many design projects fail because the team had their own language which didn't match what the users needed/understood.

<michael_fairchild> Matt: I agree, and some of the language will be sued for the backend data model.

<michael_fairchild> Matt: We have to be abstract enough to be able to compare across browsers and screen readers, but not too abstract. Testers will need concrete examples of how a specific screen reader might meet an assertion.

<mck> scribe: mck

mf: Tend to agree with most of this model.

Testers need to have concrete examples of what support looks like.

An abstract assertion will not help them much; they might not know what support would be.

As close to concrete examples as possible is good.

Hesistate to provide concrete examples because things change over time.

General expectations are better in that sense.

There is a different between minimal support and consistent experience in screen reader.

Minimal support could have impact on end user's ability.

More important to focus on minimal support than details of end user experience.

<michael_fairchild> Matt: minimal support vs full support and good experience: this came up in my report where the sub menu was too verbose (menu announced multiple times) which could make it difficult to understand and pay attention to the part of the announcement that actually matters.

<michael_fairchild> Matt: what should our minim expectations for a tester be?

<michael_fairchild> Matt: it's somewhere between new and expert screen reader user.

<michael_fairchild> JF: when you say screen reader user, do mean someone like me who is a professional who passed an exam, or someone who uses a screen reader every day?

<michael_fairchild> Matt: someone more like you

<michael_fairchild> note: Matt and JF will be out next week

<michael_fairchild> Yohta: do you want me to testing anything else using this template?

<michael_fairchild> Matt: not yet, I want to make a few more changes

<michael_fairchild> Note: no meeting next week

This is scribe.perl Revision: 1.154  of Date: 2018/09/25 16:35:56  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Present: matt-King michael_fairchild shimizuyohta
Found Scribe: MF
Found Scribe: mck
Inferring ScribeNick: mck
Scribes: MF, mck

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)