<scribe> scribe: jeanne

background

RBM: Protocols, Scoping and testing are all working on how testing might exist in WCAG3
... what is tested? Scoping. What is the smallest unit?
... might be a label, a dropdown, a phrase. The unit that we test
... then there are pages and states. We can't use page anymore, we are calling it a view
... the user process is the series of user actions and views that support the actions
... the aggregate of smallest unity, user process, and view
... Types of tests
... fully objective test - no subjectivity
... condition (subjective tests) "8 out of 10 experts would agree.

<Rachael> https://docs.google.com/presentation/d/1b5xHQWBzoYdKp7BfPgIUBCpz-yaDOx_kSq_HlQxcFh0/edit#slide=id.g115ec01aa81_0_39

RBM: test case (aka test scenario, or test statement) we can't say a universal level. The author would set the level
... all of these are pass/fail
... if they defined the definition
... if the failures are avoided
... if the site passed the conditions

MG: In the failure situation, would the algorith for reading level is set for what the level desired and that is the test, not the failure.

RBM: If there is an interactive element, there must be soemthing that is visible. We don't say what the visual interface has to be, but we can say if there is nothing, then it fails.

<mbgower> more accurately, wouldn't the condition be: use XX tool to achieve XX reading level. So failure could be 1) not using the tool 2) not achieving the reading level

<mbgower> The test would be 'use tool XX to confirm reading level 11 is met'

WF: We have two things to explore: does this fit or does this meet the reliability goal that the Reliability group is working on.

RBM: We aren't going to know if it really fits until we try to put it into the document. I would rather work on reliability

Reliability Work

<dmontalvo> https://docs.google.com/document/d/1sugAtqie_x1XqHDZo1Im7ftDNllWeRV_ty4PULeoTV0/edit#

WF: Testable Outcome:
... It addresses a clear user need
... There is little to no room for interpretation
... It is quick and inexpensive to test (possibly using tools)
... It is easy to learn

Step 1: Come up with examples

Step 2: Create a rough outline

Step 3 Define terms

Step 4: Simplify

Step 5: Write the How-to and the Methods

Step 6: Get feedback

WF: Are you ready to start writng tests for this process?

RBM: I don't know yet, I need to think about it.

MG: We are looking at a 3rd party baseline in the reading level example. It establishes a method and an outcome. An external tester can do it.
... for Visible Controls, no one has come up with a definition of what is visible. I don't know if we can write tests without that outside measure

<dmontalvo> ACT Rules definition of visible

WF: ACT has a definition of visible

MG: that doesn't apply, because there is a component of understanding

RBM: That gives the organization the ability to determine what their standard is for a visible control, and you state how they are making it visibly actionable. You must be consistent. Let them define it.

MG: They have to publish or expose what their ruler or measurement system is.

WF: ACT rules specify what information must be provided. It can be DOM tree, or it could be a Style Guide

<mbgower> okay, great, that helps me. Sorry to stutter around on that question!

<Zakim> jeanne, you wanted to suggest a forward path

<Rachael> Jeanne: One of thign things reliability group did was pick an exisitng SC and work through it to test the details. We did language of parts.

<Rachael> ...logical WCAG 3 version of it. I would like to suggest the scoping group pick an SC and write what the tests are.

<mbgower> 1.1.1 is always a good candidate for this kind of thing

<Rachael> ...That would give a lot of information about how good it is.

<dmontalvo> Pronunciation of Text

WF: ANd please follow the outline that Test Reliability has developed for the OUtcome and the Method.
... I think this process will work for what we want to do.

RBM: SOunds good to me, it would be good to have one of you join us to answer questions
... I'm thinking about Visible Controls

JS: WHat if Reliability also works on it? We are almost done on where we are.

<mbgower> ^ Okay!

WF: I think we need to iterate on the Outcomes work, so I don't think we are ready

JS: What if we did the Methods with this kind of testing?

<Rachael> akc mbgower

WF: I don't think there is a problem that needs to be addressed.

Issues

MG: If you look at 1.1.1, there are aspects that require all three types of tests

WF: There is a sorting exercise in the edge cases. That would probably be where that fits
... If the Scoping group wants to start an example, send an invite to Reliability. Or does the Scoping group want to join Reliability? Or do you want to pull that work into the Reliability group?

MG: We did an exercise in AG and looked at breakdowns. None of them finished, there isn't a plan to finish them.
... or Scoping could pick up the pieces from that exercise for the next step

Followup from the Migration Exercise

RBM: We will be doing the MIgration Exercise again next week
... our goal is to breakdown the SC into categories by Functional Needs and Test Types
... once that is done and the sheet is filled out, then the goal is to clean up the categories based on what we learned.
... then go through that list and identify overlaps and gaps
... then compare with the ACT leist
... then look at the guidance holistically and group them
... then we can update the Silver writing process
... use the gaps to identify what new guidance should be written
... Reliability has been working bottom up
... this is a top-down approach

MG: There may be a sweet spot for when we expect that this exercise will make it easier to enter the process that Wilco has explained.

RBM: Do any of our test cases need Protocols or Test Cases?

WF: The current SC don't work this way therefore it won't apply to to existing SC

MG: I will challenge that. What about Headings and Labels? AA.

WF: (laughs) I didn't say it was handled well!

<Wilco_> https://act-rules.github.io/rules/cc0f0a

WF: In ACT, it looks for relationships; That A describes B.
... what things need labels

JS: I think there are a number of notorious examples in WCAG 2.x that testers do not agree. There are subjective evaluations, especially in the manual tests.

MG: If you had a definition of "this describes this" then it would be a better example of a test case.

RBM: We wanted to work on combined vocabulary

JS: I agree. Test cases are something different in the Quality Assurance testing world.

<Rachael> Discussion of whether WCAG 2 is verifiable by an outside party

<Rachael> Jeanne: Need to know the states and the way to do the states. When test for a bank for example, they give log in information, how to open accounts, etc.

<Rachael> Wilco: Nuanced distinction. May need help to find what is tested but testing itself can be done externally.

WF: You may need help getting the meaningful states to tests, but you don't necessarily need to get all of them. That's sampling.

MG: That is trying to get the Process part of the Scoping exercise. As soon as you look at process, you need test scripts, to even get to what you are testing.

<Rachael> Key question: Does WCAG require an organization provide the information required in order to verify? (assumption of information needed to test)

MG: I want to go back to the test case that Wilco is describing
... the Labels element is a programmatic input. You probably need context to make sure it is associated with the correct input. Then does it actually describe what it is.
... and organization would have to define what is described (use a Style Guide and type of language) to successfully describe an input.
... is it realistic to get down to that level or not?
... or that some of these are so subjective that they cannot be tested.

RBM: Why not try 3.2.4. It includes a number of aspects that could include a style guide

MG: Also 1.1.1.
... "equivalent" is very subjective

<Zakim> Rachael, you wanted to suggest 3.2.4

MG: that couild be a step in the right directions

WF: I think that is trying to solve a problem that doesn't need to be solved

<Zakim> jeanne, you wanted to mention research that debbunks 8 out 10 testers

WF: helping people understand interactive elements is more valuable than testing

<Rachael> Jeanne: 8 out of 10 testers. We ran across a study that tested 8/10 and even in most ideal conditions, the best they ever found was 7/10

<Rachael> ...8/10 shoudl be dropped.

RBM: We are looking at Library of COngress how do you describe people. We are developing a style guide to help them.

Next Steps

RBM: do we want to work together or pursue separately?

<mbgower> Useful, and we are better understanding points of consensus (and not)

WF: Propose that Mike invites Reliability to participate in Scoping and we try to write one of these new types of tests

JS: +1

MG: There are points where we have agreement and points where we don't. Working together will reduce friction

RESOLUTION: MIke will respond to this meeting with an invite for another meeting

This is scribe.perl Revision VERSION of 2020-12-31
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Succeeded: s/ACT definition/ACT Rules definition/
Succeeded: s/Labels and Intructions/Headings and Labels/
Present: mbgower, thbrunet

WARNING: Fewer than 3 people found for Present list!

Found Scribe: jeanne
Inferring ScribeNick: jeanne

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)