WCAG 2.0 Evaluation Methodology Task Force Teleconference

31 Oct 2013

See also: IRC log


Vivienne, Martijn, Shadi, Alistair, Mike, Richard, Detlev, Moe, Tim
Eric, Sarah, Kathy


SA: apologies for time confusion and lack of formal agenda. Hope to have a new version soon to put into a survey to get feedback
... it will be a new editor draft to allow for comments in a survey
... today focusing on 3F and 4A, particularly as 4A is still a bit unclear

step 3.f: Eliminate Redundancies in the Sample (Optional)

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20131014#step3f

SA: Step 3F eliminates redundancies and the idea is that we have a fairly elaborate selection process where we select different types of pages and in addition we do a random selection as well.
... we added in section 3F to eliminate redundancies that might come about - particularly as the random has a likliehood to create redundancies
... section A & B don't likely involve redundancies, but it is the random selection addition that may introduce redundancies

<shadi> http://www.w3.org/2013/09/19-eval-minutes.html#item03

SA: Peter suggested 19 September to formulte the sections in such a way that you avoid duplicates from the beginning so you wouldn't need that section.
... would eliminate an extra step, as you would eliminate them as you go along
... 19 September there was support for this, and Moe has sent an email as well

Richard: from a practical point of view it is easier. If I get duplicates I may see that I've covered the site well. It acts as a double-check, as a true random sample may give me duplicates anyway and it's not a problem
... this would narrow down what I'm doing. When I do a random sample I know I have randomly sampled the whole site. It does no harm to double-check something.

SA: There is a point here about double-checking. Once you select the random sample in addition to the structured and identify duplicates, you actually throw away the duplicates and re-select new pages for those duplicates.
... the question is not to get rid of the concept but to get rid of that section from the document. It won't change the selection procedures.

ME: I was confused also. We're talking about eliminating redundancies among all the pages we're viewing not just the random sample. Maybe we should consider at the beginning out for not choosing duplicate pages or processes and have something as a final check to make sure you don't have multiple instances of the same process, feature or page.

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Oct/0020.html

SA: also Moe says something along the same lines in her email. There is an aspect of having redundancy and that might be an indication that the website is homogeneous and that can be important to note.

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Oct/0021.html

DF: I'm aware there is no point in the sampling process or the checking process where the evaluator is told that if a page has been selected and is part of a larger process we need to track back or forward to get the entire process, simply because the page you've found is part of that process. If we interpret check all processes meaning that if you check one part of the process you'd have to

check all parts of the process. We cannot for every page/state we've picked that if it's part of a process we need to check every part of that process. You're always dealing with processes of some sort, this is lacking at the moment.

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20131014#step3d

SA: isn't that what step 3e does - if you select the page and its part of a random then you also need to include all of the pages inthe processes. Step 3A,b,c, we select pages based on a particular criteria. in Step 3E you do a random selected sample on top of that.

DF: can you expect people in the exploration part of the process that you need to tell them they need to do this. You may have to call up things repeatedly and tht can become time consuming. Does that issue just come up much later when you see tht it's part of the process and then you have to add the other pages/states. You may need to add them while testing and this changes the exploratory


DF: don't know if everyone would want to go through the process itself - time consuming and laborious

SA: need to think of what the natural process is. You're saying a lot of the selection happens while there is testing happening.
... the tests overlap in a way, but we may need to consider that more when we write the sections

DF: there are different ways of tackling that. You make a selection based on pages, but as you do it it may lead to other pages that need to be tested. You may also need to check the page initiating a search as well as the search results page which would need to be tested. You may not be able to generate a url and it is just generated and you have to list the path. We need to give advice on

that, as it's a very pracitcal issue of sampling.

Alistair: don't have a problem with taking out 3F. Problem is picking another random sample and there may not be a point in selecting yet another page. 3F becomes rather redundant as you're telling people to do what is done naturally. 3E in the 2nd sentence - 10-15% of the number of pages - is that in the website or the sample. Is that the number of pages you'd select? If there is 20 pages

in the sample you add 2 random.

SA: a minimum of 5

AG: wouldn't make a major impact then

SA: 10-15% is not clear. 3E basic idea is to have a double-check mechanism - structured selection and then a random selection which should show a similar structure to the structured selection

AG: you could say you randomly select 5 pages from the remaining pages of the website and then you can get rid of 3F

<MartijnHoutepen> +1

SA: that would eliminate this and leave the mechanism to the individual
... it seems we want to integrate the content of 3F into primarily 3E but may others as well
... replace duplicates as you go along in each of the sections and it would make the document easier to use in different situations.
... Detlev also relates to the testing and things can change while you're testing.

ME: it makes sense that we mention in section 3 (introduction) that it is an on-going process of selecting sample and auditing may be rather concurrent activity. It's not only eliminating duplicate pages, but also duplicate processes - expectation that the code that folows of the feature that is presented duplicates a function.

SA: yes that needs to be explicitly mentioned. We don't want people to waste time testing stuff that's already tested. Depends on the heterogeneity or homogeneity of the website.

AG: in terms of coming on a page in the middle of a process, and the requirement of the whole process being in it. If the process is some bit that's included in every page of the site such as search. If you're randomly finding a process and then having to check back.

SA: if you check the search function and the search results, you should be able to assume that it works the same on every single page regardless of where you've decided to search from. You don't need to test the search function on every single page.

AG: the way it reads at the moment you have to have every page in a process which would mean then every page that has the search box. Wouldn't you just need 1 clear example of the process.
... say you come to the page 3/4 of the way along and you have to check every single page in the process you'd have to check every single other page that has the front part of the process in it.
... need to clarify that you need to selection 1 path through the website that represents every way the process could be displayed to the users

Sa: focus on checking the function not necessarily all the pages

DF: there are different ways of the way processes work where you have to start at page 1 or pages that progressively unfold in the DOM based on your choices. It makes sense to define the start point and then define the input which will cause a certain output. The search result page would also be evaluated. You'd have to explain that you are going to check all 7 pages for a registration process

and it would be different for pages that change. We need to give some guidance, perhaps specific types of processes you could encounter. Otherwise it would be perceived as out-dated.

SA: you mentioned 2 types of different processes - 1 that has a defined start point that you can't jump into the middle, and the other where you can link back from the middle.
... we're now getting into 4A.

Moe: no problem integrating 3F into the other stuff, but don't want to lose the optionality. In regression testing there are times you may pick the same pages or procedures, and in that case it's not a bad thing.
... if appropriate remove redundant pages

SA: are you saying it's okay if we pick the same pages

Moe: envisioning the testing more from a user perspective. Making sure all of the pages are clean and then conduct a regression test. I'm not going to hone in on the fact that as an end-user I'm seeing a page twice.

Sa: would you test it again

Moe: in regression test, yes I would

SA: there are then situatins where you mgiht want to re-test something

AG: you're not regression testing to make sure nothing's changed.
... they are 2 different things.

Moe: it may be scenario testing. We dn't want to make it so stringent they have to remove redundancies

Step 4.a: Check web pages for WCAG2.0 conformance requirements

<shadi> [[Step 4.a: Check for the Broadest Variety of Use Cases]]

SA: the old title for 4A was something like 'broadest use cases possible'
... that was the intent at the beginning. In most situations you are not going to be able to check everything.
... in checking use cases there is a back and forth between selecting and checking. We need to check for WCAG 2 conformance for each of the pages we check. How many use cases, what kind of use cases. This section is increasingly changing back into check web pages for wcag conformance
... we keep getting back to scenarios, use cases, functions. What kind of advice can we give here and how does it relate to step 3.
... when you are doing the testing, where do you start testing the function. You end up on a page tht has a search function. How do you go about testing on a web page including the functins that will take you away from that page

AG: you take the use case and you do each of the actions in terms of the users. After each action you look at the dom and see what changes have occurred. You use the use case as a way to determine how the dom will change with each interaction the user has. It is the dom state that is changing.

<Detlev> so the use case is a particular instance of a process

SA: do you conduct use cases ahead and that's what you check for - to help you.

AG: most of the time people have an idea of how a web page is going to be used. You capture the use case and then you transfer that into a use case person with disability might have. A blind person wouldn't be using a mouse and would be tabbing. By doing those key presses they may come across changes that change the dom. We need to keep the dom in mind for accessibility. All changes have to

be assessed.

SA: in step 3 the evlauator has selected an initial sample. Not all processes might be apparent yet and there may be some modifications to that sample. I start evaluating and the advice in 4A is start constructing use cases based on visual keyboard users, screen reader users, voice command etc. and construct certain use cases and check using those use cases. While you're doing that and checking

for the use cases you might identify processes or use cases that need to be part of the sample.

AG: any new DOM state needs to be re-assessed.

SA: we call for a different state of a page, and that is a new page

DF: from my experiencemost changes in dom are really activations of buttons etc that are normally the same that will be triggered by keyboard users. Be aware during page samplilng that there may be cases where dom changes occur - holds for all types of users. We can simplify this and tell them to check the relative results of the page, what is the process they are trying to achieve e.g. booking

process. It is more important to define for the tester how you specify those processes - during sampling or later on. Do you record it with the base page, or do you take the process and it becomes more pages for the sample. It has an impact on how you send your customer a quote -for how many pages are you testing. You can have problems because you didn't charge the customer forthat.

SA: the balance is to try to keep it as light as possible (the procedure) but we want to make sure that there aren't too many holes. People will use common sense but others will only try to do the minimum.

AG: there were 2 reasons for forcing people to tab through the page. It gives people a chance to see how usable the page it - if they have to tab to the search function. With javascript you can attach events to something like an item in a tab index. You can tell people to follow the use case for amouse, but they won't find that the person using keyboard only can't get to it.

DF: that is part of the discretionof the tester - it's part of SC2.1.1 that you have to get to all actionable actions via the keyboard.

SA: in some instances tabbing through - not only getting to it but also the visual indicator
... the conclusion of the discussion is that in section 4A to provide more information - say something about use cases, functionality. Relates back to the selection of pages for the samle and the need to go back and forth and think about use cases, keyboard users and other types of users.
... would people agree we need to provide more information?


<Mike_Elledge> +1

<agarrison> +1

<richard> +1

<MoeKraft> +1

<Detlev> yes draft something - but don't overdo it...

Sa: need to thnk of how to provide more detail without it become too laborious
... other aspect is to be clear

ME: we mentioned that it's important to understand the different use cases. We can talk about being aware that no only web page types, but the type of use that web pages will receive

Sa: it all ties together - the purpose of a web page is not just picking at random - what does the page actually do and how do the user act with it?

Ag: use cases are the only way to drive mobile apps as you can't run tools over the page - the state is changng all the time.

<Detlev> agree with Alistair here!

Sa: more sohisticated applications make it impossible to test every single event

<Mike_Elledge> +1

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2013/11/06 12:03:33 $