WCAG 2.0 Evaluation Methodology Task Force Teleconference

24 Jan 2013

See also: IRC log


Shadi, Martijn, Katie, Eric, Alistair, Detlev, Kathy, Peter, Mike, Sarah, Richard, Tim
Liz, Vivienne, Roberto


New editor draft and DoC

<Detlev> nicely organised!

EV: Please review on the questionaire, I am asking for your approval or opinion
... There is a dif version
... A lot of things have changed, they are also included in the survey. After today will make a new survey #8, that will have the chamges from today

EvalTF Survey 7

<ericvelleman> <https://www.w3.org/2002/09/wbs/48225/evaltfq7/>

EV: The kinj is in the agenda and above
... Some of you have already filled out the survey
... There are some things I need to know, some things ask for some reading
... I tried to cover a lot of the comments
... I tried to cover as many of the comments as possible
... Therer is a long eda there. Next time, what you can expect on Monday, a new survry with the quesy=ions like 6 to 11
... That survey will be longer than this one. Please find time after Monday evening to eview the survey
... That is what is coming this week. Survey # 7 and next wekk Survey 8

Discuss new section Random sample

EV: New section on random sample

<Sarah_Swierenga> zakin, aacc

<Mike_Elledge> zakin, aabb

<Mike_Elledge> Mike_Elledge

<ericvelleman> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e>

EV: Randon Sample in step D
... There are many editor blocks
... It is based on the survey from last year

<MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e

EV: This is the basis for what I wrote, it is relatively short. Figure 5 we need to discuss. Some folks indicated they would like a percentage.
... In the survey must people have 25 to 30 pages. Rarely over 30 pages

PK: As far as the # of how many pages should be in th e sample. There are stats out there, I think we should use them. That recommended method

EV We did this before. We looked at liturature, it was hard to apply the lit to a methodology like this one

EV: If you take he scientific approach you end up with an enormous sample. But I agree with you if we could get a scinetific sampe to use

KW: I did some research based on those formeulas and it was a lot. And then what do you determine a page to be
... So it takes ys back to what is a page. It is going to be very difficult. The size/number needs to be cost effective
... When I do a randon sampling after they have made the site compliant after it has been remediated
... Taht way you do not find as many problem. In reality the random samle actually takes less tiem

Richard: the purpose of th Randon Sample it may include pages that you have already tested. Random should be random. To get a 95% reliabilit
... If you go to Amazon where thousands of page are all the same, you got at least one of each page type
... It doesnt ensure that you have covered the whole web site

EV: I hink we covered this somewhere
... In step 3F

Detlev: It should be an OPTIONAL part of the methodology

DF: It is a problem with web pages having a number fof state as Kathy indicated
... I hink it should be OPTIONAL, I agree with the questionaire

SS: I agree it shold be optional. Some clients limit the size for you. It is hard to defend


<richard> I do not think random sample should be optional. It is a vital exercise for a large site to be sure that the WHOLE site is accesseeble

SA: I agree that it should be OPTIONAL. But I think that should not stop us from doing that. Ye WCAG group has very strong opinions about this. Random Sapling needs an approach that is indeed representative

<MartijnHoutepen> agree with shadi

SA: Make sure that the developer does NOT know which pages will be suggested

<Detlev> so *what way* should the random sample be created, according to the methodology??

EV: I agree, but SS says that the clinet only wants you to do 9 pages

SA: Our approach does not specify a certain number of pages. Some are tempate driven may only need a few. That means if you count that with any number of pages, there is a very good chance that this type of methodology may not be possible

PK: Random Sampling I really have to believe there has to be ststa on this, as to what is ais a stats
... We need to spend time doing that research ourselves if that info is not out there

EV: I agree

PK: I realize timing is difficult. Perhaps we need to pursue research funding

EV: I could provide a survey for folks to put in their lit research

PK: Thos eof us who work at Universirty talks to there Statistics department

Issue: Funding for Samping Stats

<trackbot> Created ISSUE-11 - Funding for Samping Stats; please complete additional details at <http://www.w3.org/WAI/ER/2011/eval/track/issues/11/edit>.

Alistair: How do you collect the random sampke without your natural bias coming in to play. Second point, how do you determine the actual size of the web site for a statistical anaysis tool input?
... 3rd, how do you find the more relevant pages >

EV: There are many way, we need to provide one
... The size of a website we need to look into that
... Isnt your third point covered in the random sampe text
... Mentioning what Shadi said does that not cover it?

Alistair: the 80/20 rule
... Random Sampling is supposed to back up the actual sampke
... Random selection

<shadi> +1 to that last point

DF: I fully agree with Alistair. I think auto methods is best
... Using a crawler. The other thing, we should not raise the bar to high as to make it too difficlut and folks would not want to use it. We want to make it easy
... If there is a simpler way to do Random Sampling it would be difficult to include with all the ststes of pages

TB: I sploke to our Stats deartment it is important to write thre requirements in trem of statistical reqiremenets
... The assertions need to be statistically based

<Detlev> Tim, how would you deal with page states in a statistical way?

TB: That is the feedback I got from our Stats department. Requirements need to be statistically based. I have a paper on an entirely different sibject. But it covers that same concern, how to random sample
... I will share it
... The methods would be helpful

<Kathy> I would like to see it Tim

<Zakim> shadi, you wanted to talk about statistical and existing approaches for sample size

SA: A couple of good thngs. I like Alisters suggestion t call it Sampke Selections because if we go down the stats path
... We have a lot pf effort for fairly little gain
... Again, the assumption is evelauator may be biased in the selection, by building in a portion is randomly selected, we have a verification for that 80/20

<Detlev> If random selection cannot be independently verified (and that would be very hard), every evaluator can just *claim* to have picked pages randomly!

SA: That help to determine that it is indeed indicative and to validate
... To the best we can we need to remove the bias without too much overhead

PK: We need to have a flow chart. What category of website you are dealing with. Highly structured is going to be very different than a highly varied, verses highly dynamic sites
... Even with Amazon random sampling is still important because you dont know that prer alt etext is provided for all books
... What type of web site/app you are looking at
... Maybe we need a section early on that describes the diff classses of website and how to apply the methodolgy to that type of site

EV: That sound like afgood idea

SA: I think we do have some of that notion where we talk about different type of sites. We talk about what to do for sall website. We can look into it more
... I do not think we should seperate into web iste typres because then we have to categorize

EV: Goood dea Peter if you can come up with an idea

PK: I am happy to take that action

Alistair: Single page web apps, they are not so very different - but they do need to be seperated out

<agarrison> separate out single page web app from this document into a new web app eval doc, then drawing reference to that document if a website proper contains a single page web app

PK: Determining website types is heavy handed, we need to talk about things you see and what is an approach to deal with this when we see it
... We are not going to get into site clcification
... When you see this, do this
... without classifying
... Techniques to do to evaluate these diff things

SA: I dont mind, I was just worried about heavy handed, but we seem to agree by not trying to classify web sites
... Please propose something Peter

<shadi> http://www.w3.org/TR/WCAG-EM/#specialcases

DF: Independent view of the random sample vs the bias view. If the tester owns the tester.
... The nature of random you cannot eplicate

<Mike_Elledge> Short article: http://www.custominsight.com/articles/random-sampling.asp

AS: Face to Face at CSUN, we may have a room, not confirmed, who can participate

<Mike_Elledge> Random sample calculator: http://www.custominsight.com/articles/random-sample-calculator.asp

AS: Please fill it out as soon as you can, if we have a large enough headcount we can go forward

PK: Are you anticipating a 2 day?

AS: Yes, Monday and Tuesday

TB: Will IRC be available, not sue about speacker phone

<Kathy> do we need 2 days? Could we do just Tuesday?

EV: Defien the context of web use, I deleted in and moved the elevant info into other sections
... Please have a look and the timeline for next public Working Draft, do the surveys

EVL On Monday a new Survey will be opened

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.137 (CVS log)
$Date: 2013/01/30 13:15:34 $