See also: IRC log
<Detlev> nicely organised!
EV: Please review on the questionaire, I am
asking for your approval or opinion
... There is a dif version
... A lot of things have changed, they are also included in the survey. After
today will make a new survey #8, that will have the chamges from today
<ericvelleman> <https://www.w3.org/2002/09/wbs/48225/evaltfq7/>
EV: The kinj is in the agenda and above
... Some of you have already filled out the survey
... There are some things I need to know, some things ask for some reading
... I tried to cover a lot of the comments
... I tried to cover as many of the comments as possible
... Therer is a long eda there. Next time, what you can expect on Monday, a
new survry with the quesy=ions like 6 to 11
... That survey will be longer than this one. Please find time after Monday
evening to eview the survey
... That is what is coming this week. Survey # 7 and next wekk Survey 8
EV: New section on random sample
<Sarah_Swierenga> zakin, aacc
<Mike_Elledge> zakin, aabb
<Mike_Elledge> Mike_Elledge
<ericvelleman> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e>
EV: Randon Sample in step D
... There are many editor blocks
... It is based on the survey from last year
<MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e
EV: This is the basis for what I wrote, it is
relatively short. Figure 5 we need to discuss. Some folks indicated they would
like a percentage.
... In the survey must people have 25 to 30 pages. Rarely over 30 pages
PK: As far as the # of how many pages should be in th e sample. There are stats out there, I think we should use them. That recommended method
EV We did this before. We looked at liturature, it was hard to apply the lit to a methodology like this one
EV: If you take he scientific approach you end up with an enormous sample. But I agree with you if we could get a scinetific sampe to use
KW: I did some research based on those formeulas
and it was a lot. And then what do you determine a page to be
... So it takes ys back to what is a page. It is going to be very difficult.
The size/number needs to be cost effective
... When I do a randon sampling after they have made the site compliant after
it has been remediated
... Taht way you do not find as many problem. In reality the random samle
actually takes less tiem
Richard: the purpose of th Randon Sample it may
include pages that you have already tested. Random should be random. To get a
95% reliabilit
... If you go to Amazon where thousands of page are all the same, you got at
least one of each page type
... It doesnt ensure that you have covered the whole web site
EV: I hink we covered this somewhere
... In step 3F
Detlev: It should be an OPTIONAL part of the methodology
DF: It is a problem with web pages having a
number fof state as Kathy indicated
... I hink it should be OPTIONAL, I agree with the questionaire
SS: I agree it shold be optional. Some clients limit the size for you. It is hard to defend
+1
<richard> I do not think random sample should be optional. It is a vital exercise for a large site to be sure that the WHOLE site is accesseeble
SA: I agree that it should be OPTIONAL. But I think that should not stop us from doing that. Ye WCAG group has very strong opinions about this. Random Sapling needs an approach that is indeed representative
<MartijnHoutepen> agree with shadi
SA: Make sure that the developer does NOT know which pages will be suggested
<Detlev> so *what way* should the random sample be created, according to the methodology??
EV: I agree, but SS says that the clinet only wants you to do 9 pages
SA: Our approach does not specify a certain number of pages. Some are tempate driven may only need a few. That means if you count that with any number of pages, there is a very good chance that this type of methodology may not be possible
PK: Random Sampling I really have to believe
there has to be ststa on this, as to what is ais a stats
... We need to spend time doing that research ourselves if that info is not
out there
EV: I agree
PK: I realize timing is difficult. Perhaps we need to pursue research funding
EV: I could provide a survey for folks to put in their lit research
PK: Thos eof us who work at Universirty talks to there Statistics department
Issue: Funding for Samping Stats
<trackbot> Created ISSUE-11 - Funding for Samping Stats; please complete additional details at <http://www.w3.org/WAI/ER/2011/eval/track/issues/11/edit>.
Alistair: How do you collect the random sampke
without your natural bias coming in to play. Second point, how do you determine
the actual size of the web site for a statistical anaysis tool input?
... 3rd, how do you find the more relevant pages >
EV: There are many way, we need to provide one
... The size of a website we need to look into that
... Isnt your third point covered in the random sampe text
... Mentioning what Shadi said does that not cover it?
Alistair: the 80/20 rule
... Random Sampling is supposed to back up the actual sampke
... Random selection
<shadi> +1 to that last point
DF: I fully agree with Alistair. I think auto
methods is best
... Using a crawler. The other thing, we should not raise the bar to high as
to make it too difficlut and folks would not want to use it. We want to make it
easy
... If there is a simpler way to do Random Sampling it would be difficult to
include with all the ststes of pages
TB: I sploke to our Stats deartment it is
important to write thre requirements in trem of statistical reqiremenets
... The assertions need to be statistically based
<Detlev> Tim, how would you deal with page states in a statistical way?
TB: That is the feedback I got from our Stats
department. Requirements need to be statistically based. I have a paper on an
entirely different sibject. But it covers that same concern, how to random
sample
... I will share it
... The methods would be helpful
<Kathy> I would like to see it Tim
<Zakim> shadi, you wanted to talk about statistical and existing approaches for sample size
SA: A couple of good thngs. I like Alisters
suggestion t call it Sampke Selections because if we go down the stats path
... We have a lot pf effort for fairly little gain
... Again, the assumption is evelauator may be biased in the selection, by
building in a portion is randomly selected, we have a verification for that
80/20
<Detlev> If random selection cannot be independently verified (and that would be very hard), every evaluator can just *claim* to have picked pages randomly!
SA: That help to determine that it is indeed
indicative and to validate
... To the best we can we need to remove the bias without too much overhead
PK: We need to have a flow chart. What category
of website you are dealing with. Highly structured is going to be very
different than a highly varied, verses highly dynamic sites
... Even with Amazon random sampling is still important because you dont know
that prer alt etext is provided for all books
... What type of web site/app you are looking at
... Maybe we need a section early on that describes the diff classses of
website and how to apply the methodolgy to that type of site
EV: That sound like afgood idea
SA: I think we do have some of that notion where
we talk about different type of sites. We talk about what to do for sall
website. We can look into it more
... I do not think we should seperate into web iste typres because then we
have to categorize
EV: Goood dea Peter if you can come up with an idea
PK: I am happy to take that action
Alistair: Single page web apps, they are not so very different - but they do need to be seperated out
<agarrison> separate out single page web app from this document into a new web app eval doc, then drawing reference to that document if a website proper contains a single page web app
PK: Determining website types is heavy handed, we
need to talk about things you see and what is an approach to deal with this
when we see it
... We are not going to get into site clcification
... When you see this, do this
... without classifying
... Techniques to do to evaluate these diff things
SA: I dont mind, I was just worried about heavy
handed, but we seem to agree by not trying to classify web sites
... Please propose something Peter
<shadi> http://www.w3.org/TR/WCAG-EM/#specialcases
DF: Independent view of the random sample vs the
bias view. If the tester owns the tester.
... The nature of random you cannot eplicate
<Mike_Elledge> Short article: http://www.custominsight.com/articles/random-sampling.asp
AS: Face to Face at CSUN, we may have a room, not confirmed, who can participate
<Mike_Elledge> Random sample calculator: http://www.custominsight.com/articles/random-sample-calculator.asp
AS: Please fill it out as soon as you can, if we have a large enough headcount we can go forward
PK: Are you anticipating a 2 day?
AS: Yes, Monday and Tuesday
TB: Will IRC be available, not sue about speacker phone
<Kathy> do we need 2 days? Could we do just Tuesday?
EV: Defien the context of web use, I deleted in
and moved the elevant info into other sections
... Please have a look and the timeline for next public Working Draft, do the
surveys
EVL On Monday a new Survey will be opened