14:53:34 <RRSAgent> RRSAgent has joined #eval
14:53:34 <RRSAgent> logging to http://www.w3.org/2013/01/24-eval-irc
14:53:36 <trackbot> RRSAgent, make logs world
14:53:38 <trackbot> Zakim, this will be 3825
14:53:38 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 7 minutes
14:53:39 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
14:53:39 <trackbot> Date: 24 January 2013
14:54:25 <Detlev> Detlev has joined #eval
14:56:30 <ericvelleman> ericvelleman has joined #eval
14:56:49 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
14:56:56 <Zakim> +Shadi
14:58:15 <Zakim> +[IPcaller]
14:58:30 <Zakim> + +31.65.127.aaaa
14:58:43 <shadi> zakim, ipcaller is Alistair
14:58:43 <Zakim> +Alistair; got it
14:58:46 <Zakim> +MartijnHoutepen
14:59:28 <Zakim> +Katie_Haritos_Shea
14:59:31 <MartijnHoutepen> Zakim, aaaa is ericvelleman
14:59:31 <Zakim> +ericvelleman; got it
15:00:02 <agarrison> agarrison has joined #eval
15:00:21 <shadi> zakim, Alistair is agarrison
15:00:21 <Zakim> +agarrison; got it
15:00:30 <Kathy> Kathy has joined #eval
15:00:33 <Zakim> +Detlev
15:00:39 <Ryladog_> Ryladog_ has joined #eval
15:00:40 <Mike_Elledge> Mike_Elledge has joined #eval
15:00:55 <Detlev> Zakim, mute me
15:00:56 <shadi> scribe: Ryladog_
15:00:56 <Zakim> Detlev should now be muted
15:00:59 <Zakim> +Kathy
15:01:42 <Zakim> + +1.313.322.aabb
15:02:13 <MartijnHoutepen> Zakim, please mute me
15:02:13 <Zakim> MartijnHoutepen should now be muted
15:02:14 <Ryladog_> zakim, next item
15:02:15 <Zakim> agendum 1. "Welcome, scribe" taken up [from MartijnHoutepen]
15:02:28 <Kathy> zakim, mute me
15:02:28 <Zakim> Kathy should now be muted
15:02:33 <Ryladog_> zakim, next item
15:02:33 <Zakim> agendum 1 was just opened, Ryladog_
15:02:44 <Ryladog_> zakim, next item
15:02:46 <Zakim> agendum 1 was just opened, Ryladog_
15:02:48 <shadi> zakim, close agendum 1
15:02:48 <Zakim> agendum 1, Welcome, scribe, closed
15:02:49 <Zakim> I see 6 items remaining on the agenda; the next one is
15:02:49 <Zakim> 2. New editor draft and DoC [from MartijnHoutepen]
15:03:02 <shadi> zakim, take up next
15:03:02 <Zakim> agendum 2. "New editor draft and DoC" taken up [from MartijnHoutepen]
15:03:33 <Detlev> nicely organised!
15:03:41 <Ryladog_> EV: Please review on the questionaire, I am asking for your approval or opinion
15:03:52 <Ryladog_> EV: There is a dif version
15:04:45 <Ryladog_> EV: A lot of things have changed, they are also included in the survey. After today  will make a new survey #8, that will have the chamges from today
15:05:15 <ericvelleman> q?
15:05:28 <Ryladog_> zakim, next item
15:05:28 <Zakim> agendum 3. "EvalTF Survey 7" taken up [from MartijnHoutepen]
15:05:39 <ericvelleman> <https://www.w3.org/2002/09/wbs/48225/evaltfq7/>
15:05:49 <Ryladog_> EV: The kinj is in the agenda and above
15:06:02 <Ryladog_> EV: Some of you have already filled out the survey
15:06:28 <Ryladog_> EV: There are some things I need to know, some things ask for some reading
15:06:41 <Ryladog_> EV: I tried to cover a lot of the comments
15:07:02 <Ryladog_> EV: I tried to cover as many of the comments as possible
15:07:26 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
15:07:38 <Ryladog_> EV: Therer is a long eda there. Next time, what you can expect on Monday, a new survry with the quesy=ions like 6 to 11
15:08:10 <Zakim> + +1.517.432.aacc
15:08:15 <Ryladog_> EV: That survey will be longer than this one. Please find time after Monday evening to eview the survey
15:08:34 <richard> richard has joined #eval
15:08:42 <ericvelleman> q?
15:08:47 <Ryladog_> EV: That is what is coming this week. Survey # 7 and next wekk Survey 8
15:09:09 <Ryladog_> zakim, next item
15:09:09 <Zakim> agendum 4. "Discuss new section Random sample" taken up [from MartijnHoutepen]
15:09:12 <shadi> zakim, who is on the call?
15:09:12 <Zakim> On the phone I see Shadi, agarrison, ericvelleman, MartijnHoutepen (muted), Katie_Haritos_Shea, Detlev (muted), Kathy (muted), +1.313.322.aabb, +1.517.432.aacc
15:09:25 <Ryladog_> EV: New section on random sample
15:09:29 <Zakim> +[IPcaller]
15:09:54 <Zakim> +Peter_Korn
15:10:06 <shadi> zakim, aabb is Mike_Elledge
15:10:07 <Zakim> +Mike_Elledge; got it
15:10:08 <Sarah_Swierenga> zakin, aacc
15:10:11 <korn_> korn_ has joined #eval
15:10:16 <shadi> zakim, aacc is Sarah_Swierenga
15:10:16 <Zakim> +Sarah_Swierenga; got it
15:10:17 <Mike_Elledge> zakin, aabb
15:10:31 <korn_> Zakim, who is here?
15:10:31 <Zakim> On the phone I see Shadi, agarrison, ericvelleman, MartijnHoutepen (muted), Katie_Haritos_Shea, Detlev (muted), Kathy (muted), Mike_Elledge, Sarah_Swierenga, [IPcaller], Peter_Korn
15:10:34 <Mike_Elledge> Mike_Elledge
15:10:35 <Zakim> On IRC I see korn_, richard, Sarah_Swierenga, Mike_Elledge, Ryladog_, Kathy, agarrison, ericvelleman, Detlev, RRSAgent, MartijnHoutepen, Bim, shadi, trackbot, Zakim
15:10:46 <richard> zakim, ??ipcaller is richard
15:10:46 <Zakim> sorry, richard, I do not recognize a party named '??ipcaller'
15:11:09 <richard> zakim, ipcaller is richard
15:11:09 <Zakim> +richard; got it
15:11:18 <shadi> zakim, mute me
15:11:18 <Zakim> Shadi should now be muted
15:11:51 <ericvelleman> <http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e>
15:12:01 <Ryladog_> EV: Randon Sample in step D
15:12:04 <korn_> korn_ has joined #eval
15:12:17 <shadi> regrets: Liz, Vivienne, Roberto
15:12:28 <Ryladog_> EV: There are many editor blocks
15:12:41 <Ryladog_> EV: It is based on the survey from last year
15:13:30 <MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130122#step3e
15:13:38 <Ryladog_> EV: This is the basis for what I wrote, it is relatively short. Figure 5 we need to discuss. Some folks indicated they would like a percentage.
15:13:46 <korn_> Q+
15:14:12 <Kathy> q+
15:14:15 <ericvelleman> q?
15:14:17 <richard> q+
15:14:21 <Ryladog_> EV: In the survey must people have 25 to 30 pages. Rarely over 30 pages
15:15:04 <Detlev> q+
15:15:09 <Ryladog_> PK: As far as the # of how many pages should be in th e sample. There are stats out there, I think we should use them. That recommended method
15:15:43 <Ryladog_> EV We did this before. We looked at liturature, it was hard to apply the lit to a methodology like this one
15:16:11 <shadi> q+
15:16:15 <Kathy> ack me
15:16:17 <shadi> q- korn
15:16:22 <Ryladog_> EV: If you take he scientific approach you end up with an enormous sample. But I agree with you if we could get a scinetific sampe to use
15:16:54 <Ryladog_> KW: I did some research based on those formeulas and it was a lot. And then what do you determine a page to be
15:17:38 <Ryladog_> KW: So it takes ys back to what is a page. It is going to be very difficult. The size/number needs to be cost effective
15:18:12 <Ryladog_> KW: When I do a randon sampling after they have made the site compliant after it has been remediated
15:18:14 <shadi> q-
15:18:48 <Ryladog_> KW: Taht way you do not find as many problem. In reality the random samle actually takes less tiem
15:18:50 <Kathy> zakim, mute me
15:18:50 <Zakim> Kathy should now be muted
15:19:45 <Ryladog_> Richard: the purpose of th Randon Sample it may include pages that you have already tested. Random should be random. To get a 95% reliabilit
15:20:25 <Ryladog_> Richard: If you go to Amazon where thousands of page are all the same, you got at least one of each page type
15:20:54 <Ryladog_> Richard: It doesnt ensure that you have covered the whole web site
15:21:05 <Ryladog_> EV: I hink we covered this somewhere
15:21:19 <Ryladog_> EV: In step 3F
15:21:24 <Detlev> ack me
15:21:29 <ericvelleman> q?
15:21:31 <shadi> q- r
15:21:48 <Ryladog_> Detlev: It should be an OPTIONAL part of the methodology
15:22:26 <Ryladog_> DF: It is a problem with web pages having a number fof state as Kathy indicated
15:22:31 <ericvelleman> q?
15:22:50 <Sarah_Swierenga> q+
15:22:58 <Ryladog_> DF: I hink it should be OPTIONAL, I agree with the questionaire
15:23:06 <shadi> q- s
15:23:41 <Ryladog_> SS: I agree it shold be optional. Some clients limit the size for you. It is hard to defend
15:23:46 <shadi> q+
15:23:46 <Ryladog_> +1
15:23:48 <Tim> Tim has joined #eval
15:23:49 <richard> I do not think random sample should be optional. It is a vital exercise for a large site to be sure that the WHOLE site is accesseeble
15:24:00 <shadi> ack me
15:24:01 <ericvelleman> q?
15:24:07 <MartijnHoutepen> q+
15:25:11 <MartijnHoutepen> q-
15:25:21 <Ryladog_> SA: I agree that it should be OPTIONAL. But I think that should not stop us from doing that. Ye WCAG group has very strong opinions about this. Random Sapling needs an approach that is indeed representative
15:25:27 <MartijnHoutepen> agree with shadi
15:25:43 <Ryladog_> SA: Make sure that the developer does NOT know which pages will be suggested
15:25:55 <Detlev> so *what way* should the random sample be created, according to the methodology??
15:25:55 <korn__> korn__ has joined #eval
15:25:59 <ericvelleman> q?
15:26:04 <korn__> Q+
15:26:07 <shadi> zakim, mute me
15:26:07 <Zakim> Shadi should now be muted
15:26:13 <agarrison> q+
15:26:14 <shadi> ack me
15:26:31 <Ryladog_> EV: I agree, but SS says that the clinet only wants you to do 9 pages
15:26:31 <ericvelleman> q?
15:27:39 <ericvelleman> q?
15:27:47 <Detlev> q+
15:27:49 <Zakim> +Tim_Boland
15:27:50 <Ryladog_> SA: Our approach does not specify a certain  number of pages. Some are tempate driven may only need a few. That means if you count that with any number of pages, there is a very good chance that this type of methodology may not be possible
15:27:55 <ericvelleman> q?
15:27:57 <shadi> zakim, mute me
15:27:57 <Zakim> Shadi should now be muted
15:28:02 <shadi> q- k
15:28:21 <Tim> q+
15:28:26 <Ryladog_> PK: Random Sampling I really have to believe there has to be ststa on this, as to what is ais a stats
15:28:51 <Ryladog_> PK: We need to spend time doing that research ourselves if that info is not out there
15:28:55 <Ryladog_> EV: I agree
15:29:21 <Ryladog_> PK: I realize timing is difficult. Perhaps we need to pursue research funding
15:29:54 <Ryladog_> EV: I could provide a survey for folks to put in their lit research
15:29:54 <ericvelleman> q?
15:30:00 <shadi> q+ to talk about statistical and existing approaches for sample size
15:30:18 <Ryladog_> PK: Thos eof us who work at Universirty talks to there Statistics department
15:30:24 <shadi> zakim, unmute me
15:30:24 <Zakim> Shadi should no longer be muted
15:30:44 <Ryladog_> Issue: Funding for Samping Stats
15:30:44 <trackbot> Created ISSUE-11 - Funding for Samping Stats; please complete additional details at <http://www.w3.org/WAI/ER/2011/eval/track/issues/11/edit>.
15:30:58 <korn_> korn_ has joined #eval
15:31:37 <shadi> zakim, mute me
15:31:37 <Zakim> Shadi should now be muted
15:31:59 <shadi> ack ag
15:32:43 <Ryladog_> Alistair: How do you collect the random sampke without your natural bias coming in to play. Second point, how do you determine the actual size of the web site for a statistical anaysis tool input?
15:33:12 <korn_> Q+
15:33:23 <Ryladog_> Alistair: 3rd, how do you find the more relevant pages >
15:33:47 <korn__> korn__ has joined #eval
15:33:55 <Ryladog_> EV: There are many way, we need to provide one
15:34:14 <Ryladog_> EV: The size of a website we need to look into that
15:34:39 <Ryladog_> EV: Isnt your third point covered in the random sampe text
15:35:02 <Ryladog_> EV: Mentioning what Shadi said does that not cover it?
15:35:15 <Ryladog_> Alistair: the 80/20 rule
15:35:39 <korn> korn has joined #eval
15:35:43 <Ryladog_> Alistair: Random Sampling is supposed to back up the actual sampke
15:35:49 <korn> q?
15:35:52 <Ryladog_> Alistair: Random selection
15:36:00 <shadi> +1 to that last point
15:36:00 <korn> q+
15:36:02 <Detlev> ack me
15:36:07 <korn__> Q-
15:36:23 <korn__> q-
15:36:23 <Ryladog_> DF: I fully agree with Alistair. I think auto methods is best
15:36:27 <shadi> q- korn_
15:37:20 <Ryladog_> DF: Using a crawler. The other thing, we should not raise the bar to high as to make it too difficlut and folks would not want to use it. We want to make it easy
15:38:17 <Ryladog_> DF: If there is a simpler way to do Random Sampling it would be difficult to include with all the ststes of pages
15:38:24 <shadi> ack t
15:39:01 <Ryladog_> TB: I sploke to our Stats deartment it is important to write thre requirements in trem of statistical reqiremenets
15:39:17 <Ryladog_> TB: The assertions need to be statistically based
15:40:03 <Detlev> Tim, how would you deal with page states in a statistical way?
15:40:21 <Ryladog_> TB: That is the feedback I got from our Stats department. Requirements need to be statistically based. I have a paper on an entirely different sibject. But it covers that same concern, how to random sample
15:40:32 <Mike_Elledge> q+
15:40:48 <Ryladog_> TB: I will share it
15:41:05 <Ryladog_> TB: The methods would be helpful
15:41:08 <Kathy> I would like to see it Tim
15:41:13 <shadi> ack me
15:41:14 <Zakim> shadi, you wanted to talk about statistical and existing approaches for sample size
15:41:56 <Ryladog_> SA: A couple of good thngs. I like Alisters suggestion t call it Sampke Selections because if we go down the stats path
15:42:29 <Mike_Elledge> q-
15:42:34 <Ryladog_> SA: We have a lot pf effort for fairly little gain
15:43:44 <Ryladog_> SA: Again, the assumption is evelauator may be biased in the selection, by building in a portion is randomly selected, we have a verification for that 80/20
15:44:12 <Detlev> If random selection cannot be independently verified (and that would be very hard), every evaluator can just *claim* to have picked pages randomly!
15:44:18 <Ryladog_> SA: That help to determine that it is indeed indicative and to validate
15:44:40 <Ryladog_> SA: To the best we can we need to remove the bias without too much overhead
15:45:02 <shadi> zakim, mute me
15:45:02 <Zakim> Shadi should now be muted
15:45:07 <shadi> ack k
15:46:11 <Ryladog_> PK: We need to have a flow chart. What category of website you are dealing with. Highly structured is going to be very different than a highly varied, verses highly dynamic sites
15:47:00 <Ryladog_> PK: Even with Amazon random sampling is still important because you dont know that prer alt etext is provided for all books
15:47:15 <ericvelleman> q?
15:47:19 <Ryladog_> PK: What type of web site/app you are looking at
15:47:24 <shadi> q+
15:48:01 <Ryladog_> PK: Maybe we need a section early on that describes the diff classses of website and how to apply the methodolgy to that type of site
15:48:07 <shadi> ack me
15:48:10 <Ryladog_> EV: That sound like afgood idea
15:48:29 <agarrison> q+
15:48:56 <Ryladog_> SA: I think we do have some of that notion where we talk about different type of sites. We talk about what to do for sall website. We can look into it more
15:49:21 <korn> q+
15:49:27 <Ryladog_> SA: I do not think we should seperate into web iste typres because then we have to categorize
15:49:35 <shadi> zakim, mute me
15:49:35 <Zakim> Shadi should now be muted
15:49:52 <Ryladog_> EV: Goood dea Peter if you can come up with an idea
15:50:01 <Ryladog_> PK: I am happy to take that action
15:50:53 <Zakim> -MartijnHoutepen
15:51:10 <Ryladog_> Alistair: Single page web apps, they are not so very different - but they do need to be seperated out
15:51:13 <ericvelleman> q?
15:51:14 <agarrison> separate out single page web app from this document into a new web app eval doc, then drawing reference to that document if a website proper contains a single page web app
15:51:25 <Zakim> +MartijnHoutepen
15:51:29 <Detlev> q+
15:51:30 <shadi> q+
15:51:37 <shadi> q- ag
15:51:42 <shadi> q- ko
15:51:42 <MartijnHoutepen> Zakim, please mute me
15:51:44 <Zakim> MartijnHoutepen should now be muted
15:52:22 <Ryladog_> PK: Determining website types is heavy handed, we need to talk about things you see and what is an approach to deal with this when we see it
15:52:39 <Ryladog_> PK: We are not going to get into site clcification
15:52:54 <Ryladog_> PK: When you see this, do this
15:53:05 <Ryladog_> PK: without classifying
15:53:20 <Ryladog_> PK: Techniques to do to evaluate these diff things
15:53:21 <ericvelleman> q?
15:53:28 <shadi> ack me
15:54:04 <Ryladog_> SA: I dont mind, I was just worried about heavy handed, but we seem to agree by not trying to classify web sites
15:54:40 <Ryladog_> SA: Please propose something Peter
15:54:57 <shadi> http://www.w3.org/TR/WCAG-EM/#specialcases
15:55:48 <Ryladog_> DF: Independent view of the random sample vs the bias view. If the tester owns the tester.
15:56:15 <Ryladog_> DF: The nature of random you cannot eplicate
15:56:38 <Detlev> Zakim, mute me
15:56:38 <Zakim> Detlev should now be muted
15:56:38 <ericvelleman> q?
15:56:44 <Detlev> q-
15:56:44 <Mike_Elledge> Short article: http://www.custominsight.com/articles/random-sampling.asp
15:57:13 <Ryladog_> AS: Face to Face at CSUN, we may have a room, not confirmed, who can participate
15:57:15 <Mike_Elledge> Random sample calculator: http://www.custominsight.com/articles/random-sample-calculator.asp
15:57:45 <Ryladog_> AS: Please fill it out as soon as you can, if we have a large enough headcount we can go forward
15:58:02 <Ryladog_> PK: Are you anticipating a 2 day?
15:58:14 <Ryladog_> AS: Yes, Monday and Tuesday
15:58:35 <Ryladog_> TB: Will IRC be available, not sue about speacker phone
15:58:41 <Kathy> do we need 2 days?  Could we do just Tuesday?
15:59:30 <Ryladog_> EV: Defien the context of web use, I deleted in and moved the elevant info into other sections
16:00:10 <Ryladog_> EV: Please have a look and the timeline for next public Working Draft, do the surveys
16:00:28 <Ryladog_> EVL On Monday a new Survey will be opened
16:00:52 <Ryladog_> shadi can you close out the minutes?
16:00:59 <MartijnHoutepen> zakim, unmute me
16:00:59 <Zakim> MartijnHoutepen should no longer be muted
16:01:02 <Zakim> -Tim_Boland
16:01:03 <Zakim> -Detlev
16:01:03 <Zakim> -Peter_Korn
16:01:04 <richard> bye
16:01:04 <Zakim> -agarrison
16:01:04 <Zakim> -Kathy
16:01:06 <Zakim> -Sarah_Swierenga
16:01:07 <ericvelleman> ericvelleman has left #eval
16:01:07 <Zakim> -Mike_Elledge
16:01:07 <Zakim> -ericvelleman
16:01:08 <Sarah_Swierenga> bye
16:01:09 <Zakim> -Shadi
16:01:09 <Zakim> -MartijnHoutepen
16:01:09 <Kathy> bye
16:01:16 <richard> bye
16:01:20 <shadi> trackbot, end meeting
16:01:20 <trackbot> Zakim, list attendees
16:01:20 <Zakim> As of this point the attendees have been Shadi, +31.65.127.aaaa, MartijnHoutepen, Katie_Haritos_Shea, ericvelleman, agarrison, Detlev, Kathy, +1.313.322.aabb, +1.517.432.aacc,
16:01:23 <Zakim> ... Peter_Korn, Mike_Elledge, Sarah_Swierenga, richard, Tim_Boland
16:01:26 <Zakim> -Katie_Haritos_Shea
16:01:26 <Zakim> -richard
16:01:26 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
16:01:27 <Zakim> Attendees were Shadi, +31.65.127.aaaa, MartijnHoutepen, Katie_Haritos_Shea, ericvelleman, agarrison, Detlev, Kathy, +1.313.322.aabb, +1.517.432.aacc, Peter_Korn, Mike_Elledge,
16:01:28 <Zakim> ... Sarah_Swierenga, richard, Tim_Boland
16:01:28 <trackbot> RRSAgent, please draft minutes
16:01:28 <RRSAgent> I have made the request to generate http://www.w3.org/2013/01/24-eval-minutes.html trackbot
16:01:29 <trackbot> RRSAgent, bye
16:01:29 <RRSAgent> I see no action items