13:49:01 <RRSAgent> RRSAgent has joined #eval
13:49:02 <RRSAgent> logging to http://www.w3.org/2013/04/11-eval-irc
13:49:03 <trackbot> RRSAgent, make logs world
13:49:04 <Zakim> Zakim has joined #eval
13:49:05 <trackbot> Zakim, this will be 3825
13:49:05 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 11 minutes
13:49:06 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:49:06 <trackbot> Date: 11 April 2013
13:50:01 <MartijnHoutepen> MartijnHoutepen has changed the topic to: WCAG 2.0 Evaluation Methodology Task Force
13:56:08 <Detlev> Detlev has joined #eval
13:57:20 <richard> richard has joined #eval
13:57:27 <ericvelleman> ericvelleman has joined #eval
13:57:54 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
13:58:01 <Zakim> +[IPcaller]
13:58:58 <Zakim> + +31.30.239.aaaa
13:59:08 <MartijnHoutepen> Zakim, aaaa is me
13:59:08 <Zakim> +MartijnHoutepen; got it
13:59:17 <richard> zakim, ipcaller is richard
13:59:17 <Zakim> +richard; got it
13:59:58 <MartijnHoutepen> zakim, what is on the agenda?
13:59:59 <Zakim> I see nothing on the agenda
14:00:21 <MartijnHoutepen> agenda+ welcome and documents
14:00:37 <MartijnHoutepen> agenda+ state of comments
14:00:51 <MartijnHoutepen> agenda+ test run
14:00:58 <Zakim> + +31.30.239.aabb
14:01:24 <ericvelleman> Zakim, aabb is me
14:01:24 <Zakim> +ericvelleman; got it
14:01:40 <korn> korn has joined #eval
14:02:20 <MartijnHoutepen> zakim, who's on the phone
14:02:20 <Zakim> I don't understand 'who's on the phone', MartijnHoutepen
14:02:26 <Zakim> + +49.404.318.aacc
14:02:27 <MartijnHoutepen> zakim, who's on the phone?
14:02:27 <Zakim> On the phone I see richard, MartijnHoutepen, ericvelleman, +49.404.318.aacc
14:03:07 <MartijnHoutepen> scribenick: Detlev
14:03:08 <Detlev> scribe: Detlev
14:03:19 <Vivienne> Vivienne has joined #eval
14:03:20 <MartijnHoutepen> zakim, aacc is Detlev
14:03:20 <Zakim> +Detlev; got it
14:03:37 <MartijnHoutepen> zakim, take up next
14:03:37 <Zakim> agendum 1. "welcome and documents" taken up [from MartijnHoutepen]
14:03:58 <Zakim> +[Oracle]
14:04:03 <korn> Zakim, Oracle has Peter_Korn
14:04:03 <Zakim> +Peter_Korn; got it
14:04:08 <Kathy> Kathy has joined #eval
14:04:14 <Zakim> +[IPcaller]
14:04:19 <Mike_Elledge> Mike_Elledge has joined #eval
14:04:20 <Vivienne> zakim, IPcaller is me
14:04:20 <Zakim> +Vivienne; got it
14:05:04 <Zakim> +Kathy_Wahlbin
14:05:15 <Detlev> Eric: public working draft:   the wording of  link is wrong
14:05:35 <Zakim> + +1.313.322.aadd
14:05:46 <MartijnHoutepen> zakim, take up next
14:05:46 <Zakim> agendum 2. "state of comments" taken up [from MartijnHoutepen]
14:05:49 <ericvelleman> http://lists.w3.org/Archives/Public/public-wcag-em-comments/
14:06:13 <MartijnHoutepen> who just joined?
14:06:45 <Detlev> Eric: a few organisations are still preparing comments that should be in by the 15. April
14:06:55 <MartijnHoutepen> zakim, aadd is Mike_Elledge
14:06:55 <Zakim> +Mike_Elledge; got it
14:07:56 <MartijnHoutepen> zakim, please mute me
14:07:56 <Zakim> MartijnHoutepen should now be muted
14:08:03 <MartijnHoutepen> zakim, take up next
14:08:03 <Zakim> agendum 3. "test run" taken up [from MartijnHoutepen]
14:08:24 <ericvelleman> <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Apr/0014.html>
14:08:54 <Detlev> Eric: Discussing how to organise the test run
14:09:50 <Detlev> Eric: discussing purpose of testng
14:10:10 <korn> q+
14:10:33 <ericvelleman> q?
14:10:36 <Vivienne> q+
14:10:44 <MartijnHoutepen> q- korn
14:11:18 <Detlev> Peter: some thoughts (will provide in email): do target audiendces fnd guidance clear?
14:11:47 <Detlev> Peter: 'multiple' instead of two evaluators
14:11:59 <Vivienne> zakim, ack me
14:11:59 <Zakim> I see no one on the speaker queue
14:12:03 <Detlev> Peter: will suggest changes in a mail
14:12:34 <MartijnHoutepen> q- viv
14:12:58 <Detlev> Vivienne: Approached web site owners to ask to allow testing of their sites as objects evaluation of methodology
14:13:13 <Detlev> Vivienne: site owners were very receptive
14:14:22 <Detlev> Eric: purpose of testing in this phase is to check suitability/ applicability of our sampling process
14:15:21 <Vivienne> q+
14:15:25 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
14:15:34 <Detlev> Eric: discussed with Shadi the usefulness of the WBS system (the systwem usedfor questionnaires / surveys)
14:15:47 <korn> q+
14:15:53 <Vivienne> ack me
14:15:54 <Zakim> + +1.517.432.aaee
14:15:55 <Detlev> Eric: to be used for recording results
14:16:19 <MartijnHoutepen> Zakim, aaee is Sarah_Swierenga
14:16:19 <Zakim> +Sarah_Swierenga; got it
14:16:34 <MartijnHoutepen> q- viv
14:16:43 <Detlev> Vivienne: Other purposes: an organisation that was already evaluated would be interested if out test run would produce similar results
14:17:28 <Tim> Tim has joined #eval
14:17:49 <Detlev> Eric: Braillenet di such a comparison, looking at differences in results between their common approach and WCAG EM
14:18:09 <Vivienne> zakim, mute me
14:18:09 <Zakim> Vivienne should now be muted
14:18:13 <ericvelleman> ericvelleman has joined #eval
14:18:18 <ericvelleman> test
14:18:23 <ericvelleman> q?
14:18:36 <MartijnHoutepen> q- korn
14:19:05 <Detlev> Peter: the purpose questions should be direct questions to beta testers
14:19:49 <Detlev> Peter: is was not clear whether we would explicitly ask testers to answer these questions
14:20:25 <Detlev> Eric: introducing next part, the cautions
14:22:02 <Detlev> Eric: discussing purpose (NOT testing WCAG but the methodology, anonymising site names, etc.)
14:22:37 <ericvelleman> q?
14:22:37 <Detlev> Eric: Any comments on cautions?
14:22:40 <Vivienne> q+
14:22:48 <Vivienne> zakim, ack me
14:22:48 <Zakim> unmuting Vivienne
14:22:49 <Zakim> I see no one on the speaker queue
14:23:25 <Detlev> Vivienne: Caution for ourselves - declaration of interest / involvement with any particular website being evaluatewd
14:23:45 <korn> q+
14:23:57 <Detlev> Eric: may not be necessary here since we are just focusing on WCAG EM
14:24:22 <Vivienne> zakim, mute me
14:24:23 <Detlev> Vivienne: The bias may still influence our approach
14:24:24 <Zakim> Vivienne should now be muted
14:24:26 <ericvelleman> q?
14:24:31 <MartijnHoutepen> q- kor
14:24:52 <Detlev> Peter: if we test random sampling, intimate knowledge of site may be in the way
14:25:28 <ericvelleman> q?
14:26:03 <Detlev> Eric: discussing third topic of Email, plan
14:26:33 <Detlev> Eric: we could use two or three web sites
14:26:51 <Detlev> Eric: three or four nominated web sites
14:27:08 <Detlev> Eric: will share names soon (but only EVAL TF internal)
14:27:29 <korn> q+
14:27:32 <Detlev> Eric: First test run could be first three steps (incl. sampling)
14:28:04 <Detlev> Eric: then discuss outcome on list or the WBS system
14:28:20 <Vivienne> q+
14:28:24 <ericvelleman> q?
14:28:29 <Detlev> Eric: differneces found would be interesting input for open discussion on list
14:28:40 <MartijnHoutepen> q- korn
14:28:53 <Detlev> Peter; concerned about idea of splitting this up
14:29:27 <Detlev> Peter: important to do a full end-to-end run; the second time should go the whole way
14:29:29 <Zakim> +Tim_Boland
14:29:38 <ericvelleman> q?
14:30:10 <korn> q-
14:30:18 <Detlev> Peter: ending with sampling could leasd to a situation where the different samples that are used wmight not impact theresult
14:31:09 <Vivienne> zakim, ack me
14:31:09 <Zakim> unmuting Vivienne
14:31:10 <Zakim> I see no one on the speaker queue
14:31:17 <Detlev> Eric: also interesting if different result would occur even with the same sample
14:31:53 <Detlev> Vivienne: most of the time client will have had recommendations / constraints regarding the sample size
14:32:27 <korn> q+
14:32:31 <Vivienne> zakim, mute me
14:32:31 <Zakim> Vivienne should now be muted
14:32:37 <Detlev> Vivienne: among us we will have different notions of what an adequate sample is
14:33:00 <Vivienne> so much depends upon their budget
14:33:10 <Detlev> Eric: if we come up in first part with samples of very different sizes, this needs to be adressed
14:33:21 <ericvelleman> q?
14:33:23 <Detlev> q+
14:34:00 <Detlev> Peter: another purpose question: is size of sample in line with typical sample size the reviewer would have otherwise used?
14:34:02 <korn> q-
14:36:04 <ericvelleman> q?
14:36:18 <Detlev> Detlev: there may not be a natural size of sample - depends on quality expectations
14:37:14 <Detlev> Detlev: is it worth having a 50 page sample if it just uncovers a few minor extra problems?
14:37:46 <Detlev> Eric: a few people should do a full run at least
14:37:50 <Detlev> q+
14:38:15 <Vivienne> for part 1, the website I've provided for an example is one I've done following the methodology
14:38:46 <Vivienne> so in effect I've done the whole website's evaluation
14:40:01 <korn> q+
14:40:46 <MartijnHoutepen> q- De
14:40:50 <Detlev> Detlev: may be we should do the full rtun, but inter evaluator comparisons are difficult if the sample is not not identical
14:41:37 <Detlev> Peter: The miost important aim is to establish how close we are to 'done' with WCAG EM
14:42:05 <Detlev> Holisitc testing will be neeed to tell us how close we are to 'done'
14:42:41 <Detlev> (that was Peter)
14:43:16 <MartijnHoutepen> q+
14:43:16 <Detlev> Peter: Fine-grained info less important than a measure of how close we are to completion
14:43:21 <ericvelleman> q?
14:43:25 <MartijnHoutepen> ack me
14:44:12 <Detlev> Martijn: Split in parts will provide more information than a full run right now - we can use the input then do a holistic test
14:44:20 <MartijnHoutepen> zakim, mute me
14:44:21 <Zakim> MartijnHoutepen should now be muted
14:44:30 <MartijnHoutepen> q- korn
14:45:06 <ericvelleman> q?
14:46:10 <Detlev> Peter: if we focus on unit testing we should focus on the same sample (step 4)
14:46:14 <MartijnHoutepen> i agree
14:46:24 <Vivienne> q+
14:46:30 <Vivienne> ack me
14:46:43 <richard> I go for the full thing
14:46:44 <Detlev> Eric: Unit testing or full test run?
14:47:04 <Detlev> Vivienne: First test to see how close our  samples are
14:47:16 <Vivienne> zakim, mute me
14:47:16 <Zakim> Vivienne should now be muted
14:47:27 <Detlev> Vivienne: then compare step 4 based on same sample
14:47:39 <Detlev> Eric: asking Peter for software testing experience
14:48:20 <Detlev> Peter: Its not so much sw testing, more how testers read and apply the text of WCAG EM
14:48:48 <Detlev> Peter: better to vary just one variable (human being) not the sample
14:49:21 <Detlev> Peter: in the end we need to have all the variables varying
14:50:19 <Detlev> Eric: Let's start with part 1 (first three steps), asking Shadi what sites we will pick, out things in the WBS system
14:51:09 <Detlev> Eric: then based on selected websites, carry out first three steps, Eric posing as web site owner communicating requirements
14:51:13 <Vivienne>  q+
14:51:40 <Vivienne> zakim, ack me
14:51:40 <Zakim> unmuting Vivienne
14:51:41 <Zakim> I see no one on the speaker queue
14:51:47 <Detlev> Eric: should be out soobn (may be as early as tomorrow)
14:52:31 <Detlev> Vivienne: Sent info on organisation, wonders whether we need a memorandum of understanding so they know hoe their sites are going to be used
14:52:55 <Vivienne> zakim, mute me
14:52:55 <Zakim> Vivienne should now be muted
14:53:10 <Detlev> Eric: Let's leave that to Shadi
14:53:15 <Vivienne> zakim, ack me
14:53:15 <Zakim> unmuting Vivienne
14:53:17 <Zakim> I see no one on the speaker queue
14:53:39 <Detlev> Eric: MoU mentioned in mail to Shadi?
14:53:51 <Detlev> Vivienne: probably
14:53:58 <Detlev> Eric: will remind Shadi
14:54:18 <Mike_Elledge> q+
14:54:50 <ericvelleman> q?
14:54:51 <Detlev> Eric: will send mail with site info and ask us to conduct steps 1-3
14:54:53 <Detlev> que!
14:55:25 <Detlev> Mike: We can avoid issues with companies would be to use W3C sites...
14:55:27 <korn> q+
14:55:49 <Detlev> Eric: W3C site is gigantic heterogenous, not so suitable
14:55:51 <MartijnHoutepen> q- mike
14:56:15 <Detlev> Peter: A gigantic site might be good for step one
14:56:19 <Detlev> q+
14:57:16 <Detlev> Peter: if W3C site contains samples of bad sites and what not to do - good for sampling, problem if these would not be found
14:57:39 <Detlev> Peter: Advantage that discussion of site could be open
14:57:45 <Mike_Elledge> :^)
14:59:05 <Vivienne> zakim, ack me
14:59:05 <Zakim> I see korn, Detlev on the speaker queue
14:59:11 <korn> q+
14:59:14 <Vivienne> q+
14:59:20 <MartijnHoutepen> q- det
14:59:22 <Vivienne> q-
14:59:32 <Detlev> Detlev: prefers a more typical site
14:59:36 <Zakim> -Kathy_Wahlbin
15:00:03 <Zakim> -Tim_Boland
15:00:12 <Detlev> Peter: a large site is still useful if the result is: the site is too big to apply WCAG EM to
15:00:36 <Zakim> -Mike_Elledge
15:00:55 <MartijnHoutepen> ack me
15:01:01 <Zakim> -[Oracle]
15:01:02 <Zakim> -Sarah_Swierenga
15:01:08 <Vivienne> bye now
15:01:11 <Zakim> -richard
15:01:14 <MartijnHoutepen> trackbot, end meeting
15:01:14 <trackbot> Zakim, list attendees
15:01:14 <Zakim> As of this point the attendees have been +31.30.239.aaaa, MartijnHoutepen, richard, +31.30.239.aabb, ericvelleman, +49.404.318.aacc, Detlev, Peter_Korn, Vivienne, Kathy_Wahlbin,
15:01:16 <ericvelleman> ericvelleman has left #eval
15:01:17 <Zakim> ... +1.313.322.aadd, Mike_Elledge, +1.517.432.aaee, Sarah_Swierenga, Tim_Boland
15:01:17 <Zakim> -Vivienne
15:01:18 <Zakim> -Detlev
15:01:20 <richard> bye
15:01:22 <trackbot> RRSAgent, please draft minutes
15:01:22 <RRSAgent> I have made the request to generate http://www.w3.org/2013/04/11-eval-minutes.html trackbot
15:01:23 <trackbot> RRSAgent, bye
15:01:23 <RRSAgent> I see no action items
15:01:24 <Zakim> -ericvelleman