13:58:03 <RRSAgent> RRSAgent has joined #eval
13:58:03 <RRSAgent> logging to http://www.w3.org/2013/06/13-eval-irc
13:58:05 <trackbot> RRSAgent, make logs world
13:58:05 <Zakim> Zakim has joined #eval
13:58:07 <trackbot> Zakim, this will be 3825
13:58:07 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 2 minutes
13:58:08 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:58:08 <trackbot> Date: 13 June 2013
14:00:15 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
14:00:16 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
14:00:22 <Zakim> +[IPcaller]
14:00:28 <Vivienne> Zakim, IPcaller is me
14:00:28 <Zakim> +Vivienne; got it
14:00:45 <Zakim> + +31.30.239.aaaa
14:00:52 <ericvelleman> ericvelleman has joined #eval
14:00:55 <Zakim> + +1.510.334.aabb
14:00:57 <MartijnHoutepen> Zakim, aaaa is me
14:00:57 <Detlev> Detlev has joined #eval
14:00:57 <Zakim> +MartijnHoutepen; got it
14:01:46 <Liz> Liz has joined #eval
14:01:59 <Zakim> + +49.404.318.aacc
14:02:01 <ericvelleman> Joining in a minute, trying to find the phone
14:02:11 <Zakim> + +1.517.432.aadd
14:02:25 <Detlev> Zakim, aacc is Detlev
14:02:25 <Zakim> +Detlev; got it
14:02:32 <MartijnHoutepen> Zakim, who is on the phone?
14:02:32 <Zakim> On the phone I see Vivienne, MartijnHoutepen, +1.510.334.aabb, Detlev, +1.517.432.aadd
14:02:45 <Zakim> + +1.301.975.aaee
14:03:20 <Sarah_Swierenga> aadd is me
14:03:22 <Zakim> +Eric_Velleman
14:03:34 <richard> richard has joined #eval
14:03:41 <MartijnHoutepen> Zakim, aadd is Sarah
14:03:41 <Zakim> +Sarah; got it
14:03:46 <MartijnHoutepen> Zakim, who is on the phone?
14:03:46 <Zakim> On the phone I see Vivienne, MartijnHoutepen, +1.510.334.aabb, Detlev, Sarah, +1.301.975.aaee, Eric_Velleman
14:03:48 <Zakim> + +1.313.322.aaff
14:03:48 <Detlev> Zakim, mute me
14:03:48 <Zakim> Detlev should now be muted
14:04:02 <Mike_Elledge> Mike_Elledge has joined #eval
14:04:06 <Liz> zakim, aaee is Liz
14:04:06 <Zakim> +Liz; got it
14:04:17 <Zakim> +[IPcaller]
14:04:55 <MartijnHoutepen> Zakim, [IPcaller] is richard
14:04:55 <Zakim> +richard; got it
14:05:32 <Zakim> + +1.650.506.aagg
14:05:34 <MartijnHoutepen> scribe: Sarah
14:05:43 <korn> korn has joined #eval
14:05:46 <Zakim> - +1.510.334.aabb
14:05:59 <korn> Zakim, who is here?
14:05:59 <Zakim> On the phone I see Vivienne, MartijnHoutepen, Detlev (muted), Sarah, Liz, Eric_Velleman, +1.313.322.aaff, richard, +1.650.506.aagg
14:06:01 <Zakim> On IRC I see korn, Mike_Elledge, richard, Liz, Detlev, ericvelleman, Sarah_Swierenga, Zakim, RRSAgent, Vivienne, MartijnHoutepen, shadi, Bim, trackbot
14:06:18 <korn> Zakim, aagg is Peter_Korn
14:06:18 <Zakim> +Peter_Korn; got it
14:06:41 <ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611
14:07:01 <MartijnHoutepen> Zakim, aaff is probably Mike
14:07:01 <Zakim> +Mike?; got it
14:07:36 <MartijnHoutepen> Zakim, mute me
14:07:36 <Zakim> MartijnHoutepen should now be muted
14:08:11 <MartijnHoutepen> diff version: http://services.w3.org/htmldiff?doc1=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20130219&doc2=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20130611
14:08:24 <Zakim> +Katie_Haritos-Shea
14:08:36 <Sarah_Swierenga> eric: 'common functionality' to 'core functionality' discussion
14:09:23 <Vivienne> Eric, do you have a link to the disposition of comments?
14:09:27 <Vivienne> zakim, mute me
14:09:27 <Zakim> Vivienne should now be muted
14:09:45 <ericvelleman> q?
14:09:48 <Sarah_Swierenga> http://www.w3.org/WAI/ER/conformance/comments-20130226 disposition of commetns
14:09:52 <MoeKraft> MoeKraft has joined #eval
14:09:54 <Detlev> q+
14:09:59 <Detlev> ack me
14:10:17 <Zakim> +MoeKraft
14:10:28 <Ryladog> Ryladog has joined #eval
14:10:54 <Sarah_Swierenga> detlev: thinks that Step 3 does not include enough dynamic states
14:11:07 <Vivienne> q+
14:11:15 <Vivienne> zakim, ack me
14:11:15 <Zakim> unmuting Vivienne
14:11:16 <Zakim> I see no one on the speaker queue
14:12:16 <Sarah_Swierenga> vivienne: additional comments in email from Ryladog(?) should be added to disposition of comments
14:12:33 <Vivienne> Sarah, that was comments from Giorgio Brajnik
14:12:58 <Sarah_Swierenga> eric: will add to comments doc
14:13:18 <Detlev> Zakim, mute me
14:13:18 <Zakim> Detlev should now be muted
14:13:45 <Detlev> fine!
14:13:50 <MartijnHoutepen> +1
14:13:55 <Liz> +1
14:13:57 <Vivienne> +1
14:13:57 <ericvelleman> +1
14:14:01 <Vivienne> zakim, mute me
14:14:01 <Zakim> Vivienne should now be muted
14:14:04 <Mike_Elledge> +1
14:14:06 <Sarah_Swierenga> eric: proposing to add to comments even though public comment period is over
14:14:09 <Sarah_Swierenga> +1
14:14:15 <MoeKraft> +1
14:14:29 <Detlev> yes!
14:14:30 <richard> +1
14:15:03 <ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226
14:15:09 <Zakim> +Shadi
14:15:17 <shadi> zakim, mute me
14:15:17 <Zakim> Shadi should now be muted
14:17:05 <ericvelleman> q?
14:17:11 <Sarah_Swierenga> eric: Disposition of Comments id#3 replace 'common functionality' with 'core functionality' - also mentioned a couple of other places in comments; there will be a survey to decide this
14:17:13 <korn> q+
14:18:08 <Sarah_Swierenga> peter: appreciates example of going from common to core. is the expectation that all comments will have a proposed resolution in the next survey?
14:19:29 <Sarah_Swierenga> eric: yes, proposed resolutions will be in the survey linked to the editor draft. Eric will group more of the comments for the survey.
14:19:45 <ericvelleman> q?
14:21:31 <ericvelleman> : <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0046.html>
14:21:35 <Detlev> ack me
14:21:42 <Sarah_Swierenga> eric: agenda pt 3 Design support evaluation versus conformance evaluation
14:22:03 <Vivienne> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0046.html
14:22:52 <shadi> q+
14:23:09 <korn> q+
14:23:40 <Vivienne> q+
14:23:49 <Sarah_Swierenga> detlev: three different items on goals (basic, detailed, and in-depth report); initial vs final evaluation for finished content. maybe reorder and rephrase report names.
14:24:01 <ericvelleman> q?
14:24:04 <shadi> ack k
14:24:47 <Sarah_Swierenga> peter: agrees with primary two uses, but having 3 choices are the wrong reports to have. Having use cases dictate the report type is useful.
14:24:58 <Detlev> Zakim, mute me
14:24:58 <Zakim> Detlev should now be muted
14:25:12 <Sarah_Swierenga> peter: still concerned with the word, 'conformance' since we aren't looking at every page
14:25:42 <Ryladog> q+
14:25:46 <Sarah_Swierenga> peter: talk about sampling, problems found, and confidence in results found.
14:25:51 <shadi> ack me
14:26:38 <Sarah_Swierenga> shadi: it would be an accurate representation of how well the site conforms overall.
14:27:40 <Sarah_Swierenga> shadi: agrees with use case driven reports. how about when product owner wants to do an interim review to find out how the site is progressing.
14:27:50 <shadi> zakim, mute me
14:27:50 <Zakim> Shadi should now be muted
14:27:50 <korn> +1 to that use case
14:27:56 <Sarah_Swierenga> shadi: +1 use case
14:27:58 <Vivienne> ack me
14:28:00 <MartijnHoutepen> +1
14:29:29 <Detlev> "Define the Goal of the Evaluation" -> 1. Development support evaluation 2. Conformance evaluation?
14:29:32 <Sarah_Swierenga> vivienne: agrees with Detlev. might want to do a quick pass during development (basic report), but also could have an assessment of functionality after it's completed. wants to emphasize that this protocol is useful during development too.
14:29:48 <korn> q+
14:29:56 <Detlev> q+
14:30:07 <MartijnHoutepen> q- ryla
14:30:10 <Sarah_Swierenga> katie: conformance discussion - it's ok that this protocol is associated with performance, but want to say that it's for the pages tested.
14:30:15 <Vivienne> zakim, mute me
14:30:15 <Zakim> Vivienne should now be muted
14:31:10 <Tim> Tim has joined #eval
14:31:10 <Sarah_Swierenga> peter: concern is the language of WCAG conformance is that you can only make a claim if it's perfect. we need to have language that is specific to the parameters of the sample.
14:31:14 <Ryladog> I agree with this be  associated with conformance for the pages tested. WCAG Conformance is per page
14:31:46 <korn> q-
14:32:03 <Detlev> ack me
14:32:04 <Vivienne> q+
14:33:05 <ericvelleman> :-)
14:33:11 <Sarah_Swierenga> detlev: agrees with peter, not to focus so much on conformance. the value is showing that the site is pretty darn good. define the goal of evaluation and report types is a bit of a mismatch.
14:33:46 <ericvelleman> q?
14:33:53 <Sarah_Swierenga> detlev: development support evaluations and full/formal evaluations
14:34:35 <Sarah_Swierenga> detlev: evaluation of a legacy site is another variation. they know the site is bad, but still want to do the test
14:34:35 <ericvelleman> q?
14:34:46 <Vivienne> ack me
14:34:49 <Detlev> Zakim, mute me
14:34:49 <Zakim> Detlev should now be muted
14:35:42 <richard> q+
14:35:51 <Sarah_Swierenga> vivienne: can't the product owner still say that the site was evaluated according to WCAG-EM with the specific sample?
14:36:21 <shadi> q+
14:36:27 <Vivienne> zakim, mute me
14:36:27 <Zakim> Vivienne should now be muted
14:36:30 <Detlev> Conformance clains according to WCAG 2.0 apply to individual pages anyway...
14:36:41 <shadi> zakim, unmute me
14:36:41 <Zakim> Shadi should no longer be muted
14:36:55 <Vivienne> ack me
14:37:12 <Sarah_Swierenga> peter: asks for clarification of Vivienne's comment
14:37:35 <shadi> zakim, mute me
14:37:35 <Zakim> Shadi should now be muted
14:38:56 <Detlev> q+
14:39:09 <Sarah_Swierenga> vivienne: if i evaluate the website according to WCAG-EM, and we produce a report with the problems identified, then the site is retested a new sample later after the problems were fixed, can't they make a claim that the website for that sample was compliant with WCAG A, AA or whatever?
14:39:50 <Sarah_Swierenga> peter: can't claim conformance for the site from the collection of pages
14:39:50 <Zakim> +Tim_Boland
14:40:09 <Ryladog> You can say for the pages tested on that date
14:40:14 <shadi> +1 to Eric
14:40:18 <Sarah_Swierenga> eric: can't ever say everything is perfect, but you can say that the pages tested are good.
14:40:45 <Vivienne> ack me
14:40:54 <Sarah_Swierenga> peter: have to look at every page before conformance for the site can be claimed.
14:41:41 <Sarah_Swierenga> vivienne: when a very indicative sample of pages is chosen, then the testing/retesting is done, they should be able to make a claim about the site based on the sampling.
14:42:02 <Vivienne> zakim, mute me
14:42:02 <Zakim> Vivienne should now be muted
14:42:26 <Sarah_Swierenga> eric: we can say something like this, but it is all about the confidence in the sampling. maybe we need to adjust/clarify this in the draft
14:42:50 <shadi> ack me
14:42:57 <MartijnHoutepen> q- richard
14:42:57 <shadi> q- rich
14:42:58 <Sarah_Swierenga> richard: agrees with parking this discussion since it's not on the agenda
14:44:46 <Sarah_Swierenga> shadi: need to provide a template where evaluators can indicate the number of pages, sampling x amount of pages, etc., that can be filled out. 50 pages vs 1 page review will give a strong indication of the confidence in the sampling.
14:45:01 <shadi> zakim, mute me
14:45:01 <Zakim> Shadi should now be muted
14:45:27 <Sarah_Swierenga> eric: 'confidence' needs to be expanded in the document
14:45:49 <Detlev> ack me
14:46:14 <shadi> q+ to talk about when an evaluation is done, and how rigorous it is
14:47:14 <Sarah_Swierenga> detlev: basic and detailed evaluation report types; support evaluation could be used for legacy sites
14:47:21 <shadi> ack me
14:47:22 <Zakim> shadi, you wanted to talk about when an evaluation is done, and how rigorous it is
14:48:24 <shadi> zakim, mute me
14:48:24 <Zakim> Shadi should now be muted
14:48:27 <Sarah_Swierenga> shadi: use cases for evaluation - evaluations done at different types in time. is there another dimension regarding the 'depth' of the evaluation?
14:48:38 <Vivienne> q+
14:48:40 <Vivienne> ack me
14:48:50 <shadi> ack me
14:49:42 <Detlev> q+
14:49:49 <Vivienne> q-
14:50:15 <shadi> zakim, mute me
14:50:15 <Zakim> Shadi should now be muted
14:50:16 <Sarah_Swierenga> shadi: agrees with detlev and peter that having use cases is good, but what about other variations other than just having a basic report. depth vs point in time when eval is taking place
14:50:16 <Detlev> ack me
14:50:44 <korn> q+
14:51:31 <Sarah_Swierenga> detlev: true for different levels of depth or to show that the site is terrible, but normally the aim is to cover the main templates and functionality. want to avoid making this too complex or complicated. likes the idea of a score for level of conformance.
14:52:14 <Detlev> Zakim, mute me
14:52:14 <Zakim> Detlev should now be muted
14:52:42 <Sarah_Swierenga> peter: likes shadi's third use case related to regression, i.e., this site has improved a little, a lot, or has gone backwards. likes the idea of showing progress.
14:52:43 <korn> q-
14:53:22 <Sarah_Swierenga> eric: agenda #4. To sample or not to sample - <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0028.html>
14:54:07 <Vivienne> q+
14:54:20 <Vivienne> ack me
14:54:57 <MartijnHoutepen> q+
14:54:58 <Sarah_Swierenga> vivienne: we cover it fairly well by saying that sometimes the sample is the whole website, but when it's not feasible, then a sample is appropriate.
14:55:41 <Vivienne> zakim, mute me
14:55:41 <Zakim> Vivienne should now be muted
14:56:04 <Sarah_Swierenga> vivienne: recommends researching the effect of sample size and it's agreement with results from evaluating every page.
14:56:14 <richard> q+
14:56:22 <MartijnHoutepen> ack me
14:56:31 <Sarah_Swierenga> sarah: i meant 'its' rather than 'it's'
14:57:12 <Sarah_Swierenga> martijn: this is covered well already in the document. likes the way it is in the doc right now.
14:58:35 <Sarah_Swierenga> richard: just came through a series of evaluations where it was cheaper to do the whole site, but there are formulae for determining the proper sampling size. evaluator has to do a cost-benefit analysis of conducting the eval on the whole site vs coming up with a sampling strategy.
14:59:09 <Sarah_Swierenga> richard: when doing a random selection, you'll end up with some pages that you had already pre-selected, which is okay.
14:59:49 <Sarah_Swierenga> q+
14:59:50 <richard> q+
15:00:24 <Zakim> -Katie_Haritos-Shea
15:00:30 <Sarah_Swierenga> richard: evaluator needs to decide on the sampling/whole site eval strategy
15:01:28 <Sarah_Swierenga> sarah: agrees with leaving the sampling up to the evaluator - and leaving the doc alone.
15:02:04 <Liz> Bye
15:02:09 <Sarah_Swierenga> eric: will make a editor draft, survey, etc. for the next meeting, which will be on June 27 (not June 20). meeting in two weeks.
15:02:09 <Vivienne> okay talk toyou all then
15:02:10 <Sarah_Swierenga> bye
15:02:10 <Zakim> -richard
15:02:11 <Detlev> bye
15:02:11 <Zakim> -MoeKraft
15:02:13 <Zakim> -Liz
15:02:15 <Zakim> -Eric_Velleman
15:02:16 <ericvelleman> ericvelleman has left #eval
15:02:16 <Zakim> -Vivienne
15:02:16 <shadi> trackbot, end meeting
15:02:16 <trackbot> Zakim, list attendees
15:02:17 <Mike_Elledge> bye1
15:02:17 <Zakim> As of this point the attendees have been Vivienne, +31.30.239.aaaa, +1.510.334.aabb, MartijnHoutepen, +49.404.318.aacc, +1.517.432.aadd, Detlev, +1.301.975.aaee, Eric_Velleman,
15:02:17 <Zakim> ... Sarah, +1.313.322.aaff, Liz, richard, +1.650.506.aagg, Peter_Korn, Mike?, Katie_Haritos-Shea, MoeKraft, Shadi, Tim_Boland
15:02:20 <Zakim> -Detlev
15:02:20 <Zakim> -MartijnHoutepen
15:02:20 <Zakim> -Sarah
15:02:21 <Zakim> -Mike?
15:02:23 <Zakim> -Shadi
15:02:24 <Zakim> -Peter_Korn
15:02:24 <trackbot> RRSAgent, please draft minutes
15:02:24 <RRSAgent> I have made the request to generate http://www.w3.org/2013/06/13-eval-minutes.html trackbot
15:02:25 <korn> korn has left #eval
15:02:25 <trackbot> RRSAgent, bye
15:02:25 <RRSAgent> I see no action items