14:52:39 <RRSAgent> RRSAgent has joined #eval
14:52:39 <RRSAgent> logging to http://www.w3.org/2014/01/16-eval-irc
14:52:41 <trackbot> RRSAgent, make logs world
14:52:41 <Zakim> Zakim has joined #eval
14:52:43 <trackbot> Zakim, this will be 3825
14:52:43 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 8 minutes
14:52:44 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
14:52:44 <trackbot> Date: 16 January 2014
14:52:55 <shadi> zakim, this is eval
14:52:55 <Zakim> shadi, I see WAI_ERTWG(Eval TF)10:00AM in the schedule but not yet started.  Perhaps you mean "this will be eval".
14:54:42 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
14:54:49 <Zakim> +[IPcaller]
14:54:52 <Vivienne> zakim, IPcaller is me
14:54:52 <Zakim> +Vivienne; got it
14:54:57 <Vivienne> zakim, mute me
14:54:57 <Zakim> sorry, Vivienne, muting is not permitted when only one person is present
14:59:02 <Zakim> +Shadi
15:00:00 <agarrison> agarrison has joined #eval
15:00:16 <ericvelleman> ericvelleman has joined #eval
15:00:16 <agarrison> Going to be 5 minutes late...
15:01:55 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
15:02:00 <Zakim> +EricVelleman
15:02:49 <Zakim> +Sarah_Swierenga
15:03:21 <Kathy> Kathy has joined #eval
15:04:46 <MoeKraft> MoeKraft has joined #eval
15:05:02 <Kathy> I can't get into the conference call... it says it is restricted
15:05:30 <shadi> not sure why :(
15:05:41 <shadi> zakim, code?
15:05:41 <Zakim> the conference code is 3825 (tel:+1.617.761.6200 sip:zakim@voip.w3.org), shadi
15:05:43 <Zakim> +[IPcaller]
15:05:46 <agarrison> zakim, ipcaller is agarrison
15:05:46 <Zakim> +agarrison; got it
15:06:08 <Zakim> +MoeKraft
15:06:29 <Zakim> +Kathy_Wahlbin
15:06:40 <Kathy> Shadi - I just got in
15:06:59 <shadi> can you scribe? :)
15:07:05 <Kathy> yes
15:07:09 <Kathy> scribe: Kathy
15:07:19 <shadi> how's the arm?
15:07:27 <Kathy> much better
15:07:54 <Kathy> Eric:  Did anyone look at the CSUN questionairre?
15:07:54 <shadi> Topic: Possible Face-to-Face Meeting
15:08:33 <shadi> https://www.w3.org/2002/09/wbs/48225/EvalTF_CSUN2013/results
15:09:08 <shadi> https://www.w3.org/2002/09/wbs/48225/EvalTF_CSUN2014/results
15:10:01 <Kathy> Shadi: 17 people responded
15:11:06 <Kathy> Shadi:  WCAG working group is looking at the 18th
15:12:38 <Kathy> Shadi: more turn out on Tuesday
15:14:07 <Kathy> Shadi: A few people may have a conflict with WCAG working group if we meet on Tuesday
15:14:50 <Kathy> Shadi: will confirm in the next week
15:16:37 <Kathy> Eric: We could consider later on Monday afternoon or evening
15:17:43 <Kathy> Shadi: We could try that
15:18:01 <Kathy> Vivienne: Could people dial in>
15:18:15 <Kathy> Shadi: we will need to look at timezones
15:20:18 <Kathy> Shadi: some overlap for parts
15:20:38 <Kathy> Eric: we could do a summary and discussion at particular times
15:21:07 <Kathy> Vivienne:  4pm summary would work for Australia
15:21:17 <Kathy> Shadi:  We should have a flexible agenda
15:21:30 <Kathy> Shadi:  We have people coming and going
15:21:42 <agarrison> q+
15:22:03 <Kathy> Allistair: What are the other opportunities
15:22:18 <Kathy> Shadi: Paris in July
15:23:44 <agarrison> q-
15:24:33 <Kathy> Alistair: When will the work wrap up?
15:25:19 <Tim> Tim has joined #eval
15:25:24 <Kathy> Shadi:  This could be the last draft.  Timeline to publish end of January, 4 weeks review period, 2 months to wrap and finalize the draft (April/May).  By July we should be done
15:25:50 <Kathy> TOPIC: WCAG working group meeting
15:26:01 <shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129
15:27:12 <shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment18
15:27:18 <Kathy> Shadi: WCAG working group has been busy with the publications.  We did get through some things. Raised some of the open comments (#18 - term change and 27)
15:27:34 <Kathy> Shadi:  No opposition, we can go ahead
15:28:02 <Kathy> Shadi:  Changed it from open to review
15:28:35 <Kathy> Shadi: couple of other comments.  Look at the minutes of the WCAG working group
15:29:00 <shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment43
15:29:45 <Kathy> Shadi: #43 Scoring, same concerns were raised - misleading, dangerous.  At the same time there is a need and has some use.  Maybe the turn scoring is misleading.  We are doing a tally or sum
15:29:51 <Vivienne> q+
15:29:55 <Kathy> Shadi:  Changing the name may be good
15:30:28 <Kathy> Shadi: One of the comments is that there is no data to show that this works.  If there is nothing for anyone to use then we will not have data
15:30:40 <Kathy> q+
15:30:59 <Kathy> Shadi: some people think it is good and others are opposed
15:31:35 <Kathy> Shadi:  look at the options; misuse vs. benefits
15:32:25 <Kathy> Vivienne: heard the term aggregation rather than score
15:33:06 <Kathy> Shadi: that is what I am thinking about, expectation would change with term change and would clear up the concerns
15:33:17 <shadi> q- viv
15:33:29 <Kathy> Eric: do we want to change it before we publish?
15:33:37 <Kathy> Shadi: Change it
15:33:54 <Kathy> q-
15:34:04 <Kathy> Kathy: Detlev may have data
15:34:47 <Kathy> Shadi: it is weighted scoring; tim is also presenting on scoring.  There will be studies that say things in both directions
15:35:00 <Kathy> Shadi: we are doing something similiar than scoring
15:35:25 <Kathy> Eric: Should we change to aggregation rather than score?
15:35:26 <Vivienne> +1
15:35:30 <Kathy> +1
15:35:32 <ericvelleman> +1
15:36:02 <Kathy> Alistair: It has come down to a simplistic calculation.  Do we need to tell people to do this?
15:36:03 <Sarah_Swierenga> +1
15:36:15 <Kathy> Shadi: We could drop the section
15:36:58 <Kathy> Shadi:  we could keep it and then drop it later
15:37:16 <Kathy> Shadi: it would help those who don't have a scoring mechanism
15:37:55 <Kathy> Alistair: Could be a separate document
15:38:57 <Kathy> Shadi: This could be applied to many of the sections.  Usability testing refers out to a document.  We don't have the resource yet
15:39:10 <Kathy> Eric: there are a lot of people who would like a score
15:39:53 <Kathy> Shadi: benefit to having it in the methodology, cannot compare scores if there are more ways of doing the score
15:40:13 <Vivienne> q+
15:40:36 <Kathy> Eric: we could change to aggregation for this version and it could take out in the last version if needed
15:40:45 <shadi> q- viv
15:41:39 <Kathy> Vivienne: would like to see it stay in.  It helps justifies the costs and how much they have improved
15:41:55 <Kathy> Eric: they don't really need the methodology to do a score
15:43:08 <shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment28
15:43:13 <Kathy> Shadi: comment 28 on providing sample size
15:43:34 <Kathy> Shadi: David suggested what the government of Canada recommends
15:44:14 <Zakim> +Tim_Boland
15:44:19 <Kathy> Shadi: we don't know how those numbers were calculated and it is hard to know how many pages are in the website.  Suggestion from David was to add in typical sizes
15:44:31 <Vivienne> But what is a typical website?
15:45:07 <Vivienne> We offer a variety of sample sizes depending upon budget -10,25,50,75,100
15:45:21 <Vivienne> depends largely upon the client's budget
15:45:29 <Kathy> Shadi: what does the group think?
15:46:32 <Kathy> Vivienne: Australia says 10% of the website; we recommend that people choose the largest size the budget will allow 10-100; most common is 25 pages
15:46:40 <Sarah_Swierenga> q+
15:46:51 <Kathy> Vivienne: most likely will not have to go over 30
15:47:03 <ericvelleman> Our research shows that for level A you do not need more than 23 pages to find 98 percent of the problems
15:47:09 <Kathy> Vivienne:  benchmark a site, then could review 10 pages
15:47:34 <ericvelleman> We could include this question in the test-run
15:48:52 <ericvelleman> The minimum of the current WCAG-EM is 49 pages + 5 random pages = 54 pages minimum sample
15:49:06 <Kathy> Shadi: We agreed that the methodology is independent of budget.  It may not be feasible but the sample size should be appropriate for the website.  Need a high degree of confidence
15:49:06 <ericvelleman> That is a lot
15:50:02 <Kathy> Shadi:  In order to achieve high level of confidence and analysis there may be typical sizes
15:50:03 <ericvelleman> Q?
15:50:05 <Kathy> q+
15:50:46 <shadi> q+
15:51:38 <Kathy> Sarah: agree with Vivienne, we don't want to base on budget it is reality and the typical client pages is 10 or 20.  We have not gone above 30.  Methodology should be able to be used regardless of number of pages.
15:52:02 <shadi> q-
15:52:11 <shadi> q- sarah
15:52:18 <Kathy> Shadi: we are not suggesting a percentage or a fix number of pages
15:52:26 <agarrison> q+
15:52:31 <Kathy> Shadi: is there a ball park number of pages
15:52:32 <ericvelleman> If we want WCAG-EM to be used on 10 pages, then we should provide some guidance. Maybe in an appendix? Now the number required is closer to 54 if you take a seperate page for each element
15:52:43 <Vivienne> what about a range e.g 10 - 50?
15:53:11 <ericvelleman> The methodology gives you the possibility to add more pages
15:53:11 <Kathy> Shadi: what do you do if the site does need more pages for proper analysis but there is no more budget
15:53:27 <Kathy> Sarah: cannot give the assurance
15:53:57 <Kathy> Vivienne: we give recommended sample size, if they don't do that then there is a disclaimer on the report
15:54:32 <ericvelleman> q+
15:55:09 <shadi> ack kath
15:55:59 <Kathy> Kathy:  Range would be ok, but I would not want hard numbers because there is alot of variables to consider
15:56:16 <Kathy> Alistair:  Eric, how did you come up with 54?
15:56:33 <Kathy> Eric:  Took the pages in the methodology it comes to 54
15:56:48 <Kathy> Alistair: it could be a problem since 54 is more than most reviews
15:57:11 <Kathy> Alistair: sample size should be about 25 pages or more
15:57:19 <shadi> q+
15:57:27 <Kathy> Eric:  it depends on how you read the sampling section
15:57:30 <shadi> ack ag
15:57:31 <agarrison> q-
15:57:46 <shadi> ack er
15:57:59 <Sarah_Swierenga> q+
15:58:07 <shadi> q- later
15:58:23 <Kathy> Eric:  can combine items in the page.  So then you could get 24 pages for the minimum including the random size.  Depends on interpretation of the sampling section
15:58:29 <shadi> ack sarah
15:58:37 <Kathy> Sarah: in the methodology we may want to set the minimim number
15:59:27 <Vivienne> q+
15:59:31 <shadi> ack me
15:59:33 <Kathy> Sarah: could be 10 pages; clients look at this like site optimization; the minimum number of pages could be useful in guiding clients
16:00:18 <Zakim> -MoeKraft
16:00:23 <Kathy> Shadi:  running out of time; it wasn't as clear that you can combine pages to reduce the number of pages required for the audit.
16:01:05 <Kathy> Shadi: maybe set a range for the number of pages and clarifying the sampling methology
16:01:16 <Sarah_Swierenga> 'combining pages' must be here. it would not have occurred to me to propose a new page for every item....
16:02:06 <Kathy> Shadi: Should we put an editor note or clarify before publication?  We will take this up in the next meeting
16:02:42 <Kathy> Shadi: the other piece is to review the comments and how we are addressing them.  If you have questions or comments, use the mailing list
16:03:31 <Kathy> Vivienne: sometimes the site is small and may not meet the minimum
16:04:12 <Zakim> -Kathy_Wahlbin
16:04:34 <Zakim> -Tim_Boland
16:04:35 <Vivienne> Goodnight
16:04:35 <Zakim> -agarrison
16:04:39 <Zakim> -Shadi
16:04:41 <Zakim> -EricVelleman
16:04:43 <Zakim> -Sarah_Swierenga
16:04:45 <Zakim> -Vivienne
16:04:46 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
16:04:46 <Zakim> Attendees were Vivienne, Shadi, EricVelleman, Sarah_Swierenga, agarrison, MoeKraft, Kathy_Wahlbin, Tim_Boland
16:05:17 <ericvelleman> ericvelleman has left #eval
16:08:58 <Kathy> Kathy has left #eval
16:16:08 <shadi> trackbot, end meeting
16:16:08 <trackbot> Zakim, list attendees
16:16:08 <Zakim> sorry, trackbot, I don't know what conference this is
16:16:16 <trackbot> RRSAgent, please draft minutes
16:16:16 <RRSAgent> I have made the request to generate http://www.w3.org/2014/01/16-eval-minutes.html trackbot
16:16:17 <trackbot> RRSAgent, bye
16:16:17 <RRSAgent> I see no action items