13:53:35 <RRSAgent> RRSAgent has joined #eval
13:53:35 <RRSAgent> logging to http://www.w3.org/2013/05/30-eval-irc
13:53:37 <trackbot> RRSAgent, make logs world
13:53:39 <trackbot> Zakim, this will be 3825
13:53:39 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 7 minutes
13:53:40 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:53:40 <trackbot> Date: 30 May 2013
13:54:18 <shadi> chair: Eric
13:54:47 <shadi> regrets: Kathy, Tim, Kostas, Sarah
13:58:28 <ericvelleman> ericvelleman has joined #eval
13:58:46 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
13:58:51 <Vivienne> Vivienne has joined #eval
13:58:53 <Zakim> +Shadi
13:59:21 <Zakim> +??P4
13:59:36 <MartijnHoutepen> Zakim, ??P4 is me
13:59:36 <Zakim> +MartijnHoutepen; got it
13:59:41 <Zakim> +Chaaaaaaals
14:00:34 <Vivienne> zakim, +Chaaaaaaals is me
14:00:34 <Zakim> sorry, Vivienne, I do not recognize a party named '+Chaaaaaaals'
14:00:48 <Vivienne> zakim, Chaaaaaaals is me
14:00:48 <Zakim> +Vivienne; got it
14:00:50 <Zakim> +Eric_Velleman
14:01:36 <Liz> Liz has joined #eval
14:01:43 <Detlev> Detlev has joined #eval
14:02:19 <Vivienne> scribe: vivienne
14:02:41 <MoeKraft> MoeKraft has joined #eval
14:02:44 <Zakim> +Liz
14:02:48 <MartijnHoutepen> Zakim, mute me
14:02:48 <Zakim> MartijnHoutepen should now be muted
14:02:51 <korn2> korn2 has joined #eval
14:02:52 <Zakim> +Peter_Korn
14:03:15 <Vivienne> zakim, take up agendum 2
14:03:15 <Zakim> agendum 2. "Current state of comments" taken up [from MartijnHoutepen]
14:03:17 <Zakim> +MoeKraft
14:03:28 <ericvelleman> <http://www.w3.org/WAI/ER/conformance/comments-20130226>
14:04:27 <Vivienne> Eric: disposition of comments - looking at the way it is formatted - same format as last time and has tried to group items
14:05:03 <Zakim> +Detlev
14:05:21 <Zakim> +Katie_Haritos-Shea
14:05:34 <Vivienne> Eric: received many comments now - 96 in total, and others that are editorial that aren't included in this list
14:06:03 <ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226
14:06:27 <Mike_Elledge> Mike_Elledge has joined #eval
14:07:51 <Mike_Elledge> Mike_Elledge has joined #eval
14:08:00 <Vivienne> Shadi: EOG wikki had background discussion along with the comments
14:09:07 <Detlev> Zakim, mute me
14:09:07 <Zakim> Detlev should now be muted
14:09:09 <Vivienne> Eric: explained the structure of the Disposition of Comments document
14:09:42 <Zakim> +Mike
14:10:03 <Vivienne> Eric: approach last time was to split them into larger groups and address with a proposal in an edited draft with a request for review for the TF - suggest same approach
14:10:40 <ericvelleman> q?
14:10:51 <Vivienne> q+
14:11:12 <Detlev> Vivienne: happy the way its set up
14:11:18 <Mike_Elledge> +1
14:11:30 <MartijnHoutepen> q- viv
14:11:59 <Vivienne> Eric: will make an edited draft based on the public working draft and ask for TF to review
14:12:08 <Vivienne> anyone against this?
14:12:16 <MoeKraft> +1
14:12:20 <MartijnHoutepen> +1
14:12:27 <Vivienne> Eric: close this point
14:12:27 <Liz> +1
14:12:33 <Vivienne> Zakim, take up agendum 3
14:12:34 <Zakim> agendum 3. "Test run" taken up [from MartijnHoutepen]
14:14:01 <Vivienne> Eric: changed the survey questions for 2&3 to make it more clear what was required.  Survey 2 had been answered by a number of people, so I left the questions there at question 15 or so.  You can complete questions after that if you want to.  Both surveys are now open.
14:14:27 <Vivienne> Eric: you have the opportunity to fill in the information until next week when we'll have a summary of website 2 and hopefully website 3
14:14:40 <Detlev> q+
14:14:49 <Vivienne> Eric: website 1: easy - gaming, 2: more complex, 3: library
14:15:00 <ericvelleman> q?
14:15:02 <Detlev> ack me
14:15:03 <Vivienne> Eric: has anyone looked at the changed surveys
14:15:13 <ericvelleman> <https://www.w3.org/2002/09/wbs/48225/testrun2/results>
14:15:54 <Detlev> Zakim, mute me
14:15:54 <Zakim> Detlev should now be muted
14:15:59 <Vivienne> Eric: url for the website 3 now has a direct link to the url
14:16:32 <Vivienne> Eric: link to survey 2 in the answer page after the introduction you can go down to question 11, optional questions below
14:17:15 <ericvelleman> https://www.w3.org/2002/09/wbs/48225/testrun3/
14:17:55 <Vivienne> Eric: testrun 3 page in the introduction there is a direct link to the library part of the website
14:18:07 <Vivienne> Eric: survey 1 is closed now
14:18:27 <Vivienne> Eric: need the conclusions to add to the disposition of comments about the outcomes of the surveys
14:18:55 <Vivienne> Eric: adding it to the disposition of comments makes it easy to see if we've fixed them
14:19:00 <Vivienne> zakim, mute me
14:19:00 <Zakim> Vivienne should now be muted
14:19:33 <Vivienne> Eric: if you want to address survey 2&3, please do it as quickly as possible
14:19:41 <MartijnHoutepen> q+
14:19:46 <MartijnHoutepen> ack me
14:20:07 <MartijnHoutepen> zakim, mute me
14:20:07 <Zakim> MartijnHoutepen should now be muted
14:20:10 <Vivienne> Eric: Martijn will do the comment list for survey 2
14:20:26 <Vivienne> Eric: who can summarize the outcome of survey 3?
14:20:31 <Detlev> q+
14:20:34 <Detlev> ack me
14:21:01 <Vivienne> Detlev: should we wait till we've got some replies first?  There is sufficient to talk about from survey 2.
14:21:26 <Vivienne> Eric: we'll be working on the disposition of comments, so we have enough work and can give this more time
14:21:28 <Detlev> Zakim, mute me
14:21:29 <Zakim> Detlev should now be muted
14:21:33 <Detlev> yes
14:21:34 <Mike_Elledge> +1
14:21:35 <MartijnHoutepen> +1
14:21:38 <Vivienne> Eric: we can do website 3 in a couple of weeks
14:21:39 <Liz> yes
14:21:40 <Vivienne> +1
14:22:04 <Vivienne> Eric: next week we'll discuss the outcome of website 2 and will send a reminder to people to complete the survey
14:22:17 <ericvelleman> q?
14:22:22 <Vivienne> zakim, take up agendum 4
14:22:22 <Zakim> agendum 4. "Welcome" taken up [from MoeKraft]
14:22:40 <Vivienne> zakim, close agendum 4
14:22:40 <Zakim> agendum 4, Welcome, closed
14:22:41 <Zakim> I see 1 item remaining on the agenda:
14:22:41 <Zakim> 3. Test run [from MartijnHoutepen]
14:23:10 <Vivienne> eric: Discussion: when to decide not to sample, when to sample
14:23:33 <Vivienne> Eric: Discussion: hererogeneity and homogeneity
14:24:15 <MartijnHoutepen> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0042.html
14:24:59 <korn2> q+
14:25:04 <Vivienne> Eric: discussion around developers using different coding on different pages of the website and so taking perhaps 1 or 2 tables would not be enough - homogeneity of the website
14:25:19 <Vivienne> Eric: We need a good definition
14:25:34 <Vivienne> Eric: what causes are there for this situation
14:25:41 <Detlev> q+
14:25:54 <Vivienne> Eric: how can an evaluator determine if there is homogeneity/hetergeneity on the website
14:26:42 <Vivienne> Peter: what has tables got to do with it - also in coding it can be just the html
14:27:03 <Vivienne> Peter: this isn't a hard & fast or precise thing, but more of an indicator as to the confidence level of the sampling
14:27:40 <Mike_Elledge> q+
14:28:09 <korn2> q-
14:28:11 <Vivienne> Peter: you could have 100 pages all coded in the same style, which will indicate that probably all 10,000 are similar.  However we can have a host of coding styles within the sample, and this would decrease the confident of our sample.
14:28:33 <shadi> q+
14:28:34 <Vivienne> Eric: tables is just an example of something you look for in an evaluation
14:28:42 <Detlev> ack me
14:29:26 <Vivienne> Detlev: not sure what makes that difference in the methodology, how much would it help developers.  You can have many differences such as script base, it's down to the evaluator's sampling
14:30:05 <MartijnHoutepen> q+
14:30:19 <korn2> q_
14:30:20 <korn2> q+
14:30:24 <Detlev> Zakim, mute me
14:30:24 <Zakim> Detlev should now be muted
14:31:51 <Vivienne> Mike: one of the criteria for a website being acceptable is that there is consistency throughout the site.  Whether 1 developer has used a different format for say a widget or there are several developers, this is not the key thing.  The key thing is the consistency.  There is value to pointing it out to evaluators to watch for it.  Maybe change 'style' to 'design patterns' or similar.  'Style'
14:31:51 <Vivienne> may refer more to coding.
14:32:11 <shadi> ack me
14:32:17 <shadi> ack mike
14:33:31 <Vivienne> Shadi: regarding Detlev's comments - don't make it too complicated.  Looking at Step 3 Sampling - intro paragraph talks about authoring mechanisms rather than styles - maybe we can be more specific with our wording.  Not only the types of pages dictate the sampling size, but perhaps also the way the page is coded.
14:33:35 <shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130128#step3b
14:34:28 <Vivienne> Shadi: 3(b) (link above) change from selected 2 distinct pages to 1 distinct page.  Maybe we can reflect from the test runs to see how large a sample is needed.
14:35:52 <Vivienne> Shadi: maybe in that same step we should enumerate the different types of pages that an evaluator should select as in step 2(c) and talk about the coding style as one of the parameters which would grow or shrink the sample size.  It is difficult to put objective criteria to decided what a homogeneous or heterogeneous website.
14:36:05 <korn2> q?
14:36:11 <shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130128#step2c
14:36:27 <MartijnHoutepen> ack me
14:36:51 <shadi> [[Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people.]]
14:37:15 <Vivienne> MH: agree with Mike & Shadi.  Can add something more in the general procedure - also see email.
14:37:41 <Vivienne> Eric: is it so important that we have to add more text about it.
14:37:51 <MartijnHoutepen> zakim, mute me
14:37:51 <Zakim> MartijnHoutepen should now be muted
14:38:27 <Detlev> q+
14:38:40 <shadi> ack ko
14:38:58 <Vivienne> Peter: the confidence you have in your sampling size needs to be adequate and may be influenced by the homogeneity of the coding used.  This may influence the likliehoood that your sample is capturing everythng.
14:39:24 <Detlev> ack me
14:39:24 <korn2> q-
14:41:02 <shadi> +1 to Detlev
14:41:03 <Vivienne> Detlev: need to consider the need for sampling.  Need to think about the purpose of the evaluation. YOu may gain little in the overall result if you add more different bad pages, but it's good for the designer.
14:41:04 <shadi> q+
14:41:22 <Detlev> Zakim, mute me
14:41:22 <Zakim> Detlev should now be muted
14:41:29 <shadi> ack me
14:41:30 <Vivienne> Eric: let's keep discussing this on the list
14:41:34 <shadi> http://www.w3.org/TR/WCAG-EM/#step2c
14:41:59 <Vivienne> Shadi: we can sharpen the text here - also 2(c) where we talk about what constitutes a different type of page.
14:42:45 <korn2> q+
14:43:22 <Vivienne> Shadi: it's not about building larger and larger samples.  In some cases you come across something that is fairly accessible and you want to make sure that this is representative.  We need to sharpen the terminology and reiterate that the coverage needs to get bigger depending upon the purpose of the evaluation and the type of the website
14:44:00 <shadi> q+
14:44:27 <Vivienne> Peter: if the website is so awful that you've seen enough, it turns the focus of our work on its head.  We're trying to help someonemake a solid compliance claim - aiming for perfection.  "most websites are so bad you've seen enough", then we need to focus a lot more work on how you report the lack of perfection.
14:44:29 <korn2> q-
14:44:40 <shadi> ack me
14:45:51 <Vivienne> Shadi: in the document at the beginning we talk about the use of the document.  In cases where you're looking for perfection (accessible website) and you want to verify that or issue a conformance claim is oneof the use cases that this methodology is targeted for.  Other use case - how good or how bad - what do I need to fix in order to conform
14:47:05 <Vivienne> Shadi:  We need to think about this in regard to reporting.  Depends upon purposes - conformance - you can stop early if conformance is the goal once there are a certain number of errors.  Even though you know a table is inaccessible and you've already realized they fail conformance you may continue to check to show the different types
14:48:05 <shadi> +1 agree that we need to look more deeply on the reporting aspect
14:49:08 <Vivienne> Eric: words 'reasonable confidence' step 3 and in introduction.  The concept of using this as a way to conclude what the confidence is for the results that you've gathered.
14:50:11 <Vivienne> Eric: we should discuss the question of 'to sample or not to sample' on the list
14:50:17 <Vivienne> Eric: any other issues?
14:50:18 <Detlev> q+
14:50:22 <Detlev> ack me
14:52:01 <Vivienne> Detlev: are we open to changing the question of evaluation purposes?  developer/designer for new websites so they ask for testing and another situation - 2 main scenarious for testing and they don't seem to be so well reflected in the 3 different reporting types - detailed info to the designers or informing/challenging a conformance claim
14:53:12 <Detlev> Can do
14:53:40 <Vivienne> Eric: will put it on the agenda for the next telco
14:53:41 <Detlev> Zakim, mute me
14:53:41 <Zakim> Detlev should now be muted
14:54:32 <Vivienne> Shadi: are there only the 3 different use cases - for future discussion.  We may want to think about other use cases.
14:54:40 <Vivienne> Eric: we could put it on the list
14:54:56 <Detlev> I will trigger that yes
14:55:21 <Detlev> conscientious
14:55:45 <MartijnHoutepen> ack me
14:55:46 <Liz> bye
14:55:47 <Detlev> bye
14:55:48 <Mike_Elledge> bye!
14:55:48 <Zakim> -Peter_Korn
14:55:50 <Zakim> -MoeKraft
14:55:50 <Vivienne> zakim, ack me
14:55:51 <Zakim> unmuting Vivienne
14:55:51 <Zakim> I see no one on the speaker queue
14:55:53 <Zakim> -Detlev
14:55:54 <Zakim> -Mike
14:55:55 <Zakim> -MartijnHoutepen
14:56:00 <Zakim> -Liz
14:56:04 <korn2> korn2 has left #eval
14:56:06 <shadi> trackbot, end meeting
14:56:06 <trackbot> Zakim, list attendees
14:56:06 <Zakim> As of this point the attendees have been Shadi, MartijnHoutepen, Vivienne, Eric_Velleman, Liz, Peter_Korn, MoeKraft, Detlev, Katie_Haritos-Shea, Mike
14:56:07 <ericvelleman> ericvelleman has left #eval
14:56:11 <Zakim> -Eric_Velleman
14:56:12 <Zakim> -Shadi
14:56:14 <trackbot> RRSAgent, please draft minutes
14:56:14 <RRSAgent> I have made the request to generate http://www.w3.org/2013/05/30-eval-minutes.html trackbot
14:56:14 <Zakim> -Vivienne
14:56:15 <trackbot> RRSAgent, bye
14:56:15 <RRSAgent> I see no action items
14:56:16 <Zakim> -Katie_Haritos-Shea
14:56:16 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
14:56:16 <Zakim> Attendees were Shadi, MartijnHoutepen, Vivienne, Eric_Velleman, Liz, Peter_Korn, MoeKraft, Detlev, Katie_Haritos-Shea, Mike