IRC log of eval on 2013-05-30

Timestamps are in UTC.

13:53:35 [RRSAgent]

RRSAgent has joined #eval

13:53:35 [RRSAgent]

logging to http://www.w3.org/2013/05/30-eval-irc

13:53:37 [trackbot]

RRSAgent, make logs world

13:53:39 [trackbot]

Zakim, this will be 3825

13:53:39 [Zakim]

ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 7 minutes

13:53:40 [trackbot]

Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference

13:53:40 [trackbot]

Date: 30 May 2013

13:54:18 [shadi]

chair: Eric

13:54:47 [shadi]

regrets: Kathy, Tim, Kostas, Sarah

13:58:28 [ericvelleman]

ericvelleman has joined #eval

13:58:46 [Zakim]

WAI_ERTWG(Eval TF)10:00AM has now started

13:58:51 [Vivienne]

Vivienne has joined #eval

13:58:53 [Zakim]

+Shadi

13:59:21 [Zakim]

+??P4

13:59:36 [MartijnHoutepen]

Zakim, ??P4 is me

13:59:36 [Zakim]

+MartijnHoutepen; got it

13:59:41 [Zakim]

+Chaaaaaaals

14:00:34 [Vivienne]

zakim, +Chaaaaaaals is me

14:00:34 [Zakim]

sorry, Vivienne, I do not recognize a party named '+Chaaaaaaals'

14:00:48 [Vivienne]

zakim, Chaaaaaaals is me

14:00:48 [Zakim]

+Vivienne; got it

14:00:50 [Zakim]

+Eric_Velleman

14:01:36 [Liz]

Liz has joined #eval

14:01:43 [Detlev]

Detlev has joined #eval

14:02:19 [Vivienne]

scribe: vivienne

14:02:41 [MoeKraft]

MoeKraft has joined #eval

14:02:44 [Zakim]

+Liz

14:02:48 [MartijnHoutepen]

Zakim, mute me

14:02:48 [Zakim]

MartijnHoutepen should now be muted

14:02:51 [korn2]

korn2 has joined #eval

14:02:52 [Zakim]

+Peter_Korn

14:03:15 [Vivienne]

zakim, take up agendum 2

14:03:15 [Zakim]

agendum 2. "Current state of comments" taken up [from MartijnHoutepen]

14:03:17 [Zakim]

+MoeKraft

14:03:28 [ericvelleman]

<http://www.w3.org/WAI/ER/conformance/comments-20130226>

14:04:27 [Vivienne]

Eric: disposition of comments - looking at the way it is formatted - same format as last time and has tried to group items

14:05:03 [Zakim]

+Detlev

14:05:21 [Zakim]

+Katie_Haritos-Shea

14:05:34 [Vivienne]

Eric: received many comments now - 96 in total, and others that are editorial that aren't included in this list

14:06:03 [ericvelleman]

http://www.w3.org/WAI/ER/conformance/comments-20130226

14:06:27 [Mike_Elledge]

Mike_Elledge has joined #eval

14:07:51 [Mike_Elledge]

Mike_Elledge has joined #eval

14:08:00 [Vivienne]

Shadi: EOG wikki had background discussion along with the comments

14:09:07 [Detlev]

Zakim, mute me

14:09:07 [Zakim]

Detlev should now be muted

14:09:09 [Vivienne]

Eric: explained the structure of the Disposition of Comments document

14:09:42 [Zakim]

+Mike

14:10:03 [Vivienne]

Eric: approach last time was to split them into larger groups and address with a proposal in an edited draft with a request for review for the TF - suggest same approach

14:10:40 [ericvelleman]

q?

14:10:51 [Vivienne]

q+

14:11:12 [Detlev]

Vivienne: happy the way its set up

14:11:18 [Mike_Elledge]

+1

14:11:30 [MartijnHoutepen]

q- viv

14:11:59 [Vivienne]

Eric: will make an edited draft based on the public working draft and ask for TF to review

14:12:08 [Vivienne]

anyone against this?

14:12:16 [MoeKraft]

+1

14:12:20 [MartijnHoutepen]

+1

14:12:27 [Vivienne]

Eric: close this point

14:12:27 [Liz]

+1

14:12:33 [Vivienne]

Zakim, take up agendum 3

14:12:34 [Zakim]

agendum 3. "Test run" taken up [from MartijnHoutepen]

14:14:01 [Vivienne]

Eric: changed the survey questions for 2&3 to make it more clear what was required. Survey 2 had been answered by a number of people, so I left the questions there at question 15 or so. You can complete questions after that if you want to. Both surveys are now open.

14:14:27 [Vivienne]

Eric: you have the opportunity to fill in the information until next week when we'll have a summary of website 2 and hopefully website 3

14:14:40 [Detlev]

q+

14:14:49 [Vivienne]

Eric: website 1: easy - gaming, 2: more complex, 3: library

14:15:00 [ericvelleman]

q?

14:15:02 [Detlev]

ack me

14:15:03 [Vivienne]

Eric: has anyone looked at the changed surveys

14:15:13 [ericvelleman]

<https://www.w3.org/2002/09/wbs/48225/testrun2/results>

14:15:54 [Detlev]

Zakim, mute me

14:15:54 [Zakim]

Detlev should now be muted

14:15:59 [Vivienne]

Eric: url for the website 3 now has a direct link to the url

14:16:32 [Vivienne]

Eric: link to survey 2 in the answer page after the introduction you can go down to question 11, optional questions below

14:17:15 [ericvelleman]

https://www.w3.org/2002/09/wbs/48225/testrun3/

14:17:55 [Vivienne]

Eric: testrun 3 page in the introduction there is a direct link to the library part of the website

14:18:07 [Vivienne]

Eric: survey 1 is closed now

14:18:27 [Vivienne]

Eric: need the conclusions to add to the disposition of comments about the outcomes of the surveys

14:18:55 [Vivienne]

Eric: adding it to the disposition of comments makes it easy to see if we've fixed them

14:19:00 [Vivienne]

zakim, mute me

14:19:00 [Zakim]

Vivienne should now be muted

14:19:33 [Vivienne]

Eric: if you want to address survey 2&3, please do it as quickly as possible

14:19:41 [MartijnHoutepen]

q+

14:19:46 [MartijnHoutepen]

ack me

14:20:07 [MartijnHoutepen]

zakim, mute me

14:20:07 [Zakim]

MartijnHoutepen should now be muted

14:20:10 [Vivienne]

Eric: Martijn will do the comment list for survey 2

14:20:26 [Vivienne]

Eric: who can summarize the outcome of survey 3?

14:20:31 [Detlev]

q+

14:20:34 [Detlev]

ack me

14:21:01 [Vivienne]

Detlev: should we wait till we've got some replies first? There is sufficient to talk about from survey 2.

14:21:26 [Vivienne]

Eric: we'll be working on the disposition of comments, so we have enough work and can give this more time

14:21:28 [Detlev]

Zakim, mute me

14:21:29 [Zakim]

Detlev should now be muted

14:21:33 [Detlev]

yes

14:21:34 [Mike_Elledge]

+1

14:21:35 [MartijnHoutepen]

+1

14:21:38 [Vivienne]

Eric: we can do website 3 in a couple of weeks

14:21:39 [Liz]

yes

14:21:40 [Vivienne]

+1

14:22:04 [Vivienne]

Eric: next week we'll discuss the outcome of website 2 and will send a reminder to people to complete the survey

14:22:17 [ericvelleman]

q?

14:22:22 [Vivienne]

zakim, take up agendum 4

14:22:22 [Zakim]

agendum 4. "Welcome" taken up [from MoeKraft]

14:22:40 [Vivienne]

zakim, close agendum 4

14:22:40 [Zakim]

agendum 4, Welcome, closed

14:22:41 [Zakim]

I see 1 item remaining on the agenda:

14:22:41 [Zakim]

3. Test run [from MartijnHoutepen]

14:23:10 [Vivienne]

eric: Discussion: when to decide not to sample, when to sample

14:23:33 [Vivienne]

Eric: Discussion: hererogeneity and homogeneity

14:24:15 [MartijnHoutepen]

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0042.html

14:24:59 [korn2]

q+

14:25:04 [Vivienne]

Eric: discussion around developers using different coding on different pages of the website and so taking perhaps 1 or 2 tables would not be enough - homogeneity of the website

14:25:19 [Vivienne]

Eric: We need a good definition

14:25:34 [Vivienne]

Eric: what causes are there for this situation

14:25:41 [Detlev]

q+

14:25:54 [Vivienne]

Eric: how can an evaluator determine if there is homogeneity/hetergeneity on the website

14:26:42 [Vivienne]

Peter: what has tables got to do with it - also in coding it can be just the html

14:27:03 [Vivienne]

Peter: this isn't a hard & fast or precise thing, but more of an indicator as to the confidence level of the sampling

14:27:40 [Mike_Elledge]

q+

14:28:09 [korn2]

q-

14:28:11 [Vivienne]

Peter: you could have 100 pages all coded in the same style, which will indicate that probably all 10,000 are similar. However we can have a host of coding styles within the sample, and this would decrease the confident of our sample.

14:28:33 [shadi]

q+

14:28:34 [Vivienne]

Eric: tables is just an example of something you look for in an evaluation

14:28:42 [Detlev]

ack me

14:29:26 [Vivienne]

Detlev: not sure what makes that difference in the methodology, how much would it help developers. You can have many differences such as script base, it's down to the evaluator's sampling

14:30:05 [MartijnHoutepen]

q+

14:30:19 [korn2]

q_

14:30:20 [korn2]

q+

14:30:24 [Detlev]

Zakim, mute me

14:30:24 [Zakim]

Detlev should now be muted

14:31:51 [Vivienne]

Mike: one of the criteria for a website being acceptable is that there is consistency throughout the site. Whether 1 developer has used a different format for say a widget or there are several developers, this is not the key thing. The key thing is the consistency. There is value to pointing it out to evaluators to watch for it. Maybe change 'style' to 'design patterns' or similar. 'Style'

14:31:51 [Vivienne]

may refer more to coding.

14:32:11 [shadi]

ack me

14:32:17 [shadi]

ack mike

14:33:31 [Vivienne]

Shadi: regarding Detlev's comments - don't make it too complicated. Looking at Step 3 Sampling - intro paragraph talks about authoring mechanisms rather than styles - maybe we can be more specific with our wording. Not only the types of pages dictate the sampling size, but perhaps also the way the page is coded.

14:33:35 [shadi]

http://www.w3.org/WAI/ER/conformance/ED-methodology-20130128#step3b

14:34:28 [Vivienne]

Shadi: 3(b) (link above) change from selected 2 distinct pages to 1 distinct page. Maybe we can reflect from the test runs to see how large a sample is needed.

14:35:52 [Vivienne]

Shadi: maybe in that same step we should enumerate the different types of pages that an evaluator should select as in step 2(c) and talk about the coding style as one of the parameters which would grow or shrink the sample size. It is difficult to put objective criteria to decided what a homogeneous or heterogeneous website.

14:36:05 [korn2]

q?

14:36:11 [shadi]

http://www.w3.org/WAI/ER/conformance/ED-methodology-20130128#step2c

14:36:27 [MartijnHoutepen]

ack me

14:36:51 [shadi]

[[Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people.]]

14:37:15 [Vivienne]

MH: agree with Mike & Shadi. Can add something more in the general procedure - also see email.

14:37:41 [Vivienne]

Eric: is it so important that we have to add more text about it.

14:37:51 [MartijnHoutepen]

zakim, mute me

14:37:51 [Zakim]

MartijnHoutepen should now be muted

14:38:27 [Detlev]

q+

14:38:40 [shadi]

ack ko

14:38:58 [Vivienne]

Peter: the confidence you have in your sampling size needs to be adequate and may be influenced by the homogeneity of the coding used. This may influence the likliehoood that your sample is capturing everythng.

14:39:24 [Detlev]

ack me

14:39:24 [korn2]

q-

14:41:02 [shadi]

+1 to Detlev

14:41:03 [Vivienne]

Detlev: need to consider the need for sampling. Need to think about the purpose of the evaluation. YOu may gain little in the overall result if you add more different bad pages, but it's good for the designer.

14:41:04 [shadi]

q+

14:41:22 [Detlev]

Zakim, mute me

14:41:22 [Zakim]

Detlev should now be muted

14:41:29 [shadi]

ack me

14:41:30 [Vivienne]

Eric: let's keep discussing this on the list

14:41:34 [shadi]

http://www.w3.org/TR/WCAG-EM/#step2c

14:41:59 [Vivienne]

Shadi: we can sharpen the text here - also 2(c) where we talk about what constitutes a different type of page.

14:42:45 [korn2]

q+

14:43:22 [Vivienne]

Shadi: it's not about building larger and larger samples. In some cases you come across something that is fairly accessible and you want to make sure that this is representative. We need to sharpen the terminology and reiterate that the coverage needs to get bigger depending upon the purpose of the evaluation and the type of the website

14:44:00 [shadi]

q+

14:44:27 [Vivienne]

Peter: if the website is so awful that you've seen enough, it turns the focus of our work on its head. We're trying to help someonemake a solid compliance claim - aiming for perfection. "most websites are so bad you've seen enough", then we need to focus a lot more work on how you report the lack of perfection.

14:44:29 [korn2]

q-

14:44:40 [shadi]

ack me

14:45:51 [Vivienne]

Shadi: in the document at the beginning we talk about the use of the document. In cases where you're looking for perfection (accessible website) and you want to verify that or issue a conformance claim is oneof the use cases that this methodology is targeted for. Other use case - how good or how bad - what do I need to fix in order to conform

14:47:05 [Vivienne]

Shadi: We need to think about this in regard to reporting. Depends upon purposes - conformance - you can stop early if conformance is the goal once there are a certain number of errors. Even though you know a table is inaccessible and you've already realized they fail conformance you may continue to check to show the different types

14:48:05 [shadi]

+1 agree that we need to look more deeply on the reporting aspect

14:49:08 [Vivienne]

Eric: words 'reasonable confidence' step 3 and in introduction. The concept of using this as a way to conclude what the confidence is for the results that you've gathered.

14:50:11 [Vivienne]

Eric: we should discuss the question of 'to sample or not to sample' on the list

14:50:17 [Vivienne]

Eric: any other issues?

14:50:18 [Detlev]

q+

14:50:22 [Detlev]

ack me

14:52:01 [Vivienne]

Detlev: are we open to changing the question of evaluation purposes? developer/designer for new websites so they ask for testing and another situation - 2 main scenarious for testing and they don't seem to be so well reflected in the 3 different reporting types - detailed info to the designers or informing/challenging a conformance claim

14:53:12 [Detlev]

Can do

14:53:40 [Vivienne]

Eric: will put it on the agenda for the next telco

14:53:41 [Detlev]

Zakim, mute me

14:53:41 [Zakim]

Detlev should now be muted

14:54:32 [Vivienne]

Shadi: are there only the 3 different use cases - for future discussion. We may want to think about other use cases.

14:54:40 [Vivienne]

Eric: we could put it on the list

14:54:56 [Detlev]

I will trigger that yes

14:55:21 [Detlev]

conscientious

14:55:45 [MartijnHoutepen]

ack me

14:55:46 [Liz]

bye

14:55:47 [Detlev]

bye

14:55:48 [Mike_Elledge]

bye!

14:55:48 [Zakim]

-Peter_Korn

14:55:50 [Zakim]

-MoeKraft

14:55:50 [Vivienne]

zakim, ack me

14:55:51 [Zakim]

unmuting Vivienne

14:55:51 [Zakim]

I see no one on the speaker queue

14:55:53 [Zakim]

-Detlev

14:55:54 [Zakim]

-Mike

14:55:55 [Zakim]

-MartijnHoutepen

14:56:00 [Zakim]

-Liz

14:56:04 [korn2]

korn2 has left #eval

14:56:06 [shadi]

trackbot, end meeting

14:56:06 [trackbot]

Zakim, list attendees

14:56:06 [Zakim]

As of this point the attendees have been Shadi, MartijnHoutepen, Vivienne, Eric_Velleman, Liz, Peter_Korn, MoeKraft, Detlev, Katie_Haritos-Shea, Mike

14:56:07 [ericvelleman]

ericvelleman has left #eval

14:56:11 [Zakim]

-Eric_Velleman

14:56:12 [Zakim]

-Shadi

14:56:14 [trackbot]

RRSAgent, please draft minutes

14:56:14 [RRSAgent]

I have made the request to generate http://www.w3.org/2013/05/30-eval-minutes.html trackbot

14:56:14 [Zakim]

-Vivienne

14:56:15 [trackbot]

RRSAgent, bye

14:56:15 [RRSAgent]

I see no action items

14:56:16 [Zakim]

-Katie_Haritos-Shea

14:56:16 [Zakim]

WAI_ERTWG(Eval TF)10:00AM has ended

14:56:16 [Zakim]

Attendees were Shadi, MartijnHoutepen, Vivienne, Eric_Velleman, Liz, Peter_Korn, MoeKraft, Detlev, Katie_Haritos-Shea, Mike