14:54:24 RRSAgent has joined #eval 14:54:24 logging to http://www.w3.org/2012/03/22-eval-irc 14:54:26 RRSAgent, make logs world 14:54:26 Zakim has joined #eval 14:54:28 Zakim, this will be 3825 14:54:28 ok, trackbot; I see WAI_ERTWG(Eval TF)11:00AM scheduled to start in 6 minutes 14:54:29 Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference 14:54:29 Date: 22 March 2012 14:54:41 ericvelleman has joined #eval 14:55:39 agenda: http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0049.html 14:55:45 chair: Eric 14:55:57 agenda+ Discussion of Changelog and new Methodology 14:56:11 agenda+ Next Steps 14:56:24 WAI_ERTWG(Eval TF)11:00AM has now started 14:56:31 + +1.520.271.aaaa 14:57:47 +Shadi 14:58:08 +??P42 14:58:13 regrets: Liz, Alistair 14:58:15 zakim, ??P42 is me 14:58:15 +vivienne; got it 14:58:36 Detlev has joined #eval 14:59:09 zakim, aaaa is Don 14:59:09 +Don; got it 14:59:24 + +94.04.318.aabb 14:59:31 MartijnHoutepen has joined #eval 14:59:51 +MartijnHoutepen 15:00:00 zakim, mute me 15:00:00 vivienne should now be muted 15:00:17 +MartijnHoutepen.a 15:00:31 Zakim aabb is Detlev 15:00:46 zakim, MartijnHoutepen.a is really Eric 15:00:46 +Eric; got it 15:00:48 Zakim, aabb is Detlev 15:00:56 +Detlev; got it 15:00:58 zakim, who is on the phone? 15:01:08 Zakim, mute me 15:01:18 On the phone I see Don, Shadi, vivienne (muted), Detlev, MartijnHoutepen, Eric 15:01:19 regrets: Liz, Alistair, Kathy 15:01:23 Zakim, mute me 15:01:39 Detlev should now be muted 15:02:03 MartijnHoutepen should now be muted 15:02:21 zakim, who is making noise? 15:02:56 shadi, listening for 10 seconds I heard sound from the following: Shadi (0%) 15:03:08 +Mike_Elledge 15:03:38 Mike_Elledge has joined #eval 15:04:13 scribe: Mike 15:04:25 scribenick: Mike_Elledge 15:04:26 shadi: let's get started. 15:04:35 zakim, take up agendum 1 15:04:54 Erik: One or two items, change log and new methodology. 15:05:06 http://www.w3.org/WAI/ER/conformance/comments 15:05:10 agendum 1. "Discussion of Changelog and new Methodology" taken up [from shadi] 15:05:12 http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff 15:05:17 http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320 15:05:23 Eric: Looked at list of comments we discussed in alst call, tried to make changes to each item. Change log and what we did is available. 15:05:51 Sh: Dispostion of comments and Dif mark version. Will also send separate version. 15:06:19 E: Most interesting to look up diff mark and comments to make sure okay for people who commented. 15:06:50 E: My mail has been out, dk if have seen most recent comments. Have tried to track however. Can look at dif version for changes. 15:07:22 E: Hope to have covered almost all comments seriously. In way that won't lead to add'l comments. Thanks Shadi. 15:07:35 +q 15:07:53 E: Is someone on call have comments that what was wrong and needs revising later? 15:07:55 ack me 15:07:57 +??P7 15:08:04 I was happy with the changes so far 15:08:17 Kerstin has joined #eval 15:08:28 zakim, ??p7 is Kerstin 15:08:28 +Kerstin; got it 15:08:40 zakim, mute me 15:08:40 Kerstin should now be muted 15:08:43 http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff#step1e 15:09:03 D: Still wonder what requirements step 1e, where try to specify sources for evaluation alog with eval commissioner. What does it mean in practice? 100s of techniques available. Many redundant. 15:09:20 D: Sure not what was intended. Elaborate? 15:10:00 E: Don't want techniaues to become new SC, but good to say what they are in different stages of eval process. 15:10:07 D: Why woudl that be helpful? 15:10:38 E: Sometimes want some techniques to be looked at, others not. Example: Just look at Flash. Need to workout more delicately in following steps. 15:10:41 D: Ok 15:11:32 Sh: U R not a nuisance. Do you have concersn about adding this. Maybe needs to be expalined more on list. Procurement settings where techniques are defined for developer, what should be eval'd against. 15:12:03 D: Concern is parsimony. May lead people to document all sorts of techniques wo real reason to do that. Would be nonsense. 15:12:23 richard has joined #eval 15:12:32 D: Example, Eric's comment. Techniques that may be used when others should. 15:12:43 mute me 15:12:48 q+ 15:12:56 Sh: Relates to reporting aspect. Which comes later. 15:13:10 +??P3 15:13:41 E: What should and should not be reported. This is stage when looking at site and deciding which techniquest to be used. Perhaps can limit Dev's concerns. Not list all the techniques that can be found on site. 15:13:44 zakim, ??p3 is Richard 15:13:44 +Richard; got it 15:14:00 q? 15:14:01 ack me 15:14:11 E: Answer to step 1e sufficient? 15:14:25 q? 15:14:48 zakim, unmute me 15:14:48 Kerstin should no longer be muted 15:14:58 D: Optional. Not too worried. Agree with Kathy that particular technique is not required. Not necessarily choose technique ahead of time. 15:15:04 mute me 15:15:12 K: Not happy w/ requrement, can claify later. 15:15:34 Kathy should be Kerstin in my comment above 15:15:38 q? 15:15:38 E: Have to provide more explanation in next stage. Briefly, of course. 15:15:39 zakim, mute me 15:15:40 Kerstin should now be muted 15:15:54 Sh: Eric keeping log so can go back to these? 15:16:09 fine 15:16:19 zakim, unmute me 15:16:19 Kerstin should no longer be muted 15:16:31 E: Yes, my view that wouldn't get in way of publishing draft, will take care of first thing in next draft. Kerstin Ok? 15:16:38 q? 15:16:41 K: Yes. Goodness criteria? 15:16:41 http://www.w3.org/WAI/ER/conformance/comments#c21 15:16:48 E: Where in document? 15:17:02 http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0055.html 15:17:07 K: Not in document. Very concerned bec of it. 15:17:24 +Don.a 15:17:28 -Don 15:17:32 zakim, unmute me 15:17:32 Kerstin was not muted, Kerstin 15:17:35 Sh: Comment #21. Comment to focus on. Also linked. Brief discussion on line this afternoon. 15:17:39 zakim, mute me 15:17:39 Kerstin should now be muted 15:17:41 zakim, Don.a is Don 15:17:41 +Don; got it 15:17:52 E: What was in discussion? 15:18:10 zakim, unmute me 15:18:10 Kerstin should no longer be muted 15:18:17 Sh: Between Kerestin and det lev. Maybe discuss? 15:18:39 http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0058.html 15:19:11 q+ 15:19:15 ack ker 15:19:44 K: I try. Use more words than last month? Quantitative evaluation calls for Validity, etc. No question about their relevance. Detlev posted article on qualitative methods. If goodness criteria is relevant for qualitative research. 15:19:49 that article referenced was not so important - the comments before it were! 15:19:57 K: But we're doing quantitative, so relevant. 15:20:15 Sh: What are you suggesting? 96% reliability? 15:20:28 q- 15:20:32 q+ 15:20:36 TimBoland has joined #eval 15:20:56 K: would be okay if worked it out later. How much reliable, etc. But for me it is important that have 3 goodness criteria: objective, reliable, . 15:21:08 E: Isn't already in document? Have to put in scope? 15:21:28 K: Yes, important for clients and users. 15:21:46 E: Needs more discussion. 15:22:05 K: Can put one sentence in abstract in one of first chapters. Work it out later. 15:22:28 E: In abstract...can put in that we strive for this. 15:22:32 q? 15:23:03 K: Maybe one add'l sentence. Wonder that this is a question. Bec all I knew from eval method in quantiative field, no question. 15:23:36 q+ 15:23:38 E: Agree, just where and how to put it. Put it in abstract. At least in document. Something discussing. Also, put in status of document? 15:23:50 Sh: I'd like more discussion before decide where to put it. 15:23:50 q+ 15:23:52 q? 15:23:53 zakim, mute me 15:23:53 Kerstin should now be muted 15:23:55 ack me 15:23:59 E: Discussion on goodness criteria. 15:24:02 +Tim_Boland 15:24:25 http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0055.html 15:24:53 D: Added 3 comments on obj, rel, and val, clear that looking for reilibe and valid results. Objectivitly something we can aim for. 15:25:28 q+ 15:25:28 D: Didn't find clear undisputed link to three criteria, always some subjectivity. It's in mailing list. 15:25:36 Sh: Let R go. 15:25:38 ack richard 15:26:03 zakim, unmute me 15:26:03 Kerstin should no longer be muted 15:26:04 R: Listening to K, goodness phrase confusing. Instead use level of confidence? Clearer? 15:26:09 ack ker 15:26:17 mute me 15:26:37 q+ 15:26:43 K: Goodness criteria. Think should use Goodness criteria. Since it's the term. Especially think about translations, should use specific terms. 15:26:48 ack viv 15:26:49 ack me 15:26:52 zakim, mute me 15:26:52 Kerstin should now be muted 15:27:17 V: Actually when reading comments. Have never run across Goodness term. 15:27:23 I haven't either. 15:27:37 V: Maybe it's in translation you're coming from. 15:27:37 zakim, mute me 15:27:37 vivienne should now be muted 15:27:41 ack me 15:27:41 E: New to me too. 15:27:57 there are 11 Million hits on google for "goodness criteria" 15:29:14 q+ 15:29:18 Sh: Part of my concern is that discussion of whether method is qualitative, quantitative and citing scientific defs, don't want to be orthogonal. Agree that overall want reliablitily between evaluators, but binding them to quant values have to think about nature of values. Ex: how good is alt atrtriburte. Is subjective. 15:29:19 agree with Shadi 15:29:55 Sh: A lengthy discussion. Let's not add it hastily. Could draw attention away from document. 15:29:57 zakim, unmute me 15:29:57 Kerstin should no longer be muted 15:30:01 q+ 15:30:07 ack ker 15:30:40 K: Sorry.Don't agree. Discussing sometime for months. Lots of emails on these issues. Also said reliability objectivity, reliablity never 100%, are ranges. 15:30:40 Level of confidence: Statistical measure of the number of times out of 100 that test results can be expected to be within a specified range. For example, a confidence level of 95% means that the result of an action will probably meet expectations 95% of the time. Most analyses of variance or correlation are described in terms of some level of confidence. 15:31:00 Sh: But specific number is concerning. Even though not feasible. 15:31:00 -Don 15:31:13 K: Intention not specific number. 15:31:14 ack me 15:31:26 zakim, mute me 15:31:26 Kerstin should now be muted 15:31:27 +Don 15:32:27 D: Think it's fair to say method should rely on reliability and objectivity, fair to try to define, But have to be more specific. Validity rely on use on web, reliability on evaluators replicating. Can be spelled out. But how much should be presciriptive? 15:32:44 q+ 15:32:52 q? 15:32:59 zakim, unmute me 15:32:59 Kerstin should no longer be muted 15:33:01 D: Wonder if that's possible. Validity, also outside methodology. Depends on timing and procedures used. Change over time. 15:33:37 K: Think that's a misunderstanding. Validity means testing WCAG 2, then nothing else. Not an interpretation. 15:33:47 mute me 15:34:01 zakim, mute me 15:34:03 Kerstin should now be muted 15:34:05 I agree with Detlev 15:35:34 Sh: Maybe in next week work out specific proposal for criteria, how prescirptive, what we understand about each. Validity...have in requirements already, dk how to get into specification. Shouldn't add in anything that we'er 15:35:43 still discussion. 15:36:07 fine, give it more time... 15:36:39 E: Selected sample part of report need to make decisions on what is compliant. We'll get into this discussion there. Like level of confidence. We'll get into it in section 5 and 4. Too early. 15:36:43 q+ 15:36:55 q+ to add to what Eric just said 15:36:56 E: Go into discussion on list. What we mean. Where to place it in document. 15:37:00 I agree. 15:37:12 q? 15:37:14 zakim, umute me 15:37:14 I don't understand 'umute me', Kerstin 15:37:20 Validity would have to go throughout document. Need more discussion before we put it in. 15:37:21 ack ker 15:38:11 ack me 15:38:11 shadi, you wanted to add to what Eric just said 15:38:21 K: Need more discussion. But don't understand why we can't put in sentence. Maybe we want to increase valididyt and objectively. Don't understand why we should leave these very improtant things out. 15:38:25 zakim, mute me 15:38:25 Kerstin should now be muted 15:38:57 someone's puffing quite loudly out there... 15:39:09 Sh: Want to echo E's comments. Plan is first version this December. Then put into practice. in second year revise according to feedback. 15:39:22 Someone is breathing heavily. ;^) 15:40:15 Sh: Have validity in document already. Adding as requirement into document would have implications to entire document. Would hold us back from publication. 15:41:11 E: To stay in middle. What we could do is in status of document ask public to give feedback on definition, scope and interpretation os WCAG 2, look at requirements to make sure document meets what we've set. 15:41:15 q+ 15:41:16 q? 15:41:21 zakim, unmute me 15:41:21 Kerstin should no longer be muted 15:41:26 ack ker 15:41:29 E: Your issue could be a good thing for comments. 15:42:03 K: Not sure. Also look for solution. Okay for me if we increase objectivity and reliability and then work it out. Criteria are important, however. 15:42:19 E: Don't deny they're important. Just how to put in document. 15:42:21 [[It addresses the need for a standardized approach ...]] 15:42:35 E: Too much to do before publishing draft. 15:42:47 -Don 15:42:51 K; But coud do it now (?). 15:43:23 E: How about point to it in document and ask for comment. Asking for public to comment on it. 15:44:05 Sh: Agree that point to requirements, but singling out specific requirements, not sure a good idea. Abstract does say need for std approach, intent is there. 15:44:26 Sh: Any suggestions? Do we need to vote on it? 15:45:03 E: Would propose that we add in what we want people to look at, ask them to look at requirements. Replicability, reliability are in there. Could generate valuable input. 15:45:21 +1 to Eric's suggestion 15:45:22 zakim, mute me 15:45:22 Kerstin should now be muted 15:45:33 E: Already discussing in mailing list. Is mailing list noted in document? Can we do that? 15:45:44 Sh: Yes. Discussion or what? 15:46:28 E: People can see how we got to this. Valuable point. Need to take a decision. Push document into draft or stop for more changes. Can we close this now? 15:46:32 +1 15:46:48 q+ 15:46:55 zakim, unmute me 15:46:55 Kerstin should no longer be muted 15:47:01 K; How about Eric's suggestion: pointing to document to get more comments. 15:47:10 K: Would depend on form of edit. 15:47:31 http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff#sotd 15:47:35 Sh: Three bullets asking for feedback--also looking for feedback looking for requirements. 15:47:44 E: Along with link to document. 15:48:09 K: But don't have criteria in document. Don't want to bother you, but very important to me. 15:48:17 E: Specific requriement 15:48:21 "x"? 15:48:35 zakim, mute me 15:48:35 Kerstin should now be muted 15:48:37 q+ 15:48:39 Sh: Again, what do others think. 15:48:49 zakim, ack me 15:48:49 unmuting vivienne 15:48:50 I see Kerstin on the speaker queue 15:48:54 Sh: Single out specific requirements. 15:48:56 zakim, mute me 15:48:56 Shadi should now be muted 15:49:01 q+ 15:49:19 q- ker 15:49:51 V: Think we're being...agree they're critical, but don' understand importance putting into document at this point. Don't understand requirement to spell it out now. It's a work in progress. 15:50:07 ack me 15:50:11 V: To me getting comments would be sufficient. 15:50:12 zakim, mute me 15:50:12 vivienne should now be muted 15:51:08 M: Don't see explicit need to explain goals. 15:51:18 zakim, mute me 15:51:18 MartijnHoutepen should now be muted 15:51:20 q+ 15:51:22 q+ 15:51:26 zakim, unmute me 15:51:26 Kerstin should no longer be muted 15:51:27 ack ker 15:52:14 K: As M said, clear that we are looking for methodology that is reliable, but when remember discussions on mailing list it's a critical point. If we had all agreed there wouldn't be discussion. 15:52:38 K: But discussed from the beginning also had ? in the draft, don't find it now. 15:53:54 E: Is it numbered in list? Conforms to guidelines, reliability, requirement 3 would cover in part. What we could do in status of document ask people to compare with requirements. Yielding of equivialent results. Sections 4 and 5 will be interesting. 15:54:00 http://www.w3.org/WAI/ER/conformance/ED-methodology-reqs-20111012#rq03 15:54:17 K; Remeber discussion of equivalent results. Reliability and equivalent results. 15:54:53 zakim, mute me 15:54:53 Kerstin should now be muted 15:54:54 E:High degree of correlation, with different web pages and tools. Maybe have to go back. But propose not to do it now. 15:55:03 ack me 15:55:22 E: Just small part of task force. How to proceed? Vote for draft? 15:55:52 Sh: Do we need to resolve issue now or after release as draft. Hold up or resolve. 15:55:58 Release the doc. 15:56:05 Release the doc 15:56:10 release the doc 15:56:11 release the doc 15:56:19 Is full consensus of the group a prerequisite for publication? 15:56:20 release the doc 15:56:24 Sh: Feel free to type what you want to see. 15:56:37 E: Doesn't mean we won't discuss in future. 15:57:25 release the doc 15:57:33 release the doc with mentioning the criteria in one of the first chapter and asking the public on these issue 15:57:46 Sh: Consensus is not that everyone is in agreement. Document can go out with filed objection. Ultimately up to directors. Happened in WCAG 2. 15:58:13 K: Add request for public to review. 15:58:19 There seems to be a strong majority in favour of publishing (Richard and I had reservations for other reasons but have given in) 15:58:29 E: Add specifically pointer to requirements. 15:58:46 Thanks Detlev ! 15:58:54 E: Point to mailing list. 15:59:12 Sh: URL will change. 15:59:22 http://www.w3.org/WAI/ER/conformance/comments#c74 15:59:49 after pub 15:59:51 after publication 16:00:01 can wait till after publication I think 16:00:01 Sh: From martin 5b to 5f unclear what benefit would be...type before or after pub 16:00:03 after pub 16:00:05 After pub 16:00:05 after pub comment 74 16:00:14 after pub 16:00:18 http://www.w3.org/WAI/ER/conformance/comments#c89 16:00:48 Sh: Comment #89 trickier: Amy chen, remove essence 1D minimum set of browsers, usual practice of evaluation... 16:00:51 After pub 16:00:54 comment 89 16:00:54 after pub 16:00:57 after publication 16:00:59 after pub 16:01:02 after pub 16:01:08 after pub for the last two issues 16:01:14 after pub 16:01:24 zakim, take up next 16:01:25 I see a speaker queue remaining and respectfully decline to close this agendum, shadi 16:01:31 q? 16:01:32 q? 16:01:36 q- m 16:01:40 zakim, take up next 16:01:41 Sh: in absence of mailing list will proceed to publication. Again K, E and I will address specific concerns. 16:01:43 agendum 2. "Next Steps" taken up [from shadi] 16:01:55 No problem :^) 16:02:49 Sh: This doc goig as is with minor reference to reliability. Congratualtions. Already have several issues to come back to and discuss. will give us valuable input. 16:03:00 E: Give to working groups for feedback. 16:03:24 E: Meet next week for follow-up discussion on next draft. 16:03:25 zakim, unmute me 16:03:25 MartijnHoutepen should no longer be muted 16:03:27 okay, bye now 16:03:28 bye bye 16:03:28 bye 16:03:29 -Tim_Boland 16:03:29 I will also be out next week. 16:03:30 -Shadi 16:03:30 -vivienne 16:03:31 Bye. 16:03:31 -MartijnHoutepen 16:03:35 -Mike_Elledge 16:03:36 -vivienne 16:03:38 -Detlev 16:03:39 -Richard 16:03:41 -Kerstin 16:03:46 -Eric 16:03:48 WAI_ERTWG(Eval TF)11:00AM has ended 16:03:48 Attendees were +1.520.271.aaaa, Shadi, vivienne, Don, +94.04.318.aabb, MartijnHoutepen, Eric, Detlev, Mike_Elledge, Kerstin, Richard, Tim_Boland 16:03:50 quit 16:09:59 ericvelleman has left #eval 16:10:24 trackbot, end meeting 16:10:24 Zakim, list attendees 16:10:24 sorry, trackbot, I don't know what conference this is 16:10:32 RRSAgent, please draft minutes 16:10:32 I have made the request to generate http://www.w3.org/2012/03/22-eval-minutes.html trackbot 16:10:33 RRSAgent, bye 16:10:33 I see no action items