13:59:24 <RRSAgent> RRSAgent has joined #eval
13:59:24 <RRSAgent> logging to http://www.w3.org/2012/08/09-eval-irc
13:59:26 <trackbot> RRSAgent, make logs world
13:59:26 <Zakim> Zakim has joined #eval
13:59:28 <trackbot> Zakim, this will be 3825
13:59:28 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 1 minute
13:59:29 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:59:29 <trackbot> Date: 09 August 2012
13:59:36 <Detlev> Zakim, mute me
13:59:36 <Zakim> sorry, Detlev, I don't know what conference this is
13:59:46 <shadi> zakim, this is eval
13:59:46 <Zakim> ok, shadi; that matches WAI_ERTWG(Eval TF)10:00AM
13:59:48 <MoeKraft> MoeKraft has joined #eval
13:59:51 <Zakim> +??P4
13:59:56 <vivienne> zakim, ??P4 is me
13:59:56 <Zakim> +vivienne; got it
14:00:00 <shadi> zakim, who is on the phone?
14:00:00 <Zakim> On the phone I see +1.301.975.aaaa, +49.404.318.aabb, +31.30.239.aacc, vivienne
14:00:12 <Detlev> Zakim, mute me
14:00:12 <Zakim> sorry, Detlev, I do not know which phone connection belongs to you
14:00:16 <MartijnHoutepen> zakim, aacc is me
14:00:16 <Zakim> +MartijnHoutepen; got it
14:00:23 <Zakim> - +49.404.318.aabb
14:00:25 <shadi> zakim, aabb is Detlev
14:00:25 <Zakim> +Moe_Kraft
14:00:29 <MartijnHoutepen> zakim, mute me
14:00:29 <Zakim> sorry, shadi, I do not recognize a party named 'aabb'
14:00:31 <Zakim> MartijnHoutepen should now be muted
14:00:45 <Zakim> +Shadi
14:00:48 <shadi> zakim, who is on the phone?
14:00:48 <Zakim> On the phone I see +1.301.975.aaaa, MartijnHoutepen (muted), vivienne, Moe_Kraft, Shadi
14:00:50 <efong> aaaa is Liz Fong
14:01:21 <Zakim> + +49.404.318.aadd
14:01:23 <efong> zakim, aaaa is Liz Fong
14:01:23 <Zakim> I don't understand 'aaaa is Liz Fong', efong
14:01:37 <Detlev> Zakim, aadd is Detlev
14:01:37 <Zakim> +Detlev; got it
14:01:48 <Detlev> Zakim, mute me
14:01:48 <Zakim> Detlev should now be muted
14:02:31 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
14:02:37 <ericvelleman> ericvelleman has joined #eval
14:03:00 <shadi> chair: Eric
14:03:07 <Zakim> + +1.517.432.aaee
14:03:43 <shadi> agenda: http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0004.html
14:03:58 <Zakim> +Eric_Velleman
14:04:58 <Sarah_Swierenga> zakin, aaee is me
14:05:18 <Sarah_Swierenga> zakim, aaee is me
14:05:18 <Zakim> +Sarah_Swierenga; got it
14:06:07 <MoeKraft> scribe: Moe
14:06:13 <MoeKraft> scribe: MoeKraft
14:06:23 <MoeKraft> agenda+ Welcome
14:06:43 <MoeKraft> agenda+ Review of New documents
14:07:14 <MoeKraft> agenda+ Discussion of Open Comments
14:07:52 <MoeKraft> agenda+ TPAC in Lyon
14:08:12 <MoeKraft> Zakim, take up next
14:08:12 <Zakim> agendum 1. "Welcome" taken up [from MoeKraft]
14:08:37 <shadi> zakim, mute me
14:08:37 <Zakim> Shadi should now be muted
14:08:58 <MoeKraft> Eric: Welcome Everybody, then review of new documents
14:09:11 <MoeKraft> Zakim, take up next
14:09:11 <Zakim> agendum 1 was just opened, MoeKraft
14:09:16 <shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0002.html
14:09:29 <shadi> zakim, close agendum 1
14:09:29 <Zakim> agendum 1, Welcome, closed
14:09:31 <Zakim> I see 3 items remaining on the agenda; the next one is
14:09:31 <Zakim> 2. Review of New documents [from MoeKraft]
14:09:38 <shadi> zakim, take up next
14:09:38 <Zakim> agendum 2. "Review of New documents" taken up [from MoeKraft]
14:09:41 <MoeKraft> Eric: Survey open until next Wednesday, 8/15
14:09:42 <shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0002.html
14:10:36 <MoeKraft> Eric: Please if there are any further comments before publication please put these into the survey
14:11:01 <MoeKraft> Eric: Can also put discuss into mailing list. 
14:11:05 <shadi> q+
14:11:09 <shadi> ack me
14:11:10 <MoeKraft> s/discuss/discussion
14:11:43 <MoeKraft> Shadi: Keep in mind which comments need to be addressed now and which to be addressed later on. 
14:12:02 <shadi> zakim, mute me
14:12:02 <Zakim> Shadi should now be muted
14:12:11 <MoeKraft> Zakim , close agendum 2
14:12:19 <MoeKraft> Zakim, take up next
14:12:19 <Zakim> agendum 3. "Discussion of Open Comments" taken up [from MoeKraft]
14:12:31 <shadi> q+
14:12:32 <MoeKraft> Eric: 8 comments remain open.
14:12:34 <shadi> ack me
14:13:34 <MoeKraft> Shadi: We reworked entire disposition of comments. Many issues are closed as addressed by group. Some have been addressed somewhat differently but kept in spirit of group. Marked review needed
14:13:49 <MoeKraft> Shadi: Then there are some open ones that need to be addressed more clearly. 
14:14:13 <MoeKraft> Shadi: Only 8 open comments that still need to be resolved. Hope to resolve on today's call. 
14:14:52 <MoeKraft> Shadi: Review document first and then review disposition of comments to see if you agree or have feedback. There is room for comments in the survey. 
14:15:03 <shadi> zakim, mute me
14:15:03 <Zakim> Shadi should now be muted
14:15:15 <vivienne> zakim, mute me
14:15:15 <Zakim> vivienne should now be muted
14:15:16 <MoeKraft> Topic: Comment #19: Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>
14:16:35 <Detlev> q+
14:16:41 <Detlev> ack me
14:16:55 <MoeKraft> Eric: Proposed resolution to open an action item to further discuss. 
14:17:04 <shadi> q+
14:17:11 <shadi> ack me
14:17:13 <MoeKraft> Detlev: Would be good to discuss a proposal to help us further agree. 
14:17:42 <MoeKraft> Eric: Idea is that if we put in tracker we start discussing in the tracker including preparing text. 
14:18:00 <MoeKraft> Detlev: Issue will be discussing whehter we should address at all?
14:18:18 <shadi> q+
14:18:26 <MoeKraft> Eric: Everything is a fail if there is an issue. Do we want this to be like this or do we want tolerance metrics?
14:18:48 <shadi> ack me
14:18:51 <MoeKraft> Eric: Tracker item to discuss if needed and how. 
14:18:59 <Detlev> Zakim, mute me
14:18:59 <Zakim> Detlev should now be muted
14:19:01 <shadi> [[3rd bullet objective from http://www.w3.org/WAI/ER/2011/eval/eval-ws - that of "Aggregating individual results into an overall conformance statement; this includes defining approaches for assessing the relative impact on failures, potentially through incorporating tolerance metrics."]]
14:19:12 <MoeKraft> Shadi: Agree with Eric. Let's open a tracker issue. 
14:20:01 <MoeKraft> Shadi: Issue is How do we want to address tolerance metrics?
14:20:31 <MoeKraft> +1
14:20:34 <Sarah_Swierenga> +1
14:20:37 <shadi> zakim, mute me
14:20:37 <Zakim> Shadi should now be muted
14:20:37 <vivienne> I'm okay with oening it in the tracker
14:20:40 <Detlev> fine, as I said
14:20:40 <MartijnHoutepen> +1
14:20:42 <MoeKraft> Eric: Is everyone okay with opening an issue?
14:21:05 <ericvelleman> Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)
14:21:10 <shadi> ack me
14:21:24 <MoeKraft> Resolution: Everyone is okay with opening an issue in tracker to discuss tolerance metrics. 
14:21:31 <shadi> zakim, mute me
14:21:31 <Zakim> Shadi should now be muted
14:21:36 <MoeKraft> Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)
14:21:51 <MoeKraft> Topic: Comment #12: Need to agree on update in editor draft. There is a clarification of the concept of use-case in the updated section 3.4.1 Step 4.a (Check for the Broadest Variety of Use Cases). Does this clarify use case sufficiently for the moment or should we open an issue on use cases and come back tot that in a later draft. <http://www.w3.org/WAI/ER/conformance/comments#c12>
14:22:23 <shadi> ack me
14:23:32 <MoeKraft> Shadi: Not sure if 12 needs that much discussion. 15 really needs to be discussed. 
14:23:48 <MoeKraft> Shadi: 15 is linked from comment 19
14:24:51 <MoeKraft> Shadi: If you agree with edit on Comment #12 then this does not need further discussion. 
14:24:58 <vivienne> 15 isn't in the agenda though
14:24:59 <Detlev> q+
14:25:04 <Detlev> ack me
14:25:41 <MoeKraft> Shadi: Review edits on comment #12 in survey. This is homework. No further discussion. 
14:26:17 <MoeKraft> Detlev: What is the order in which we are processing the comments? 
14:26:25 <Tim> Tim has joined #eval
14:26:31 <richard> richard has joined #eval
14:26:33 <MoeKraft> Shadi: The numbers of the comments were reorder based on where they fit in the document. 
14:27:02 <MoeKraft> Detlev: Having trouble finding the right issue in the disposition of comments. 
14:27:27 <MoeKraft> Shadi: We are not going through the entire list of comments. We are only discussing the 8 listed in Agenda as open comments. 
14:30:21 <Zakim> +Tim_Boland
14:30:22 <vivienne> q+
14:30:30 <vivienne> ack me
14:30:32 <MoeKraft> Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>
14:30:52 <MoeKraft> Topic: Comment#15 Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>
14:31:07 <MoeKraft> Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>
14:31:23 <MartijnHoutepen> +1
14:31:24 <vivienne> zakim, mute me
14:31:24 <Zakim> vivienne should now be muted
14:31:27 <Sarah_Swierenga> +1
14:31:32 <vivienne> +1
14:31:38 <ericvelleman> +1
14:31:46 <efong> +1
14:31:59 <shadi> zakim, mute me
14:31:59 <Zakim> Shadi should now be muted
14:32:13 <MoeKraft> Eric: Does everyone agree for comment #15 that we open an issue in Tracker to address Tolerance metrics?
14:32:15 <MoeKraft> +1
14:32:34 <Detlev> fine
14:33:22 <Tim> +1
14:33:29 <MoeKraft> Resolution: Agree that Comment #15 should be addressed by an issue in Tracker. Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #19)
14:33:50 <ericvelleman> http://www.w3.org/WAI/ER/conformance/comments#c16 
14:34:13 <MoeKraft> Topic: Comment #16: Need to resolve an objection. Is this objection solved by the updated section 3.5.3 Step 5.c (Provide a Performance Score (optional)? The section adds different scores. This methodology is not limited to automated evaluation or related sub-scores. <http://www.w3.org/WAI/ER/conformance/comments#c16>
14:34:39 <Detlev> q+
14:34:46 <MoeKraft> Eric: We currently have 3 different types of scores
14:34:48 <ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120730.html#step5c
14:34:57 <Detlev> ack me
14:35:31 <vivienne> q+
14:35:38 <MoeKraft> Detlev: Would be useful to have the score for items that can be evaluated automatically. This could be quite valuable. Could cause some confusion. Not sure how to rectify the text. 
14:36:15 <MoeKraft> Eric: There is a section on machine evaluation. The text is limited. Maybe it is not enough. 
14:36:18 <vivienne> ack me
14:36:21 <Detlev> Zakim, mute me
14:36:21 <Zakim> Detlev should now be muted
14:36:45 <Detlev> ack me
14:36:57 <MoeKraft> Vivienne: Are we saying that the only time we should use a performance score for automated testing?
14:37:09 <shadi> https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2647
14:37:26 <MoeKraft> Detlev: Comment #16 talks about only limiting to automated evaluation. 
14:37:35 <shadi> q+
14:37:49 <MoeKraft> Detlev: Could be quite valuable for benchmarking
14:38:12 <MoeKraft> Detlev: Could have  layer on top of conformance evaluation
14:38:56 <MoeKraft> Detlev: Maybe need to have another score on top of conformance, maybe performance metrics. May be need two score.
14:39:30 <Detlev> q+
14:39:41 <shadi> q- last
14:39:47 <shadi> q+ last
14:39:52 <shadi> q- last
14:39:59 <shadi> q-
14:40:01 <shadi> q+
14:40:37 <vivienne> zakim, mute me
14:40:37 <Zakim> vivienne should now be muted
14:40:43 <MoeKraft> Vivienne: Need to have some method for comparison. Need to have a metric developed to compare on an equal plane. Step 3.5.3 5.c is an optional thing but would be good to be part of methodology. Incredible amount of value in that.
14:41:21 <shadi> s/method for comparison/method for comparison in large-scale evaluation
14:41:34 <MoeKraft> Detlev: Agree with Vivienne that a score would be useful. Would be good to know how many pass/fail in evaluation. Automatic evaluation meet all or some criteria gives us an automatic score. 
14:41:37 <shadi> ack me
14:42:00 <Detlev> zakim, mute me
14:42:00 <Zakim> Detlev should now be muted
14:42:22 <vivienne> q+
14:42:30 <MoeKraft> Shadi: Agrees with all of the above. Definitely a need for metrics and scores and for automated testing. Research and Development working group are looking at what is out there. In Eval TF we are looking at performance criteria. 
14:42:36 <Detlev> q-
14:42:36 <MoeKraft> Shadi: What has been met and not met. 
14:43:25 <MoeKraft> Shadi: We would have to analyze which criteria are fully automatable. And then the score would be confusing. The score would be simply for the automatable test. Maybe we should look at this in the future. 
14:43:31 <ericvelleman> q?
14:43:34 <shadi> zakim, mute me
14:43:34 <Zakim> Shadi should now be muted
14:43:34 <vivienne> ack me
14:44:19 <shadi> q+ to suggest opening an issue to further discuss "automated testing"
14:44:21 <MoeKraft> Vivienne: Jury still out. Vivienne uses metrics because she needs something to make evaluation quantifiable. Working on a conformance score to compare. 
14:45:00 <Detlev> q+
14:45:05 <MoeKraft> Vivienne: Using methodolgy as we do now but add in a score to give it some metrics for comparison. Can keep this optional but would like to keep this part of the methodology .
14:45:33 <Detlev> ack me
14:45:38 <shadi> ack me
14:45:39 <Zakim> shadi, you wanted to suggest opening an issue to further discuss "automated testing"
14:46:43 <vivienne> q+
14:46:53 <MoeKraft> Shadi: Believes there is a misunderstanding. The idea is not to remove what we have. The request is to add another score for automated testing only in addition to the full conformance score. Looking to add a separate score when you only run an automated tool on a website. 
14:47:39 <MoeKraft> Shadi: Do we want to open an issue in Tracker and come back to this comment for a future draft?
14:47:41 <vivienne> ack me
14:48:16 <vivienne> zakim, mute me
14:48:16 <Zakim> vivienne should now be muted
14:48:17 <MoeKraft> Vivienne: Yes. I misunderstood. Quite happy with the resolution. 
14:49:48 <MoeKraft> Resolution: Tentative. Peter Korn is not on the call. Resolution: No change  Rationale: WCAG-EM is only for full evaluation, not for tool measurement only. This is not covered in the current requirements and therefore not in the document.  Related discussion: Eval TF survey #4 (ID 16)
14:50:14 <MoeKraft> Shadi: Need to inform Peter while he was the only one who objected from the survey. 
14:50:48 <Sarah_Swierenga> +1 to the approach
14:50:48 <ericvelleman> q?
14:50:52 <vivienne> sounds good to me Shadi
14:50:59 <MoeKraft> Shadi: If Peter objects, we will open an issue in Tracker to further discuss after draft is out. 
14:51:02 <MartijnHoutepen> both options are fine to me
14:51:12 <efong>  +1 for the approach
14:51:13 <MoeKraft> +1
14:51:30 <Sarah_Swierenga> +1
14:51:31 <Tim> +1
14:51:34 <shadi> zakim, mute me
14:51:34 <Zakim> Shadi should now be muted
14:51:50 <MoeKraft> Topic: Comment #48: Need to resolve an objection. The rationale is as indicated in the objection very much stressing the need to make it non-optional. We can decide to make no change and keep it optional (1) because we want to stay on the SC level as in WCAG2.0 and/or (2) open an issue to discuss making Step 1.e non-optional. Please note the open ended nature of the techniques. <http://www.w3.org/WAI/ER/conformance/comments#c48>
14:52:55 <MartijnHoutepen> It was alistair
14:52:55 <shadi> https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2630
14:53:49 <shadi> ack me
14:54:51 <Detlev> the rationale (imported from Alistair)  will have caused the confusion...
14:56:25 <shadi> "techniques are not required for conformance"
14:56:42 <Detlev> q+
14:56:43 <vivienne> +1
14:56:45 <MartijnHoutepen> +1 
14:56:45 <Detlev> ack me
14:56:51 <ericvelleman> +1
14:56:54 <shadi> zakim, mute me
14:56:54 <Zakim> Shadi should now be muted
14:57:35 <MoeKraft> Detlev: Commissioner would be interested in seeing the technique is addressed by methodology. 
14:57:44 <shadi> ack me
14:57:45 <MoeKraft> Eric: Yes. It is useful but it is optional. 
14:58:14 <MoeKraft> Detlev: Right rationale in many circumstances techniques may not be able to be provided and evaluation would have to proceed with out them. 
14:58:43 <MoeKraft> Detlev: Techniques may not be available to the evaluator
14:58:58 <MoeKraft> Detlev: So we cannot make this required. 
14:59:29 <MoeKraft> Shadi: I can agree to Detlev's rationale. It addresses Allistair's concern more directly. 
14:59:29 <Detlev> Zakim, mute me
14:59:29 <Zakim> Detlev should now be muted
14:59:38 <shadi> zakim, mute me
14:59:38 <Zakim> Shadi should now be muted
14:59:39 <shadi> ack me
14:59:47 <MartijnHoutepen> +1
14:59:53 <Sarah_Swierenga> +1
14:59:53 <efong> +1
14:59:53 <vivienne> +1
14:59:54 <ericvelleman> +1
14:59:57 <MoeKraft> +1
15:00:08 <shadi> zakim, mute me
15:00:08 <Zakim> Shadi should now be muted
15:00:28 <MoeKraft> Resolution: Update the rationale to reflect Detlev's argument. 
15:00:48 <MoeKraft> Eric: We still have 4 open comments. 
15:01:08 <MoeKraft> Eric: No use starting discussions now. Will send out to mailing list. 
15:01:10 <shadi> ack me
15:01:47 <MoeKraft> Shadi: There is one from Kirsten which she put in the survey as well. Others please review these 4 remaining comments. Send by email or put in survey. 
15:02:06 <shadi> zakim, take up next
15:02:06 <Zakim> agendum 4. "TPAC in Lyon" taken up [from MoeKraft]
15:02:07 <Detlev> q+
15:02:15 <Detlev> ack me
15:02:26 <MoeKraft> Comment #A1: Need to agree on the proposed resolution. The tekst says “..minimum set of web browsers and assistive technology to evaluate for shall be defined”. This is important because it is an important part of defining "accessibility support" Proposed resolution is therefore to make no change. <http://www.w3.org/WAI/ER/conformance/comments#c001>  Comment #A4: Need to agree on the proposed resolution. See: <http://www.w3.org/WAI/ER/conformance/comments#
15:02:33 <MoeKraft> Zakim, take up next
15:02:33 <Zakim> agendum 4 was just opened, MoeKraft
15:03:02 <MoeKraft> Detlev: Will there be enough members attending?
15:03:21 <MoeKraft> Shadi: If you are planning on attending, please register and make travel arrangements. 
15:04:00 <MoeKraft> Shadi: Certain folks did indicate that they are able to come. 
15:04:25 <shadi> https://www.w3.org/2002/09/wbs/myQuestionnaires
15:04:48 <Sarah_Swierenga> I'm sorry I won't be able to meet all of you in person in Lyon. I have a research participant coming in now, so I need to go. Take care, Sarah
15:04:55 <Zakim> -Sarah_Swierenga
15:04:57 <MoeKraft> Eric: Are there any other issue?
15:04:59 <efong> bye
15:05:12 <richard> bye
15:05:16 <vivienne> good night all
15:05:16 <MoeKraft> Eric: Will put the 4 remaining comments in the discussion list. 
15:05:17 <MartijnHoutepen> zakim, unmute me
15:05:17 <Zakim> MartijnHoutepen should no longer be muted
15:05:18 <Zakim> -Tim_Boland
15:05:26 <Zakim> -Detlev
15:05:26 <MoeKraft> bye
15:05:28 <Zakim> -Eric_Velleman
15:05:29 <Zakim> -Shadi
15:05:30 <Zakim> -Moe_Kraft
15:05:31 <Zakim> -vivienne
15:05:34 <Zakim> -MartijnHoutepen
15:05:34 <Zakim> - +1.301.975.aaaa
15:05:35 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
15:05:37 <Zakim> Attendees were +1.301.975.aaaa, +49.404.318.aabb, +31.30.239.aacc, vivienne, MartijnHoutepen, Moe_Kraft, Shadi, +49.404.318.aadd, Detlev, +1.517.432.aaee, Eric_Velleman,
15:05:39 <Zakim> ... Sarah_Swierenga, Tim_Boland
15:05:45 <ericvelleman> ericvelleman has left #eval
15:39:01 <shadi> trackbot, end meeting
15:39:01 <trackbot> Zakim, list attendees
15:39:01 <Zakim> sorry, trackbot, I don't know what conference this is
15:39:09 <trackbot> RRSAgent, please draft minutes
15:39:09 <RRSAgent> I have made the request to generate http://www.w3.org/2012/08/09-eval-minutes.html trackbot