IRC log of eval on 2012-08-09

Timestamps are in UTC.

13:59:24 [RRSAgent]

RRSAgent has joined #eval

13:59:24 [RRSAgent]

logging to http://www.w3.org/2012/08/09-eval-irc

13:59:26 [trackbot]

RRSAgent, make logs world

13:59:26 [Zakim]

Zakim has joined #eval

13:59:28 [trackbot]

Zakim, this will be 3825

13:59:28 [Zakim]

ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 1 minute

13:59:29 [trackbot]

Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference

13:59:29 [trackbot]

Date: 09 August 2012

13:59:36 [Detlev]

Zakim, mute me

13:59:36 [Zakim]

sorry, Detlev, I don't know what conference this is

13:59:46 [shadi]

zakim, this is eval

13:59:46 [Zakim]

ok, shadi; that matches WAI_ERTWG(Eval TF)10:00AM

13:59:48 [MoeKraft]

MoeKraft has joined #eval

13:59:51 [Zakim]

+??P4

13:59:56 [vivienne]

zakim, ??P4 is me

13:59:56 [Zakim]

+vivienne; got it

14:00:00 [shadi]

zakim, who is on the phone?

14:00:00 [Zakim]

On the phone I see +1.301.975.aaaa, +49.404.318.aabb, +31.30.239.aacc, vivienne

14:00:12 [Detlev]

Zakim, mute me

14:00:12 [Zakim]

sorry, Detlev, I do not know which phone connection belongs to you

14:00:16 [MartijnHoutepen]

zakim, aacc is me

14:00:16 [Zakim]

+MartijnHoutepen; got it

14:00:23 [Zakim]

- +49.404.318.aabb

14:00:25 [shadi]

zakim, aabb is Detlev

14:00:25 [Zakim]

+Moe_Kraft

14:00:29 [MartijnHoutepen]

zakim, mute me

14:00:29 [Zakim]

sorry, shadi, I do not recognize a party named 'aabb'

14:00:31 [Zakim]

MartijnHoutepen should now be muted

14:00:45 [Zakim]

+Shadi

14:00:48 [shadi]

zakim, who is on the phone?

14:00:48 [Zakim]

On the phone I see +1.301.975.aaaa, MartijnHoutepen (muted), vivienne, Moe_Kraft, Shadi

14:00:50 [efong]

aaaa is Liz Fong

14:01:21 [Zakim]

+ +49.404.318.aadd

14:01:23 [efong]

zakim, aaaa is Liz Fong

14:01:23 [Zakim]

I don't understand 'aaaa is Liz Fong', efong

14:01:37 [Detlev]

Zakim, aadd is Detlev

14:01:37 [Zakim]

+Detlev; got it

14:01:48 [Detlev]

Zakim, mute me

14:01:48 [Zakim]

Detlev should now be muted

14:02:31 [Sarah_Swierenga]

Sarah_Swierenga has joined #eval

14:02:37 [ericvelleman]

ericvelleman has joined #eval

14:03:00 [shadi]

chair: Eric

14:03:07 [Zakim]

+ +1.517.432.aaee

14:03:43 [shadi]

agenda: http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0004.html

14:03:58 [Zakim]

+Eric_Velleman

14:04:58 [Sarah_Swierenga]

zakin, aaee is me

14:05:18 [Sarah_Swierenga]

zakim, aaee is me

14:05:18 [Zakim]

+Sarah_Swierenga; got it

14:06:07 [MoeKraft]

scribe: Moe

14:06:13 [MoeKraft]

scribe: MoeKraft

14:06:23 [MoeKraft]

agenda+ Welcome

14:06:43 [MoeKraft]

agenda+ Review of New documents

14:07:14 [MoeKraft]

agenda+ Discussion of Open Comments

14:07:52 [MoeKraft]

agenda+ TPAC in Lyon

14:08:12 [MoeKraft]

Zakim, take up next

14:08:12 [Zakim]

agendum 1. "Welcome" taken up [from MoeKraft]

14:08:37 [shadi]

zakim, mute me

14:08:37 [Zakim]

Shadi should now be muted

14:08:58 [MoeKraft]

Eric: Welcome Everybody, then review of new documents

14:09:11 [MoeKraft]

Zakim, take up next

14:09:11 [Zakim]

agendum 1 was just opened, MoeKraft

14:09:16 [shadi]

http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0002.html

14:09:29 [shadi]

zakim, close agendum 1

14:09:29 [Zakim]

agendum 1, Welcome, closed

14:09:31 [Zakim]

I see 3 items remaining on the agenda; the next one is

14:09:31 [Zakim]

2. Review of New documents [from MoeKraft]

14:09:38 [shadi]

zakim, take up next

14:09:38 [Zakim]

agendum 2. "Review of New documents" taken up [from MoeKraft]

14:09:41 [MoeKraft]

Eric: Survey open until next Wednesday, 8/15

14:09:42 [shadi]

http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0002.html

14:10:36 [MoeKraft]

Eric: Please if there are any further comments before publication please put these into the survey

14:11:01 [MoeKraft]

Eric: Can also put discuss into mailing list.

14:11:05 [shadi]

q+

14:11:09 [shadi]

ack me

14:11:10 [MoeKraft]

s/discuss/discussion

14:11:43 [MoeKraft]

Shadi: Keep in mind which comments need to be addressed now and which to be addressed later on.

14:12:02 [shadi]

zakim, mute me

14:12:02 [Zakim]

Shadi should now be muted

14:12:11 [MoeKraft]

Zakim , close agendum 2

14:12:19 [MoeKraft]

Zakim, take up next

14:12:19 [Zakim]

agendum 3. "Discussion of Open Comments" taken up [from MoeKraft]

14:12:31 [shadi]

q+

14:12:32 [MoeKraft]

Eric: 8 comments remain open.

14:12:34 [shadi]

ack me

14:13:34 [MoeKraft]

Shadi: We reworked entire disposition of comments. Many issues are closed as addressed by group. Some have been addressed somewhat differently but kept in spirit of group. Marked review needed

14:13:49 [MoeKraft]

Shadi: Then there are some open ones that need to be addressed more clearly.

14:14:13 [MoeKraft]

Shadi: Only 8 open comments that still need to be resolved. Hope to resolve on today's call.

14:14:52 [MoeKraft]

Shadi: Review document first and then review disposition of comments to see if you agree or have feedback. There is room for comments in the survey.

14:15:03 [shadi]

zakim, mute me

14:15:03 [Zakim]

Shadi should now be muted

14:15:15 [vivienne]

zakim, mute me

14:15:15 [Zakim]

vivienne should now be muted

14:15:16 [MoeKraft]

Topic: Comment #19: Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>

14:16:35 [Detlev]

q+

14:16:41 [Detlev]

ack me

14:16:55 [MoeKraft]

Eric: Proposed resolution to open an action item to further discuss.

14:17:04 [shadi]

q+

14:17:11 [shadi]

ack me

14:17:13 [MoeKraft]

Detlev: Would be good to discuss a proposal to help us further agree.

14:17:42 [MoeKraft]

Eric: Idea is that if we put in tracker we start discussing in the tracker including preparing text.

14:18:00 [MoeKraft]

Detlev: Issue will be discussing whehter we should address at all?

14:18:18 [shadi]

q+

14:18:26 [MoeKraft]

Eric: Everything is a fail if there is an issue. Do we want this to be like this or do we want tolerance metrics?

14:18:48 [shadi]

ack me

14:18:51 [MoeKraft]

Eric: Tracker item to discuss if needed and how.

14:18:59 [Detlev]

Zakim, mute me

14:18:59 [Zakim]

Detlev should now be muted

14:19:01 [shadi]

[[3rd bullet objective from http://www.w3.org/WAI/ER/2011/eval/eval-ws - that of "Aggregating individual results into an overall conformance statement; this includes defining approaches for assessing the relative impact on failures, potentially through incorporating tolerance metrics."]]

14:19:12 [MoeKraft]

Shadi: Agree with Eric. Let's open a tracker issue.

14:20:01 [MoeKraft]

Shadi: Issue is How do we want to address tolerance metrics?

14:20:31 [MoeKraft]

+1

14:20:34 [Sarah_Swierenga]

+1

14:20:37 [shadi]

zakim, mute me

14:20:37 [Zakim]

Shadi should now be muted

14:20:37 [vivienne]

I'm okay with oening it in the tracker

14:20:40 [Detlev]

fine, as I said

14:20:40 [MartijnHoutepen]

+1

14:20:42 [MoeKraft]

Eric: Is everyone okay with opening an issue?

14:21:05 [ericvelleman]

Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)

14:21:10 [shadi]

ack me

14:21:24 [MoeKraft]

Resolution: Everyone is okay with opening an issue in tracker to discuss tolerance metrics.

14:21:31 [shadi]

zakim, mute me

14:21:31 [Zakim]

Shadi should now be muted

14:21:36 [MoeKraft]

Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)

14:21:51 [MoeKraft]

Topic: Comment #12: Need to agree on update in editor draft. There is a clarification of the concept of use-case in the updated section 3.4.1 Step 4.a (Check for the Broadest Variety of Use Cases). Does this clarify use case sufficiently for the moment or should we open an issue on use cases and come back tot that in a later draft. <http://www.w3.org/WAI/ER/conformance/comments#c12>

14:22:23 [shadi]

ack me

14:23:32 [MoeKraft]

Shadi: Not sure if 12 needs that much discussion. 15 really needs to be discussed.

14:23:48 [MoeKraft]

Shadi: 15 is linked from comment 19

14:24:51 [MoeKraft]

Shadi: If you agree with edit on Comment #12 then this does not need further discussion.

14:24:58 [vivienne]

15 isn't in the agenda though

14:24:59 [Detlev]

q+

14:25:04 [Detlev]

ack me

14:25:41 [MoeKraft]

Shadi: Review edits on comment #12 in survey. This is homework. No further discussion.

14:26:17 [MoeKraft]

Detlev: What is the order in which we are processing the comments?

14:26:25 [Tim]

Tim has joined #eval

14:26:31 [richard]

richard has joined #eval

14:26:33 [MoeKraft]

Shadi: The numbers of the comments were reorder based on where they fit in the document.

14:27:02 [MoeKraft]

Detlev: Having trouble finding the right issue in the disposition of comments.

14:27:27 [MoeKraft]

Shadi: We are not going through the entire list of comments. We are only discussing the 8 listed in Agenda as open comments.

14:30:21 [Zakim]

+Tim_Boland

14:30:22 [vivienne]

q+

14:30:30 [vivienne]

ack me

14:30:32 [MoeKraft]

Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>

14:30:52 [MoeKraft]

Topic: Comment#15 Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>

14:31:07 [MoeKraft]

Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>

14:31:23 [MartijnHoutepen]

+1

14:31:24 [vivienne]

zakim, mute me

14:31:24 [Zakim]

vivienne should now be muted

14:31:27 [Sarah_Swierenga]

+1

14:31:32 [vivienne]

+1

14:31:38 [ericvelleman]

+1

14:31:46 [efong]

+1

14:31:59 [shadi]

zakim, mute me

14:31:59 [Zakim]

Shadi should now be muted

14:32:13 [MoeKraft]

Eric: Does everyone agree for comment #15 that we open an issue in Tracker to address Tolerance metrics?

14:32:15 [MoeKraft]

+1

14:32:34 [Detlev]

fine

14:33:22 [Tim]

+1

14:33:29 [MoeKraft]

Resolution: Agree that Comment #15 should be addressed by an issue in Tracker. Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #19)

14:33:50 [ericvelleman]

http://www.w3.org/WAI/ER/conformance/comments#c16

14:34:13 [MoeKraft]

Topic: Comment #16: Need to resolve an objection. Is this objection solved by the updated section 3.5.3 Step 5.c (Provide a Performance Score (optional)? The section adds different scores. This methodology is not limited to automated evaluation or related sub-scores. <http://www.w3.org/WAI/ER/conformance/comments#c16>

14:34:39 [Detlev]

q+

14:34:46 [MoeKraft]

Eric: We currently have 3 different types of scores

14:34:48 [ericvelleman]

http://www.w3.org/WAI/ER/conformance/ED-methodology-20120730.html#step5c

14:34:57 [Detlev]

ack me

14:35:31 [vivienne]

q+

14:35:38 [MoeKraft]

Detlev: Would be useful to have the score for items that can be evaluated automatically. This could be quite valuable. Could cause some confusion. Not sure how to rectify the text.

14:36:15 [MoeKraft]

Eric: There is a section on machine evaluation. The text is limited. Maybe it is not enough.

14:36:18 [vivienne]

ack me

14:36:21 [Detlev]

Zakim, mute me

14:36:21 [Zakim]

Detlev should now be muted

14:36:45 [Detlev]

ack me

14:36:57 [MoeKraft]

Vivienne: Are we saying that the only time we should use a performance score for automated testing?

14:37:09 [shadi]

https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2647

14:37:26 [MoeKraft]

Detlev: Comment #16 talks about only limiting to automated evaluation.

14:37:35 [shadi]

q+

14:37:49 [MoeKraft]

Detlev: Could be quite valuable for benchmarking

14:38:12 [MoeKraft]

Detlev: Could have layer on top of conformance evaluation

14:38:56 [MoeKraft]

Detlev: Maybe need to have another score on top of conformance, maybe performance metrics. May be need two score.

14:39:30 [Detlev]

q+

14:39:41 [shadi]

q- last

14:39:47 [shadi]

q+ last

14:39:52 [shadi]

q- last

14:39:59 [shadi]

q-

14:40:01 [shadi]

q+

14:40:37 [vivienne]

zakim, mute me

14:40:37 [Zakim]

vivienne should now be muted

14:40:43 [MoeKraft]

Vivienne: Need to have some method for comparison. Need to have a metric developed to compare on an equal plane. Step 3.5.3 5.c is an optional thing but would be good to be part of methodology. Incredible amount of value in that.

14:41:21 [shadi]

s/method for comparison/method for comparison in large-scale evaluation

14:41:34 [MoeKraft]

Detlev: Agree with Vivienne that a score would be useful. Would be good to know how many pass/fail in evaluation. Automatic evaluation meet all or some criteria gives us an automatic score.

14:41:37 [shadi]

ack me

14:42:00 [Detlev]

zakim, mute me

14:42:00 [Zakim]

Detlev should now be muted

14:42:22 [vivienne]

q+

14:42:30 [MoeKraft]

Shadi: Agrees with all of the above. Definitely a need for metrics and scores and for automated testing. Research and Development working group are looking at what is out there. In Eval TF we are looking at performance criteria.

14:42:36 [Detlev]

q-

14:42:36 [MoeKraft]

Shadi: What has been met and not met.

14:43:25 [MoeKraft]

Shadi: We would have to analyze which criteria are fully automatable. And then the score would be confusing. The score would be simply for the automatable test. Maybe we should look at this in the future.

14:43:31 [ericvelleman]

q?

14:43:34 [shadi]

zakim, mute me

14:43:34 [Zakim]

Shadi should now be muted

14:43:34 [vivienne]

ack me

14:44:19 [shadi]

q+ to suggest opening an issue to further discuss "automated testing"

14:44:21 [MoeKraft]

Vivienne: Jury still out. Vivienne uses metrics because she needs something to make evaluation quantifiable. Working on a conformance score to compare.

14:45:00 [Detlev]

q+

14:45:05 [MoeKraft]

Vivienne: Using methodolgy as we do now but add in a score to give it some metrics for comparison. Can keep this optional but would like to keep this part of the methodology .

14:45:33 [Detlev]

ack me

14:45:38 [shadi]

ack me

14:45:39 [Zakim]

shadi, you wanted to suggest opening an issue to further discuss "automated testing"

14:46:43 [vivienne]

q+

14:46:53 [MoeKraft]

Shadi: Believes there is a misunderstanding. The idea is not to remove what we have. The request is to add another score for automated testing only in addition to the full conformance score. Looking to add a separate score when you only run an automated tool on a website.

14:47:39 [MoeKraft]

Shadi: Do we want to open an issue in Tracker and come back to this comment for a future draft?

14:47:41 [vivienne]

ack me

14:48:16 [vivienne]

zakim, mute me

14:48:16 [Zakim]

vivienne should now be muted

14:48:17 [MoeKraft]

Vivienne: Yes. I misunderstood. Quite happy with the resolution.

14:49:48 [MoeKraft]

Resolution: Tentative. Peter Korn is not on the call. Resolution: No change Rationale: WCAG-EM is only for full evaluation, not for tool measurement only. This is not covered in the current requirements and therefore not in the document. Related discussion: Eval TF survey #4 (ID 16)

14:50:14 [MoeKraft]

Shadi: Need to inform Peter while he was the only one who objected from the survey.

14:50:48 [Sarah_Swierenga]

+1 to the approach

14:50:48 [ericvelleman]

q?

14:50:52 [vivienne]

sounds good to me Shadi

14:50:59 [MoeKraft]

Shadi: If Peter objects, we will open an issue in Tracker to further discuss after draft is out.

14:51:02 [MartijnHoutepen]

both options are fine to me

14:51:12 [efong]

+1 for the approach

14:51:13 [MoeKraft]

+1

14:51:30 [Sarah_Swierenga]

+1

14:51:31 [Tim]

+1

14:51:34 [shadi]

zakim, mute me

14:51:34 [Zakim]

Shadi should now be muted

14:51:50 [MoeKraft]

Topic: Comment #48: Need to resolve an objection. The rationale is as indicated in the objection very much stressing the need to make it non-optional. We can decide to make no change and keep it optional (1) because we want to stay on the SC level as in WCAG2.0 and/or (2) open an issue to discuss making Step 1.e non-optional. Please note the open ended nature of the techniques. <http://www.w3.org/WAI/ER/conformance/comments#c48>

14:52:55 [MartijnHoutepen]

It was alistair

14:52:55 [shadi]

https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2630

14:53:49 [shadi]

ack me

14:54:51 [Detlev]

the rationale (imported from Alistair) will have caused the confusion...

14:56:25 [shadi]

"techniques are not required for conformance"

14:56:42 [Detlev]

q+

14:56:43 [vivienne]

+1

14:56:45 [MartijnHoutepen]

+1

14:56:45 [Detlev]

ack me

14:56:51 [ericvelleman]

+1

14:56:54 [shadi]

zakim, mute me

14:56:54 [Zakim]

Shadi should now be muted

14:57:35 [MoeKraft]

Detlev: Commissioner would be interested in seeing the technique is addressed by methodology.

14:57:44 [shadi]

ack me

14:57:45 [MoeKraft]

Eric: Yes. It is useful but it is optional.

14:58:14 [MoeKraft]

Detlev: Right rationale in many circumstances techniques may not be able to be provided and evaluation would have to proceed with out them.

14:58:43 [MoeKraft]

Detlev: Techniques may not be available to the evaluator

14:58:58 [MoeKraft]

Detlev: So we cannot make this required.

14:59:29 [MoeKraft]

Shadi: I can agree to Detlev's rationale. It addresses Allistair's concern more directly.

14:59:29 [Detlev]

Zakim, mute me

14:59:29 [Zakim]

Detlev should now be muted

14:59:38 [shadi]

zakim, mute me

14:59:38 [Zakim]

Shadi should now be muted

14:59:39 [shadi]

ack me

14:59:47 [MartijnHoutepen]

+1

14:59:53 [Sarah_Swierenga]

+1

14:59:53 [efong]

+1

14:59:53 [vivienne]

+1

14:59:54 [ericvelleman]

+1

14:59:57 [MoeKraft]

+1

15:00:08 [shadi]

zakim, mute me

15:00:08 [Zakim]

Shadi should now be muted

15:00:28 [MoeKraft]

Resolution: Update the rationale to reflect Detlev's argument.

15:00:48 [MoeKraft]

Eric: We still have 4 open comments.

15:01:08 [MoeKraft]

Eric: No use starting discussions now. Will send out to mailing list.

15:01:10 [shadi]

ack me

15:01:47 [MoeKraft]

Shadi: There is one from Kirsten which she put in the survey as well. Others please review these 4 remaining comments. Send by email or put in survey.

15:02:06 [shadi]

zakim, take up next

15:02:06 [Zakim]

agendum 4. "TPAC in Lyon" taken up [from MoeKraft]

15:02:07 [Detlev]

q+

15:02:15 [Detlev]

ack me

15:02:26 [MoeKraft]

Comment #A1: Need to agree on the proposed resolution. The tekst says “..minimum set of web browsers and assistive technology to evaluate for shall be defined”. This is important because it is an important part of defining "accessibility support" Proposed resolution is therefore to make no change. <http://www.w3.org/WAI/ER/conformance/comments#c001">http://www.w3.org/WAI/ER/conformance/comments#c001> Comment #A4: Need to agree on the proposed resolution. See: <http://www.w3.org/WAI/ER/conformance/comments#

15:02:33 [MoeKraft]

Zakim, take up next

15:02:33 [Zakim]

agendum 4 was just opened, MoeKraft

15:03:02 [MoeKraft]

Detlev: Will there be enough members attending?

15:03:21 [MoeKraft]

Shadi: If you are planning on attending, please register and make travel arrangements.

15:04:00 [MoeKraft]

Shadi: Certain folks did indicate that they are able to come.

15:04:25 [shadi]

https://www.w3.org/2002/09/wbs/myQuestionnaires

15:04:48 [Sarah_Swierenga]

I'm sorry I won't be able to meet all of you in person in Lyon. I have a research participant coming in now, so I need to go. Take care, Sarah

15:04:55 [Zakim]

-Sarah_Swierenga

15:04:57 [MoeKraft]

Eric: Are there any other issue?

15:04:59 [efong]

bye

15:05:12 [richard]

bye

15:05:16 [vivienne]

good night all

15:05:16 [MoeKraft]

Eric: Will put the 4 remaining comments in the discussion list.

15:05:17 [MartijnHoutepen]

zakim, unmute me

15:05:17 [Zakim]

MartijnHoutepen should no longer be muted

15:05:18 [Zakim]

-Tim_Boland

15:05:26 [Zakim]

-Detlev

15:05:26 [MoeKraft]

bye

15:05:28 [Zakim]

-Eric_Velleman

15:05:29 [Zakim]

-Shadi

15:05:30 [Zakim]

-Moe_Kraft

15:05:31 [Zakim]

-vivienne

15:05:34 [Zakim]

-MartijnHoutepen

15:05:34 [Zakim]

- +1.301.975.aaaa

15:05:35 [Zakim]

WAI_ERTWG(Eval TF)10:00AM has ended

15:05:37 [Zakim]

Attendees were +1.301.975.aaaa, +49.404.318.aabb, +31.30.239.aacc, vivienne, MartijnHoutepen, Moe_Kraft, Shadi, +49.404.318.aadd, Detlev, +1.517.432.aaee, Eric_Velleman,

15:05:39 [Zakim]

... Sarah_Swierenga, Tim_Boland

15:05:45 [ericvelleman]

ericvelleman has left #eval

15:39:01 [shadi]

trackbot, end meeting

15:39:01 [trackbot]

Zakim, list attendees

15:39:01 [Zakim]

sorry, trackbot, I don't know what conference this is

15:39:09 [trackbot]

RRSAgent, please draft minutes

15:39:09 [RRSAgent]

I have made the request to generate http://www.w3.org/2012/08/09-eval-minutes.html trackbot

15:39:10 [trackbot]

RRSAgent, bye

15:39:10 [RRSAgent]

I see no action items