WCAG 2.0 Evaluation Methodology Task Force Teleconference

Eric: Welcome Everybody, then review of new documents

Review of New documents

Eric: Survey open until next Wednesday, 8/15

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0002.html

Eric: Please if there are any further comments before publication please put these into the survey
... Can also put discussion into mailing list.

Shadi: Keep in mind which comments need to be addressed now and which to be addressed later on.

Discussion of Open Comments

Eric: 8 comments remain open.

Shadi: We reworked entire disposition of comments. Many issues are closed as addressed by group. Some have been addressed somewhat differently but kept in spirit of group. Marked review needed
... Then there are some open ones that need to be addressed more clearly.
... Only 8 open comments that still need to be resolved. Hope to resolve on today's call.
... Review document first and then review disposition of comments to see if you agree or have feedback. There is room for comments in the survey.

Comment #19: Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>

Eric: Proposed resolution to open an action item to further discuss.

Detlev: Would be good to discuss a proposal to help us further agree.

Eric: Idea is that if we put in tracker we start discussing in the tracker including preparing text.

Detlev: Issue will be discussing whehter we should address at all?

Eric: Everything is a fail if there is an issue. Do we want this to be like this or do we want tolerance metrics?
... Tracker item to discuss if needed and how.

<shadi> [[3rd bullet objective from http://www.w3.org/WAI/ER/2011/eval/eval-ws - that of "Aggregating individual results into an overall conformance statement; this includes defining approaches for assessing the relative impact on failures, potentially through incorporating tolerance metrics."]]

Shadi: Agree with Eric. Let's open a tracker issue.
... Issue is How do we want to address tolerance metrics?

+1

<Sarah_Swierenga> +1

<vivienne> I'm okay with oening it in the tracker

<Detlev> fine, as I said

<MartijnHoutepen> +1

Eric: Is everyone okay with opening an issue?

<ericvelleman> Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)

Resolution: Everyone is okay with opening an issue in tracker to discuss tolerance metrics.
... No change; open an issue to further discuss "tolerance metrics" (see also Comment #15)

Comment #12: Need to agree on update in editor draft. There is a clarification of the concept of use-case in the updated section 3.4.1 Step 4.a (Check for the Broadest Variety of Use Cases). Does this clarify use case sufficiently for the moment or should we open an issue on use cases and come back tot that in a later draft. <http://www.w3.org/WAI/ER/conformance/comments#c12>

Shadi: Not sure if 12 needs that much discussion. 15 really needs to be discussed.
... 15 is linked from comment 19
... If you agree with edit on Comment #12 then this does not need further discussion.

<vivienne> 15 isn't in the agenda though

Shadi: Review edits on comment #12 in survey. This is homework. No further discussion.

Detlev: What is the order in which we are processing the comments?

Shadi: The numbers of the comments were reorder based on where they fit in the document.

Detlev: Having trouble finding the right issue in the disposition of comments.

Shadi: We are not going through the entire list of comments. We are only discussing the 8 listed in Agenda as open comments.

Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>

Comment#15 Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c19>

Need to agree on opening an issue on "tolerance metrics" More explanation of the issue and proposed resolution: <http://www.w3.org/WAI/ER/conformance/comments#c15>

<MartijnHoutepen> +1

<Sarah_Swierenga> +1

<vivienne> +1

<ericvelleman> +1

<efong> +1

Eric: Does everyone agree for comment #15 that we open an issue in Tracker to address Tolerance metrics?

+1

<Detlev> fine

<Tim> +1

Resolution: Agree that Comment #15 should be addressed by an issue in Tracker. Resolution: No change; open an issue to further discuss "tolerance metrics" (see also Comment #19)

<ericvelleman> http://www.w3.org/WAI/ER/conformance/comments#c16

Comment #16: Need to resolve an objection. Is this objection solved by the updated section 3.5.3 Step 5.c (Provide a Performance Score (optional)? The section adds different scores. This methodology is not limited to automated evaluation or related sub-scores. <http://www.w3.org/WAI/ER/conformance/comments#c16>

Eric: We currently have 3 different types of scores

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120730.html#step5c

Detlev: Would be useful to have the score for items that can be evaluated automatically. This could be quite valuable. Could cause some confusion. Not sure how to rectify the text.

Eric: There is a section on machine evaluation. The text is limited. Maybe it is not enough.

Vivienne: Are we saying that the only time we should use a performance score for automated testing?

<shadi> https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2647

Detlev: Comment #16 talks about only limiting to automated evaluation.
... Could be quite valuable for benchmarking
... Could have layer on top of conformance evaluation
... Maybe need to have another score on top of conformance, maybe performance metrics. May be need two score.

Vivienne: Need to have some method for comparison in large-scale evaluation. Need to have a metric developed to compare on an equal plane. Step 3.5.3 5.c is an optional thing but would be good to be part of methodology. Incredible amount of value in that.

Detlev: Agree with Vivienne that a score would be useful. Would be good to know how many pass/fail in evaluation. Automatic evaluation meet all or some criteria gives us an automatic score.

Shadi: Agrees with all of the above. Definitely a need for metrics and scores and for automated testing. Research and Development working group are looking at what is out there. In Eval TF we are looking at performance criteria.
... What has been met and not met.
... We would have to analyze which criteria are fully automatable. And then the score would be confusing. The score would be simply for the automatable test. Maybe we should look at this in the future.

Vivienne: Jury still out. Vivienne uses metrics because she needs something to make evaluation quantifiable. Working on a conformance score to compare.
... Using methodolgy as we do now but add in a score to give it some metrics for comparison. Can keep this optional but would like to keep this part of the methodology .

<Zakim> shadi, you wanted to suggest opening an issue to further discuss "automated testing"

Shadi: Believes there is a misunderstanding. The idea is not to remove what we have. The request is to add another score for automated testing only in addition to the full conformance score. Looking to add a separate score when you only run an automated tool on a website.
... Do we want to open an issue in Tracker and come back to this comment for a future draft?

Vivienne: Yes. I misunderstood. Quite happy with the resolution.

Resolution: Tentative. Peter Korn is not on the call. Resolution: No change Rationale: WCAG-EM is only for full evaluation, not for tool measurement only. This is not covered in the current requirements and therefore not in the document. Related discussion: Eval TF survey #4 (ID 16)

Shadi: Need to inform Peter while he was the only one who objected from the survey.

<Sarah_Swierenga> +1 to the approach

<vivienne> sounds good to me Shadi

Shadi: If Peter objects, we will open an issue in Tracker to further discuss after draft is out.

<MartijnHoutepen> both options are fine to me

<efong> +1 for the approach

+1

<Sarah_Swierenga> +1

<Tim> +1

Comment #48: Need to resolve an objection. The rationale is as indicated in the objection very much stressing the need to make it non-optional. We can decide to make no change and keep it optional (1) because we want to stay on the SC level as in WCAG2.0 and/or (2) open an issue to discuss making Step 1.e non-optional. Please note the open ended nature of the techniques. <http://www.w3.org/WAI/ER/conformance/comments#c48>

<MartijnHoutepen> It was alistair

<shadi> https://www.w3.org/2002/09/wbs/48225/evaltfq4/results#x2630

<Detlev> the rationale (imported from Alistair) will have caused the confusion...

<shadi> "techniques are not required for conformance"

<vivienne> +1

<MartijnHoutepen> +1

<ericvelleman> +1

Detlev: Commissioner would be interested in seeing the technique is addressed by methodology.

Eric: Yes. It is useful but it is optional.

Detlev: Right rationale in many circumstances techniques may not be able to be provided and evaluation would have to proceed with out them.
... Techniques may not be available to the evaluator
... So we cannot make this required.

Shadi: I can agree to Detlev's rationale. It addresses Allistair's concern more directly.

<MartijnHoutepen> +1

<Sarah_Swierenga> +1

<efong> +1

<vivienne> +1

<ericvelleman> +1

+1

Resolution: Update the rationale to reflect Detlev's argument.

Eric: We still have 4 open comments.
... No use starting discussions now. Will send out to mailing list.

Shadi: There is one from Kirsten which she put in the survey as well. Others please review these 4 remaining comments. Send by email or put in survey.

TPAC in Lyon

Comment #A1: Need to agree on the proposed resolution. The tekst says “..minimum set of web browsers and assistive technology to evaluate for shall be defined”. This is important because it is an important part of defining "accessibility support" Proposed resolution is therefore to make no change. <http://www.w3.org/WAI/ER/conformance/comments#c001> Comment #A4: Need to agree on the proposed resolution. See: <http://www.w3.org/WAI/ER/conformance/comments#

Detlev: Will there be enough members attending?

Shadi: If you are planning on attending, please register and make travel arrangements.
... Certain folks did indicate that they are able to come.

<shadi> https://www.w3.org/2002/09/wbs/myQuestionnaires

<Sarah_Swierenga> I'm sorry I won't be able to meet all of you in person in Lyon. I have a research participant coming in now, so I need to go. Take care, Sarah

Eric: Are there any other issue?

Eric: Will put the 4 remaining comments in the discussion list.