WCAG 2.0 Evaluation Methodology Task Force Teleconference

Discussion of Changelog and new Methodology

Eric: One or two items, change log and new methodology.

<shadi> http://www.w3.org/WAI/ER/conformance/comments

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320

Eric: Looked at list of comments we discussed in alst call, tried to make changes to each item. Change log and what we did is available.

Sh: Dispostion of comments and Dif mark version. Will also send separate version.

E: Most interesting to look up diff mark and comments to make sure okay for people who commented.
... My mail has been out, dk if have seen most recent comments. Have tried to track however. Can look at dif version for changes.
... Hope to have covered almost all comments seriously. In way that won't lead to add'l comments. Thanks Shadi.

E: Is someone on call have comments that what was wrong and needs revising later?

<vivienne> I was happy with the changes so far

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff#step1e

D: Still wonder what requirements step 1e, where try to specify sources for evaluation alog with eval commissioner. What does it mean in practice? 100s of techniques available. Many redundant.
... Sure not what was intended. Elaborate?

E: Don't want techniaues to become new SC, but good to say what they are in different stages of eval process.

D: Why woudl that be helpful?

E: Sometimes want some techniques to be looked at, others not. Example: Just look at Flash. Need to workout more delicately in following steps.

D: Ok

Sh: U R not a nuisance. Do you have concersn about adding this. Maybe needs to be expalined more on list. Procurement settings where techniques are defined for developer, what should be eval'd against.

D: Concern is parsimony. May lead people to document all sorts of techniques wo real reason to do that. Would be nonsense.
... Example, Eric's comment. Techniques that may be used when others should.

Sh: Relates to reporting aspect. Which comes later.

E: What should and should not be reported. This is stage when looking at site and deciding which techniquest to be used. Perhaps can limit Dev's concerns. Not list all the techniques that can be found on site.
... Answer to step 1e sufficient?

D: Optional. Not too worried. Agree with Kerstin that particular technique is not required. Not necessarily choose technique ahead of time.

K: Not happy w/ requrement, can claify later.

E: Have to provide more explanation in next stage. Briefly, of course.

Sh: Eric keeping log so can go back to these?

<Detlev> fine

E: Yes, my view that wouldn't get in way of publishing draft, will take care of first thing in next draft. Kerstin Ok?

K: Yes. Goodness criteria?

<shadi> http://www.w3.org/WAI/ER/conformance/comments#c21

E: Where in document?

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0055.html

K: Not in document. Very concerned bec of it.

Sh: Comment #21. Comment to focus on. Also linked. Brief discussion on line this afternoon.

E: What was in discussion?

Sh: Between Kerestin and det lev. Maybe discuss?

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0058.html

K: I try. Use more words than last month? Quantitative evaluation calls for Validity, etc. No question about their relevance. Detlev posted article on qualitative methods. If goodness criteria is relevant for qualitative research.

<Detlev> that article referenced was not so important - the comments before it were!

K: But we're doing quantitative, so relevant.

Sh: What are you suggesting? 96% reliability?

K: would be okay if worked it out later. How much reliable, etc. But for me it is important that have 3 goodness criteria: objective, reliable, .

E: Isn't already in document? Have to put in scope?

K: Yes, important for clients and users.

E: Needs more discussion.

K: Can put one sentence in abstract in one of first chapters. Work it out later.

E: In abstract...can put in that we strive for this.

K: Maybe one add'l sentence. Wonder that this is a question. Bec all I knew from eval method in quantiative field, no question.

E: Agree, just where and how to put it. Put it in abstract. At least in document. Something discussing. Also, put in status of document?

Sh: I'd like more discussion before decide where to put it.

E: Discussion on goodness criteria.

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Mar/0055.html

D: Added 3 comments on obj, rel, and val, clear that looking for reilibe and valid results. Objectivitly something we can aim for.
... Didn't find clear undisputed link to three criteria, always some subjectivity. It's in mailing list.

Sh: Let R go.

R: Listening to K, goodness phrase confusing. Instead use level of confidence? Clearer?

K: Goodness criteria. Think should use Goodness criteria. Since it's the term. Especially think about translations, should use specific terms.

V: Actually when reading comments. Have never run across Goodness term.

I haven't either.

V: Maybe it's in translation you're coming from.

E: New to me too.

<Kerstin> there are 11 Million hits on google for "goodness criteria"

Sh: Part of my concern is that discussion of whether method is qualitative, quantitative and citing scientific defs, don't want to be orthogonal. Agree that overall want reliablitily between evaluators, but binding them to quant values have to think about nature of values. Ex: how good is alt atrtriburte. Is subjective.

<Detlev> agree with Shadi

Sh: A lengthy discussion. Let's not add it hastily. Could draw attention away from document.

K: Sorry.Don't agree. Discussing sometime for months. Lots of emails on these issues. Also said reliability objectivity, reliablity never 100%, are ranges.

<richard> Level of confidence: Statistical measure of the number of times out of 100 that test results can be expected to be within a specified range. For example, a confidence level of 95% means that the result of an action will probably meet expectations 95% of the time. Most analyses of variance or correlation are described in terms of some level of confidence.

Sh: But specific number is concerning. Even though not feasible.

K: Intention not specific number.

D: Think it's fair to say method should rely on reliability and objectivity, fair to try to define, But have to be more specific. Validity rely on use on web, reliability on evaluators replicating. Can be spelled out. But how much should be presciriptive?
... Wonder if that's possible. Validity, also outside methodology. Depends on timing and procedures used. Change over time.

K: Think that's a misunderstanding. Validity means testing WCAG 2, then nothing else. Not an interpretation.

<MartijnHoutepen> I agree with Detlev

Sh: Maybe in next week work out specific proposal for criteria, how prescirptive, what we understand about each. Validity...have in requirements already, dk how to get into specification. Shouldn't add in anything that we'er

still discussion.

<Detlev> fine, give it more time...

E: Selected sample part of report need to make decisions on what is compliant. We'll get into this discussion there. Like level of confidence. We'll get into it in section 5 and 4. Too early.
... Go into discussion on list. What we mean. Where to place it in document.

I agree.

Validity would have to go throughout document. Need more discussion before we put it in.

<Zakim> shadi, you wanted to add to what Eric just said

K: Need more discussion. But don't understand why we can't put in sentence. Maybe we want to increase valididyt and objectively. Don't understand why we should leave these very improtant things out.

Sh: Want to echo E's comments. Plan is first version this December. Then put into practice. in second year revise according to feedback.

Sh: Have validity in document already. Adding as requirement into document would have implications to entire document. Would hold us back from publication.

E: To stay in middle. What we could do is in status of document ask public to give feedback on definition, scope and interpretation os WCAG 2, look at requirements to make sure document meets what we've set.
... Your issue could be a good thing for comments.

K: Not sure. Also look for solution. Okay for me if we increase objectivity and reliability and then work it out. Criteria are important, however.

E: Don't deny they're important. Just how to put in document.

<shadi> [[It addresses the need for a standardized approach ...]]

E: Too much to do before publishing draft.

K; But coud do it now (?).

E: How about point to it in document and ask for comment. Asking for public to comment on it.

Sh: Agree that point to requirements, but singling out specific requirements, not sure a good idea. Abstract does say need for std approach, intent is there.
... Any suggestions? Do we need to vote on it?

E: Would propose that we add in what we want people to look at, ask them to look at requirements. Replicability, reliability are in there. Could generate valuable input.

<shadi> +1 to Eric's suggestion

E: Already discussing in mailing list. Is mailing list noted in document? Can we do that?

Sh: Yes. Discussion or what?

E: People can see how we got to this. Valuable point. Need to take a decision. Push document into draft or stop for more changes. Can we close this now?

+1

K; How about Eric's suggestion: pointing to document to get more comments.

K: Would depend on form of edit.

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120320-diff#sotd

Sh: Three bullets asking for feedback--also looking for feedback looking for requirements.

E: Along with link to document.

K: But don't have criteria in document. Don't want to bother you, but very important to me.

E: Specific requriement

"x"?

Sh: Again, what do others think.
... Single out specific requirements.

V: Think we're being...agree they're critical, but don' understand importance putting into document at this point. Don't understand requirement to spell it out now. It's a work in progress.
... To me getting comments would be sufficient.

M: Don't see explicit need to explain goals.

K: As M said, clear that we are looking for methodology that is reliable, but when remember discussions on mailing list it's a critical point. If we had all agreed there wouldn't be discussion.
... But discussed from the beginning also had ? in the draft, don't find it now.

E: Is it numbered in list? Conforms to guidelines, reliability, requirement 3 would cover in part. What we could do in status of document ask people to compare with requirements. Yielding of equivialent results. Sections 4 and 5 will be interesting.

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-reqs-20111012#rq03

K; Remeber discussion of equivalent results. Reliability and equivalent results.

E: High degree of correlation, with different web pages and tools. Maybe have to go back. But propose not to do it now.
... Just small part of task force. How to proceed? Vote for draft?

Sh: Do we need to resolve issue now or after release as draft. Hold up or resolve.

Release the doc.

<richard> Release the doc

<vivienne> release the doc

<MartijnHoutepen> release the doc

<Detlev> Is full consensus of the group a prerequisite for publication?

<TimBoland> release the doc

Sh: Feel free to type what you want to see.

E: Doesn't mean we won't discuss in future.

<ericvelleman> release the doc

<Kerstin> release the doc with mentioning the criteria in one of the first chapter and asking the public on these issue

Sh: Consensus is not that everyone is in agreement. Document can go out with filed objection. Ultimately up to directors. Happened in WCAG 2.

K: Add request for public to review.

<Detlev> There seems to be a strong majority in favour of publishing (Richard and I had reservations for other reasons but have given in)

E: Add specifically pointer to requirements.

<richard> Thanks Detlev !

E: Point to mailing list.

Sh: URL will change.

<shadi> http://www.w3.org/WAI/ER/conformance/comments#c74

<Detlev> after pub

<MartijnHoutepen> after publication

<vivienne> can wait till after publication I think

Sh: From martin 5b to 5f unclear what benefit would be...type before or after pub

<TimBoland> after pub

After pub

<ericvelleman> after pub comment 74

<richard> after pub

<shadi> http://www.w3.org/WAI/ER/conformance/comments#c89

Sh: Comment #89 trickier: Amy chen, remove essence 1D minimum set of browsers, usual practice of evaluation...

After pub

<ericvelleman> comment 89

<Detlev> after pub

<MartijnHoutepen> after publication

<ericvelleman> after pub

<vivienne> after pub

<Kerstin> after pub for the last two issues

<richard> after pub

Sh: in absence of mailing list will proceed to publication. Again K, E and I will address specific concerns.

Next Steps

Sh: This doc goig as is with minor reference to reliability. Congratualtions. Already have several issues to come back to and discuss. will give us valuable input.

E: Give to working groups for feedback.
... Meet next week for follow-up discussion on next draft.