See also: IRC log
<shadi> http://www.w3.org/WAI/ER/methodology/
<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Jul/0003.html
<shadi> https://www.w3.org/2002/09/wbs/1/WCAG-EM-20140623/results
Shadi: This is the last meeting for the WCAG EM
working. Some comments remain but not too controversial.
... Link to overall comments. Some very positive.
... Number of comments and their severity is dropping which is good news.
Under 100 comments.
... Provide an editor draft to the WCAG WG next Tuesday and propose for
publish
... Once published will ask WCAG WG to close down this task force.
... Most folks are active in WCAG WG. I encourage you to continue to
participate in WCAG or other W3C working groups
Detlev: One question: What about other comments
that are not on the list? For example, random pages.
... Does this mean it will be addressed with the commenter.
Shadi: Refresh and you will now see comments marked as objections and marked Do not publish with out it. Should be in there now.
Detlev: Thanks. Couldn't find it earlier.
Shadi: We'll try to get to that on this call as
well.
... We will not be able to cover every comment today. Next step will be. Get
more input on comments and Shadi can address fairly easily and a final draft
could be developed.
... Any other questions on steps, process and covering last hurdle?
<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Jul/0003.html
Shadi: Link to email circulating on selection of
comments marked as objections or needing to resolve.
... All comments relate to 3 areas of document
... Comments 9, 10 and 11: Area of expertise
... Edited section so that we do not make a statement that a team is required
but can be done by 1 person but skills may be necessary. Some folks feel we are
still implicitly referring to teams.
... Suggested change
<shadi> This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise). Using the combined expertise of different evaluators may provide an effective way to evaluate content when some of the required expertise is missing from one team member but is possessed by another on the team. While not required for using this methodology, the use of review teams may sometimes be necessary and/or beneficial. Using
<shadi> Combined Expertise to Evaluate Web Accessibility provides further guidance on using the combined expertise of review teams, which is beyond the scope of this document.
<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140623#combined
KW: Are these together?
Shadi: Yes. Together. Not replacing.
<MaryJo> +1 for newly proposed text.
Vivienne: I little concerned that we don't water
down the notion that using more than 1 person to review is better than 1 single
person.
... Literature recommends that it takes 3 people to capture most errors.
... I'm concerned that we are watering down that multiple reviewers is
better.
Shadi: W3C material doesn't say it as clearly and
critique that they are referring to an old resource. We would need to qualify
that 1 experienced evaluator may be better than 2 inexperienced evaluators.
... We get into a whole complexity that we are trying to avoid.
... Need skills. Sometimes you may need more than 1 person to cover skills.
Vivienne: Georgio Brajnik equated 3 skilled evaluators to 14 novices. Our statement goes against the research.
Shadi: We want to stress that it is about the skill level and not the number.
DetLev: Don't mind original text. Even experienced evaluators may miss things. Combining results is good. But maybe don't see the problem as long as the methodology states that multiple evaluators may product better results.
Shadi: Correct. We do build in the "may"
<alistair> This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise) or a team with collective expertise - See combined expertise.
<Vivienne> we use 3 evaluators which provides what we feel is a more robust approach and better reliability. I think the fact that it is optional already covers it without having to dilute it further.
Alistair: Recommend that it is beyond our reach to state how many but that it is depended upon the reader to decide. We do direct them to Using Combined Expertise document. It is up to the reader to decide.
<Detlev> Can live with that
<EricVelleman> +1
<Detlev> 1+
Moe: It is beyond the scope of this document that is why we refer the reader to Using Combined Expertise document
<alistair> +1
<martijnhoutepen_> +1
+1
<Vivienne> not really but I can live with it
<kw> +1
<EricVelleman> I like the text that David proposes. It seems to address the concerns by Vivienne
Shadi: It might be better addressing what Vivienne is talking about by redirecting the reader to the combined expertise document, http://www.w3.org/WAI/eval/reviewteams
<EricVelleman> +1
<Detlev> Sounds good
<MaryJo> +1
<martijnhoutepen_> +1
<Vivienne> +1
Shadi: Is it okay for Shadi to take an editorial stab at the wording?
+1
<alistair> +1
Shadi: Moving on to Comments 33, 34 & 35
... Comment 33 wants to add specifically the scheme of pass/fail.
... Success Criteria can only be met or not met
<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140623#step4a
Shadi: Commenter wants this to be super clear
... How can we add this to not convolute the document?
Vivienne: Are we saying we cannot have N/A? For example, Video and Audio controls. If they do not have video do these pass?
Shadi: Yes. According to WCAG 2.0
<Detlev> As you know I have always argued pass/fail does not work well when judging content against WCAG SC so I would opt not make that explicit
Shadi: Pass by virtue that there is no multimedia
content
... WCAG WG agreed to allow Not Present but this is equivalent to Pass
... See WCAG 2 Success Criteria. There is a note talking about Not Present.
Vivienne: Some times we put Not Tested. Can only reliably test when on premise.
Shadi: Not Test is ambiguous. What is the reason
for not testing as opposed to Not present.
... WCAG is written in a way to be applicable.
... Let's not get into this discussion again. Let's review the wording to make
it more clear
Detlev: It is difficult some success criteria to have just pass or fail. For that reason it is not a good idea to have pass / fail. Causes a problem for schemes that have a more granular testing scheme.
<shadi> [[There are typically several ways to determine whether WCAG 2.0 Success Criteria have been met or not met.]]
Detlev: Personally should not be mentioned explicitly.
<shadi> [[Evaluators might use other approaches to evaluate whether WCAG 2.0 Success Criteria have been met or not met.]]
Shadi: We do only talk about met and not met
throughout the document. We do not talk about grading schemes. This may be were
the confusion comes from by the commented. WCAG does follow a pass/fail
scheme
... Do we need to clarify?
<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140623#success-criteria
Shadi: Take a moment to read through this section
Alistair: The reference to not applicable that it passes. Would it help to put a reference to that position so the reader can make their on conclusion.
Shadi: We do link to the success criteria being
satisfied.
... Not sure if it is stated that clearly in WCAG.
... Does anyone know if there is a clear statement in WCAG? I will look for
it.
<martijnhoutepen_> suggestion for first sentence ¨There are typically several ways to determine whether WCAG 2.0 Success Criteria pass or fail¨
<Detlev> I prefer the current text
Marijn: Change the first sentence to indicate there are multiple ways to determine if SC pass/fail.
+1 @ Detlev. I too prefer the current text
<shadi> http://www.w3.org/TR/WCAG/#cc1
Shadi: WCAG talks about SC being satisfied.
<EricVelleman> I like the current text and think it is clear, so +1 to keep the text
<Vivienne> I don't think that is terribly clear that you can't have n/a
Shadi: I will take a pass at small adjustments based on what Martijn was saying to clarify. Might still be misunderstandings.
Detlev: Vivienne is worried about Not Present.
Having been met or not met allows more room for interpretation and assessing
success criteria.
... I prefer to keep the text right now.
Shadi: That would be a concern by the WCAG WG while we are stating it is ambiguous. The Note is good and we could start off the sentence WCAG 2.0 success criteria can only be met or not met when there is no content presented to the evaluator.
<EricVelleman> WCAG2.0 is also not very clear... the link to satisfied does not say pass or fail, but 'false'
Shadi: Make more explicit. SC apply or do not apply.
<martijnhoutepen_> +1 to adding it in the note
+1
<EricVelleman> +1
And find a reference link if possible
Alistair: I would like to see a link to WCAG that states that.
Shadi: Alistair can you take an action item to find the location in WCAG that states clearly the requirement.
<Vivienne> I've looked through WCAG 2.0 and can't find anywhere it says it can't have n/a in this document, even though everyone feels it is implied
Alistair: Having trouble finding the statement by WCAG WG that if an item is not present that it passes SC.
<martijnhoutepen_> probably here: http://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html
Shadi: We should kick this back to the WCAG
WG.
... If there is a statement in WCAG that you can only pass or fail we link to
it otherwise we need to require WCAG WG to add this.
Vivienne: I think I found it.
Can you post the link in IRC?
@Vivienne
<Vivienne> http://www.w3.org/TR/UNDERSTANDING-WCAG20/
<EricVelleman> True or false ... or pass or fail?
<Vivienne> true or false
<shadi> http://www.w3.org/TR/UNDERSTANDING-WCAG20/complete.html#abstract
<Vivienne> " A success criterion is a testable statement that will be either true or false when applied to specific content."
Shadi: Let's move on. We still have a couple
items.
... Next set of comments.
Shadi: Comments 37, 38, 39 & 40
... We took the approach to remove the actual calculation, the algorithm to
calculate a score. We just left a comment that a score could be ambiguous and
misleading.
... Do we need to be stronger about this?
<EricVelleman> More disclaimers would be strange
Shadi: How do people feel about this?
<Vivienne> http://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html This has further reinforcement of the WCAG group's concerns: "Note: this means that if there is no content to which a success criterion applies, the success criterion is satisfied."
Shadi: remove "widely recognized" and change to
there is "no single metric". I think this goes too far. There is no metric we
know of per se.
... I do not see this as a big issue.
... Any thoughts?
I agree with you Shadi. I don't think this is a big deal
<shadi> http://www.w3.org/WAI/ER/conformance/comments-20140623#step5
Sarah: Sorry, my meeting ran over but didn't want to miss this one.
<Detlev> I thik the text can stay the way it is
<EricVelleman> Agree with Shadi
<martijnhoutepen_> +1 to shadi: not a big issue i think
Sarah: Dropping "widely recognized" is probably appropriate.
<EricVelleman> How do we know there is not single metric?
Sarah: We may be asked for references to metric.
<Detlev> A consensual metric might still be developed!
Shadi: I see. If we say "widely recognized" we might imply that some exist. If we say that there are no metrics, we cannot prove that either.
<Vivienne> I'm okay with the statement the way it stands
Shadi: How about? To date we are not aware of a metric...
Alistair: What about? Remove the first two sentences.
<EricVelleman> I have spoken to people who are convoked that they have a good metric that reflects it all. That is why the addition 'widely recognized' is a safe thing to say
Alistair: Let's not make an explicit statement.
Shadi: We have been asked to make a statement.
<Detlev> Prefer the text to stay as it is
Alistair: It's been reduced from a paragraph to a statement. Prefer to get rid of it.
Shadi: I see your point but we would be undoing previous comments and previous revisions
How about something like, At the date of publication of this document there are no known metrics...
Shadi: Let's move on to Comment 28
<shadi> http://www.w3.org/WAI/ER/conformance/comments-20140623#comment28
Shadi: We don't provide selection of how to do a random sample in any case. Random can be as much as random clicking so appreciate that it is difficult for applications but we cannot get into providing a clear directive
Detlev: If there are 1 page websites, then that 1 page is your sample.
<EricVelleman> Yes, but we also cover a description instead of a URL
Detlev: Not meaningful to have 5 web pages there
Shadi: Even if there is 1 page,1 url, there would be different states.
Detlev: Not necessarily true. I am testing a 1
page document right now that has no dynamic content.
... We should not exclude these sites from our methodology.
<EricVelleman> We could add " if available"
Shadi: At start of Step 3, if your sample is the entire site, that your sample is the entire site.
Detlev: Ok. If so, that resolves this issue.
Shadi: Please review Step 3 where we have the statement regarding the entire web site
<EricVelleman> Agree with Detlev. Would this be solved by adding "if available" to the requirement of 5 random pages?
Detlev: Maybe need an exception that we may not
be able to come up with a Random sample.
... Not a big issue. We can move on.
<Vivienne> If I'm looking at a 25 page website and have a 10 page sample that is targeted, should I be forced to select 5 random page?
Shadi: Random Sample is not specific. It is a safety check.
Eric: It is a valid claim that Detlev makes.
Maybe we can add a phrase , "if available"
... I too tested a case where I was testing a site of 1 page.
Alistair: If you can test the entire site, then the random sampling can be skipped.
Shadi: A clarification is easy to add.
<Detlev> +1 to Vivienne!!
Vivienne: If I am looking at 25 page web site and we only have a 10 page sample because this is what was allowed by the commissioner. I wouldn't require another 5 page sample. This would be a waste of client's money.
<EricVelleman> The use of the methodology is optional
Vivienne: Recommend a random sample but do not insist it be 5 pages
<Detlev> 5 extra pages is not doable on our typical budget
<Detlev> 10% is fine!
Shadi: If we make this optional, we completely the document. If we remove the 5 pages minimum and say 10%, then you would still need a random sample.
Vivienne: To insist 5 pages is unreasonable. Maybe instead recommend 10%
Shadi: 10% would be a requirement.
<Detlev> 10% OF THE SAMPLE
<kw> sample or out of the total number of pages
Mary Jo: When you get to enterprise sites, 10% is enormous
Shadi: Would be 10% of sample
<Vivienne> I'm totally fine with 10% of the structured sample, I don't want to see us insisting on 5 random.
<EricVelleman> This is quite a change proposed now.. We did not receive comments on this one from the public review
<Detlev> 1+ tio Shadi
Shadi: Remove 5 pages but 10% has always been in
the draft
... 10% requirement already exists. 10% of structured sample
<Sarah_Swierenga> It seems late in the game to make a significant change like this, although I also have a concern about the 5 page/10% requirement.
Shadi: There is no cap. We could have 100s of
pages needing to evaluate.
... Issue is scaling down to the small websites.
<Detlev> Still a significant changer o make WCAG EM workable!!
Current text: The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5 instances of web pages and web page states
<EricVelleman> Ok to take the 5 random pages minimum out it that is improvement of document.
<Detlev> Let's vote to remove the 5 pages requirement
<Vivienne> can we vote on it?
<EricVelleman> Ok for vote, as long as we agree :-)
+1
<Vivienne> +1
<Detlev> 1+
<MaryJo> +1
<EricVelleman> Propose to remove minimum of 5 pages
<EricVelleman> +1
<Sarah_Swierenga> +1
<kw> +1
<martijnhoutepen_> +1
<alistair> +1 ok
Shadi: Voting to remove 5 pages minimum requirement
<EricVelleman> By acclamation! congrats and thanks all for the great work.
Shadi: This is probably our last call. Thank you everyone! I am quite sentimental. : )
<Detlev> Thanks to you and Eric as wel!
<kw> yes thank you!
Thanks Shadi and Eric! Nice work.
<Vivienne> It's been great working with all of you and hope we meet up in another group.
Shadi: Next step. Updated editor draft with changes we discussed by Monday maybe earlier. That will be sent out to WCAG EM and WCAG WG. Please take a quick skim through it. No new comments but ensure that we addressed comments.
Shadi: Hoping for approval of publication of this draft on Tuesday's WCAG WG call.