WCAG 2.0 Evaluation Methodology Task Force Teleconference

08 May 2014

See also: IRC log


Mike, Kathy, Vivienne, Martijn, Eric, Detlev, Liz, Tim
Shadi, Alistair, Moe, Gavin, Sarah



Welcome everyone. No call last week, only 2 people there. Three surveys out. Eval questionaire #14, 15, 16.

<martijnhoutepen> #2. Proposed rewrite of Step 1.d (evaluation methods)

<martijnhoutepen> Link: https://www.w3.org/2002/09/wbs/48225/evaltfq15/

Idea to keep open until next Tuesday, then discuss next meeting. Can discuss 5D today.

#15 rewrite on 1D, evaluation methods to be used. Change current title or rewrite the section.

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/evaltfq15/results

Proposed rewrite of Step 1.d (evaluation methods)

<Vivienne> Sorry I haven't been too active with travelling, will get the surveys done this week

Results: Seven answers, so far. Can try to discuss on here.

First change to title. Two--accepted change 4--keep 1: do not accept for following reasons. Don't understand what we are asking evaluator to say?

Detlev: Like the new title. don't need to enumerate all procedures using, concentrate on techniques for testing particular areas that owner has new method
... Change the title to "Define Evaluation Procedures for particulal success criteria.

EV: Like two reasons you gave. Retweak the re-write for comment.
... Comments?
... Think it would be more clear.

<Vivienne> +1

<martijnhoutepen> +1

K: Fine.

<Liz> change to new title

L: Fine.

M: Fine.

<ericvelleman> +1

EV: Will change title to Detlev's suggestion.

Det: Make them go well with text. Keep it short.

EV: Important to add D's two points. Will add to the text.

D: Easier to draft those things when they're in the text.

Ev: Add to comments field or list?

<Vivienne> +1


<ericvelleman> +1

<Liz> +1

K; Yep

<martijnhoutepen> +1 for adding two points from detlevs comment

D: ? Would object. Need for more enumeration. Site owner has to describe what is needed.

<Vivienne> Mike, I think Detlev said Alistair would object

EV: LIke Richard's comment. That evaluator might have own way fo doing things.

M: Thanks Vivienne.

EV: Will send around for further comment.
... Can you change your decision?

<Detlev> I think "Describe Evaluation Procedures for particular Success Criteria" is better vs. "Define Evaluation Procedures for particular Success Criteria"

<Detlev> Richards suggestion

EV: Would be nice, not have to do another survey.
... Anything else on this one?

Proposed rewrite of Step 5.d

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/evaltfq16

EV: Rewrite of 1C.

rewrite of 1c

Six answers so far. Really difficult one.

Two questions: Current text and Guidance for baseline.

Ev: Current text: 1 Add qeustions: 4

D: Discussing second question.

EV: Stick this first. Survey #16 first.
... Can come back to #15.

D: Pointless to list all adaptive technologies, browsers, etc. "When a particular choice of user agents and assistive technologies is specified by the site owner to achieve conformance and these technologies are known to the evaluator, they should be listed here."

Same as what Richard said.

EV: Like D's proposal. See this as restricted lsit.

D: Similar to what Michael said. Make it clear that you do this step when reviewing a particular site that is tuned to specific browsers and tech.
... Main thiing to avoid confusion that don't have to list every conceivable combination of browsers and tech.

EV: Good Change?

<Liz> Yes

<ericvelleman> +1

<Vivienne> yes, the change is good - I just recorded my answers for this question

<martijnhoutepen> +1

K: Feeling a little lost. Have to look at closer.


EV: Will come up with rewrite with this proposal.
... Main thing is that we discussed a list of questions. Conclude that instead of list cover it with this change?



<Detlev> +1

<Vivienne> +1

<Liz> +1

<ericvelleman> +1

<martijnhoutepen> +1

<Tim> +1

Never too quick!

Ev: BAck to 15. Second question.

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/evaltfq15/results

Rewrite of 1d.

EV: will rewrite and add D's two points.
... Do not accept by Richard. I am not happy with "This step is optional because WCAG 2.0 conformance can be determined without the use of documented evaluation procedures". I think that, when reporting an evaluation, it is important to explain how it was done in order to give the recipient confidence that the work was carried out properly. I can live with this step being optional but do not like the implication that it is unecessary.
... Don't want to make it seem as unnecessary.
... D: Talks about other techniques but not custom. Just a technicality.

D: Pointless to list all adaptive technologies, browsers, etc. "When a particular choice of user agents and assistive technologies is specified by the site owner to achieve conformance and these technologies are known to the evaluator, they should be listed here."
... A good practice to refer to documented procedures.

EV: would look into the text. Would add a lot of bulk.

D: Not really clear what is meant.

EV: Trying to make clearer. Not sure what we meant.
... Now looks like have to do all the work all the time.

D: Similar to earlier question where specify particular circumstances.
... 1D
... Maybe can be merged.

EV: Will come up with another text proposal.
... and send around.
... Covered 1 D and 1C.

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/evaltfq17/results

EV: Also on later section: scoring. #16--5D
... Lots of previous discussion. One question always comes back: no one agrees on it. No widely used metric, lots of disclaimers.
... One outcome, let's look at metrics document and see if we can come up with a few examples, with same disclaimers.
... Problem is looks like we're still proposing scoring. My suggestion is to keep the first paragraph and delete teh rest.
... Seven answered: 6 accept. 1 accept with changes. Vivian says does not meet need for metric, but okay for now.
... Frederick suggests that research is still going on, not yet closed.


V: Happy with F's suggestion about adding note.
... As organization we're really looking for help with this. But it's a controversial subject. Can add something that there is ongoing research.


D: Two minds. Try to keep methodolgy short. If we don't have advice will use whatever they use. But don't mind that.

F: I think it needs to be mentioned. Can be a note. People are looking for it.
... should be indicated on document that it's being addressed.

<Liz> +1

M: Agree. Keep link to page with metrics document.

<ericvelleman> +1

<Vivienne> +1

<martijnhoutepen> +1

EV: Add ongoing research and keep link (it is previously in document).
... will lead to changes to current survey. Good work today.
... Surveys open until Tuesday, so still have opportunity to respond. Will make changes to a few, another rewrite and propose again. For next week would like to try to make editor draft of new document.
... Start with re-writes we've agreed to. Will start with latest batch and mark for review. Hope to have by Tuesday. If you have oppty on Wed or Thur have a look. Will keep it open. Work on this and next week.
... Will mark them all. So you can comment on them during telco.

<Vivienne> that's fine thanks

<Detlev> fine

<martijnhoutepen> +1

EV: Okay?

<Liz> +1


<Tim> +1

<Detlev> -q

EV: Results of test run survey...answered by a number of people.. but anything to say Martijn

M: Yes. Interesting feedback. Tried to put on mailing list. Baseline very different.
... With coming re-writes may get to them. People found guidance easy, comments positive. We like that. Some confusion about exploring the website. Order in which you explore different from pages you found for sample. Confusing.
... Also, minor text issues. Processes have different understanding of: urls, explain processes a bit more...some ahve user processes, some have website processes. What users can do vs. what can be done on website.
... Comments on that also. Can define more clearly.

EV: Step 2 order in disposition of comments already. People generally enthusiastic about methodology. Not everyone finished entire questionnaire. Not everyone had enough time. Maybe something needs to be done to ensure confidence in using our method.

M: Not in dispositioin of comments since they didn't get to end of questionnaire.

EV: Will do a mapping of the comments in survey to disposition of comments. Will cover every comment from public and test run.

M: Around 10 gave results we can use, five finished questionnaire.

EV: Any comments? Other than hundreds we expected.

ME: Any surprises?

M: People found easy to use. No, big surprises. Were some big differences in how much time they took to make sample. But overall people generally agreed with it. Results as expected to me.

EV: Sounds good.
... Leads us to end of telco. Anything else?

K: Can you send me copies of questions?

EV: Yes.
... Have good days, nights, etc. Please find time Wednesday and Thursday to review revisions.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2014/05/14 17:44:15 $