WCAG 2.0 Evaluation Methodology Task Force Teleconference

New Editor Draft, DoC and survey

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0037.html

https://www.w3.org/2002/09/wbs/48225/evaltfq10/

http://www.w3.org/WAI/ER/methodology/

http://www.w3.org/WAI/ER/conformance/comments-20130226

EV: circulated documents, including new Editor Draft and survey
... comments addressed in batches
... please complete survey by next Tuesday
... still have a number of large issues that need further discussion

<Vivienne> I haven't had time to look at the edited draft or other documents yet

<Sarah_Swierenga> Sorry, I haven't either.... too many fires for me, too.

<Vivienne> I'll set the time aside and make sure that I do it

<Sarah_Swierenga> i'll give it a try for next Tuesday, too

EV: survey links to the corresponding sections in the Editor Draft

Rewrite of section 1.b

http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#step1b

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0035.html

EV: tried to explain in the email
... propose to change the title of the section
... the goal of the evaluation is set by the document
... not really the goal being determined here

PK: maybe "purpose" or "focus" instead?

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130712#audience

PK: assumption is that there are multiple usecases
... don't want to discourage use in other settings
... for example product team during development
... to say the only goal is full conformance evaluation would be a mistakre

SAZ: think the document describes a process for conformance evaluation
... like the terms "purpose" or "focus"

PK: seems we are not sufficiently expressing the dichtomy between absolute conformance
... and levels of conformance or performance

EV: have purposes already listed in the introduction section
... would be nice to get to a minimum requirement that is flexible

SAZ: agree on dichotomy issue, but this is separate
... discussion not tying back into this section

MH: agree with Shadi
... not best title for this section
... purpose and goal should be in abstract or introduction

PK: not commenting on title per se

EV: agree, risk of what is sometimes understood by "conformance"

PK: maybe section explaining what we mean by "conformance"
... as the extent to which, or something like that

<Ryladog> +1

<Liz> +1

<ericvelleman> +1

VC: like the way Peter put this
... asked often by clients if they conform and what they need to fix it

<korn> "Representative Conformance"?

VC: then need to explain that we only evaluate a sample

<Sarah_Swierenga> I agree with Peter and Vivienne on making sure we clearly define 'conformance' - 'representative conformance' would be a term, but maybe appropriate for our document/approach

VC: different people have different understanding of what conformance is
... several other terms that also need explanation
... to avoid legal issues
... we put disclaimer on our evaluations
... to explain this

SAZ: agree with Peter's suggestion
... believe we even have the wording in the document
... but just not pulled out and clearly highlighted

<korn> +1 to the new section.

<Vivienne> +1 from me too

<MartijnHoutepen> +1

<Liz> +1 to add a new section on conformance.

EV: strong support, will add to document
... but different from our initial discussion

PK: on Step 1.b, agree that title does not match the text
... text is about report results
... all about how much detail you want
... think need to be synched

<ericvelleman> So instead of branching out the current section 1.b. we narrow it down to scoping additional wishes by evaluation commissioners that would normally not fall within the minimum requirements for WCAG-EM. This will make the methodology more flexible.

EV: thinking of it as "extra things that a commissioner asks for beyond the minimum of what you would do to evaluate the website"
... think goal is to do a conformance evaluation
... this document shold define the minimum requirements to do so
... evaluation commissioner may want more information than that minimum
... these additions need to be in the scope at first

MH: maybe add this information in the "definition of scope" section?
... could work with the basic report
... and extended based on the request

SAZ: think relates to the depth rather than the breadth
... it is about detail rather than area scope

PK: think may need a new 1.x section
... one that describes scope and detail
... and one about goal

SAZ: how does this impact the remaining steps as opposed to the detail?

PK: in a regression test, I will look at previous sample and add new pages
... that purpose impacts the sampling technique
... for development evaluation, I will prioritize areas of development
... so it influences the scope

SAZ: think have something on "re-testing" in sampling
... wonder if current "Baic Report", "Detailed Report", and "In-Depth Analysis" works?

PK: "Basic Report" to me is what I can give to a senior executive
... "Detailed" contains more about the issues
... and "In-Depth" is very useful for website owners and developers who are new to accessibility
... depends on if you are looking for consultancy on "what is wrong", or on "what is wrong and how to fix it"

VC: think need separate sections for goal and reporting
... also have other extras that we sometimes offer
... for example testing with users if the client wants
... don't offer three levels of reporting but mostly do "In-Depth"
... that is the service we offer
... but like the suggestion of "additions requested by the commissioner"

<Vivienne> I think we need to remember the difference between the purpose of the document and the purpose of the evaluation

SAZ: hearing agreement that title and text are a mismatch

SAZ: also that "level of detail" seems to be one axis
... then also "additional services" like testing with users or providing repair suggestions

Parameters of the sample size

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jul/0028.html

<korn> Eric - I would add to the parameter list any additional requests from the eval commissioner

EV: looking for parameters that influence the sample size
... please send feedback on this

Planning of next meetings

EV: holiday season
... intending to meet next two weeks

[25.7 and 1.8]

scribe: then two weeks break

<Sarah_Swierenga> I will be presenting at HCII next week and then will be on vacation the following week. Good luck in the next couple of meetings!

[8.8 and 15.8]