WCAG 2.0 Evaluation Methodology Task Force Teleconference

eric: describing the changes in the draft. Added possible action items to each item.

detlev: comments from wcag worked into document

<ssirois> http://www.w3.org/WAI/ER/conformance/ED-methodology-20111207.html

detlev: proposed changes by Michael Cooper - scope of document. Also changed appendicies to make doc more in line with W3C

<vivienne> yes, I read through it - looking good

<Kathy> yes, I read it

<Detlev> yes

<houtepen> yes

<Detlev> http://www.w3.org/WAI/ER/conformance/ED-methodology-20111207

<Detlev> zakim unmute me

detlev; question about 3rd party sites and conformance claims

eric: good solution by allister

allister: summarized concepts - as an example; eric will add to doc

eric: changed numbers in section 3, questioning where the definitions should go

mike: key functionalities phrase might be changed to important functionaities. concerned that it might be interpreted as keyboard keys

allister: maybe we should revise doc before going through this

detlev: large online retailer discussion - difficult to evaluate 100s of widgets with wcag. too many to evaluate for conformance claims.

eric: asks for an example page, but project is not public

<Nethermind> I think this is valuable, because a lot of design is moving towards a modular consumption method (social media, 3rd party content, aggregated data, e-Commerce, etc)

detlev: consider shopping functions in a large retailing website. is it possible to have an eval procedure for one form/widget

<agarrison> From WCAG 2.0 - Conformance (and conformance level) is for full Web page(s) only

<Mike_Elledge> +1

eric: will place in document as an editor note

kathy: often finds with large retailer and large acadmeic institution, defining use cases and user stories is often done by QA. User stories often based on what developer groups are working on.
... developer groups responsible for different parts of the page, so getting feedback back to the correct team is important to consider in the eval process

eric: will also add this as an editor note - supporting developer teams in their work based on use cases and user stories

<Mike_Elledge> agree; scripts can be a useful method for defining what to review

<Detlev> Agree

AmyChen: likes user stories and use cases approach. we define business processes (and then find out which development team owns specific parts), but then evaluation should be on the whole process, e.g. the whole shopping process.
... probably don't need to make too big of a deal about supporting evaluations of parts of processes.

eric: agrees with amychen

amychen: could do conformance by team, but the end report needs to be about whole process

eric: we strive to document the end result of the eval, not who did what

martyn: could fall into 3.8

allistair: wants to see more harmonization with what's been written in conformance section

<Detlev> I was just raising the issue because it is a frequent use case

leonie: good point about harmonizing with conformance from wcag. need to consider how parts of organization will deal with conformance before the whole product is put together.

eric: looking at procedure to express the scope, agrees with allistair

mike: mention in doc that when starting on a project, the dev groups could decide who's doing what to make the intermediate evals easier.

kathy: developer groups are internal, but user stories are task-based, which are useful for establishing conformance claim evals

<Nethermind> +1 agreed, many user stories are more discrete than a complete process

kathy: could still be under 3.4, could also fall under 3.8 because each user story/task could be evaluated separately. Could have multiple user stories for the same shopping process, various user views.

eric: maybe kathy could figure out where to add this idea

elle: asking about excluding specific pages from the scope, e.g., secure pages

eric: 3.6

elle: this might be misunderstood as only requiring non-secure pages for conformance reviews

eric: logins for intranet - this could be separate from the scope of the eval. these separations need to be clarified in this doc

elle: we want to make sure secure pages are clearly in scope

<Nethermind> -q

allistair: scope - say someone has done all of the work to set up for the conformance evaluation, but then finds out that more parts need to be included. (referring to an email) Website owners would want to know what is minimally inside the evaluation scope.
... why must they then go have a whole website evaluated? Proposes that we use the website owners conformance claim as the claim.

<Detlev> I'd lbe happy to adress Alistair's point directly

<AmyChen> me too

amychen: allistair's point - top of document should indicate what parts are claimed for conformance, if it's not the whole site.
... kathy's question - how to fit user stories and tasks into complete process. Example would be software for expense report, but user profile might be employee vs manager, but the product we sell is just expense reports. wants to propose that the complete process/product named in the conformance claim is written at the top, and then we could indicate the user profiles used.

eric: we are looking at 2 sections at the same time, but where can we express the scope of the evaluation?

<agarrison> For clarity - my question was "should the methodology evaluate a website owners own conformance claim (so the conformance claim becomes the scope) or should we say this is what we want people to evaluate and this is the conformance claim they can then make (disregarding their own claim)"

amychen: maybe separate scope from complete process. doc should then indicate whether it includes the complete process or not.

detlev: we as evaluators should make sure that we know whether the scope of the conformance refers to a few bits or the whole site. need to make sure that the important processes are included in the claim.

leonie: maybe we should be more clear in section 2 who is responsible for the evaluation of conformance for the overall process.

eric: this methodology is meant for the whole website, not just parts or specific steps

mike: amychen had a good idea, but if we are talking about an eval done on a finished site, we need to say that very clearly. Seems like a lot of this would be important for reviewing specific parts of a site. Thirdly, we don't want to exclude people who want to evaluate parts of sites, rather than the whole site.

<LeonieW> +1

eric: methodology would have to do very different things if it were for a full site vs unfinished parts of a site.

mike: would it be possible to have a doc for partial reviews?

<Detlev> the same nethodology should suppoort both partial reviews (nor conformance-oriented) and final reviews

eric: will make a note of it, but it would be a completely different methodology.
... will open a discussion on a few of the items, e.g., 3.4 and other parts of section 3. will also look at the proposal that allistair made.

<Tim> need to also address time dependencies in evaluations at some point..

shadi: how development process/organization impacts the evaluation process. we are concerned with how it all comes together.

shadi: face-to-face meetings conferences, e.g. CSUN, to see if we had enough people to have a face-to-face meeting.