WCAG 2.0 Evaluation Methodology Task Force Teleconference

ev: distribution of comments circulated

<ericvelleman> http://www.w3.org/WAI/ER/conformance/comments

disposition of comments

ev: id5: 1.1, 2.1 - scores better defined - add a note about website parts and clarification of including/excluding parts

also to add a note in 5c a method for aggregating performance scores

Anyone against?

SA: not seeing anythng about exclusion. The case is about different portlets and what we can say about the entire portal. This may need to be addressed in 3.1 or nearby. Disagree with excluding parts - that is a different discussion

EV: will take out the comment about excluding. Definition is a bit unclear. We will add a clarification in 5c on aggregation.

SA: define the performance score first in 5c

EV: add a note about website parts to be explained in 3.1
... clarify the performance scores in 5c and then how to aggregate

Detlev: the issue is that you can single out a part of the site like a library for evaluation, but you can't use the overall university evaluation and exclude library. Regarding the performance score, we can't make any decision on that until we decide on the score function. Putting in a note now could be confusing about putting in information about score when there isn't anything there yet.

We can't say anything about aggregation methods yet.

Richard: Are we talking about aggregate eg Level A or Level AA, we can't have bits being one and bits being another. There is no inbetween, so not sure what we mean about aggregating

<Aurelien> maybe we can ask him some more details on his comment

EV: Francois' comment is about 3 sections, which is making it confusing. You cannot aggregate A, AA, AAA, but we'll come back to that in 5c. We could say if part of a website conform to A and others to AAA, you can't say the website is AAA. You would say the website is A with parts at AAA

SA: agree with Detlev. We need to have the algorithm for scoring in 5C that we provide optional, in addition to the Level A, AA, AAA conformance claim. Before that we can't talk about aggregation till we have pieces. What belongs in 3.1 is the note as Detlev suggests. As to aggregating n the levels basis, we can do that easily by talking about the least common denominator - if one is A and the

others AA, then the overall accessibility is single A - could add a note that parts are better. But the aggregated value is only A.

<Detlev> agree with Shadi

<richard> I agree with Shadi

SA: 5C is required to be looked at separately

<Detlev> fine

EV: in 3.1 we don't have anything about the conformance aggregation - propose then to just clarify the definition of website parts and to make a note to continue discussion on the performance scores later. Is that okay with everyone?

SA: let's talk about clarification of the website parts separately
... propose to add all of this to section 5
... we can say that aggregation on a performance value is the least common denominator. With regard to aggregating the performance score we need a note saying we need to address this later. Not sure where the 2 pieces belong - more in section 5

EV: 5C says option to provide a performance score
... need to decide what you want to score when you then want to re-aggregate - hasn't been described yet

SA: 1 defines a portal - maybe a note saying that 1.a - if you already know the accessibilit of a website part, then you don't need to re-evaluate that, you can consider that outcome along with your entire evaluation
... section 5: need 1 step that talks about the conformance claim itself - 5.8 provide documention for each step and part of that is to provide the actual outcome and in there will be the aggregation on a conformance status.

<richard> The conformance claim for the whole site can only claim the lowest score of any and all parts

SA: there will then be more stuff in 5c

<agarrison> It should be noted that you would only be able to aggregate evaluations which are done within a certain time period

I agree with what Shadi said, along with Alistair's comment - that's good

Aurelien: agree with Shadi.

EV: if you have looked at website parts that you do not need to re-evaluate them as long as they are within a certain time frame and in step 5 add a short proposed text about how to aggregate the parts. Start putting in info about the lowest common denominator eg A

<Aurelien> yes that's it shadi

Detlev: lowest common denominator - if you have a site which has 15 portlets and you evaluated 14 and they were all AA and 1 was single A, if you aggregated it, that would be single A. Is that meaningful?
... you need the detail so you need to work on the one that wasn't AA

SA: agree with Detlev - use cases are difficult - same as when we evaluate a website and it 'nearly meets AA on all occasions' but the conformance reallity is that it is A. The additional documentation allows you to add more information - but we only have the 4 discrete values of conformance defined by WCAG. The report gives more information, but the performance score provides additional

information so that they can clean up the last one

Richard: you need to be really clear about who the performance claim is aimed at - if it's the user then we go with the WCAG - lowest common. If you're looking at the developer's point of view, but it's different - not a conformance claim but a report on the progress towards compliance. Aimed at the owner/developer and not the user.

<shadi> +1 to Kathy

Kathy: agree with Richard. Regarding not having to review a portlet that we'd done in the past within a certain time - but need to add a restriction on whether there was a major change - need to be careful on wording

+1 also

Detlev: benchmark might be - not aggregated score where you try to put the performance on a level - if conformance has not been reached on that level, then how many of the success criteria have been met - eg say 3 failed and the rest passed. You could do that for any scope

EV: report on more than the AA or AAA. This is feasible - maybe add something in section 5

VC: it is good to add information about what has been met for each of the levels - can be encouraging

SA: Canadian Treasury Board's aggregation was circulated. This was simple and quite effective, and there is an on-going counter. In addition to the methodology we have resources to develop an evaluator interface. Similar to "How to Meet..." the idea is to develop this interface.
... we could make this more clear in our document

Alistair: not convinced that we should be looking at AAA, when we are asked to look at AA. If people ask for AA, then I stick with AA and not look for AAA, this is too big an increase in time required.
... you have to add sign language for AAA, that is too expensive for most

EV: currently marked as optional?

Alistair: don't want to encourage people to expect AAA when we put that in

EV: in 3.1c there is a sentence about that it can be useful to do AAA - need to clarify

<Aurelien> no

EV: what do we want to do with the remark by Francois - wants the scores better defined. We need more discussion about that. I can write a new proposal on the mailing list and see if we can agree on a sentence to add for the scores n section 5.c
... all agree that I make a clarification on website parts?

SA: will write on some actions

<shadi> [Suggested actions: (1) provide a section that is more clear about "determining conformance outcome" (Level A, AA, AAA); (2) add a note in there about aggregating such conformance outcomes from individual website parts; (3) add an editor note to section 5.c about aggregation of individual website parts (for future, after we agree on the actual algorithms); (4) add pointers to these concepts early in the document (eg. step 1.a)]

<ericvelleman> +1

<Detlev> fine

<Aurelien> +1

+1 from me

<MartijnHoutepen> +1

<Kathy> +1

Ev: ID 20 - other audiences -

SA: 1.3 says that it is expected that the reader is aware of the resources

EV: could add it just about 1.4 - a short and concise pointing to the business case

SA: seems to be out of scope of the document - should assume they already want to do an evaluation - this is the guidance on how to do it. Maybe reference this later in the document.

Detlev: Peter was getting at the need for business to progress and making that comparable at different stages. We can re-write it later and upgrade it later when we know what we can offer for the business side.

EV: it is in the scoring - for comparison

Detlev: if you do the evaluations 3 times, then you have a score you can compare it yourself

<shadi> http://lists.w3.org/Archives/Public/public-wcag-em-comments/2012Apr/0001.html

SA: Peter is talking about the progress - not only does it meet/not meet, but how is the support for WCAG improving over time. This is the performance score section that is still to be developed. We need to finalise that first and talk about how we can motivate and inform developers even if they have minimal support for WCAG 2. It will become later what we can say at the front. Need a resolution

to come back to it later after 5C has been developed.

EV: ID:21 Add definition of "ancillary functionality" we agreed that as a concept it might be useful. But just hold the thought and not currently add it to the document.

<Detlev> Fine, can be added later

SA: mark the comment as 'closed' - first one!

<agarrison> I don't know whether allowing people to say they meet 59 out of the 61 checkpoints is encouraging them - or allow a status quo to develop. If you only talk about A or AA that might be more encouraging...

EV: ID 27: add partial conformance. It is already inside WCAG 2.0 - could refer to it in WCAG 2.0 - add 3.5b and c

Detlev: can you explain this?

EV: in WCAG 2 there is partial conformance for third party and language - just point to the existence of these

<shadi> http://www.w3.org/TR/WCAG20/#conformance-partial

<shadi> http://www.w3.org/TR/WCAG20/#conformance-partial-lang

EV: the clarification would be good - thoughts?

SA: this could be a longer discussion. There is a bit of a miss-match. Partial conformance does not mean excluding a part from the evalution. Part of the comment seems to say 'exclude the ability to remove part of a website' . It is more about how you write a conformance statement, but it does assume you have done the evaluation. We'll need more in section 5 about the documentation. Removing

part of a website would not allow you to make a statement of partial conformance because you don't know what that missing piece does or does not meet.

Ev: thoughts/

<Aurelien> good for me

+1

<Detlev> Vivienne you are a brillant scribe!

TA!

who was just speaking?

Thanks Shadi. Missed Tim's comment - Tim can you please write in your comment?

EV: it is a clarification that you can't leave something out - partial conformance is meant in a different way
... will see if we can speed it up by doing part of the discussion on the mailing list and put comments and resolutions in that mailing list. All please put your comments down for discussion. We can go through these in the next call.

<Tim> how does partial conformance apply to websites instead of webpages as WCAG defines?

F2F Meeting

<agarrison> CSUN would be better - given the choice

SA: news about Lyon. Survey - we have 13 likely attending, 13 can't likely attend. Some might be able to travel from the US, but also another maybe a CSUN in the US and we would have the opposite figures there. Would still recommend that 13 people would be a good turnout for a sub-group meeting - a good idea still. We could do a lot of ironing out over the course of 2 days. Are there any

objections?

I'm still going to see if I can get there.

Sa: we will still have some dial-in discussion
... any objections to this sub-meeting?

EV: no objections noted - will try to organise it
... other issues?

Alistair: re techniques being optional? Can we put this forward at a later date?

Ev: added ID 47 which was a missing comment in the disposition. So I'll add another one for Alistair's comment.