WCAG 2.0 Evaluation Methodology Task Force Teleconference

EV: has not been able to get latest editor draft ready yet
... disposition of comments
... last week's use case study and also question in the disposition of comments on how methdology decide which states to consider
... also missing consistency with the diagram - naming steps / stage
... status of use case revision - section 1(b)

SA: have 4 use cases according to thread - discussion has stopped - what is required - should it go out to review again?

EV: seems to have skipped over this section and its comments
... also detailed overview of reports which is to be replaced with the use cases

SA: tie in with remaining sections is important
... primary thing to address is how this all relates throughout the document - that isn't very well worked out yet
... in the end in the report, there is no tie-in with the other sections

EV: shall we work offline and get a proposal to the task force

SA: yes, that's the suggestion

EV: zakim, take up agendum 3

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611#specialcases

EV: types of websites
... 'common stages' of the websites - 3 times
... generated states of the web applications can really differ - what does this mean - what more significant place can we give it?

Shadi: did you want to take over scribing?

EV: easiest way would be to add it to one of the last sections - 'considerations for particular situations' at the end of the document
... this is right at the end - Step 5.d
... it comes after 5.d

SA: what are the comments that we're trying to address?

EV: the comment is that the methodology doesn't help you decide which state to consider

SA: talking more about the sampling aspect and how pages/applications and states should be addressed
... we're talking about the states of pages and how to select them

EV: 2c identify the variety of web pages, then 3b include exemplar as identified in 2c

PK: we're really talking about having a web application that without changing the url can do different things - how do we track that and where in the steps do we do that. Is it more important for a longer discourse about the use of testing software and how you record the steps you took in the software to get to various places. It makes me start to think that stages is a better term than steps

- that you took through the ap to record various paths - may be a new body of text

VC: we could also put it in the complete processes as each step needs to be recorded

EV: You could get that when the user's profile is known and they get dynamically generated pages according to that preference

Richard: government sites offer different processes according to your needs - you get a different set of pages
... with a large set of pages you could then sample - including a random sample. You can apply the same idea to these multi-state sets of pages - we need to know the range of profiles and then select a random sample. We'd need to talk to the web developer/owner and ask about those profiles - which are your priority targets etc. We need to talk to the owner and then once you've got the

range you can use your standard random sampling technicque.

EV: then you do it in the scoping of the website and testing - getting the range of profiles

Richard: it would be one of the loop-backs that you discover when scoping the analysis
... you'd identify that they are using profiles and you would contact the web owner to clarify the list of profiles

SA: recall that we had a section on web applications that we talked about the different states of an application and when it is considered to be a web page. That might have got lost in the beginning. It was also focused on types of websites that are called applications and doesn't really cover the different deliveries of content based on who you are. Do we need to spell that out within the

sections themselves. "Also make sure that you identify how the web pages are generated and try to find with the help of the developer the range of different pages that can be generated, acknowledging that this might not be exhaustive". You can see how the web pages are being generated, being put together." in some cases it might be part of a complete process or part of an individual state.

SA: we have the ideas, but it's getting lost because it's in the very beginning because we don't explain it too well how to do it

EV: yes, that sounds good - it has be an integral part of what we're doing

Richard: it's actually covered in the very first part, agree with Shadi

EV: propose that we start with what's already there and which sections should have some information added. Shadi can we prepare this together?

SA: quite a bit of editing to do throughout - can work on that

EV: maybe not in the next edited draft - we need to see what has changed to get it into the document

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611#procedure

EV: next agenda point - section performance evaluation procedure - the diagram
... In thei intorduction to the conformance evaluation procedure - intro. top part says 'stages' and the lower part says 'steps'. Should 1a,b,c, be steps or stages?

PK: like the idea of renaming the stages - wonder about reserving the idea of 'step' for 'steps through a web application' but that might be going a bit far. The stages are significantly different from each other and I'm happy calling them stages.

EV: in the diagram - step 1 - change to stage 1,2,3?

PK: yes

<shadi> +1 to Sarah's comment

SS: like the way that it is now in terms of steps - clearer from the unfamiliar user's perspection - the first step is this. Stages implies they are independent. That way the numbering can stay with step 1,2

Richard: keep it as steps - it is a user process

<richard> steps +1

<Liz> steps +1

<MoeKraft> +1 steps

I'm okay with steps, but I'm not concerned too much

EV: in the survey I'll put this in to see if everyone agrees
... you saw a difference in stage and steps, and you wanted to reserve steps?

PK: use steps through an application - record the 5 steps I took to get to this screen where I had a problem etc. When I think about the amount of work in these 5 steps, I like to call them stages. But whether it is stages or steps, it is more a question of having a variety of nouns for the different things to stop it becoming confusing. I personally like stage better for this, but it is not

a significant issue. More significant that we use a different word for different things.

SA: the issue is that we're trying to fix is that we're using the word steps too much. Keep them in the title as that's easy to communicate. We can say using the technique 4.1... and not use the word 'step' within the context of the document itself.

EV: we agreed that we can keep steps as it keeps it clear and understandable. In this text we use 'stage' and 'steps' - should we drop the word 'stage'?
... if you go through a web application you follow a route to get to a point or screen - what do you call this if we already use 'steps' for this, we cannot use 'steps' for the things you do to get through a web application.

What about path?

<shadi> +1 to path (+ parameters?)

You can have the points on a path

SA: for the sections of this document - we should keep it to steps. Do we use 'steps' of the document and also 'steps' of the application where it is confusing?
... otherwise it might be okay to say the 'steps that you took to get to this page'
... depends on how we write it and don't use it in an ambiguous way

<shadi> +1 to Liz

Liz: I had the feeling that step is an action - it is what you should do and stage is an environment. In the diagram you are talking about the proceduring - they are action, exploring. After you've done that you set the stage for the next step. They are different, not synonyms

EV: it is important to keep this in the back of our minds, have to do more to describe the ways you found the pages - the way that you go to a destination - depends on the decisions you make. Maybe the path, or the stpes on that path
... if they see the word step -do they get confused
... we can just make sure we describe it clearly

PK: it's a small issue compared to some of those we still have to deal with. We haven't figured out what a good statistical sampling method is and how you ensure you have sampled enough and how you have described the stages through a web app. This is small compared to those bigger issues

EV: agree
... regarding statistical sampling we've got some things from Giorgio that I'm putting into the dissolution of comments
... we haven't described the states far enough - even in the reporting section. It currently looks like you don't have to report how you got somewhere. We need to add more sections to make this clearer.
... other issues. Anyone have anything to add or discuss?

SA: yes, can we pick up on the statistical sampling?

EV: we have added comments from Giorgio and have tried to add it into the comments and we still have to decide on

SA: I think coming up with a number of statistical will be close to impossible - depends on type of website etc and lots of parameters. Depends on how many authors, tools used etc. If we use a different approach - Detlev commented on this. Relates to the purpose of the evaluation.
... after a couple of web pages you usually know if it's compliant or not. If the first few are already so few of errors it might already answer the question the evaluation is trying to address.
... but if we say that there is a website that we don't know how it is going to fare and you're selecting and evaluating pages and continuing to find errors. How repeated are those errors - you'd expect the number of types of errors to level off sometime. If you go to someone and they ask how much it will cost, you need to be able to say how many pages. You should be able to come up with a

number of pages that should give you a reliable indication of the status of the website.

SA: the process where you have a selection of validated pages that is a cross-section with randomly generated pages. The evaluator will be making those decisions - they know when they have seen enough. It needs to be clear in the reporting.

PK: purpose if very important, but we don't have different types of reports in our document based on the different purposes. If you're going to report on the basis of imperfection it's easy to stop on the first imperfection. If you're trying to get an accurate overall sense you need to get a better sense of the homgeneity of heterogeneity of the site.
... sometimes the pages look visually similar, you notice that there are differences that aren't apparent visually. There is something to be said for some analysis of the heterogeneity of your sample. If in a random sample you see that the pages are coded in the same way, you have confidence to say that the pages are similar. If there is more heterogeneity, you need to look further to see how

much deviation there is the sample types.

PK: Statistically this may be very important

SA: Peter has a good point. Calculating the deviation may be important, but the difficult thing is getting the parameters. What constitutes hetergeneity?

EV: this is probably the most important question about the sampling
... one of the questions could be the homogeneity or hetergeneity of the website. The other is the purpose of the evaluation - acceptance, compliance. Lots of parameters to consider - perhaps we need to make a list.

<Sarah_Swierenga> have a good week. i need to head to another meeting.....

PK: also points out the important of the what we're calling the report. It is critical we review the use of the word 'conformance'. Are you just checking the pulse to see if there is anything major in the acceptance test you don't know if it conforms to WCAG. We need to be careful in what we call the output of this - don't think that 'conformance' is the right word.

EV: these parameters are important to get in a list and see what we have to work on and what is important or not.
... let's start this on the list this week and get some short explanations there
... hope to have editors' draft and survey by the weekend if possible. That's the homework for next week.