WCAG 2.0 Evaluation Methodology Task Force Teleconference

25 Apr 2013

See also: IRC log


Shadi, Martijn, Liz, Mike, Eric, Kathy, Peter, Richard, Vivienne
Tim, Moe, Detlev, Alistair, Sarah


<ericvelleman> <http://lists.w3.org/Archives/Public/public-wcag-em-comments/>

current comments

Eric: Welcome. Not a large agenda. We are still working on the comments. We can look at the current state of comments. Not alot of change yet. There are 9 comments
... We can encourage people to submit comments

Peter: Another person from Oracle interested in providing comments, we will get this within a week

Testing Methodology

Eric: may be some issues with the survey. Kathy and Tim are having problems with the results being posted.

Eric: We will try to locate the error. Kathy will send comments to Eric

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/testrun/results

Eric: What has been your general impression for Steps 1-3

Martijn: Took a bit more time for defining the scope in the evaluation. Sample was slightly larger than usual. May be due to the website selected

<ericvelleman> Went through it too. Sometimes hare because you have your won methodology. We need to tighten up the language. This is not well for applications. Ok for websites, but sample also lot larger.. Common page elements adds a lot. Last step replace with .. NO: take them out and not replaceā€¦ three times larger sample than normal or needed. We need more guidance depending on type of review (do you need random sample? Maybe link this to three types of ev[CUT]

<ericvelleman> Says Kathy

Liz: Did not take too much time selecting a random sample. Looked diagram that Vivienne sent. Came up with a smaller sample.
... Guide did not say how much is enough

Kathy; May want to look at the language random sample.

Liz: started with Vivienne's list. Then selected a list of pages based on the different types of items that were different

Vivienne - the document was based on Step 1 and Step 2. It was to find out the types of web pages and exploring the website so you can see the different key pages. Automated scan was run and identified the problem areas

Vivienne - Automated tool gave a starting point for identifying the sample. Looking at sitemap and key pages will help with Step 3

Vivienne - need more language about how to explore the target website

Eric: We have the feedback from some people, we should walk though the questions in the survey

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/testrun/results

Eric: Questions are asking for more clarification
... Some questions are not answered

Martijn: Some questions do not apply to this website. We need to review other sites

Vivienne: Will the survey be open

Eric: Will be open for 2 more weeks. We will open a new survey for site #2
... New website is more of an application

Eric: We may take a different step from the methodology for the other sites
... I will send a link around. We will focus on a different step

Vivienne: Step 2 is where you explore the website and know what technologies they have used, where they come from, landing areas, use cases with key functionality. This is where you have to find out about the site. You need to get the information about the websites
... fair bit that needs to be done in this step

Eric: Is this the output we should have for this step?

Vivienne: I would have put more information there

Vivienne: Add more metrics about the site. The report should have information about the website

Mike: Do we have to save everything all at once or as we go through

Eric; Yes, you can go back. It will be open for 2 weeks

Martijn: Only thing in step 2 we ask to be reported is the types of technologies used. We should have more information in step 2. That way we have a clear picture of what was done
... We should rephrase what we are asking

Richard: First sentence asks to understand what the site is supposed to do is the most important part
... What is that the site does .. the purpose of the site is what we need to define
... Important for the tester but also those who are reading the report

Eric: I don't think this is in Step 5
... we should think about adding.

Richard: It is in Step 2 but you skip to 2a

Vivienne: that is why I did the diagrams to understand the website

Eric: this would be a valuable information for reporting
... also determine the pages

Vivienne: Need to know this to be able to select the pages. To create use cases and complete process you need to know what the website is

Shadi: good point, we need to document what the evaluator sees as the main functions. Do these need to be steps?

<Vivienne> I rather think we have that in Step 2 already, but it could be better explained

Eric: This is not in section 5, we should add it maybe as optional

Shadi: Should this be all in step 5 or as a concrete step in section 2. As Richard pointed out, it is gets lost at the top of step 2

Peter: What purpose of these new steps? How does that clearly help the evaluation? We need be clear on how this helps or supports the purpose of the evaluation

Vivienne: we already have this in Step 2 but evaluator needs to understand the site in order to do select the sample. In particular it is hard to define complete processes

Peter: I don't disagree with the logic. We need to describe how this will help. Maybe hard for those not familar with evaluating websites
... We need to give them enough to be effective

<korn> [And the question is, what is "enough", what is "effective"]

Shadi: Agree with Peter. What do we need for those who are not as experience?

Peter: And the question is, what is "enough", what is "effective"
... what is the reproducability of the methodology? How likely are they to come up with the same results

Shadi: if there is a different sample than the other person, this could be because of what they did during Step 2

<ericvelleman> sorry for not leading this more strict :)

<shadi> Kathy: main thing throughout step 2 and step 3 is clear points without telling people they need to use specific tools

<shadi> ...different ways of achieving these points

<shadi> ...can work with the website owner, or commissioner or whatever

<shadi> ...should ideally be done collaboratively, not by the evaluator alone

<shadi> ...information is very important

<shadi> ...but if we require too much information it may not be practical

<shadi> ...may also interfere with privacy and legal requirements

<shadi> ...it is part of what we should be doing, but maybe addressed differently in the reporting

<shadi> ...not about collecting data because may be sensitive

<shadi> ...also can't dictate the way to do this, there many different approaches

<shadi> ...usually best done with input from the website owners

<ericvelleman> How about adding a step specifically for the evaluator to define the main purpose of the website (for the evaluation)..Could be more of a collaborative step.. Shall we prepare this for a next editor draft?

Eric: propose to prepare the next step for review in the next editor draft

<MartijnHoutepen> +1


<Vivienne> +1

<Liz> +1

<Mike_Elledge> +1 But perhaps as optional item

<shadi> +1

Eric: This finishes the call. We can pick this up on the next call. We will keep the WBS survey open. A new one will be opened for Site #2

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2013-04-26 13:37:31 $