WCAG 2.0 Evaluation Methodology Task Force Teleconference

Disposition of comments and editor draft

EV: editor draft is in progress, probably before next meeting
... we will discuss this in the next telcon

testrun 2

EV: 4 people responded

<ericvelleman> <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jun/0000.html>

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jun/0000.html

https://www.w3.org/2002/09/wbs/48225/testrun2/results

<ericvelleman> Martijn presents the summary of website#2

<Detlev> Just peeking a bit over IRC today

PK: regular expressions maybe not needed, too formal

MH: agrees

<MoeKraft> Korn: It is confusing to talk about regular expressions. Using this as an example of formalizations.

<korn> We don't need the full glory of regular expressions

<MoeKraft> Korn: Wouldn't say regular expressions aren't needed. Just don't need a whole lot of them.

MK: we don't require them, but suggest re

<MoeKraft> Moe: Regular expressions are not required but formalizations are recommended to identify the web pages in scope

<MoeKraft> Martijn: One thing to mention about Step 1, when we have a big website such as sub-domains you would need more contact with evaluation commissioner to define scope of evaluation.

<MoeKraft> Martijn: Since webiste is so big it is hard to know when to stop. Need input from commissioner.

<MoeKraft> MoeKraft: Martijn: Step 2, one evaluator listed common pages, others listed templates and how used. One evaluator looked at functionality, another looked a visual appearance, other evaluation looked at structure and sections of web pages.

<MoeKraft> MoeKraft: Martijn: not sure if guidance is clear since we have different results

<MoeKraft> Kathy: The comment regarding that the website is too big and need more help from commissioner, maybe we need to have guidance to break site into smaller pieces.

<MoeKraft> Kathy: Maybe have guidance on how to define scope of review.

<MoeKraft> Korn: I think the problem is that exactly the problem we are tackling in the first place, We indicate that you have to sample.

<MoeKraft> MoeKraft: We did test this site and worked closely with the commissioner on defining the scope

<MoeKraft> Kathy: Maybe we need more guidance on communicating with the commissioner for example providing a set of questions to ask.

ack me q- moe

<MoeKraft> Martijn: Sometimes the scope of the website is simple and you do not need to interact with commissioner but sometimes the website is so big that we do need to involve the commissioner and I think providing more guidance on this would be helpful.

<MoeKraft> Martijn: For instance asking questions of the commissioner.

<ericvelleman> Note: Involvement of the website owner and/or website developer can be useful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough evaluation.

<ericvelleman> help

<MoeKraft> Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

<MoeKraft> Eric: This is very general. Is this enough?

<MoeKraft> Eric: Do we need more information here?

<MartijnHoutepen> +1

Should we add a statement that indicates based upon the size and complexity of the website that it would be good to ask questions of the commissioner to define the scope.

<Detlev> How do you meet the requirement for 'objective assessment', given that 4 evaluators arrived at different results?

Richard: No good to have components that are accessible and others that are not. It is important that the commissioner define what the elements are that belong in the scope of the test.
... Concerned that we are getting away from our guidance to look at all components.

Kathy: Not clear when to start and stop. Good idea to have questions to come up with representative example of all page.s
... There is a lot of information we can get from the commissioner before evaluating a site

Eric: This is used regularly in our testing in the Netherlands when to start and stop
... Need help from the commissioner

Just to respond to Richard is that we can get pulled into many different websites both owned by the same organization and not and need guidance from commissioner as to what belongs to support website.

Korn: It is entirely appropriate for team to scope what pages it will look at in a given moment in time.

Martijn: Common functionality. Section 2b. Responses are pretty much the same from all respondents.
... Step 2b
... Step 2c Variety of web pages. One evaluator referred back to Step 2a where pages were already mentioned. Not sure if this means we don't really need 2c. Other responders described templates used.
... Step 2d is found to be confusing
... Technologies Used

<Detlev> Richard and I seem to agree that it would be the *custom* techniques that need to be listed

Moe: When looking at common functionality want to be careful to not test functionality but to test the access to the functionality

Richard: Be careful not to call Common Functionality but instead Important Functionality
... Receiving many comments on this section from the public comments.

<MartijnHoutepen> in the disposition of comments

Eric: Is there an alternative proposal in the results page?

Richard: No. In the disposition of comments.

<ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226

<MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/comments-20130226#comment3

http://www.w3.org/WAI/ER/conformance/comments-20130226#section1cont

Look at ID 3

Eric: Concerned that people don't see Section 2 as the Exploration of the website.
... The third step should product the sample.
... Maybe not clear for everyone.
... Let's just flag this one.

<richard> +1

<ericvelleman> Eric: Look into results of step 2. Do we have to put down everything there? flag this for review if we want to put everything there. Should it not be basis for sample?

Martijn: Maybe partly because in the results of the questionnaire we were asked what everyone did maybe we didn't have to note the list but this was asked of the questionnaire.

Moe: Step 2a states "During this step the common web pages of the website are identified and documented."

Martijn: Section 3: Selecting a sample. Two people gave URIs another person selected all pages. Last evaluator selected a small sample and guidance how to get to pages.
... People who gave URIs gave them to pages identified in sample 2. Comment regarding small sample, describes pages but no URIs.

Eric: Do we have an idea that these samples would miss things? Or would they cover the same things?

Martijn: Different scope produced different samples.
... Detlev had a much more in depthy scope and produced different results.

<Detlev> My hunch is that we would uncover many of the non-conformant things regardless of the exact shape of the sample...

Martijn: May need to be more explicit.

Korn: Peter is looking at answers. A couple answers give a lot of pause. 1 evaluator is expecting to test all pages of web site.

Richard: Self contained element should be tested.

Korn: "All pages in the application will be evaluated. "

Richard: Scope was already defined in Step 1. So all pages in application were already defined.
... Portal gave access to all support website. Did not identify all of the web site. Went by scope indicated by URL.
... Only checked the portal.
... 9 pages

Eric: Home, Support, Downloads, etc. Tabs at top of pages

Korn: Find the product and other links on this page. Were these in scope or out of scope of the test?

Richard: Looked at horizontal menu, 7 tabs, and opened each tab, and followed links that went to portal. Downloads and Troubleshoot went to external areas.
... Essentially looked at application as a portal.

Martijn: I agree with Richard. It is a portal. I did the same as Richard but also included some more static pages and inclued some functionality from main page, signing in, look for products.

Eric: Interesting to see the differences in sample.

Moe: This brings us back to our first discussion regarding communicating to the evaluation commissioner to define the scope.

Eric: Looking for volunteer to write a summary of results for website 3.
... Will search for a volunteer outside this telco.