WCAG 2.0 Evaluation Methodology Task Force Teleconference

23 May 2013

See also: IRC log


Vivienne, Moe, Martijn, Liz, Richard, Eric, Kathy, Peter, Tim
Shadi, Kostas, Detlev, Sarah, Alistair



Eric: Welcomes everyone to the teleconference. Shadi is in Brussels presenting on W3C. He will not be able to join the call.

<ericvelleman> <http://lists.w3.org/Archives/Public/public-wcag-em-comments/>

Eric: No new comments received
... Just finished disposition of comments
... Shadi will put this online probably over the weekend
... Will get more information from agenda item 3 regarding surveys
... Ask folks to work as much as they can on the Game accessibility website and Richard has evaluated the results.

Website #1: <https://www.w3.org/2002/09/wbs/48225/testrun/results>

Eric: Please take us through the results

<ericvelleman> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0023.html

Richard: Only 4 people gave feedback in results. Small set.
... Need to look at adding to step 1, please confirm what you have decided as your scope is what the client wants.
... Can state Step 1 is fairly okay

Eric: Anyone else have feedback on conclusion to step 1?
... Did receive public feedback on Step 1

Richard: Step 2 we had a very small sample and wonder why. Was Step 2 too difficult? It is hard to analyze. Why did people not complete all 4 steps?

Martijn: I took the guidance for reporting from Step 5
... In second questionairre added more information for step 2

Vivienne: Was unable to finish steps due to travel.

Eric: Thank you for being on the call after all that travel.

Vivienne: Chatted about methodolgy at conference and has feedback to share.

Liz: Did not finish Step #2 because she saw a very comprehensive diagram of site in a picturesque way that she liked. She did not understand how to answer the question. Likes how Vivienne has put together diagram.
... Not sure what to say.

Richard: Need result of exploration on questionnaire so we can figure out if we had similar results or differing. Did we understand methodology and step?
... Also identify technologies used.
... Basically get results of what was done.

Eric: Need to explain clarly in the survey text what is expected.

Richard: Step 3 was to select a representative sample.
... 2 out of 3 tested entire site while it was a small site.
... 1 person did an analysis with selected pages using the PayPal section which should have been excluded.
... Need to look clearly at the guidelines for Step 3. How do you decide if the site is so small that it is not worth sampling.
... How do decide to sample or not sample?

Eric: Obvious on small sites we do not have to go through sampling

Korn: This is one of the most challenging parts of our work. How do you come up with a sampling methodology that is going to yield similar results across different folks performing test.
... There are probably tecniques at universities or elsewhere on doing this type of sampling work.

<Vivienne> https://staffmail.ecu.edu.au/owa/attachment.ashx?attach=1&id=RgAAAABI7j1wgS3dSaYJ6jJKs4DjBwCK%2bnd0GxHbR7JBMfHjgiepAG7AfT4PAACK%2bnd0GxHbR7JBMfHjgiepASPU4MxzAAAJ&attid0=EAAAycCB1a9PTolHYVBhMIRT&attcnt=1

Korn: How do we have confidence that we will have similar results person to person?

<Vivienne> Link above to to Giorgio Brajnik's paper on sampling

<Vivienne> paper is entitled "Effects of Sampling Methods on Web Accessibility Evaluations"

Kathy: Agree with Peter that this is the most difficult part. What is interesting is that folks chose full site. Issues on representative sample pages includes all different types of elements in a website.
... Also just as important to document how this gets applied to pages not reviewed.
... Definitely a bigger challenge if we have a representative sample client is not sure how this applies to all the different pages on the site.
... Need to provide information on how the representative sample applies to rest of site.

Eric: This is where I see alot of comments in the disposition of comments on how do we guide people on how to define the sample.
... We need a much better description on how to define sample in such a way to give us a level of confidence that we can get to the same end results.
... Is there statistical work that can help us?

Kathy: Only other comment is that we often in a real world situation there is a lot of going back and forth with business owner. It is important to get a demo of the site or application. It is knowing how the application has been developed or grown overtime.
... e.g. Website/application that has grown overtime where no code was reused. Couldn't look at representative sample because of the diversity of the web site. One table might differ from another because they are not coded in the same manner.

Eric: Good point. We hadn't thought of different developers developing tables in different ways.

<Kathy> what is the username/password for the link

Vivienne: Has link for paper from Georgio. He has done evaluation of a website with 32,000 pages. May be able to refer to this paper in which he has collected a lot of data. I can do a synopsis of the paper.

<MartijnHoutepen> Vivienne, the link is pointing to your e-mail, so we can't access it

Eric: Best to discuss with Shadi and ask Georgio to provide comments to the methodology.

<Vivienne> okay, will try to get a different link

Korn: In talking about number of developers and difference in coding style, in more conformance to coding style the better confidence you have in the sample. If the coding style is more heterogenous then need more sampling.
... Could have survey and do a code inspection.

<Vivienne> link to that paper: http://users.dimi.uniud.it/~giorgio.brajnik/papers/sampling-assets.pdf

Korn: Use automated framework and tools. If every tab pane is same component, then you don't need to examine every menu. If you find menu components are heterogenous need a larger sample and check each more carefully.

<MartijnHoutepen> Thanks vivienne!

Korn: May be good to build this into methodology as well.

Kathy: Difficult to do when a consultant because looking at generated DOM.

Korn: Could still look at generate DOM
... Will be differences that are telling

Kathy: Only have so much time to generate sample. Sampling can take an enormous amount of time.

Korn: Should be looking at how to conserve time by creating the sample. If we cannot assure ourselves given the time allotted then need to say that.
... Need to indicate in the report our level of confidence based upon the time alotted

Kathy: Much more efficient if you work with website owner to determine the sample

Korn: Not opposed to relationship with web site owner but would also like to suggest techniques that can flush out the heterogenous coding style.

Kathy: Agrees this should be a collaborative effort.

Eric: Agrees this would be a good addition to the methodology. Will make a first phase at adding this guidance and will send out to the mailing list.

Richard: Before we leave Step #3, When you decide not to do a sample but instead test the whole site, it might actually save you time to test the whole site.
... Once you've analyzed related pages, you know that you can then skip additional pages.
... Need to explain the decision that has to be made at the start of Step 3

Eric: Are there other questions?

Richard: The general theme was that it was generally clear. May not be perfect yet but we are on the right track.
... Need to remember that our target audience is not just us but a much wide audience and audience that has less technical background.
... We are very knowledgeable in the area of website accessibility.
... Let's get everybody trying at least one test so we can get a better picture.
... Does this produce plausible results? Not a big enough sample yet. Cannot draw any additional conclusions.

Eric: Richard has a proposal for additional questions. Please send to mailing list.
... This takes us through the summary of website #1. Richard, thank you very much!
... Any other comments regarding summary?
... Everyone work on website #2. We will pick up comments to that the week after next.
... Looking for a volunteer to do a summary of the results in the survey for website #2
... this would be due on Thursday, 6/6
... Any other issues?

<Vivienne> bye all

Eric: If not, have a beautiful holiday. Speak to everyone next Thursday.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2013/06/09 11:55:57 $