See also: IRC log
Eric: Welcomes everyone to the teleconference. Shadi is in Brussels presenting on W3C. He will not be able to join the call.
<ericvelleman> <http://lists.w3.org/Archives/Public/public-wcag-em-comments/>
Eric: No new comments received
... Just finished disposition of comments
... Shadi will put this online probably over the weekend
... Will get more information from agenda item 3 regarding surveys
... Ask folks to work as much as they can on the Game accessibility website
and Richard has evaluated the results.
Website #1: <https://www.w3.org/2002/09/wbs/48225/testrun/results>
Eric: Please take us through the results
<ericvelleman> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0023.html
Richard: Only 4 people gave feedback in results.
Small set.
... Need to look at adding to step 1, please confirm what you have decided as
your scope is what the client wants.
... Can state Step 1 is fairly okay
Eric: Anyone else have feedback on conclusion to
step 1?
... Did receive public feedback on Step 1
Richard: Step 2 we had a very small sample and wonder why. Was Step 2 too difficult? It is hard to analyze. Why did people not complete all 4 steps?
Martijn: I took the guidance for reporting from
Step 5
... In second questionairre added more information for step 2
Vivienne: Was unable to finish steps due to travel.
Eric: Thank you for being on the call after all that travel.
Vivienne: Chatted about methodolgy at conference and has feedback to share.
Liz: Did not finish Step #2 because she saw a
very comprehensive diagram of site in a picturesque way that she liked. She did
not understand how to answer the question. Likes how Vivienne has put together
diagram.
... Not sure what to say.
Richard: Need result of exploration on
questionnaire so we can figure out if we had similar results or differing. Did
we understand methodology and step?
... Also identify technologies used.
... Basically get results of what was done.
Eric: Need to explain clarly in the survey text what is expected.
Richard: Step 3 was to select a representative
sample.
... 2 out of 3 tested entire site while it was a small site.
... 1 person did an analysis with selected pages using the PayPal section
which should have been excluded.
... Need to look clearly at the guidelines for Step 3. How do you decide if
the site is so small that it is not worth sampling.
... How do decide to sample or not sample?
Eric: Obvious on small sites we do not have to go through sampling
Korn: This is one of the most challenging parts
of our work. How do you come up with a sampling methodology that is going to
yield similar results across different folks performing test.
... There are probably tecniques at universities or elsewhere on doing this
type of sampling work.
Korn: How do we have confidence that we will have similar results person to person?
<Vivienne> Link above to to Giorgio Brajnik's paper on sampling
<Vivienne> paper is entitled "Effects of Sampling Methods on Web Accessibility Evaluations"
Kathy: Agree with Peter that this is the most
difficult part. What is interesting is that folks chose full site. Issues on
representative sample pages includes all different types of elements in a
website.
... Also just as important to document how this gets applied to pages not
reviewed.
... Definitely a bigger challenge if we have a representative sample client is
not sure how this applies to all the different pages on the site.
... Need to provide information on how the representative sample applies to
rest of site.
Eric: This is where I see alot of comments in the
disposition of comments on how do we guide people on how to define the
sample.
... We need a much better description on how to define sample in such a way to
give us a level of confidence that we can get to the same end results.
... Is there statistical work that can help us?
Kathy: Only other comment is that we often in a
real world situation there is a lot of going back and forth with business
owner. It is important to get a demo of the site or application. It is knowing
how the application has been developed or grown overtime.
... e.g. Website/application that has grown overtime where no code was reused.
Couldn't look at representative sample because of the diversity of the web
site. One table might differ from another because they are not coded in the
same manner.
Eric: Good point. We hadn't thought of different developers developing tables in different ways.
<Kathy> what is the username/password for the link
Vivienne: Has link for paper from Georgio. He has done evaluation of a website with 32,000 pages. May be able to refer to this paper in which he has collected a lot of data. I can do a synopsis of the paper.
<MartijnHoutepen> Vivienne, the link is pointing to your e-mail, so we can't access it
Eric: Best to discuss with Shadi and ask Georgio to provide comments to the methodology.
<Vivienne> okay, will try to get a different link
Korn: In talking about number of developers and
difference in coding style, in more conformance to coding style the better
confidence you have in the sample. If the coding style is more heterogenous
then need more sampling.
... Could have survey and do a code inspection.
<Vivienne> link to that paper: http://users.dimi.uniud.it/~giorgio.brajnik/papers/sampling-assets.pdf
Korn: Use automated framework and tools. If every tab pane is same component, then you don't need to examine every menu. If you find menu components are heterogenous need a larger sample and check each more carefully.
<MartijnHoutepen> Thanks vivienne!
Korn: May be good to build this into methodology as well.
Kathy: Difficult to do when a consultant because looking at generated DOM.
Korn: Could still look at generate DOM
... Will be differences that are telling
Kathy: Only have so much time to generate sample. Sampling can take an enormous amount of time.
Korn: Should be looking at how to conserve time
by creating the sample. If we cannot assure ourselves given the time allotted
then need to say that.
... Need to indicate in the report our level of confidence based upon the time
alotted
Kathy: Much more efficient if you work with website owner to determine the sample
Korn: Not opposed to relationship with web site owner but would also like to suggest techniques that can flush out the heterogenous coding style.
Kathy: Agrees this should be a collaborative effort.
Eric: Agrees this would be a good addition to the methodology. Will make a first phase at adding this guidance and will send out to the mailing list.
Richard: Before we leave Step #3, When you decide
not to do a sample but instead test the whole site, it might actually save you
time to test the whole site.
... Once you've analyzed related pages, you know that you can then skip
additional pages.
... Need to explain the decision that has to be made at the start of Step 3
Eric: Are there other questions?
Richard: The general theme was that it was
generally clear. May not be perfect yet but we are on the right track.
... Need to remember that our target audience is not just us but a much wide
audience and audience that has less technical background.
... We are very knowledgeable in the area of website accessibility.
... Let's get everybody trying at least one test so we can get a better
picture.
... Does this produce plausible results? Not a big enough sample yet. Cannot
draw any additional conclusions.
Eric: Richard has a proposal for additional
questions. Please send to mailing list.
... This takes us through the summary of website #1. Richard, thank you very
much!
... Any other comments regarding summary?
... Everyone work on website #2. We will pick up comments to that the week
after next.
... Looking for a volunteer to do a summary of the results in the survey for
website #2
... this would be due on Thursday, 6/6
... Any other issues?
<Vivienne> bye all
Eric: If not, have a beautiful holiday. Speak to everyone next Thursday.