13:40:22 <RRSAgent> RRSAgent has joined #eval
13:40:22 <RRSAgent> logging to http://www.w3.org/2013/05/23-eval-irc
13:40:24 <trackbot> RRSAgent, make logs world
13:40:24 <Zakim> Zakim has joined #eval
13:40:26 <trackbot> Zakim, this will be 3825
13:40:26 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 20 minutes
13:40:27 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:40:27 <trackbot> Date: 23 May 2013
13:40:42 <MartijnHoutepen> Zakim, what's on the agenda?
13:40:42 <Zakim> I see nothing on the agenda
13:41:00 <MartijnHoutepen> agenda+ Welcome
13:41:10 <MartijnHoutepen> agenda+ Current state of comments
13:41:29 <MartijnHoutepen> agenda+ Test run
13:41:36 <MartijnHoutepen> Ok, thank you!
13:52:37 <Vivienne> Vivienne has joined #eval
13:56:22 <MoeKraft> MoeKraft has joined #eval
13:59:00 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
13:59:07 <Zakim> +[IPcaller]
13:59:12 <Vivienne> zakim, IPcaller is me
13:59:12 <Zakim> +Vivienne; got it
13:59:27 <Zakim> +??P7
13:59:40 <Liz> Liz has joined #eval
13:59:44 <MartijnHoutepen> Zakim, ??Py is me
13:59:44 <Zakim> sorry, MartijnHoutepen, I do not recognize a party named '??Py'
13:59:47 <Zakim> +MoeKraft
13:59:49 <MartijnHoutepen> Zakim, ??P7 is me
13:59:49 <Zakim> +MartijnHoutepen; got it
14:00:20 <Zakim> +Liz
14:00:32 <Kathy> Kathy has joined #eval
14:00:51 <Kathy> I will be late joining the conference call
14:00:52 <richard> richard has joined #eval
14:01:35 <ericvelleman> ericvelleman has joined #eval
14:01:45 <Zakim> +[IPcaller]
14:02:16 <MartijnHoutepen> scribeNick: MoeKraft
14:02:24 <richard> zakim, IPcaller is Richard
14:02:24 <Zakim> +Richard; got it
14:02:35 <Zakim> +??P20
14:02:37 <MartijnHoutepen> Zakim, what's on the agenda?
14:02:37 <Zakim> I see 3 items remaining on the agenda:
14:02:38 <Zakim> 1. Welcome [from MartijnHoutepen]
14:02:38 <Zakim> 2. Current state of comments [from MartijnHoutepen]
14:02:38 <Zakim> 3. Test run [from MartijnHoutepen]
14:02:40 <MoeKraft> agenda+ Welcome
14:03:04 <ericvelleman> Zakim, ??P20 is me
14:03:05 <Zakim> +ericvelleman; got it
14:03:08 <MoeKraft> Zakim takeup next
14:03:26 <MoeKraft> Zakim, take up next
14:03:26 <Zakim> agendum 1. "Welcome" taken up [from MartijnHoutepen]
14:03:47 <MartijnHoutepen> Zakim, please mute me
14:03:47 <Zakim> MartijnHoutepen should now be muted
14:04:07 <MoeKraft> Eric: Welcomes everyone to the teleconference. Shai is in Brussels presenting on W3C. He will not be able to join the call.
14:04:13 <MoeKraft> Zakim, take up next
14:04:13 <Zakim> agendum 1 was just opened, MoeKraft
14:04:24 <MoeKraft> Zakim, take up next
14:04:24 <Zakim> agendum 1 was just opened, MoeKraft
14:04:27 <MartijnHoutepen> ack mr
14:04:30 <MartijnHoutepen> ack me
14:04:52 <MoeKraft> Zakim, close agendum 1
14:04:52 <Zakim> agendum 1, Welcome, closed
14:04:54 <Zakim> I see 3 items remaining on the agenda; the next one is
14:04:54 <Zakim> 2. Current state of comments [from MartijnHoutepen]
14:04:58 <ericvelleman> <http://lists.w3.org/Archives/Public/public-wcag-em-comments/>
14:05:09 <MoeKraft> Eric: No new comments received
14:05:12 <MartijnHoutepen> Zakim, mute me
14:05:12 <Zakim> MartijnHoutepen should now be muted
14:05:28 <MoeKraft> Eric: Just finished disposition of comments
14:05:29 <korn> korn has joined #eval
14:05:38 <Zakim> +Kathy
14:05:48 <MoeKraft> Eric:  Shadi will put this online probably over the weekend
14:06:00 <Bim> Bim has joined #eval
14:06:13 <MoeKraft> Eric: Will get more information from agenda item 3 regarding surveys
14:06:44 <ericvelleman> q?
14:06:50 <MoeKraft> Zakim, close agendum 2
14:06:50 <Zakim> agendum 2, Current state of comments, closed
14:06:51 <Zakim> I see 2 items remaining on the agenda; the next one is
14:06:51 <Zakim> 3. Test run [from MartijnHoutepen]
14:07:38 <MoeKraft> Eric: Ask folks to work as much as they can on the Game accessibility website and Richard has evaluated the results.
14:07:42 <MoeKraft> Website #1: <https://www.w3.org/2002/09/wbs/48225/testrun/results>
14:07:56 <MoeKraft> Eric: Please take us through the results
14:07:58 <ericvelleman> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0023.html
14:08:48 <MoeKraft> #3825
14:08:58 <shadi> zakim, code?
14:08:58 <Zakim> the conference code is 3825 (tel:+1.617.761.6200 sip:zakim@voip.w3.org), shadi
14:09:11 <Zakim> +Peter_Korn
14:09:24 <MoeKraft> Richard: Only 4 people gave feedback in results. Small set.
14:09:51 <MoeKraft> Richard: Need to look at adding to step 1, please confirm what you have decided as your scope is what the client wants.
14:10:11 <MoeKraft> Richard: Can state Step 1 is fairly okay
14:10:30 <MoeKraft> Eric: Anyone else have feedback on conclusion to step 1?
14:11:00 <MoeKraft> Eric: Did receive public feedback on Step 1
14:11:56 <MoeKraft> Richard: Step 2 we had a very small sample and wonder why. Was Step 2 too difficult? It is hard to analyze. Why did people not complete all 4 steps?
14:11:56 <MartijnHoutepen> q+
14:12:02 <MartijnHoutepen> ack me
14:12:17 <Vivienne> q+
14:12:23 <MoeKraft> Martijn: I took the guidance for reporting from Step 5
14:13:12 <ericvelleman> q?
14:13:15 <Vivienne> zakim, ack me
14:13:17 <MoeKraft> Martijn: In second questionairre added more information for step 2
14:13:18 <Zakim> I see no one on the speaker queue
14:13:21 <MartijnHoutepen> Zakim, mute me
14:13:21 <Zakim> MartijnHoutepen should now be muted
14:13:44 <MoeKraft> Vivienne: Was unable to finish steps due to travel.
14:14:37 <MoeKraft> Eric: Thank you for being on the call after all that travel.
14:14:57 <MoeKraft> Vivienne: Chatted about methodolgy at conference and has feedback to share.
14:15:07 <Liz> q+
14:15:36 <Vivienne> zakim, mute me
14:15:36 <Zakim> Vivienne should now be muted
14:15:46 <MartijnHoutepen> q- liz
14:16:33 <MoeKraft> Liz: Did not finish Step #2 because she saw a very comprehensive diagram of site in a picturesque way that she liked. She did not understand how to answer the question. Likes how Vivienne has put together diagram.
14:16:42 <MoeKraft> Liz: Not sure what to say.
14:17:27 <ericvelleman> q?
14:17:29 <MoeKraft> Richard: Need result of exploration on questionnaire so we can figure out if we had similar results or differing. Did we understand methodology and step?
14:18:22 <MoeKraft> Richard: Also identify technologies used.
14:18:36 <MoeKraft> Richard: Basically get results of what was done.
14:19:01 <MoeKraft> Eric: Need to explain clarly in the survey text what is expected.
14:20:34 <ericvelleman> q?
14:20:51 <MoeKraft> Richard: Step 3 was to select a representative sample.
14:21:07 <MoeKraft> Richard: 2 out of 3 tested entire site while it was a small site.
14:22:13 <MoeKraft> Richard: 1 person did an analysis with selected pages using the PayPal section which should have been excluded.
14:22:40 <MoeKraft> Richard: Need to look clearly at the guidelines for Step 3. How do you decide if the site is so small that it is not worth sampling.
14:23:10 <MoeKraft> Richard: How do decide to sample or not sample?
14:23:23 <ericvelleman> q?
14:23:24 <korn> q+
14:23:28 <MoeKraft> Eric: Obvious on small sites we do not have to go through sampling
14:24:00 <Kathy> q+
14:24:15 <MoeKraft> Korn: This is one of the most challenging parts of our work. How do you come up with a sampling methodology that is going to yield similar results across different folks performing test.
14:24:37 <MoeKraft> Korn: There are probably tecniques at universities or elsewhere on doing this type of sampling work.
14:24:42 <korn> q-
14:24:42 <Tim> Tim has joined #eval
14:24:48 <Vivienne> https://staffmail.ecu.edu.au/owa/attachment.ashx?attach=1&id=RgAAAABI7j1wgS3dSaYJ6jJKs4DjBwCK%2bnd0GxHbR7JBMfHjgiepAG7AfT4PAACK%2bnd0GxHbR7JBMfHjgiepASPU4MxzAAAJ&attid0=EAAAycCB1a9PTolHYVBhMIRT&attcnt=1
14:24:49 <ericvelleman> q?
14:24:53 <Kathy> ack me
14:24:57 <MoeKraft> Korn: How do we have confidence that we will have similar results person to person?
14:25:03 <Vivienne> Link above to to Giorgio Brajnik's paper on sampling
14:25:44 <Vivienne> paper is entitled "Effects of Sampling Methods on Web Accessibility Evaluations"
14:25:47 <MoeKraft> Kathy: Agree with Peter that this is the most difficult part. What is interesting is that folks chose full site. Issues on representative sample pages includes all different types of elements in a website.
14:26:06 <MoeKraft> Kathy: Also just as important to document how this gets applied to pages not reviewed.
14:26:19 <ericvelleman> q?
14:26:40 <MoeKraft> Kathy: Definitely a bigger challenge if we have a representative sample client is not sure how this applies to all the different pages on the site.
14:27:08 <MoeKraft> Kathy: Need to provide information on how the representative sample applies to rest of site.
14:28:40 <Vivienne> q+
14:29:09 <MoeKraft> Eric: This is where I see alot of comments in the disposition of comments on how do we guide people on how to define the sample.
14:29:40 <MoeKraft> Eric: We need a much better description on how to define sample in such a way to give us a level of confidence that we can get to the same end results.
14:29:52 <MoeKraft> Eric: Is there statistical work that can help us?
14:30:15 <ericvelleman> q?
14:31:32 <MoeKraft> Kathy: Only other comment is that we often in a real world situation there is a lot of going back and forth with business owner. It is important to get a demo of the site or application. It is knowing how the application has been developed or grown overtime.
14:32:04 <korn> q+
14:32:32 <MoeKraft> Kathy: e.g. Website/application that has grown overtime where no code was reused. Couldn't look at representative sample because of the diversity of the web site. One table might differ from another because they are not coded in the same manner.
14:32:55 <Zakim> +Tim_Boland
14:33:32 <MoeKraft> Eric: Good point. We hadn't thought of different developers developing tables in different ways.
14:33:36 <ericvelleman> q?
14:33:43 <Vivienne> zakim, ack me
14:33:43 <Zakim> unmuting Vivienne
14:33:44 <Zakim> I see korn on the speaker queue
14:34:16 <Kathy> what is the username/password for the link
14:35:08 <MoeKraft> Vivienne: Has link for paper from Georgio. He has done evaluation of a website with 32,000 pages. May be able to refer to this paper in which he has collected a lot of data. I can do a synopsis of the paper.
14:35:11 <MartijnHoutepen> Vivienne, the link is pointing to your e-mail, so we can't access it
14:35:50 <MoeKraft> Eric: Best to discuss with Shadi and ask Georgio to provide comments to the methodology.
14:36:04 <ericvelleman> q?
14:36:06 <korn> q?
14:36:18 <Vivienne> okay, will try to get a different link
14:36:50 <Kathy> q+
14:37:16 <MoeKraft> Korn: In talking about number of developers and difference in coding style, in more conformance to coding style the better confidence you have in the sample. If the coding style is more heterogenous then need more sampling.
14:37:32 <MoeKraft> Korn: Could have survey and do a code inspection.
14:37:49 <Vivienne> link to that paper: http://users.dimi.uniud.it/~giorgio.brajnik/papers/sampling-assets.pdf
14:38:34 <korn> q-
14:38:41 <MoeKraft> Korn: Use automated framework and tools. If every tab pane is same component, then you don't need to examine every menu. If you find menu components are heterogenous need a larger sample and check each more carefully.
14:38:49 <Kathy> ack me
14:38:56 <MartijnHoutepen> Thanks vivienne!
14:39:07 <MoeKraft> Korn: May be good to build this into methodology as well.
14:39:26 <MoeKraft> Kathy: Difficult to do when a consultant because looking at generated DOM.
14:39:39 <MoeKraft> Korn: Could still look at generate DOM
14:39:52 <MoeKraft> Korn: Will be differences that are telling
14:40:20 <MoeKraft> Kathy: Only have so much time to generate sample. Sampling can take an enormous amount of time.
14:40:57 <MoeKraft> Korn: Should be looking at how to conserve time by creating the sample. If we cannot assure ourselves given the time allotted then need to say that.
14:41:47 <MoeKraft> Korn: Need to indicate in the report our level of confidence based upon the time alotted
14:42:01 <MoeKraft> Kathy: Much more efficient if you work with website owner to determine the sample
14:43:08 <MoeKraft> Korn: Not opposed to relationship with web site owner but would also like to suggest techniques that can flush out the heterogenous coding style.
14:43:21 <MoeKraft> Kathy: Agrees this should be a collaborative effort.
14:44:24 <MoeKraft> Eric: Agrees this would be a good addition to the methodology. Will make a first phase at adding this guidance and will send out to the mailing list.
14:45:09 <MoeKraft> Richard: Before we leave Step #3, When you decide not to do a sample but instead test the whole site, it might actually save you time to test the whole site.
14:46:01 <MoeKraft> Richard:  Once you've analyzed related pages, you know that you can then skip additional pages.
14:46:24 <MoeKraft> Richard: Need to explain the decision that has to be made at the start of Step 3
14:47:02 <MoeKraft> Eric: Are there other questions?
14:47:35 <MoeKraft> Richard: The general theme was that it was generally clear. May not be perfect yet but we are on the right track.
14:48:05 <MoeKraft> Richard: Need to remember that our target audience is not just us but a much wide audience and audience that has less technical background.
14:48:32 <MoeKraft> Richard: We are very knowledgeable in the area of website accessibility.
14:48:49 <MoeKraft> Richard: Let's get everybody trying at least one test so we can get a better picture.
14:49:31 <MoeKraft> Richard: Does this produce plausible results? Not a big enough sample yet. Cannot draw any additional conclusions.
14:50:15 <MoeKraft> Eric: Richard has a proposal for additional questions. Please send to mailing list.
14:50:32 <MoeKraft> Eric: This takes us through the summary of website #1. Richard, thank you very much!
14:50:48 <MoeKraft> Eric: Any other comments regarding summary?
14:52:50 <Vivienne> zakim, ack me
14:52:50 <Zakim> I see no one on the speaker queue
14:52:53 <Vivienne> q+
14:53:05 <MoeKraft> Eric: Everyone work on website #2. We will pick up comments to that the week after next.
14:54:02 <Vivienne> zakim, ack me
14:54:02 <Zakim> I see no one on the speaker queue
14:54:29 <Vivienne> zakim, mute me
14:54:29 <Zakim> Vivienne should now be muted
14:55:22 <MoeKraft> Eric: Looking for a volunteer to do a summary of the results in the survey for website #2
14:56:02 <MoeKraft> Eric: this would be due on Thursday, 6/6
14:56:12 <MoeKraft> Eric: Any other issues?
14:56:26 <Zakim> -Peter_Korn
14:56:28 <Vivienne> bye all
14:56:29 <MartijnHoutepen> ack me
14:56:30 <Zakim> -Tim_Boland
14:56:31 <Zakim> -Kathy
14:56:31 <Zakim> -Richard
14:56:32 <MoeKraft> Eric: If not, have a beautiful holiday. Speak to everyone next Thursday.
14:56:33 <Liz> bye
14:56:33 <Zakim> -Vivienne
14:56:34 <Zakim> -ericvelleman
14:56:35 <Zakim> -MartijnHoutepen
14:56:41 <Zakim> -MoeKraft
14:56:47 <MartijnHoutepen> trackbot, end meeting
14:56:47 <trackbot> Zakim, list attendees
14:56:47 <Zakim> As of this point the attendees have been Vivienne, MoeKraft, MartijnHoutepen, Liz, Richard, ericvelleman, Kathy, Peter_Korn, Tim_Boland
14:56:55 <trackbot> RRSAgent, please draft minutes
14:56:55 <RRSAgent> I have made the request to generate http://www.w3.org/2013/05/23-eval-minutes.html trackbot
14:56:56 <trackbot> RRSAgent, bye
14:56:56 <RRSAgent> I see no action items