13:55:19 <RRSAgent> RRSAgent has joined #eval
13:55:19 <RRSAgent> logging to http://www.w3.org/2011/10/06-eval-irc
13:55:21 <trackbot> RRSAgent, make logs world
13:55:21 <Zakim> Zakim has joined #eval
13:55:23 <trackbot> Zakim, this will be 3825
13:55:23 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 5 minutes
13:55:24 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:55:24 <trackbot> Date: 06 October 2011
13:55:40 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
13:55:47 <Zakim> + +1.301.975.aaaa
13:55:59 <ssirois> ssirois has joined #eval
13:56:53 <Zakim> + +1.514.312.aabb
13:56:56 <Zakim> - +1.301.975.aaaa
13:56:58 <Zakim> + +1.301.975.aaaa
13:57:06 <ssirois> zakim, aabb is me
13:57:06 <Zakim> +ssirois; got it
13:57:23 <ssirois> Zakim, mute me
13:57:23 <Zakim> ssirois should now be muted
13:57:24 <Liz> zakim, aaaa is Liz
13:57:25 <Zakim> +Liz; got it
13:57:58 <Zakim> + +1.978.443.aacc
13:58:01 <AmyChen> AmyChen has joined #eval
13:58:46 <Kathy> zakim, aacc is Kathy
13:58:46 <Zakim> +Kathy; got it
13:58:59 <shadi> zakim, call shadi-617
13:59:01 <Zakim> ok, shadi; the call is being made
13:59:05 <Zakim> +Shadi
13:59:31 <Zakim> + +1.925.694.aadd
13:59:41 <AmyChen> zakim, aadd is me
13:59:47 <Zakim> +AmyChen; got it
13:59:49 <Kathy> zakim, mute me
13:59:53 <AmyChen> zakim, mute me
13:59:54 <shadi> zakim, who is on the phone?
13:59:57 <Zakim> Kathy should now be muted
14:00:01 <Zakim> AmyChen should now be muted
14:00:07 <Zakim> On the phone I see Liz, ssirois (muted), Kathy (muted), Shadi, AmyChen (muted)
14:00:33 <Zakim> +Katie_Haritos-Shea
14:01:33 <richard> richard has joined #eval
14:02:04 <Zakim> +??P38
14:02:06 <Ryladog_> Ryladog_ has joined #eval
14:02:35 <Zakim> + +666028aaee
14:02:39 <shadi> zakim, ??p38 is Richard
14:02:39 <Zakim> +Richard; got it
14:03:01 <shadi> zakim, aaee is Emmanuelle
14:03:01 <Zakim> +Emmanuelle; got it
14:03:07 <shadi> zakim, who is on the phone?
14:03:07 <Zakim> On the phone I see Liz, ssirois (muted), Kathy (muted), Shadi, AmyChen (muted), Katie_Haritos-Shea, Richard, Emmanuelle (muted)
14:03:28 <Tim> Tim has joined #eval
14:03:44 <shadi> http://www.w3.org/WAI/ER/2011/eval/minutes.html
14:03:51 <shadi> Topic: Welcome
14:03:59 <LeonieWatson> LeonieWatson has joined #eval
14:04:20 <AmyChen> I can try scribing. Do I just type notes in IRC?
14:04:40 <shadi> scribe: AmyChen
14:05:10 <AmyChen> ok
14:05:26 <Zakim> +Tim_Boland
14:05:38 <sinarmaya> sinarmaya has joined #eval
14:06:16 <AmyChen> Shadi: top of list are candidates for scribing
14:06:43 <shadi> http://www.w3.org/WAI/ER/2011/eval/eval-tf
14:06:54 <Zakim> + +1.517.353.aaff
14:07:53 <shadi> zakim, aaff is Mike
14:07:53 <Zakim> +Mike; got it
14:08:00 <shadi> Topic: Requirements
14:08:01 <AmyChen> Shadi: some progress on list, transition from requirements phase to content development phase
14:08:28 <shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Oct/0010.html
14:08:32 <AmyChen> Shadi: Richard proposed alternative concise requirements
14:08:53 <shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Oct/0025.html
14:09:10 <Zakim> + +30231125aagg
14:10:07 <kostas> zakim, aagg is kostas
14:10:07 <Zakim> +kostas; got it
14:10:12 <AmyChen> Shadi: Note that it is not engraved in stone, initial set of requirements for better understanding, good discussions
14:11:24 <AmyChen> Shadi: From my perspective, req pretty stable. Kristen raised Objective and Validity. Open that up for discussion.
14:11:29 <Tim> q+
14:11:48 <shadi> ack tim
14:12:49 <Zakim> +??P59
14:13:14 <AmyChen> Tim: Intents of requirement, when possible should be the same. Present active tense, testability, partial results. Better to be consistent. Will employ versus "should" get equivalent results. General comments. Better to use declarative present tense to evaluate when see fit, pass/fail.
14:13:34 <agarrison> agarrison has joined #eval
14:14:02 <shadi> zakim, ??59 is agarrison
14:14:02 <Zakim> sorry, shadi, I do not recognize a party named '??59'
14:14:09 <shadi> zakim, ??P59 is agarrison
14:14:09 <Zakim> +agarrison; got it
14:14:21 <richard> q+
14:14:28 <shadi> ack r
14:15:28 <AmyChen> Richard: Agree about the tense. Word "should" to try and limit it, discussions instead of exact comparison to be less strict. Not identical results. Used the word "should." Don't mind if changed. 
14:15:46 <Mike_Elledge> q+
14:16:13 <AmyChen> Shadi: We want as much correlation as possible btw different evaluators, but how to find the wording and measure. Don't know if we can sort out at this early phase. Try to find wording we are all comfortable with, then get methodlogy where we want it to be.
14:16:17 <shadi> ack m
14:16:34 <AmyChen> Mike: Think we should have consistent headings in their structure and formatting. 
14:17:15 <AmyChen> Shadi: Leave this for discussion for editorial work. Does help to have consistency.
14:17:41 <shadi> [[RQ 01 : Define methods for evaluating WCAG 2.0 conformance
14:17:41 <shadi> The Methodology provides methods to measure conformance with WCAG 2.0 that 
14:17:41 <shadi> can be used by the target audience (see section 2 above) for evaluating 
14:17:43 <shadi> small and large websites, sections of websites or web-based applications.]]
14:17:50 <AmyChen> Shadi: Req 1 - note that we did want to look at the heading for each of the req and tense. Any other comments on Req 1?
14:18:43 <AmyChen> Question from phone: what is def of large website? 
14:18:57 <AmyChen> Shadi: proper wording, website regardless of size
14:19:16 <shadi> s/Question from phone:/Tim:
14:20:20 <shadi> [[RQ 02 – Clear, understandable, and translatable language
14:20:20 <shadi> The methodology is written in clear language, understandable to the target 
14:20:20 <shadi> audience and capable of translation to other languages.]]
14:21:29 <ssirois> rq2 is just fine with me
14:21:44 <Kathy> capable is fine
14:21:54 <Mike_Elledge> q+
14:21:56 <shadi> [[RQ 04 - Tool and browser independent
14:21:56 <shadi> The use and application of the Methodology is independent of any particular 
14:21:56 <shadi> evaluation tools, browsers, and assistive technology. It requires combined 
14:21:56 <shadi> use of manual and automated testing approaches to carry out a full 
14:21:56 <shadi> evaluation according to the Methodology.]]
14:22:27 <Mike_Elledge> q+
14:22:36 <shadi> ack m
14:23:05 <agarrison> should we use the term evaluation or assessment
14:23:09 <AmyChen> Mike: Question about Req 2. Objectivity could be addressed in Req 2 to resolve some of the issues brought up.
14:23:31 <AmyChen> Shadi: Please re-raise if forget.
14:24:07 <shadi> [[RQ 05 - QA framework specification guidelines
14:24:07 <shadi> The Methodology will conform to the Quality Assurance framework 
14:24:07 <shadi> specification guidelines as set in: http://www.w3.org/TR/qaframe-spec/.]]
14:24:28 <AmyChen> Shadi: Good question. (Agarrison) Keep for later.
14:24:41 <AmyChen> Shadi: R5 pretty straightforward.
14:24:44 <shadi> [[RQ 06 - Reporting
14:24:44 <shadi> The Methodology includes recommendations for reporting evaluation findings. 
14:24:44 <shadi> It will be based on the [href=http://www.w3.org/WAI/eval/template.html 
14:24:44 <shadi> standard template] and supplemented with machine-readable 
14:24:45 <shadi> [href=http://www.w3.org/WAI/intro/earl reports using Evaluation and Report 
14:24:47 <shadi> Language (EARL)].]]
14:24:48 <Ryladog_> q+
14:24:57 <shadi> ack r
14:25:32 <Mike_Elledge> q+
14:25:33 <richard> q+
14:25:57 <AmyChen> Katie: Back to Req 4. Whoever recommened use and application language. independent of any browser, tools, AT. Like that wording. 
14:26:01 <shadi> ack m
14:26:34 <AmyChen> Mike: Req 6 - difficulty understanding the statement. Methodology include info on how to report the findings, or recommendations based on the eval findings?
14:26:59 <AmyChen> Shadi: Should include both. What do others think?
14:27:09 <shadi> ack r
14:27:55 <AmyChen> Richard: Req 6, the template talks about WAI 1, standard template to look from for report, know what questions we have to answer, such as define website to make it valid and reliable.
14:28:51 <AmyChen> Shadi: Tried to separate the two points. We can update the template. Group that maintains that is EOWG. Base intial thoughts from there and update template with methology that we have. Template could be part of methology. Standardized reporting template.
14:29:19 <AmyChen> Shadi: Methodology includes "how", woudl that make it clearer?
14:29:46 <AmyChen> sorry, can you repeat the changes?
14:29:46 <shadi> [[RQ 07 - Use of existing WCAG 2.0 processes
14:29:46 <shadi> Wherever possible the Methodology will employ existing testing procedures in 
14:29:46 <shadi> the WCAG 2.0 Techniques documents rather than replicate them.]]
14:30:16 <shadi> The Methodology includes recommendations for howe to report evaluation findings based on ...
14:30:39 <Kathy> RQ 07 is good
14:31:06 <agarrison> is testing procedures the right term?
14:31:42 <AmyChen> Shadi: "The methology employes existing testing procedures."
14:32:12 <shadi> Methodology employs existing WCAG 2.0 Techniques rather than replicate them
14:32:19 <AmyChen> Allistair: There was a testing procedure at the end of every technique.
14:32:30 <shadi> The Methodology employs existing testing procedures from WCAG 2.0 Techniques rather than replicate them
14:32:37 <agarrison> q?
14:32:44 <Ryladog_> q+
14:33:08 <Ryladog_> +1 test procedure
14:33:25 <AmyChen> Allistair: What about test procedure? nice to have one for each technique to limit it to that technique
14:33:42 <shadi> ack r
14:33:45 <AmyChen> Shadi: Diff to expalin lower level testing and the evaluation. interrelationship.
14:34:02 <shadi> The Methodology employs existing testing procedures within WCAG 2.0 Techniques rather than replicate them
14:34:25 <AmyChen> Katie: Agree with Test Procedure. Not sure about limiting it to one technique. Test Procedure covers overall intent of the requirement, instead of identifying specific techniques.
14:34:38 <AmyChen> Shadi: Tried changing one word. Instead of "from" to "within"
14:34:40 <Kathy> that is better
14:34:46 <AmyChen> +1
14:34:49 <agarrison> within is better
14:34:53 <Tim> better
14:36:05 <AmyChen> Q from phone: methodology for manual eval. Testing and tech not complete, only for automatic eval? 
14:36:25 <AmyChen> Shadi: Techniques not only for automatic testing. 
14:36:46 <AmyChen> Shadi: Techniques on WCAG working group. 
14:36:46 <Ryladog_> shadi is echoing
14:37:08 <shadi> zakim, mute emmanuelle
14:37:08 <Zakim> Emmanuelle should now be muted
14:37:56 <LeonieWatson> LeonieWatson has joined #eval
14:38:09 <AmyChen> Shadi: Techniques not exhaustive. This group not focus on the individual testing procedures, but rather on overall procedure aspect.
14:39:27 <AmyChen> Shadi: Work being done outside W3C. Vendors, tools, testing techniques. Open initiatives for testing procedures. Some work to compile that. Fit into methodology.
14:39:31 <shadi> q?
14:39:59 <shadi> [[RQ 08 - Recommendations for scope and sampling
14:39:59 <shadi> It includes recommendations for methods of sampling web pages and how to 
14:39:59 <shadi> ensure that complete processes (such as for a shopping site where all the 
14:39:59 <shadi> pages that are part of the steps in an ordering process) are included. Such 
14:39:59 <shadi> selections would be reflected in any conformance claim.]]
14:40:02 <AmyChen> Shadi: Title of Req 7 needs to be updates
14:40:11 <kostas> q+
14:40:30 <shadi> ack k
14:41:03 <agarrison> "It includes recommendations" might be "it defines"
14:41:14 <AmyChen> Kostas: Concerned about req about sampling. General methods, but give examples? How can we cover all of the possibilities? Effective sampling? General recommendations and methods? How can we ensure the complete process?
14:41:20 <AmyChen> q+
14:42:19 <Kathy> q+
14:42:25 <AmyChen> Shadi: good example, give some numbers, based on website size. Need to develop. One of the big parts. Req 8 and 9 the biggest parts of the methodology. Depending on how well we do 8 and 9, then number 3, replicability will emerge
14:43:01 <AmyChen> Kostas: Maybe need to be more specific.
14:43:04 <agarrison> q+
14:43:04 <AmyChen> ack me
14:43:08 <shadi> ack amy
14:43:33 <shadi> Amy: sampling will be a big part but we have lots of experience at Oracle
14:43:39 <Kathy> ack me
14:43:47 <shadi> ...but ok with the wording as it stands now
14:44:07 <AmyChen> Kathy: Suggest taking out the example. "Such as for a shopping site" Scenarios, concerned with throwing one specific thing in there.
14:44:21 <AmyChen> Shadi: Better to define complete process in terminology section and link to it
14:44:25 <Kathy> zakim, mute me
14:44:25 <Zakim> Kathy should now be muted
14:44:29 <shadi> ack a
14:44:34 <Ryladog_> +1
14:44:45 <Mike_Elledge> +1
14:45:15 <Tim> q+
14:45:20 <AmyChen> Allistair: Recommendations, methods have to be consistent. Doesn't mean they have to be using those methods.
14:45:38 <AmyChen> Allistair: Change to recommendations for scope and sampling
14:46:05 <Mike_Elledge> +1
14:46:06 <AmyChen> +1
14:46:10 <Tim> +1
14:46:11 <kostas> +1
14:46:12 <shadi> ack t
14:46:15 <agarrison> should be change from recommendations to just scope and sampling
14:46:15 <Kathy> +1
14:46:17 <richard> +1
14:46:31 <Ryladog_> +1
14:46:52 <AmyChen> Sorry, Allistair means, take out recommendations
14:47:29 <AmyChen> Tim: WCAG and ATAG discussions on what would be in a conformance claim. 
14:47:41 <AmyChen> Shadi: could go in reporting section on conformance claim
14:47:43 <Ryladog_> +1
14:47:50 <richard> +1
14:47:50 <Mike_Elledge> +1
14:47:52 <Kathy> +1
14:47:53 <sinarmaya> +1
14:48:07 <AmyChen> q+
14:48:17 <shadi> [[RQ 09 - Includes tolerance metrics
14:48:17 <shadi> It includes calculation methods for determining nearness of conformance. 
14:48:17 <shadi> Depending on the amount of tolerance, a failure could fall within a certain 
14:48:17 <shadi> tolerance level meaning that the page or website might be considered 
14:48:17 <shadi> conformant even though there is a failure. Such tolerances would be 
14:48:18 <shadi> reflected in any conformance claim.]]
14:48:48 <agarrison> remove includes from title, and change includes to defines
14:48:49 <AmyChen> Shadi: Remove the last sentence, but any other thoughts?
14:49:04 <Kathy> I agree
14:49:30 <shadi> [[RQ 10 - Support documentation
14:49:30 <shadi> The document will give a short description of the knowledge necessary for 
14:49:30 <shadi> using the Methodology for evaluations.]]
14:49:32 <AmyChen> Change Includes Tolerance metrics to Tolerance Metrics.
14:50:06 <shadi> [[RQ 03 – Reliable
14:50:06 <shadi> Different Web accessibility evaluators using the same methods on the same 
14:50:06 <shadi> website(s) should get equivalent results. Evaluation process and results are 
14:50:06 <shadi> documented to support independent verification.]]
14:50:24 <AmyChen> Shadi: From Kriten on mailing list, that this does not go far enough. 
14:50:53 <ssirois> ssirois has joined #eval
14:50:59 <Ryladog_> +q
14:51:02 <AmyChen> Shadi: diluted the wording somewhat, will there really be 100%, same results. What would be reliable?
14:51:03 <Mike_Elledge> q+
14:51:18 <AmyChen> Shadi: Tim also raised the concern "should"
14:51:25 <AmyChen> sorry, from earlier, nevermind
14:51:31 <shadi> q- amy
14:51:36 <shadi> ack r
14:51:57 <agarrison> The wording for R03 sounds nice and scientific - I support current form (should is fine)
14:52:18 <AmyChen> Katie: 100% not going to happen, on the 80/20 reviewers. Will probably need something better than that when we get to the actual requirement.
14:52:18 <ssirois> sorry, irc connection trouble, but was on the phone all alone and haven't have something to disagry on that would have urge me to find the way to unmute myself over touchtone! ;)
14:52:40 <shadi> ack m
14:53:03 <AmyChen> Mike: "Should get equivalent and equally valid results." and define valid
14:53:11 <Kathy> q+
14:53:25 <AmyChen> Shadi: what diff btw equivalent and equally valid?
14:53:43 <Kathy> ack me
14:53:43 <shadi> ack k
14:53:45 <AmyChen> Mike: Equivalent results, but make sure valid. Maybe don't want to say equally. Maybe valid results.
14:54:08 <AmyChen> Kathy: Comment on not 100% possible, if we define equivalent results. 
14:54:32 <AmyChen> Action: Equivalent results need to be defined. High correlation. Weave in validity.
14:54:32 <trackbot> Sorry, couldn't find user - Equivalent
14:54:51 <Kathy> zakim, mute me
14:54:51 <Zakim> Kathy should now be muted
14:55:15 <AmyChen> Shadi: Mike earlier raised objectiveness and objectivity. Fits in req 2
14:55:55 <AmyChen> Shadi: Reliable, R3, define equiv results and link to it
14:56:00 <Mike_Elledge> +1
14:56:04 <agarrison> +1
14:56:07 <Kathy> +1
14:56:09 <Ryladog_> +1
14:56:12 <AmyChen> +1
14:56:14 <ssirois> +1
14:56:15 <Tim> +1
14:56:17 <sinarmaya> +1
14:56:37 <Mike_Elledge> +1
14:56:43 <AmyChen> Shadi: go back to Objective R2, how about Clear language that is unambiguous and understandable to audience
14:57:05 <agarrison> clear = unambiguous
14:57:11 <AmyChen> Mike: Yes, seems to get at it to me.
14:57:22 <agarrison> q+
14:57:28 <shadi> ack a
14:57:43 <AmyChen> Allistair: How about language that is clear and understandable to the audience.
14:57:56 <Ryladog_> clear is not testable
14:58:37 <richard> Q+
14:58:37 <AmyChen> sorry, repeat please?
14:58:59 <AmyChen> Allistair: Unambigous might be harder to understand
14:59:22 <AmyChen> Shadi: Replace clear rather than unambigous
14:59:51 <Mike_Elledge> <clapping hands>
15:00:00 <shadi> Topic: Naming
15:01:37 <AmyChen> Methodology for Assessing Website Conformity 
15:02:06 <AmyChen> Kostas: Sampling, Inspection, and Technical Evaluation (SITE) has the word "site" not to be confused with websites
15:02:28 <AmyChen> (Kostas - was that what you were saying?)
15:02:37 <kostas> What about MEAC? INSTEAD OF MAC?
15:02:37 <AmyChen> Shadi: Let the creative juices flow for naming
15:02:58 <AmyChen> Shadi: take it to the list and talk about it next week.
15:03:11 <Kathy> Thanks!  Bye
15:03:14 <ssirois> Zakim, unmute me
15:03:18 <Ryladog_> cioa
15:03:20 <AmyChen> Thanks everyone, bye! 
15:03:20 <shadi> trackbot, end meeting
15:03:20 <trackbot> Zakim, list attendees
15:03:21 <trackbot> RRSAgent, please draft minutes
15:03:21 <RRSAgent> I have made the request to generate http://www.w3.org/2011/10/06-eval-minutes.html trackbot
15:03:21 <sinarmaya> sinarmaya has left #eval
15:03:22 <trackbot> RRSAgent, bye
15:03:22 <RRSAgent> I see 1 open action item saved in http://www.w3.org/2011/10/06-eval-actions.rdf :
15:03:22 <RRSAgent> ACTION: Equivalent results need to be defined. High correlation. Weave in validity. [1]
15:03:22 <RRSAgent>   recorded in http://www.w3.org/2011/10/06-eval-irc#T14-54-32
15:03:27 <Mike_Elledge> Bye, bye!