14:53:22 <RRSAgent> RRSAgent has joined #wcag-act
14:53:22 <RRSAgent> logging to http://www.w3.org/2017/02/16-wcag-act-irc
14:53:24 <trackbot> RRSAgent, make logs public
14:53:27 <trackbot> Zakim, this will be
14:53:27 <trackbot> Meeting: Accessibility Conformance Testing Teleconference
14:53:27 <trackbot> Date: 16 February 2017
14:53:27 <Zakim> I don't understand 'this will be', trackbot
14:54:10 <Wilco> agenda?
14:54:32 <Wilco> zakim, take up next
14:54:32 <Zakim> agendum 2. "Draft section 4.2 Accessibility Support Data https://github.com/w3c/wcag-act/pull/32/files?diff=split" taken up [from Wilco]
14:54:52 <Wilco> zakim, take up next
14:54:52 <Zakim> agendum 2 was just opened, Wilco
14:55:06 <Wilco> zakim, close item 2
14:55:06 <Zakim> agendum 2, Draft section 4.2 Accessibility Support Data https://github.com/w3c/wcag-act/pull/32/files?diff=split, closed
14:55:08 <Zakim> I see 5 items remaining on the agenda; the next one is
14:55:08 <Zakim> 3. Draft section 6.2 Rule Aggregation https://github.com/w3c/wcag-act/pull/46/files?diff=split [from Wilco]
14:55:13 <Wilco> zakim, take up item 3
14:55:13 <Zakim> agendum 3. "Draft section 6.2 Rule Aggregation https://github.com/w3c/wcag-act/pull/46/files?diff=split" taken up [from Wilco]
15:00:36 <Wilco> agenda+ Benchmarking, https://github.com/w3c/wcag-act/pull/47
15:01:05 <Wilco> agenda?
15:02:04 <agarrison> agarrison has joined #wcag-act
15:03:56 <maryjom> maryjom has joined #wcag-act
15:05:52 <Wilco> present+ Wilco
15:05:53 <shadi> present+
15:06:04 <agarrison> present+
15:06:32 <agarrison> zakim, take up next
15:06:32 <Zakim> agendum 4. "Current week's survey results https://www.w3.org/2002/09/wbs/93339/ACTTF15Feb2017/results" taken up [from Wilco]
15:06:35 <maryjom> present+MaryJoMueller
15:06:53 <MoeKraft> MoeKraft has joined #wcag-act
15:07:28 <agarrison> zakim, take up item 3
15:07:28 <Zakim> agendum 3. "Draft section 6.2 Rule Aggregation https://github.com/w3c/wcag-act/pull/46/files?diff=split" taken up [from Wilco]
15:08:17 <agarrison> wilco: shadi's question - do we need aggregation
15:08:33 <agarrison> wilco: have you seen my response on that?
15:09:23 <agarrison> My call's dropped
15:12:36 <agarrison> Can someone take over minuting - I have an issue with phone.
15:13:39 <MoeKraft> Wilco: A rule can have multiple results for each match in the content.
15:14:14 <MoeKraft> Shadi: I still think that this section needs to be, at least the last sentence, combining all the results to aggregate a final result for the rule is confusing.
15:14:47 <MoeKraft> Shadi: The rule is run several times or can return several results. We need to return a single result or combined into a single result. Is aggregation the correct word here?
15:15:06 <MoeKraft> Shadi: I think of aggregation is an overall score.
15:15:31 <MoeKraft> Wilco: Might be too much of a loaded term.
15:16:02 <MoeKraft> Shadi: Yes. Aggregation could be combining results of all rules into a single result
15:16:14 <MoeKraft> Wilco: Yes. It could be that.
15:16:33 <MoeKraft> Shadi: If we're okay with aggregation then I am fine with it. Don't have a better word.
15:16:35 <agarrison> Shadi: If aggregation is the right word, then OK.
15:16:46 <MoeKraft> Wilco: Alistair, have you had a chance to review this pull request?
15:16:56 <MoeKraft> Alistair: No. I have not.
15:17:39 <agarrison> Alistair: I have not have a chance to look at aggregation.
15:18:01 <agarrison> Wilco: Is there another way to word this; or another way to do this.
15:18:10 <agarrison> Shadi: What is the this?
15:18:23 <agarrison> Wilco: The list of pass / fails into a higher result.
15:18:36 <agarrison> Moe: What about cummulative.
15:18:55 <agarrison> Wilco: Does that not imply things happening over time.
15:19:35 <agarrison> Shadi: It's also how you do the logic.  With earl you had multiple assertions.  You had one result with multiple pointers.
15:19:51 <agarrison> Shadi: It passed and here are the things we checked.
15:20:22 <agarrison> Shadi: The same rule could be fired several times.
15:20:34 <agarrison> Moe: There could be several test cases per rule.
15:21:29 <agarrison> Wilco: We haven't looked at test cases yet.
15:21:50 <agarrison> Wilco: Maybe we should look at that when we review test cases.
15:22:23 <agarrison> Wilco: Want to point out - output format being proposed is different from earl.
15:22:44 <agarrison> Wilco: Earl is set up to provide 1 result with a bunch of pointers.
15:22:56 <agarrison> Wilco: This new method has one pointer.
15:23:50 <agarrison> Shadi: If the same content is tested you should be one set of results.
15:24:25 <agarrison> Wilco: Shadi, you are not opposed to making this change.
15:24:34 <agarrison> Wilco: What about the format.
15:24:46 <agarrison> Wilco: JSON link data format.
15:26:13 <Wilco> https://github.com/w3c/wcag-act/pull/46/files?diff=split%22#diff-7e52b0c9a47dd6941c123f45c8f6d1f5R171
15:33:53 <agarrison> Wilco: This would be a good way to compare tools.
15:34:57 <agarrison> Wilco: If a client wants to see the tool does what we say it does they could use this format.
15:36:18 <shadi> q+
15:37:01 <agarrison> Shadi: We are talking about output format.
15:37:13 <shadi> ack me
15:37:25 <agarrison> Wilco: If I understand Alistair, no we don't need a common format.
15:37:49 <agarrison> Wilco: We can either get all tools to output in the same format.
15:38:17 <agarrison> Wilco: Or, if you are developing software you would use assertion libraries.
15:38:34 <agarrison> Wilco: You don't need a format to do that.
15:39:37 <agarrison> Wilco: I work with aXe.  We use unit tests; run a single rule on a code snippet; and look at the expected result.
15:42:06 <agarrison> Alistair: We take the unit test approach.
15:43:25 <MoeKraft> I have to drop. ttyl
15:45:29 <agarrison> Shadi: Having a format for the output of test is possibly outside of the rule format.
15:45:50 <agarrison> Wilco: The reason we need aggregation is that it allows benchmarking.
15:46:40 <agarrison> Wilco: Tester tests a page.  Page fails on this, passes this, others non-applicable.  We run the tools against that page.  How do we check the rule is accurate.
15:47:20 <agarrison> Wilco: We can't compare them without a mapping to a success criteria.
15:55:24 <agarrison> Alistair: Using a test suite which we all agree on is the best way from my perspective to benchmark.
15:58:36 <agarrison> Wilco: We don't know if a test catches all the issues.  How do we find these issue.
16:00:08 <agarrison> Wilco: If we compare the results from loads of people writing reports - we could see where they disagree with our rules.
16:04:12 <agarrison> Shadi: Good discussion.  We have different thoughts in the group, which is good.