See also: IRC log
wilco: shadi's question - do we need aggregation
... have you seen my response on that?
My call's dropped
Can someone take over minuting - I have an issue with phone.
<MoeKraft> Wilco: A rule can have multiple results for each match in the content.
<MoeKraft> Shadi: I still think that this section needs to be, at least the last sentence, combining all the results to aggregate a final result for the rule is confusing.
<MoeKraft> Shadi: The rule is run several times or can return several results. We need to return a single result or combined into a single result. Is aggregation the correct word here?
<MoeKraft> Shadi: I think of aggregation is an overall score.
<MoeKraft> Wilco: Might be too much of a loaded term.
<MoeKraft> Shadi: Yes. Aggregation could be combining results of all rules into a single result
<MoeKraft> Wilco: Yes. It could be that.
<MoeKraft> Shadi: If we're okay with aggregation then I am fine with it. Don't have a better word.
Shadi: If aggregation is the right word, then OK.
<MoeKraft> Wilco: Alistair, have you had a chance to review this pull request?
<MoeKraft> Alistair: No. I have not.
Alistair: I have not have a chance to look at aggregation.
Wilco: Is there another way to word this; or another way to do this.
Shadi: What is the this?
Wilco: The list of pass / fails into a higher result.
Moe: What about cummulative.
Wilco: Does that not imply things happening over time.
Shadi: It's also how you do the logic. With earl you had
multiple assertions. You had one result with multiple pointers.
... It passed and here are the things we checked.
... The same rule could be fired several times.
Moe: There could be several test cases per rule.
Wilco: We haven't looked at test cases yet.
... Maybe we should look at that when we review test cases.
... Want to point out - output format being proposed is different from
earl.
... Earl is set up to provide 1 result with a bunch of pointers.
... This new method has one pointer.
Shadi: If the same content is tested you should be one set of results.
Wilco: Shadi, you are not opposed to making this change.
... What about the format.
... JSON link data format.
Wilco: This would be a good way to compare tools.
... If a client wants to see the tool does what we say it does they
could use this format.
Shadi: We are talking about output format.
Wilco: If I understand Alistair, no we don't need a common
format.
... We can either get all tools to output in the same format.
... Or, if you are developing software you would use assertion
libraries.
... You don't need a format to do that.
... I work with aXe. We use unit tests; run a single rule on a code
snippet; and look at the expected result.
Alistair: We take the unit test approach.
<MoeKraft> I have to drop. ttyl
Shadi: Having a format for the output of test is possibly outside of the rule format.
Wilco: The reason we need aggregation is that it allows
benchmarking.
... Tester tests a page. Page fails on this, passes this, others
non-applicable. We run the tools against that page. How do we check the
rule is accurate.
... We can't compare them without a mapping to a success criteria.
Alistair: Using a test suite which we all agree on is the best way from my perspective to benchmark.
Wilco: We don't know if a test catches all the issues. How
do we find these issue.
... If we compare the results from loads of people writing reports - we
could see where they disagree with our rules.
Shadi: Good discussion. We have different thoughts in the group, which is good.