Accessibility Conformance Testing Teleconference

19 Jun 2017

See also: IRC log


MaryJo, Kathy, Manoj, Charu, Anne, Alistair, Wilco, Chris, Shadi
Shadi, Wilco, MaryJo


Book and register for TPAC: Thurs-Fri is the ACT meeting

Shadi: TPAC meeting on Thurs / Friday for this group.
... Tech plenary is in SF in the USA.
... Please book now.
... Hope some of you will be able to attend.

Survey on availability for calls https://www.w3.org/2002/09/wbs/93339/availability/

Shadi: Reminder - let us know your availability.
... Please keep it up to date.

Pull request 94 - Merge change log with version history

<shadi> https://github.com/w3c/wcag-act/pull/94

<shadi> https://github.com/w3c/wcag-act/pull/94/files

Shadi: Changes - versioning.
... It could have been that we more interested in moving to a change log, rather than semantic versioning.

Moe: Should be use a better adjective than huge.

<MoeKraft> It is recommended that for significant changes, such as those that change the scope of a rule, a new rule is created and the old rule is deprecated.

Shadi: Any thoughts on this.
... Moe that looks simpler.

<cpandhi> + 1 to Moe's change

Shadi: Kathy - changes that Moe suggested are better.
... Alistair as you have concerns on how versioning is done, please look at that.
... Any other comments.

Alistair: I think the initial comment was people don't like versions to change to quickly.

Shadi: In the new concept, minor changes would be added to the change list. But major changes would change the test number.

Anne: Would a change to the rule change, by putting in a change to the process in the test, would change the number.

Alistair: But, what would happen if a person tests now, and then tests 3 months from now - and loads of ids have changed.

Wilco: If you break a rule up, then you'd have to give the sub-rules a different id.

Anne: What is an update and what is a significant change.
... Wilco things only really big changes should change the id; and Shadi suggests smaller changes might change the id.
... Initially the simplification is OK; but we need to be clear on when something needs a changed id number.
... Tracking things over time is really important, so I wouldn't want to change ids too much.

Charu: it is good to be able to have history; so we have to be clear about what would cause a significant change.
... What if it was a bug fix, would that force a new id?

Shadi: We need a way of capturing changes to tests; and we need to properly understand "significant" in terms of something which would require a new id.

Wilco: It seems that that section about ids is common sense. If it comes at the cost of loosing your history.

Shadi: Sounds to me like we need more clarity in what to do.

Wilco: take out the rule about create a new id if there are big changes.

Charu: Issue we see in our rules. One of complaints is, a result is changed through a bug fix.
... People's reports being changed is something they don't like.
... What we do is to have a rule set. Then we release one or two rule sets per year.

Alistair: Bug fixes in tests are a good thing.

Shadi: If you test a point x and y you don't know if the content changed; or the rule changed.
... So we need the log.
... You'd need to go to the change log to see if a test has changed.

<MoeKraft> Significant changes, for example those that change the scope of a rule, may require the creation of a new rule and deprecation of the current rule.

Moe: Thinking about to Wilco's suggestion. Maybe we want to soften the second paragraph.
... The change is "may"
... Its not prescriptive.

Shadi: Thoughts.

Wilco: I like the change.

<Wilco_> +1 to example

<cpandhi> +1 to example

Alistair: With the word "may" you would need an example.
... Example: if you split a test as you have discovered it is too broad. The two new tests would then each get a new id, and the old broad test would be deprecated.

Shadi: Do we need an example of a minor change.

Wilco: Don't think its necessary.

Anne: When developers read through ACT framework they wanted more examples, and two examples when possible.
... We would like a minor change example.

Alistair: Let the minor change example be a bug fix.

Wilco: I'll put the major and minor examples in.

Moe: Pull request added.

Shadi: Any other comments.
... Will you notify mailing lists of the changes.

ACT Review Process

<shadi> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Review_Process

Shadi: Take a moment to look through it.
... Have we all read it?
... We want enough incentives so that the community starts to review their own tests.
... You ask them to get a certain number of implementations.
... Once the contributor can show this validation, then it is easier.
... Basic idea behind this.
... What do people think?
... Is it workable, or too optimistic.

Alistair: How do people show the rule to someone who implements it.

Shadi: They would put the rule in the GitHub repo with a flag marked untested.
... You keep on updating your rule in GitHub.
... We have the rule, test cases and implementations.

Alistair: What if loads of people place rules which are similar. How do they work together. Then how many rules might we expect.

<Wilco_> On this point, what we discussed previously was that we might publish 'draft' rules that haven't been implemented in enough places yet, as a way to get that ball rolling

Shadi: People would have to try to collaborate.

Charu: In GitHub we can have beta rules.
... We could have loads of tests in there... there needs to be a way for people to comb through rules.

Shadi: We could weed out certain tests through formats.
... What if different companies submitted overlapping tests - they each have their reasons for having their own tests.

Shadi - discussion would have to be peer-to-peer.

Shadi: 4 more minutes.

Shadi - Let's give people more time to work through this - shall we use mailing list, GitHub.

Wilco: Let's use mailing list.

Shadi: You could help move this forward. What would incentivise you to put rules forwards.

Wilco: Worth us checking back with our colleagues.

Alistair: How do we discuss the tests on GitHub.

Shadi: Implementations would be what we'd be interested in.

Kathy: Our testing is different from the tests that aXe can run.

Shadi: Either we have a group that checks rules centrally; or you try and get people to implement your rules in other tools.

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2017/06/19 17:12:56 $