Accessibility Conformance Testing Teleconference

ACT TF at CSUN 2017

Wilco: you guys wanted to talk about CSUN. From our side we are going to send in a proposal to CSUN about the deliverables we have been working on. Deadline is tomorrow

Stein Erik: we just wanted it on the agenda to make sure that we promote the work we are doing here. As long as there is something going on, that is great

Wilco: we will send out the proposal to the mailing list
... Mary Jo, can you send it out to the mailing list? With the changed title...

<Kathy> yes

Mary Jo: I will send it

Wilco: Are everyone going to be at CSUN?

yes...

Reviewing ACT Rules Format https://www.w3.org/2002/09/wbs/93339/ACT_Sep2017/

<Wilco> https://www.w3.org/2002/09/wbs/93339/ACT_Sep2017/results

Wilco: We put up a survey to get the conversation going on the Rules Format. Got a couple of responses so far

<Wilco> https://github.com/w3c/wcag-act/issues/109

Tobias responded. First comment is on an open issue regarding the input types

Wilco: Did you have a chance to talk about how to get the ACT Rules Format into work in your organizations?

Stein Erik: We are just emerging from a big release. We are starting up the work tomorrow on how to implement the ACT Rules into our own tool and starting up work on writing some of our own rules in ACT Format. So within the next couple of months

Charu: October is going to be hectic. My personal view is that we are going to be able to focus on this after mid-November at least
... I'm working on getting our actual rule developers to participate in the rule creation group

Kathy: Had not had a chance to look at where we are in relation to test driving rules

Alistair: Had not had a chance to look at it either

Wilco: Are you still interested in doing this? And in which time frame do expect to see it done?

Alistair: Any large chance, and this would be a large chance, requires a business case first

Wilco: The purpose would be to see whether it is feasible and then use that as feedback

Stein Erik: Are there any hurdles that are stopping us all? It seems that we are pretty much in the same situation. Are there any commen blockers that could be identified and removed? In the document or otherwise?

Shadi: I don't think so, the document has been worked on for a long time, we need to put it into to practice to refine it further

Shadi: We need to find the real issues, so we are not wasting time
... We do not require any real change from the organisations, it's the assessment that we need at this point
... Wilco, you said you had a script for transforming the rules. Maybe this is something other should have a look at, just exporting the rules

Stein Erik: It might help to know what the minimum requirements are, and yes, sharing resources will also help

<shadi> https://www.w3.org/TR/act-rules-format/

Alistair: Is there a place where I can just see a test in this format? That would help me figure out how much is required

<Wilco> https://w3c.github.io/wcag-act-rules/rules/ACT-R1.html

<shadi> https://w3c.github.io/wcag-act-rules/

Wilco: yes, we created example rules to go with the Rules Format

Shadi: There are three sample rules

Wilco: Yes, as well as test cases that go with each

Shadi: Is it linked from the Wiki?
... I will link to it from the Wiki after the call

Alistair: What level of complexity can this thing do before it falls over? e.g. a function that loops through headings and put them into a function...

Wilco: This is like documentation

<Wilco> https://github.com/dequelabs/axe-core/blob/develop/doc/act-rules-format.md

Alistair: Shadi said you had some kind of parser to do it?

Wilco: No, I wrote out how to do it
... It's fairly straightforward, naming and replacing a few things. There are some things you can do in aXe rules that you can't do in ACT and vice versa, and I tried to write this out
... you don't actually have to describe how you got the foreground color and background color

Alistair: I want to try to make people come up with the fringe cases. We don't need these kind of descriptions, people can just look at the code

Wilco: If anyone has access to the code...

Alistair: At what point do you tell me how to write the code?

Wilco: They don't tell you how to write the code, they tell in fairly high level plain text how to perform the test

Alistair: Step 1 etc. that is fairly low level

Wilco: yes, we are trying to be unambigious

Alistair: I'm worried about spending too much time writing step 1, step 2 processes. The important thing is the quick summary and the expected outcomes

Shadi: yes, and we have decided that each ACT rule will be accompanied by test cases, that will be showing the fringes of this test. Are these the things you are looking for, Alistair?

Alistair: Yes, my developers say: just give me a piece of code that should fail this test. We don't go back and explain how to write the code that is already written. It will be a major job to go back and align

<Wilco> https://w3c.github.io/wcag-act-rules/rule-tests/ACT-R2-tests.html

Shadi: We have decided that we want both audiences. Also manual testers or people who want to know what this test is doing without having access to your compiled code. But then also have the test cases for the situation you describe. Yes, it should all be test case driven

Wilco: We have test cases. There are test cases linked in the bottom from all rules. It's part of the Rules Format

<Wilco> https://github.com/dequelabs/axe-core/issues/531

Alistair: In my opinion we would be able to implement everything pretty fast, except rewriting test procedures

Wilco: I have started using them, this is a rule proposed to aXe core a few weeks ago. First describing on a high level what the rule is going to do, so we can find out if there are false positives etc. before we actually implement the rule
... and if people ask about what a rule does, giving them the code is rarely helpful, it's better to give them the description

Alistair: This is pretty simple however, what about a function that calculates if there is an accessible name? That could be quite a lengthy process

Wilco: In auto-WCAG we have referred to the accessible name calculation, which is publicly available.
... Does the rest of you have questions about the level of detail required here?

Stein Erik: From our side the point of the ACT rules as a whole is to be able to present documentation on our rules to help people to decide on the reliability of rules. Transparency and accuracy is key here

Alistair: Couldn't this not just be done by running my rules on my test suite?

Stein Erik: Yes, but if it's the same author of the rules and the test suite, it could go in one direction or the other

Shadi: You don't have to write out everything in steps unless you want to contribute rules
... Or you could just implement existing test rules

Alistair: There are a million ways to build a function, do I need to follow the steps to say that I confirm or is it enough to match the test cases?

<Skotkjerra> Have to run. thanks, all!

Wilco: If you find a more performant way of implementing it, you are free to do it

Alistair: Should there be a "my stuff passes, but I did it in my own way" declaration?
... I'm saying, we don't need to write out the steps, as long as it passes

Wilco: I believe we need it, it's documentation, it's about transparency

Alistair: I value time, we need things to go quickly, we don't want to spend time on documentation, if we can just run it on a test suite. Especially when we have more than 400 rules

Shadi: Can we write somewhere in the format that the programming code in the tool might deviate from the description, but that the description is created to be human readable. But as long as the input and test cases matches, it follows the standard

Alistair: Excactly!

Shadi: I don't see why we shouldn't add this

Wilco: Yeah, I have no problem with that
... Does this alleviate your concern about the amount of documentation?

Alistair: Yes

Wilco: I think the most valuable thing people can contribute are test cases
... Then we can always find out who should describe it. Test cases are probably more important than the descriptions here
... I will add an issue for this
... Where are we on the agenda... We sort of did test run and test case suite

Kathy: I agree with a lot f what Alistair said

Charu: I think it was good that we had this discussion, thank you Alistair for asking so many questions, it also answered questions that I had
... was I was going to do was propose this format to my developer and ask how much extra work this was going to create for him
... I personally feel that these steps might help us find fringe cases that we have missed
... I think it's a good idea going forward to compare the steps and see how we differ and if there is fringe cases we might have missed

Wilco: Thank you for bringing this up, Alistair