14:05:20 <RRSAgent> RRSAgent has joined #wcag-act
14:05:20 <RRSAgent> logging to http://www.w3.org/2016/10/05-wcag-act-irc
14:05:22 <trackbot> RRSAgent, make logs public
14:05:24 <trackbot> Zakim, this will be
14:05:24 <Zakim> I don't understand 'this will be', trackbot
14:05:25 <trackbot> Meeting: Accessibility Conformance Testing Teleconference
14:05:25 <trackbot> Date: 05 October 2016
14:05:33 <rdeltour> : scribenick: rdeltour
14:05:37 <Wilco> agenda+  Availability survey
14:05:39 <Wilco> agenda+ ISO take-aways, Alistair & Charu
14:05:42 <Wilco> agenda+ ARIA test harness take-aways, Wilco
14:05:44 <Wilco> agenda+ Web platform test harness
14:05:47 <Wilco> agenda+ Look at github frontend
14:05:51 <Wilco> agenda+ Open Actions
14:05:54 <Wilco> agenda+ ACT Planning
14:06:03 <Wilco> present+ wilco
14:06:05 <rdeltour> present+
14:06:10 <shadi> present+ shadi
14:06:18 <Wilco> zakim, next item
14:06:18 <Zakim> agendum 1. "Availability survey" taken up [from Wilco]
14:06:30 <rdeltour> Topic: Availability survey
14:06:34 <Wilco> https://www.w3.org/2002/09/wbs/93339/availability/
14:06:35 <shadi> https://www.w3.org/2002/09/wbs/93339/availability/
14:06:40 <rdeltour> wilco: shadi set up a survey for our availability
14:06:48 <rdeltour> ... please fill it out!
14:07:51 <rdeltour> shadi: you can keep updating the survey, it will keep the latest entry
14:08:01 <Wilco> zakim, next item
14:08:01 <Zakim> agendum 2. "ISO take-aways, Alistair & Charu" taken up [from Wilco]
14:08:06 <rdeltour> ... before or after the call
14:08:16 <rdeltour> Topic: ISO take-aways
14:08:49 <rdeltour> alistair: it was an overview of the good practice people provide on acceptance test
14:08:55 <rdeltour> ... it's an area we're interested in
14:09:02 <rdeltour> ... we're looking for features
14:09:13 <rdeltour> ... positive or negative features, how to write those up
14:09:29 <rdeltour> ... then best practices on how to write those tests
14:09:31 <Kathy> Kathy has joined #wcag-act
14:09:38 <rdeltour> ... not rocket science, but good advice
14:09:57 <rdeltour> ... basics that software testers learn
14:10:07 <shadi> q+
14:10:12 <rdeltour> wilco: do we have both postiive and negative feature tests?
14:10:24 <rdeltour> alistair: failure techniques are generating negatives
14:10:34 <rdeltour> ... success techniques generate positives
14:10:40 <rdeltour> ... we can look at negatives first
14:11:00 <Kathy> present+ Kathy
14:11:07 <rdeltour> wilco: looks good, we should take that into consideration
14:11:22 <shadi> present+ Alistair
14:11:22 <rdeltour> ... a11y testing does have both pos and neg testing
14:11:36 <rdeltour> ... looking at axe and other tools, there's a clear distinction
14:11:42 <rdeltour> ... it seems it makes a lot of sense
14:12:08 <rdeltour> ... what I still don't think we have is the ability to say "pass"
14:12:20 <rdeltour> ... so it's non-conformance testing instead of conformance testing
14:12:28 <rdeltour> ... is there anything about that in ISO?
14:12:41 <rdeltour> alistair: the ISO stuff doesn't go in these details
14:12:51 <rdeltour> ... it's more about writing standards
14:13:13 <rdeltour> wilco: so we'd have to decide on whether we want to do that
14:13:26 <rdeltour> alistair: in my experience you settle on one way
14:13:55 <rdeltour> ... what we want to decide is whether the end product is a claim or rather "you have not done this"
14:14:03 <rdeltour> wilco: it seems to me we're aiming at the latter
14:14:10 <rdeltour> alistair: right
14:14:27 <rdeltour> wilco: did you work with Charu?
14:14:39 <rdeltour> alistair: no, just did this 20 mins in the morning ;-)
14:14:56 <Wilco> q?
14:15:02 <shadi> ack me
14:15:03 <Wilco> ack s
14:15:21 <rdeltour> shadi: things we can start putting in requirements:
14:15:29 <rdeltour> ... atomicity: how atomic is a test
14:15:49 <rdeltour> ... another question is what we just discussed about positives or negatives
14:16:02 <rdeltour> ... I feel a lot of this is good stuff to put in some kind of requirements
14:16:19 <rdeltour> ... aomicity, relation to SC
14:16:31 <rdeltour> ... translate what we mean by atomic in the context of WCAG
14:16:41 <rdeltour> s/aomicity/atomicity/
14:16:53 <rdeltour> alistair: the 1st thing is not to build massive tests
14:17:03 <rdeltour> ... for SC, look at the techniques you can use to meet that
14:17:12 <rdeltour> ... then break it down to smaller tests
14:17:37 <rdeltour> ... that means we need to know what techniques we're following
14:17:46 <rdeltour> ... another thing is non technique tests
14:18:03 <rdeltour> ... e.g. an image: a technique test is test if there's an aria label value
14:18:23 <rdeltour> ... a non-technique test is "does the image has a computed name value"
14:18:53 <rdeltour> shadi: is this now the right time to discuss this and put that in reqs?
14:19:19 <rdeltour> ... so that we have an understanding, and detailed later in spec
14:19:24 <Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Framework_Requirements
14:19:25 <rdeltour> ... right now our reqs are very high level
14:19:52 <rdeltour> alistair: ultimately we want to achieve massive amount of test coverage
14:20:02 <rdeltour> ... we'll want both approaches
14:20:15 <rdeltour> ... top-down: you don't know the udnerlying techniques
14:20:29 <rdeltour> ... bottom-up: you know the techniques,, tests if implemented well
14:20:41 <rdeltour> wilco: I've looked at other areas, e.g. SVG
14:21:16 <rdeltour> ... if you look to make assesments about a11y support requirements
14:21:30 <rdeltour> ... you need to consider which exact attribute led to an accessible name
14:21:41 <rdeltour> ... it goes back to techniques
14:22:00 <rdeltour> ... you may have used a known technique, or another one. we need to report on these differences
14:22:13 <rdeltour> alistair: it's almost the difference between warnings and fails
14:22:32 <rdeltour> ... you won't be able to say "you've done it wrong" if you look only at aria-label for instance
14:22:58 <shadi> q+
14:23:17 <rdeltour> ... if you look at the outcome, it's about the SC
14:23:43 <rdeltour> ... if you look at the techniques, you can say "yes you pass" but maybe a better way is another technique (???)
14:23:49 <shadi> ack me
14:24:17 <rdeltour> shadi: what I don't know in our fwk is if we're expecting some kind of a11y support as input of our test
14:24:23 <Kathy> q+
14:24:24 <rdeltour> ... assumptions as input
14:24:36 <rdeltour> ... for that baseline you define which does or doesn't work
14:24:51 <rdeltour> ... I don't want to get into the specifics right now
14:25:09 <rdeltour> ... is this the stufff we want in our req document?
14:25:24 <agarrison> q+
14:25:28 <rdeltour> kathy: on the a11y support side we'll always have difficult times
14:25:38 <rdeltour> ... to say this technique is supported or not
14:25:46 <rdeltour> ... so many different components are part of it
14:25:59 <rdeltour> ... we can't really have that in here. w/b great if we could
14:26:17 <shadi> q+
14:26:19 <rdeltour> ... but I'd be concerned if we tried to put a11y support  there, if only for maintenance difficulty
14:26:24 <shadi> ack kathy
14:26:37 <rdeltour> wilco: we've discussed a11y support before, when we started ACT
14:26:49 <rdeltour> ... we don't want to have that baked into rules, for the reasones you mentioned
14:26:59 <rdeltour> ... you need to decide on granularity
14:27:21 <rdeltour> ... if we can achieve that, ideally, is to get to a point where the user has some sort of baseline
14:27:30 <rdeltour> ... (probably up to the tools developers)
14:28:04 <rdeltour> ... so you can say "given this support", maybe a matrix, as input in a rule
14:28:15 <rdeltour> ... so we're not stuck in a11y support issues
14:28:32 <rdeltour> alistair: I agree 100% with you both
14:28:37 <shadi> ack aga
14:28:56 <rdeltour> ... definitely not put that into the tests themeselves, but make sure we have the broader coverage
14:29:16 <rdeltour> ... the tools developer can apply some weightings, but that's not for us to hardcode in the tests
14:29:33 <rdeltour> wilco: what I want is to put a11y support data as input, and build results based on that
14:29:47 <rdeltour> alistair: it looks like something tools developers should put in
14:29:58 <rdeltour> ... we're talking about how to write tests
14:30:05 <rdeltour> ... how to utilize them is different
14:30:15 <rdeltour> shadi: isn't it part of how to write tests?
14:30:31 <rdeltour> ... I agree that we shouldn't hard code a11y support in tests
14:30:59 <rdeltour> ... whether we want a mechanism that tool developers or a user puts a database of assumptions in
14:31:21 <rdeltour> alistair: it's still metadata related to the test, so you need to maintain it
14:31:37 <rdeltour> shadi: let's take a simple example, aria-label
14:31:45 <rdeltour> ... right now, the test is to check for aria-label
14:32:04 <rdeltour> ... if we're not considering a11y support, it's just a test to see if aria-label exists
14:32:38 <rdeltour> ... a 2nd approach, is to have somewhere else in the test rule a statement to say "this rule relies on aria-label to be implemented"
14:32:54 <rdeltour> ... so when you implement, you can look at these statements
14:33:22 <rdeltour> ... in the reqs, we want to come to an agreement and write that down
14:33:34 <rdeltour> ... how our fwk deals with a11y support statements
14:33:53 <shadi> ack me
14:33:56 <rdeltour> alistair: I agree
14:34:16 <rdeltour> ... you need to push the information in a kind of "relies upon" section
14:34:51 <rdeltour> kathy: I think that would work
14:35:06 <rdeltour> ... I agree with shadi
14:35:10 <rdeltour> ... the more information the better
14:35:27 <rdeltour> ... we just need to be careful about not including things that would be impossible to maintain
14:35:58 <rdeltour> wilco: should we add a resolution?
14:36:14 <rdeltour> shadi: my suggestion is to put that in the requirements document, or record a resolution to do so
14:36:41 <rdeltour> ... and for people to think about other headings to put in the requirements document
14:37:04 <rdeltour> ... you can send an email and request comments
14:37:54 <shadi> romain: to what extent do we want to say in the reqs how the rules will look like
14:38:05 <shadi> q+
14:38:35 <shadi> ...like reqs for reqs
14:38:56 <rdeltour> wilco: the way I see it is: what features would I want the rules to have, what are the quality aspects
14:39:09 <rdeltour> ... the way to deal with a11y support is one of them
14:39:11 <shadi> ack me
14:39:24 <rdeltour> shadi: we can take a middle way
14:40:02 <rdeltour> ... I don't want to wordsmith, but we can find a way to turn that in a req
14:40:18 <rdeltour> ... we don't have to take a decision right now
14:40:28 <rdeltour> ... we might later have other areas that influence this decision
14:40:46 <rdeltour> ... for now, we need to gather these topics, to have a kind of skeleton for our fwk
14:40:51 <rdeltour> +1
14:40:55 <Wilco> zakim, next item
14:40:55 <Zakim> agendum 3. "ARIA test harness take-aways, Wilco" taken up [from Wilco]
14:41:08 <Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources#Take-aways_from_WPT_.26_ARIA_Test_Harness
14:41:20 <rdeltour> topic: ARIA test harness
14:41:28 <rdeltour> wilco: little documentation, but interesting ideas
14:41:49 <rdeltour> ... WPT have 5 test types
14:42:09 <rdeltour> ... some of them pretty similar to what we'll be doing
14:42:35 <rdeltour> ... ** goes over the 5 tests **
14:42:46 <rdeltour> s/tests/test types/
14:43:32 <rdeltour> ... about the requirements, we're aligned on atomicity, being short minimal, cross platform
14:43:47 <rdeltour> ... e.g. "no proprietary techniques"
14:44:04 <rdeltour> ... they also talk about self-contained tests
14:44:14 <rdeltour> ... all their tests are written in HTML
14:44:33 <rdeltour> ... we don't use a format in auto-wcag, but WPT is explicit about it
14:44:46 <rdeltour> ... do we want to say what format we want the rules to be written in?
14:45:12 <rdeltour> shadi: can you elaborate?
14:45:24 <rdeltour> wilco: all their tests are in HTML
14:46:21 <rdeltour> ... their tests are really atomic, which is very powerful
14:46:42 <rdeltour> alistair: self-contained tests is very important
14:46:52 <rdeltour> ... in auto-wcag we have some dependencies between tests
14:47:08 <rdeltour> wilco: the reason for that is atomicity
14:47:14 <rdeltour> ... it results in very big rules
14:48:13 <rdeltour> ... point 6 is about "accessible name", "role". we had that disucssion
14:48:25 <rdeltour> ... all tests  are "pass" or "fail" tests
14:49:04 <rdeltour> ... do we need an output format? or do we only want a pass/fail
14:49:13 <rdeltour> alistair: I think you'll need more than that
14:49:30 <rdeltour> ... non-applicable, pass/fail, review
14:49:51 <rdeltour> wilco: do we really need something like EARL or can we make it work with something simpler
14:49:59 <rdeltour> ... (boolean result)
14:50:24 <rdeltour> shadi: EARL is trying to describe results, which can come from any model (binary or not)
14:50:38 <rdeltour> ... we concluded we have 4/5 types
14:50:47 <rdeltour> ... pass, fail, untested, etc
14:51:19 <rdeltour> ... my impression is that e.g. "cannot tell" must be in there
14:51:36 <rdeltour> ... there may be triggers for a test, but the test won't be able to tell pass or fail
14:51:48 <rdeltour> ... "non-applicable" might not be needed
14:52:34 <rdeltour> ... in terms of warnings, they are usuall essentially a "pass", just ugly ones
14:52:45 <rdeltour> s/usuall/usually/
14:52:58 <rdeltour> ... we may not want a value for "warning", it's just an attrribute
14:53:11 <rdeltour> ... some tools like to say "nearly pass"
14:53:25 <rdeltour> ... if it's a very minor issue. it kinda works.
14:53:52 <rdeltour> ... it's actually a fail. you have a problem, it just doesn't have much impact
14:53:57 <rdeltour> ... these are nuances on pass
14:54:08 <rdeltour> ... that's why you have extra info in EARL
14:54:37 <agarrison> q+
14:54:37 <rdeltour> wilco: we'll definitely to have this discussion again.
14:54:48 <rdeltour> shadi: we can untangle this from EARL
14:55:04 <rdeltour> ... for now, we can say we'll need to define the results (with potential candidates)
14:55:12 <rdeltour> ... when we get into that, we can discuss it more
14:55:23 <rdeltour> alistair: it needs to be totally non-subjective
14:55:35 <rdeltour> ... "nearly pass" is very subjective
14:55:47 <rdeltour> ... "non-applicable" is not subjective, and very useful to know
14:56:02 <rdeltour> ... not having it may lead to wrong conclusions
14:56:14 <rdeltour> ... all of these have to be non-subjective
14:56:18 <rdeltour> wilco: +1
14:56:34 <rdeltour> ... shadi, have you looked at github?
14:56:43 <rdeltour> shadi: you all should have access and be able to edit
14:56:53 <rdeltour> ... I set up a team
14:57:12 <rdeltour> ... I've not set up the automated publishing process
14:57:16 <Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/track/
14:57:28 <rdeltour> wilco: quick look at action items
14:59:00 <rdeltour> ... last thing: we have our planning document
14:59:11 <Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Month_by_month_plan
14:59:59 <rdeltour> ... by december-ish we'll need agreement on the requirement, and a first draft
15:00:45 <rdeltour> trackbot, end meeting
15:00:45 <trackbot> Zakim, list attendees
15:00:47 <Zakim> As of this point the attendees have been Wilco, Katie, MaryJo, Charu, Alistair, Jemma, rdeltour, shadi, Kathy
15:00:53 <trackbot> RRSAgent, please draft minutes
15:00:53 <RRSAgent> I have made the request to generate http://www.w3.org/2016/10/05-wcag-act-minutes.html trackbot
15:00:54 <trackbot> RRSAgent, bye
15:00:54 <RRSAgent> I see no action items