See also: IRC log
saz: EC funded BenToWeb to work on benchmarking tools
they have been working with us on test samples
spent time developing tests following these procedures
Also worked on metadata for tests
e.g., whether the test a passing or failing condition etc.
Wiki repository of test samples
have explored other options for the test harness
also looked at Mobile Web test harness
One issue has been that since WCAG was changing, hard to keep up
We have a repository of unreviewed test samples
and WCAG seems to be stabalizing
Do we want to keep structure of task force or look at other options for doing this work?
Note BenToWeb project has ended, so reduced participation from that area
Also, since BenToWeb people submitted tests, didn't think appropriate for them to review them
cs: There are ~200 test samples, mostly mapped to May 2007 WD
and some mapped to April 2006 WD
There are also ~400 waiting to be uploaded
also mapped to May 2007 WD
so total of 609 in various states
bc: relationship of those test samples to the techniques?
cs: there is a mapping from test case to techniques
don't have test cases for all techniques
239 test cases don't map to any techniques
some of these were created before techniques were created
many of them of failure techniques
lgr: these test cases may be what we need for Accessibility Support evidence
Message about SC without failure techniques
bc: might be able to use test cases as well as other things for the data gathering
mc: test cases that state they should pass can be run through UAs to see if they do in fact
cs: more than half of the test cases are for failure conditions
don't know about relationship to accessibility support - some of this was developed when we had the baseline concept
lgr: this might be only a starting point, then, but a huge start
we're still trying to figure out to gather, record, and report on data about accessibility support
saz: there's a lot of metadata availalbe to help
lgr: need info about combinations of browsers and AT
wouldn't necessarily capture with test file, but would be part of overall db
cs: metadata has section for technologies used in test sample
even specific features
lgr: very helpful
mc: so we've got a lot of resources
still more to add
and need to review/approve it all
can use for accessibility support evidnece
lgr: need to get them in W3C space
and worry about persistence of resource
saz: can post easily
but what do we do with it from there
would like a harness that's easier than a wiki
that's a short-term project
validating the test samples would be an ongoing project
especially since we want people to submit test cases
cs: (noted earlier most of the test cases are in English)
lgr: would like test cases in other languages
cs: some of the unsubmitted test cases still under review for certain structural things
mc: ask about level of particpation in TSDTF given BenToWeb closed
saz: now have only 3 regular contributors
Carlos Iglesias, Christophe Strobbe, Tim Boland are the main contributors
Carlos Velasco, Shadi Abou-Zahra, Michael Cooper are peripheral contributors
We're hoping as WCAG stabailizes that some WCAG resources can come into this work
mc: Is it better to get people from WCAG WG into TF, or move TF into WCAG WG?
saz: would be ok to dissolve TF and move to WCAG WG
but also ok to work separately
lgr: This is a big chunk, and there are other big chunks
so the WG will be busy
What is procedure if WG reviews test cases and rejects some?
saz: The review process is spelled out in the process document
though it doesn't prescribe what WCAG WG would do
substantial fixes would mean test case sent back to submitter, asked to edit and resubmit
lgr: who are the submitters?
saz: BenToWeb tests submitted in bulk, but also anticipate individual submissions in the future
The BenToWeb tests all go to an email address, there will be someone at the other end
mc: Exploring structure - TSDTF working separately has allowed focus, but low on radar of WCAG WG; within WCAG WG would be higher radar but harder to organize priorities
bc: always a resource issue
lgr: there has been some exploration of resources but unsure of tieline
mc: would review of tests be doable during CR phase?
lgr: hopefully; issues about availability of WG members to work on this
saz: Joint with ERTWG means we have a different skillset available
structural reviews can be done outside WCAG WG, just need a content review
bc: ways to tie test samples to existing techniques?
saz: would be useful to share example code etc.
mc: should be able to automate references from techniques to applicable tests
bc: issues around tests in various places that aren't clearly tied to techniques
also there are tests that were published but not approved by WG, but URI persistence an issue
cs: task force following a rule that every test sample must map to a technique
Mapping to techniques and failures
bc: concern about unapproved tests
mc: TSDTF requires WCAG WG approval
lgr: need a plan for WCAG review
saz: need to get through the structural review stages as well
working within WCAG WG would simplify procedure by a step
staying in Task Force, we need to collect resources
lgr: we need to do something about accessibility supported, undrafted sufficient techniques, reviewing implementations, respoinding to comments on implementations
these are competing priorities
what happens after WG approval?
saz: we're not even really that far yet
mc: Let's check in in two weeks
MC and SAZ to float things around meanwhile
Continue this discusison in the 25 April meeting (regrets: LGR)