WAI Testing Caucus

24 Jan 2007


See also: IRC log


Michael_Cooper, David_Poehlman, Shadi_About-Zahra, Don_Evans, Andrew_Arch, Alan_Chuter, Henny_Swan, Sylvie_Duchateau, Tim_Boland, Cathy_Laws, Becky_Gibson, Shawn_Henry, Helle_Bjarnø, Jan_Richards, Jim_Allan, Rich_Schwerdtfeger, Christophe_Strobbe, Janina_Sajka, Al_Gilman
David_Poehlman, jallan, Tim_Boland, Christophe_Strobbe




<MichaelC> scribe: David_Poehlman

<MichaelC> scribeNick: dpoehlman

Agenda review Michael: goals, develop roadmap and define commonalities among groups.

test samples taskforce: sh

WCAGWG has final say We are going to collect test samples. We are going to work with Shawn and others in eo.

Metadata language called tcdl test description language. describes user senarios etc and we are using only a subset of this language.

aria test suites: Michael.

Michael briefly describes aria.

aria test suites provide evaluatlrs with knowledge of states and rolles property attachment. cl:

Jim: aria targets accessability, other specifications raise implementation questions for all including accessability.

<MichaelC> Need expected behaviour of UA for widgets; test suites validate that

<MichaelC> We don’t all agree on what the proper behaviour of widgets should be

<becka11y> http://developer.mozilla.org/en/docs/Accessible_DHTML#Sample_widgets

<MichaelC> http://www.mozilla.org/access/qa/testcases

RESOLUTION: terminology needs to be consistant and clear.

<jallan> need to use existing definitions when available

uaawg test cases: jim:

<becka11y> ARIA samples related to AJAX: http://accessibleajax.clcworld.net/

Jim describes test suites.

Shadi describes test samples and the taskforce.

Jim continues test suite description.

Shadi: We need a sharable database which can provide different views depending on what is required.

Jim: you can edite the test suite, but you are editing code in a form which will result in a page of html being developped and loaded but it is not a streightforward process.

cl: maybe a wiki?

Shadi: What about server side scripts?
... user agent stuff is different than content stuff.

au: tim:

tim: early work on an approach to testing tools.

Tim, testing the interface as well as the output.

Tim, some issues are keeping track of kinds of authoring tools.

Tim, taking quality assurance into account.

Tim: someone can emply an authoring tool they have never seen before. Would this be a valid test.
... in some cases, no.
... a tool developer could test the tool with knowledge of the test/success criteria.
... the tests and results need to be vailable in public domain.
... disclaimer in generated report and questions about submitter.
... where would ths repository be?
... Does this process encourage people to test?
... other questions.

Michael: How does a test verify that the content was developped accessibly.

tim: tests are against WCAG.
... What the tool produces would be in conformance with WCAG.

Jan: various tests.

tim: preview before Check:

david: are there are any accessible authorig tools?

Jan: there is an editor which is part of a content management system.

Shadi: an ATAG compliant authoring tool needs an evaluation tool.

qawg resources: tim:

tim: Quality assurance wg produced some materials.
... They opened in 01 and closed in 05.
... there is now a qa interest group.

Tim, there is a document "QA test faq" to guide test tool developpers.

tim: qa has a wiki.
... qa frameworks specifications guidelines.
... various other software testing resources.
... work on user agent problems, http header problems.

Michael, how do we fit into the qa group?

Tim: there is a series of requirments for test assertions test metada etc.
... QA has two chairs.


How Many members?

Tim: ten but anyone can contribute.

<Andrew> Tim: for other testing resources - http://www.softwareqatest.com/qatfaq1.html

<scribe> ACTION: tim, assemble list of non w3c related test resources. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action01]

<Tim> http://esw.w3.org/topic/QA

<Tim> QA Wiki Resource provided in preceding URL


<jallan> scribe: jallan


summary of current goals

mc: conformance to specs/recommendations
... test for content creation, rendering, content itself, content repair

bg: want to test wai guidelines, and for outside folks to test for conformatince

tb: goal? end to end accessibility - author puts content on web, and makes accessible to end user
... is what the author did realized by the user

mc: other goal-develop test cases to meet other goals

cl: consistent way to test same item across guidelines
... problem in UAAG, testing a technique, but there may be several ways to arrive at a solution.
... test cases are not normative, they are just samples,

mc: there may be normative test cases, i.e. is an authroing tool accessible yes or no

ag: within EARL there are mechanisms to bind test to a technique
... metadata to indicate sufficiency

mc: goal? joint repository for all of WAI.

ja: yes

jr: yes, for shared infrastructure, reporting, etc. even if tests are pulled from elsewhere

mc: add to goal list
... delineate test cases, help structure repository

ag: QA test taxonomy may be helpful, and test metadata

mc: ?goal to conform to QA recommendations
... start there and branch when necessary
... ?goal expect WAI specs to conform to QA spec recommendations

cl: how does collab with QA and WAI work?

tb: explains process, QA reviews spec as they are developed

** joint repository for all of WAI.

** expect WAI specs to conform to QA spec recommendations

** develop test cases to meet other goals

** test for content creation, rendering, content itself, content repair

** conformance to specs/recommendations

** end to end accessibility

** conform to QA recommendations

** test cases are not normative

** consitent way of testing same item across different areas

** share test utilities

ag: EO goals?

** EO create materials to explain testing (further down the road), and in human readable fashion

** improve communication between WAI WGs about testing

ag: in EO, goal should be, increase level of checking in site building,
... authors should check work, need to monitor
... make testing easier, authors might use tests more
... sampling, don't need a compliance suite for every provision, if there were a high level 20 items

tb: better to have something rather than everything

** encourage market uptake and use of testing procedures (key things, then all things)

cl: tests but also some reporting mechanism?

jr: works for atag and uaag, because few implementation. web content may be different

ag: a way to get traction, if people create reports, WAI can track usage

tb: CSS selectors has a report for conformance

mc: for reccommendation status, must show 2 implementations, so must have some reporting mechanism

** provide mechanism and structure for submitting implementation reports

ag: also test accounting

tb: QA has best practices that address this

mc: prioritization? or find commonality?

Commonality of test cases

mc: core and WG specific
... example Test case for images
... is alt there? y/n
... wcag - no is fail, yes is qualified pass
... atag if generated content looks like this pass, if not fail
... uaag....
... two test cases used in multiple ways
... are there many cases like this?
... are we going to be sharing similiar files?

ja: explains UAAG for image

mc: nuance, null alt, do UAAG and WCAG have same expectations
... C Ridpath alt tests have 10+ for image

tb: must have explicit expectations about requirements

ag: need to make distinctions for how a UA might repair poor alt

mc: make test case, could share, need specific test cases for each WG,
... help harmonize expectations

jr: different requirements for each WG for same test

tb: QA has info about this

mc: test procedures being different

<Tim> http://www.w3.org/QA/WG/2005/01/test-faq#good

<Tim> QA TESt FAQ What makes a good test?

mc: same test file, 3+ procedures dependent on WG requirements
... identify major types of procedures for each WG

dp: test cases? may not be files for every test,

mc: right

ag: in UAAG tests, there is a vpat test, user must be able to get conditional content, UAAG doesn't specify how to get conditional content
... create a reproducible test, through this gesture user can get content

cl: that information is in the report, not part of the procedure. the reporter must explain how they complied
... user can get content if they change this setting for perform this behavior
... very vendor/tool specific

ag: ubiquitious web, rebinding input events to access content

mc: need generic test case. use the UA mechanism to get this content
... 1 test case, multiple procedures, multiple expectiatons and outputs
... report results will be different for each vendor
... are there cases where test files are only applicable to 1 group, or just have a repository

bg: wcag would not need repair

jr: atag may have just procedures with no files

cl: chrome for UA would note have webfile or test

jr: atag has procedure, can you change system setting, does it impact renderend content

mc: use a sample file to run multiple procedures against

jr: atag-is help section of tool accessible

mc: may use wcag procedures and tests to test against help files.

cl: UAAG GL 6 http://cita.rehab.uiuc.edu/wai-eval/index-gl.php?ts_id=1&gl_id=6
... no tests

mc: but would need a test document to run against the processes

jr: uaag gl 11 http://cita.rehab.uiuc.edu/wai-eval/index-gl.php?ts_id=1&gl_id=11

ag: GL 6 would be place for WAI to shop DOM and other test suite for tests for implementation

saz: explains note>>procedure>>file>>result diagram
... files are shared

mc: procedures may not be shared

saz: all should be shared, with WG views and public view

mc: we are agreed that having a repository is good

ag: what about procedure sharing, and results

mc: files must be stimulus and response

ag: if each stimulus drive 1 assertion

mc: for everything the tool could evalutate, had average of 7 files to run against, and 1+ pass and fail condition
... for a given input, should be 1 result

saz: perhaps not
... need more specific examples of scenarios

ag: could use headers attrib to point to td, or describedby, etc.

mc: no, each condition is a separate test
... ** choosing which test to be accountable to is a different task for sufficiency
... depends on group doing the testing

---- break for lunch ----

<shawn> break-out group will come for lunch in a little bit

<shawn> shadi - michael - jim - anyone: would you ask if anyone is taking a cab to the airport this afternoon? Sylvie might want to share a cab this afternoon, and Henny later on.

<MichaelC> scribe: Tim_Boland

<MichaelC> scribeNick: Tim

How test cases might work

cs: intro to TCDL
... based on language from BentoWeb project (subset used)
... XML-based, divided into several sections
... formal metadata, technologies, testcase, rules, namespace mappings
... use CVS keywords for some of these
... Dublin Core is referenced also
... designation for terms is required or optional, with notes as needed

container elements for locations and techniques

shadi: ERT TF, instead of taking time and effort, used BentoWeb..

focussed on elements needed, minimum set of metadata..

points to WCAG SC and associated technique (different model?)

interest sought from WGs to work on metadata to describe test cases

mc: how does this relate to test files/test procedures?

cs: QA test taxonomy consulted - hard to adapt?

maybe different than what UAAG or ATAG might need?

mc: maybe need more structure for ATAG/UAAG - would it work with

this structure?

mc: want to take advantage of this work in our future efforts

maybe work out some examples to illustrate this?

ag: less structured information definitions may be more appropriate?
... common ground is more abstract in terms of description?

shadi: expert guidance should be in technique - took TCDL as is
... to JR and JA: is there interest in common metadata?

JR: would help to try out some examples

JA: useful also, need critical mass to contribute

mc: even if joint test files, still need test procedures

shadi: infrastructure/tools could still be in common

jr: accessible help files - utility that's a checker..

is that something metadata could handle?

<jallan> http://cita.rehab.uiuc.edu/wai-eval/show-test/index.php?test_id=234

ja: author bindings for accesskey

mc: are there elements of this that can be described using TCDL?

ja: multiple nuances to test case.. need to list everything needed

kathy: procedure is vague..output could be in different forms

ja: technology has changed since test was developed, so expectatations are higher now

cs: several issues with adaptation to TCDL

for instance, expected result is descriptive in this example..

shadi: both procedure and result included in WCAG technique

mc: test files should be shared in repository
... take subset of TCDL as file description, take WCAG test approach,..

but use same files and metadata language

ag: maybe appropriate for WCAG, not necessary so for UAAG/ATAG,..

but maybe more procedural description may be appropriate

shadi: files could be pointed to from different sources

mc: need distinction of what could be joint, and what needs to be separate
... need resources to exist, and test procedures need to get at them for various purposes
... what is in common besides test files?

jr: UAAG/ATAG have procedures/interaction recipes

mc: WCAG may as well?

ag: wcag text equivalent/uaag conditional content - in common?

mc: need wide deployment for wcag
... for wcag, a lot of evaluation can only be done manually
... is prerequisite tests in TCDL?

cs: not used now

shadi: conceptually possible in metadata

mc: testing framework needs to have way of expressing preconditions?

baseline questions

mc: framework for describing and storeing test procedures/files would

be things in common

kathy: and framework for reporting on test results

mc: tied in to procedure?

ag: EARL has some of this info?

shadi: how it was done, but no configuration info..

ja: this is where UAAG would come in

shadi: TCDL companion to EARL (under discussion)?

ag: archive in more detail than that specified in test procedure
... need to capture detailed experiences for richness

shadi: asserter can be both human and/or software

<jallan> http://cita.rehab.uiuc.edu/wai-eval/evals-cp.php?eval_id=13&cp_id=1

kathy: uaag many tests for checkpoint, how many do we need to pass to pass checkpoint?

shadi: EARL can provide such detailed reporting, but sequence/which tests to do..

were taken out

kathy: different from wcag perspective (based on instances) vs. uaag (based on test case)

jr: atag does both

mc: have in common: framework for storing test files, test procedures, expected results,..
... also common framework for executing test files?
... applications to facilitate execution of tests?

ja: need everything in one "package" to run tests

mc: test driver application is important - should be shared?

ag: cyclic graph of dependencies among groups?
... mutual capability to run one another's tests

jr: special uaag/atag dependency needs to be explored?

mc: resources may be available?

ja: test driver application should be self-contained

shadi: administrative development process is issue ..

support for test development within WGs is important

mc: wcag developing tests, where are other WGs?

ja: not actively developing uaag tests right now, still some issues, but fairly complete

kathy: some checkpoints don't have tests yet

mc: how much of what's there is repurposeable?

ja: a lot would be.. (for HTML4.01) (ARIA not covered)

shadi: uaag support in common infrastructure development work?

ja: resources scarce, but ua will be there..

<cklaws> what guys?

jr: nothing for atag1, some procedural stuff for atag2
... would like to help, but resources scarce

ja: maybe use atag work in ua

shadi: maybe use mobile web work also?

rich: pf taxonomy created which may be helpful?
... style guide under development also

ja: may also apply to atag as well

rich: need browser extensions to run these tests

ja: does wcag support dynamic html functionality (all ties together)?

mc: we hope wcag addresses this - some techniques, not many tests

cs: want to do, but haven't gotten there yet

mc: testing a bunch of technologies working together

cs: in TCDL, can partially describe this (doesn't go far enought for AJAX)

ag: can we see tests in accessibility APIs?

rich: possibly so (raven on alphaworks - rules based)

<cklaws> Raven on IBM alphaWorks: http://www.alphaworks.ibm.com/tech/raven

<scribe> ACTION: Rich to put Shadi and Barry Feigenbaum together on rule-based technology on raven [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action02]

<MichaelC> scribe: Christophe_Strobbe

<MichaelC> scribeNick: Christophe

Develop preliminary roadmap and timeline

mc: agreed we want common framework for testfiles and common framework for test procedures
... . Need to be able to run each other's tests.
... Way of activating tests.
... Share knowledge and resources about administrative aspects.
... Liaising with QA IG. Considered joint task force (QA - WAI WGs).
... WAI CG already co-ordinates communication; could also have task force for liasing with QA.

JR: Facilitatator needs to be someone with lots of time and/or lots of resources to do the main part of the work; could then call on busier people.

SAZ: TSD TF has two facilitators: one from WCAG WG, one from ERT WG. Co-facilitating would be an option for the liaising TF.

AG: Common frameworks: hard to get critical mass of resources to make this happen.
... Common functionality, yes; not necessarily common harness.

<JR> JR: correction: facilitator shouldn't just be a single person trying to call on other busy people

MC: TF will be responsible for narrowing the scope. Scope needs to be reasonable.

SAZ: (response to AG) ... Developing content of repository is something else.

AG: "Build it and they will come"? (..) How do we expect to see this get used?

JR: If you have test cases in repository without harness, who will run the tests? (...)

SAZ: [Tool by Jon Gunderson?] If we have someone to get this running, ... Some test cases are combinations of files. There's more to it than a single piece of HTML, so that tool would need to be changed.
... TSD TF didn't start developing a metadata format immediately; first looked at what existed. Similarly, there may be other tools to run tests. The TF would have to look for these first.

MC: TF could continue discussion and do research. Could be incubator TF?

SAZ: Lifetime for this work is too limited; should be real task force.

MC: TF picks up from the common understanding we reach today.

SAZ: WGs are in different stages regarding test files: some want them now, some later.
... Why would tney contribute if their need is still in the future?

AG: Feel need for more co-ordination at the techniques level.
... See payoff in co-ordination regarding techniques.

MC: Earlier, we saw this as a side benefit. Could make it a goal.
... Other possible outcome of meeting: continue what we were doing? Something in between?
... Long term, we would benefit from common repository and framework; short term, contributing is not obvious.
... Maybe move repository - of test files - to common location.
... E.g. test files UAAG test cases; go through TSD TF review process for WCAG.

JR: Who has programming skills etc. to create it and set it up?

MC: TSD TF has tools set up.

AG: QA has best practices regarding test suites.

SAZ: Not only a matter of programming skills but also time and resources.
... (response to RS): would need to port Jon Gunderson's ...

RS: We need to identify what we have; then build the roadmap.

JA: Stand-alone section on W3C space; application could be on UIUC servers.
... We have two different sets (see list of resources: UAAG) UAAG: XML DTD looks like early version of TCDL.

MC: Even basic task of getting test resources in one place is important to avoid duplication, but even that requires considerable resources.

SAZ: If we want to do this, we need commitment from each of the WGs to review the work.
... Need to contact chairs and other key people of these WGs.

MC: WCAG maybe not before June.

JA: Currently not enough resources...

MC: Strategies: how get people involved and keep them interested?
... If UAAG gets rechartered, it needs more people; WCAG could also use refresh of personnel.
... Maybe talk about it at conferences. Governments and generous corporations...

RS: Set format that everyone would need to convert test cases to?

MC: Would be nice in long term. First need to collect what is available, then ...

RS: E.g. Yahoo interface library: could that be used or would it need to be modified? Modification is lot of work.

MC: Agree. TCDL makes sure certain data are always available. Coding of test files: not necessary look the same.
... (...) Test cases do not need to look the same except when required for the purpose of the test.
... We have common interests and needs but appear to lack resources for a new task force.
... Maybe a report out of this meeting that describes to the WAI what we identified regarding resources and needs, what could happen, etc.
... This could ensure some kind of common purpose across WAI working groups in future.

SR: Regarding pulling this test suite together: Mozilla Foundation grants (cf AJAX ...)

CL: Accessibility of Firefox: XXX got ability to manage grants for resources working on Firefox-related topics.

MC: Who would get funded?

SR: Cf skill sets. Accumulating test suites from vendors; automate things; define testing methodology; ...

AG: Accessibility initiative may be wrong scope to take action on this. It does not matter who owns the repository. If other groups are more heavily into conformance demonstrations, ...

SAZ: Those test suites are for our specs and technologies: WCAG, UAAG,...

AG: Those are not technologies...

MC: Context is W3C validating its own specs; other people's test suites might disappear.
... Other W3C groups also have testing needs and solve them in different ways.

AG: (...) Test library is an asset in terms of overall world.

CL: We could do some referencing instead of ...
... We at IBM also refer to UAAG test cases.

SAZ: Need to look at this on case-by-case basis; some have better persistency than others.

AG: Cf VPAT declaration

MC: So there are many ideas; need more research and resources.
... Propose that we create a white paper to present to the WAI as a whole and go from there.
... This would summarize what we found out today and describe what we would like to create next.

SAZ: Cf. CL's statement about referencing test suites instead of collecting them in one place: there are benefits to both approaches.
... This would be a design choice.
... Also circulete draft of this paper to some people who could not attend.

DP: Also on WAI IG?
... Announce that we are working on this, to get input.

AG: Send to XTech instead (?) of PF list?

SAZ: Circulating to other lists would be good to get more involvement, also outside WAI.

MC: There was a somewhat similar meeting 2 years ago (Technical plenary session on Joint Test Suite; report is public). New report should not go the same way.

<scribe> ACTION: Michael and Shadi to create a report from today's discussion and to do the necessary follow up. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action03]

Meeting adjourned.

Summary of Action Items

[NEW] ACTION: Michael and Shadi to create a report from today's discussion and to do the necessary follow up. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action03]
[NEW] ACTION: Rich to put Shadi and Barry Feigenbaum together on rule-based technology on raven [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action02]
[NEW] ACTION: tim, assemble list of non w3c related test resources. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action01]
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.127 (CVS log)
$Date: 2007/01/24 22:29:17 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.127  of Date: 2005/08/16 15:12:03  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/a good ATAG tool/an ATAG compliant authoring tool/
Succeeded: s/qa group/qa interest group/
Succeeded: s/format/designation/
Found embedded ScribeOptions:  -final


Found Scribe: David_Poehlman
Found ScribeNick: dpoehlman
Found Scribe: jallan
Inferring ScribeNick: jallan
Found Scribe: Tim_Boland
Found ScribeNick: Tim
Found Scribe: Christophe_Strobbe
Found ScribeNick: Christophe
Scribes: David_Poehlman, jallan, Tim_Boland, Christophe_Strobbe
ScribeNicks: dpoehlman, jallan, Tim, Christophe
Present: Michael_Cooper David_Poehlman Shadi_About-Zahra Don_Evans Andrew_Arch Alan_Chuter Henny_Swan Sylvie_Duchateau Tim_Boland Cathy_Laws Becky_Gibson Shawn_Henry Helle_Bjarnø Jan_Richards Jim_Allan Rich_Schwerdtfeger Christophe_Strobbe Janina_Sajka Al_Gilman
Agenda: http://www.w3.org/2007/01/wai-testing
Got date from IRC log name: 24 Jan 2007
Guessing minutes URL: http://www.w3.org/2007/01/24-wai-testing-minutes.html
People with action items: michael rich shadi tim

[End of scribe.perl diagnostic output]