See also: IRC log
<MichaelC> scribe: David_Poehlman
<MichaelC> scribeNick: dpoehlman
Agenda review Michael: goals, develop roadmap and define commonalities among groups.
test samples taskforce: sh
WCAGWG has final say We are going to collect test samples. We are going to work with Shawn and others in eo.
Metadata language called tcdl test description language. describes user senarios etc and we are using only a subset of this language.
aria test suites: Michael.
Michael briefly describes aria.
aria test suites provide evaluatlrs with knowledge of states and rolles property attachment. cl:
Jim: aria targets accessability, other specifications raise implementation questions for all including accessability.
<MichaelC> Need expected behaviour of UA for widgets; test suites validate that
<MichaelC> We don’t all agree on what the proper behaviour of widgets should be
<becka11y> http://developer.mozilla.org/en/docs/Accessible_DHTML#Sample_widgets
<MichaelC> http://www.mozilla.org/access/qa/testcases
RESOLUTION: terminology needs to be consistant and clear.
<jallan> need to use existing definitions when available
uaawg test cases: jim:
<becka11y> ARIA samples related to AJAX: http://accessibleajax.clcworld.net/
Jim describes test suites.
Shadi describes test samples and the taskforce.
Jim continues test suite description.
Shadi: We need a sharable database which can provide different views depending on what is required.
Jim: you can edite the test suite, but you are editing code in a form which will result in a page of html being developped and loaded but it is not a streightforward process.
cl: maybe a wiki?
Shadi: What about server side
scripts?
... user agent stuff is different than content stuff.
au: tim:
tim: early work on an approach to testing tools.
Tim, testing the interface as well as the output.
Tim, some issues are keeping track of kinds of authoring tools.
Tim, taking quality assurance into account.
Tim: someone can emply an
authoring tool they have never seen before. Would this be a
valid test.
... in some cases, no.
... a tool developer could test the tool with knowledge of the
test/success criteria.
... the tests and results need to be vailable in public
domain.
... disclaimer in generated report and questions about
submitter.
... where would ths repository be?
... Does this process encourage people to test?
... other questions.
Michael: How does a test verify that the content was developped accessibly.
tim: tests are against
WCAG.
... What the tool produces would be in conformance with
WCAG.
Jan: various tests.
tim: preview before Check:
david: are there are any accessible authorig tools?
Jan: there is an editor which is part of a content management system.
Shadi: an ATAG compliant authoring tool needs an evaluation tool.
qawg resources: tim:
tim: Quality assurance wg
produced some materials.
... They opened in 01 and closed in 05.
... there is now a qa interest group.
Tim, there is a document "QA test faq" to guide test tool developpers.
tim: qa has a wiki.
... qa frameworks specifications guidelines.
... various other software testing resources.
... work on user agent problems, http header problems.
Michael, how do we fit into the qa group?
Tim: there is a series of
requirments for test assertions test metada etc.
... QA has two chairs.
Michael:
How Many members?
Tim: ten but anyone can contribute.
<Andrew> Tim: for other testing resources - http://www.softwareqatest.com/qatfaq1.html
<scribe> ACTION: tim, assemble list of non w3c related test resources. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action01]
<Tim> http://esw.w3.org/topic/QA
<Tim> QA Wiki Resource provided in preceding URL
break:
<jallan> scribe: jallan
summary of current goals
mc: conformance to
specs/recommendations
... test for content creation, rendering, content itself,
content repair
bg: want to test wai guidelines, and for outside folks to test for conformatince
tb: goal? end to end
accessibility - author puts content on web, and makes
accessible to end user
... is what the author did realized by the user
mc: other goal-develop test cases to meet other goals
cl: consistent way to test same
item across guidelines
... problem in UAAG, testing a technique, but there may be
several ways to arrive at a solution.
... test cases are not normative, they are just samples,
mc: there may be normative test cases, i.e. is an authroing tool accessible yes or no
ag: within EARL there are
mechanisms to bind test to a technique
... metadata to indicate sufficiency
mc: goal? joint repository for all of WAI.
ja: yes
jr: yes, for shared infrastructure, reporting, etc. even if tests are pulled from elsewhere
mc: add to goal list
... delineate test cases, help structure repository
ag: QA test taxonomy may be helpful, and test metadata
mc: ?goal to conform to QA
recommendations
... start there and branch when necessary
... ?goal expect WAI specs to conform to QA spec
recommendations
cl: how does collab with QA and WAI work?
tb: explains process, QA reviews spec as they are developed
** joint repository for all of WAI.
** expect WAI specs to conform to QA spec recommendations
** develop test cases to meet other goals
** test for content creation, rendering, content itself, content repair
** conformance to specs/recommendations
** end to end accessibility
** conform to QA recommendations
** test cases are not normative
** consitent way of testing same item across different areas
** share test utilities
ag: EO goals?
** EO create materials to explain testing (further down the road), and in human readable fashion
** improve communication between WAI WGs about testing
ag: in EO, goal should be,
increase level of checking in site building,
... authors should check work, need to monitor
... make testing easier, authors might use tests more
... sampling, don't need a compliance suite for every
provision, if there were a high level 20 items
tb: better to have something rather than everything
** encourage market uptake and use of testing procedures (key things, then all things)
cl: tests but also some reporting mechanism?
jr: works for atag and uaag, because few implementation. web content may be different
ag: a way to get traction, if people create reports, WAI can track usage
tb: CSS selectors has a report for conformance
mc: for reccommendation status, must show 2 implementations, so must have some reporting mechanism
** provide mechanism and structure for submitting implementation reports
ag: also test accounting
tb: QA has best practices that address this
mc: prioritization? or find commonality?
mc: core and WG specific
... example Test case for images
... is alt there? y/n
... wcag - no is fail, yes is qualified pass
... atag if generated content looks like this pass, if not
fail
... uaag....
... two test cases used in multiple ways
... are there many cases like this?
... are we going to be sharing similiar files?
ja: explains UAAG for image
mc: nuance, null alt, do UAAG and
WCAG have same expectations
... C Ridpath alt tests have 10+ for image
tb: must have explicit expectations about requirements
ag: need to make distinctions for how a UA might repair poor alt
mc: make test case, could share,
need specific test cases for each WG,
... help harmonize expectations
jr: different requirements for each WG for same test
tb: QA has info about this
mc: test procedures being different
<Tim> http://www.w3.org/QA/WG/2005/01/test-faq#good
<Tim> QA TESt FAQ What makes a good test?
mc: same test file, 3+ procedures
dependent on WG requirements
... identify major types of procedures for each WG
dp: test cases? may not be files for every test,
mc: right
ag: in UAAG tests, there is a
vpat test, user must be able to get conditional content, UAAG
doesn't specify how to get conditional content
... create a reproducible test, through this gesture user can
get content
cl: that information is in the
report, not part of the procedure. the reporter must explain
how they complied
... user can get content if they change this setting for
perform this behavior
... very vendor/tool specific
ag: ubiquitious web, rebinding input events to access content
mc: need generic test case. use
the UA mechanism to get this content
... 1 test case, multiple procedures, multiple expectiatons and
outputs
... report results will be different for each vendor
... are there cases where test files are only applicable to 1
group, or just have a repository
bg: wcag would not need repair
jr: atag may have just procedures with no files
cl: chrome for UA would note have webfile or test
jr: atag has procedure, can you change system setting, does it impact renderend content
mc: use a sample file to run multiple procedures against
jr: atag-is help section of tool accessible
mc: may use wcag procedures and tests to test against help files.
cl: UAAG GL 6
http://cita.rehab.uiuc.edu/wai-eval/index-gl.php?ts_id=1&gl_id=6
... no tests
mc: but would need a test document to run against the processes
jr: uaag gl 11 http://cita.rehab.uiuc.edu/wai-eval/index-gl.php?ts_id=1&gl_id=11
ag: GL 6 would be place for WAI to shop DOM and other test suite for tests for implementation
saz: explains
note>>procedure>>file>>result diagram
... files are shared
mc: procedures may not be shared
saz: all should be shared, with WG views and public view
mc: we are agreed that having a repository is good
ag: what about procedure sharing, and results
mc: files must be stimulus and response
ag: if each stimulus drive 1 assertion
mc: for everything the tool could
evalutate, had average of 7 files to run against, and 1+ pass
and fail condition
... for a given input, should be 1 result
saz: perhaps not
... need more specific examples of scenarios
ag: could use headers attrib to point to td, or describedby, etc.
mc: no, each condition is a
separate test
... ** choosing which test to be accountable to is a different
task for sufficiency
... depends on group doing the testing
---- break for lunch ----
<shawn> break-out group will come for lunch in a little bit
<shawn> shadi - michael - jim - anyone: would you ask if anyone is taking a cab to the airport this afternoon? Sylvie might want to share a cab this afternoon, and Henny later on.
<MichaelC> scribe: Tim_Boland
<MichaelC> scribeNick: Tim
cs: intro to TCDL
... based on language from BentoWeb project (subset used)
... XML-based, divided into several sections
... formal metadata, technologies, testcase, rules, namespace
mappings
... use CVS keywords for some of these
... Dublin Core is referenced also
... designation for terms is required or optional, with notes
as needed
container elements for locations and techniques
shadi: ERT TF, instead of taking time and effort, used BentoWeb..
focussed on elements needed, minimum set of metadata..
points to WCAG SC and associated technique (different model?)
interest sought from WGs to work on metadata to describe test cases
mc: how does this relate to test files/test procedures?
cs: QA test taxonomy consulted - hard to adapt?
maybe different than what UAAG or ATAG might need?
mc: maybe need more structure for ATAG/UAAG - would it work with
this structure?
mc: want to take advantage of this work in our future efforts
maybe work out some examples to illustrate this?
ag: less structured information
definitions may be more appropriate?
... common ground is more abstract in terms of description?
shadi: expert guidance should be
in technique - took TCDL as is
... to JR and JA: is there interest in common metadata?
JR: would help to try out some examples
JA: useful also, need critical mass to contribute
mc: even if joint test files, still need test procedures
shadi: infrastructure/tools could still be in common
jr: accessible help files - utility that's a checker..
is that something metadata could handle?
<jallan> http://cita.rehab.uiuc.edu/wai-eval/show-test/index.php?test_id=234
ja: author bindings for accesskey
mc: are there elements of this that can be described using TCDL?
ja: multiple nuances to test case.. need to list everything needed
kathy: procedure is vague..output could be in different forms
ja: technology has changed since test was developed, so expectatations are higher now
cs: several issues with adaptation to TCDL
for instance, expected result is descriptive in this example..
shadi: both procedure and result included in WCAG technique
mc: test files should be shared
in repository
... take subset of TCDL as file description, take WCAG test
approach,..
but use same files and metadata language
ag: maybe appropriate for WCAG, not necessary so for UAAG/ATAG,..
but maybe more procedural description may be appropriate
shadi: files could be pointed to from different sources
mc: need distinction of what
could be joint, and what needs to be separate
... need resources to exist, and test procedures need to get at
them for various purposes
... what is in common besides test files?
jr: UAAG/ATAG have procedures/interaction recipes
mc: WCAG may as well?
ag: wcag text equivalent/uaag conditional content - in common?
mc: need wide deployment for
wcag
... for wcag, a lot of evaluation can only be done
manually
... is prerequisite tests in TCDL?
cs: not used now
shadi: conceptually possible in metadata
mc: testing framework needs to have way of expressing preconditions?
baseline questions
mc: framework for describing and storeing test procedures/files would
be things in common
kathy: and framework for reporting on test results
mc: tied in to procedure?
ag: EARL has some of this info?
shadi: how it was done, but no configuration info..
ja: this is where UAAG would come in
shadi: TCDL companion to EARL (under discussion)?
ag: archive in more detail than
that specified in test procedure
... need to capture detailed experiences for richness
shadi: asserter can be both human and/or software
<jallan> http://cita.rehab.uiuc.edu/wai-eval/evals-cp.php?eval_id=13&cp_id=1
kathy: uaag many tests for checkpoint, how many do we need to pass to pass checkpoint?
shadi: EARL can provide such detailed reporting, but sequence/which tests to do..
were taken out
kathy: different from wcag perspective (based on instances) vs. uaag (based on test case)
jr: atag does both
mc: have in common: framework for
storing test files, test procedures, expected results,..
... also common framework for executing test files?
... applications to facilitate execution of tests?
ja: need everything in one "package" to run tests
mc: test driver application is important - should be shared?
ag: cyclic graph of dependencies
among groups?
... mutual capability to run one another's tests
jr: special uaag/atag dependency needs to be explored?
mc: resources may be available?
ja: test driver application should be self-contained
shadi: administrative development process is issue ..
support for test development within WGs is important
mc: wcag developing tests, where are other WGs?
ja: not actively developing uaag tests right now, still some issues, but fairly complete
kathy: some checkpoints don't have tests yet
mc: how much of what's there is repurposeable?
ja: a lot would be.. (for HTML4.01) (ARIA not covered)
shadi: uaag support in common infrastructure development work?
ja: resources scarce, but ua will be there..
<cklaws> what guys?
jr: nothing for atag1, some
procedural stuff for atag2
... would like to help, but resources scarce
ja: maybe use atag work in ua
shadi: maybe use mobile web work also?
rich: pf taxonomy created which
may be helpful?
... style guide under development also
ja: may also apply to atag as well
rich: need browser extensions to run these tests
ja: does wcag support dynamic html functionality (all ties together)?
mc: we hope wcag addresses this - some techniques, not many tests
cs: want to do, but haven't gotten there yet
mc: testing a bunch of technologies working together
cs: in TCDL, can partially describe this (doesn't go far enought for AJAX)
ag: can we see tests in accessibility APIs?
rich: possibly so (raven on alphaworks - rules based)
<cklaws> Raven on IBM alphaWorks: http://www.alphaworks.ibm.com/tech/raven
<scribe> ACTION: Rich to put Shadi and Barry Feigenbaum together on rule-based technology on raven [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action02]
<MichaelC> scribe: Christophe_Strobbe
<MichaelC> scribeNick: Christophe
mc: agreed we want common
framework for testfiles and common framework for test
procedures
... . Need to be able to run each other's tests.
... Way of activating tests.
... Share knowledge and resources about administrative
aspects.
... Liaising with QA IG. Considered joint task force (QA - WAI
WGs).
... WAI CG already co-ordinates communication; could also have
task force for liasing with QA.
JR: Facilitatator needs to be someone with lots of time and/or lots of resources to do the main part of the work; could then call on busier people.
SAZ: TSD TF has two facilitators: one from WCAG WG, one from ERT WG. Co-facilitating would be an option for the liaising TF.
AG: Common frameworks: hard to
get critical mass of resources to make this happen.
... Common functionality, yes; not necessarily common
harness.
<JR> JR: correction: facilitator shouldn't just be a single person trying to call on other busy people
MC: TF will be responsible for narrowing the scope. Scope needs to be reasonable.
SAZ: (response to AG) ... Developing content of repository is something else.
AG: "Build it and they will come"? (..) How do we expect to see this get used?
JR: If you have test cases in repository without harness, who will run the tests? (...)
SAZ: [Tool by Jon Gunderson?] If
we have someone to get this running, ... Some test cases are
combinations of files. There's more to it than a single piece
of HTML, so that tool would need to be changed.
... TSD TF didn't start developing a metadata format
immediately; first looked at what existed. Similarly, there may
be other tools to run tests. The TF would have to look for
these first.
MC: TF could continue discussion and do research. Could be incubator TF?
SAZ: Lifetime for this work is too limited; should be real task force.
MC: TF picks up from the common understanding we reach today.
SAZ: WGs are in different stages
regarding test files: some want them now, some later.
... Why would tney contribute if their need is still in the
future?
AG: Feel need for more
co-ordination at the techniques level.
... See payoff in co-ordination regarding techniques.
MC: Earlier, we saw this as a
side benefit. Could make it a goal.
... Other possible outcome of meeting: continue what we were
doing? Something in between?
... Long term, we would benefit from common repository and
framework; short term, contributing is not obvious.
... Maybe move repository - of test files - to common
location.
... E.g. test files UAAG test cases; go through TSD TF review
process for WCAG.
JR: Who has programming skills etc. to create it and set it up?
MC: TSD TF has tools set up.
AG: QA has best practices regarding test suites.
SAZ: Not only a matter of
programming skills but also time and resources.
... (response to RS): would need to port Jon Gunderson's
...
RS: We need to identify what we have; then build the roadmap.
JA: Stand-alone section on W3C
space; application could be on UIUC servers.
... We have two different sets (see list of resources: UAAG)
UAAG: XML DTD looks like early version of TCDL.
MC: Even basic task of getting test resources in one place is important to avoid duplication, but even that requires considerable resources.
SAZ: If we want to do this, we
need commitment from each of the WGs to review the work.
... Need to contact chairs and other key people of these
WGs.
MC: WCAG maybe not before June.
JA: Currently not enough resources...
MC: Strategies: how get people
involved and keep them interested?
... If UAAG gets rechartered, it needs more people; WCAG could
also use refresh of personnel.
... Maybe talk about it at conferences. Governments and
generous corporations...
RS: Set format that everyone would need to convert test cases to?
MC: Would be nice in long term. First need to collect what is available, then ...
RS: E.g. Yahoo interface library: could that be used or would it need to be modified? Modification is lot of work.
MC: Agree. TCDL makes sure
certain data are always available. Coding of test files: not
necessary look the same.
... (...) Test cases do not need to look the same except when
required for the purpose of the test.
... We have common interests and needs but appear to lack
resources for a new task force.
... Maybe a report out of this meeting that describes to the
WAI what we identified regarding resources and needs, what
could happen, etc.
... This could ensure some kind of common purpose across WAI
working groups in future.
SR: Regarding pulling this test suite together: Mozilla Foundation grants (cf AJAX ...)
CL: Accessibility of Firefox: XXX got ability to manage grants for resources working on Firefox-related topics.
MC: Who would get funded?
SR: Cf skill sets. Accumulating test suites from vendors; automate things; define testing methodology; ...
AG: Accessibility initiative may be wrong scope to take action on this. It does not matter who owns the repository. If other groups are more heavily into conformance demonstrations, ...
SAZ: Those test suites are for our specs and technologies: WCAG, UAAG,...
AG: Those are not technologies...
MC: Context is W3C validating its
own specs; other people's test suites might disappear.
... Other W3C groups also have testing needs and solve them in
different ways.
AG: (...) Test library is an asset in terms of overall world.
CL: We could do some referencing
instead of ...
... We at IBM also refer to UAAG test cases.
SAZ: Need to look at this on case-by-case basis; some have better persistency than others.
AG: Cf VPAT declaration
MC: So there are many ideas; need
more research and resources.
... Propose that we create a white paper to present to the WAI
as a whole and go from there.
... This would summarize what we found out today and describe
what we would like to create next.
SAZ: Cf. CL's statement about
referencing test suites instead of collecting them in one
place: there are benefits to both approaches.
... This would be a design choice.
... Also circulete draft of this paper to some people who could
not attend.
DP: Also on WAI IG?
... Announce that we are working on this, to get input.
AG: Send to XTech instead (?) of PF list?
SAZ: Circulating to other lists would be good to get more involvement, also outside WAI.
MC: There was a somewhat similar meeting 2 years ago (Technical plenary session on Joint Test Suite; report is public). New report should not go the same way.
<scribe> ACTION: Michael and Shadi to create a report from today's discussion and to do the necessary follow up. [recorded in http://www.w3.org/2007/01/24-wai-testing-minutes.html#action03]
Meeting adjourned.
This is scribe.perl Revision: 1.127 of Date: 2005/08/16 15:12:03 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/a good ATAG tool/an ATAG compliant authoring tool/ Succeeded: s/qa group/qa interest group/ Succeeded: s/format/designation/ Found embedded ScribeOptions: -final *** RESTARTING DUE TO EMBEDDED OPTIONS *** Found Scribe: David_Poehlman Found ScribeNick: dpoehlman Found Scribe: jallan Inferring ScribeNick: jallan Found Scribe: Tim_Boland Found ScribeNick: Tim Found Scribe: Christophe_Strobbe Found ScribeNick: Christophe Scribes: David_Poehlman, jallan, Tim_Boland, Christophe_Strobbe ScribeNicks: dpoehlman, jallan, Tim, Christophe Present: Michael_Cooper David_Poehlman Shadi_About-Zahra Don_Evans Andrew_Arch Alan_Chuter Henny_Swan Sylvie_Duchateau Tim_Boland Cathy_Laws Becky_Gibson Shawn_Henry Helle_Bjarnø Jan_Richards Jim_Allan Rich_Schwerdtfeger Christophe_Strobbe Janina_Sajka Al_Gilman Agenda: http://www.w3.org/2007/01/wai-testing Got date from IRC log name: 24 Jan 2007 Guessing minutes URL: http://www.w3.org/2007/01/24-wai-testing-minutes.html People with action items: michael rich shadi tim[End of scribe.perl diagnostic output]