Research and Development Working Group Teleconference

04 Aug 2011

See also: IRC log


Vivienne, Markel, Giorgio, Shadi, Simon, Joshue, Mate
Yeliz, Yehya, Karl


Benchmarking Web Accessibility Metrics (Automated Metrics, Evaluation, Benchmarking.)

simon: let's talk about te web accessibilitymetrics item in the agenda

<sharper> http://www.w3.org/WAI/RD/wiki/Benchmarking_Web_Accessibility_Metrics

ok, thanks

markel: wiki was updated as per actions last week.
... metrics=way to measure something.

<Joshue> am looking at it now

no i did not.

markel: accessibility wrt conformance is one thing
... accessibility in use is another one

mv: up to now most metrics are based on results produced by tools

<Joshue> Is SAMBA a tool? have you a URI?

samba is amethod. I have a paper; see from www.dimi.uniud.it/giorgio/publications.html

mv: there are many diverse challenges concerned with metrics.

<Joshue> interesting overview Marco, thanks.

<Joshue> Sorry, Markel.

simon: which open questions are more interesting?

mv: accessibility in use is interesting for me.

<vconway> I am also interested in how we can include the user aspect

<Joshue> I think a very interesting question is "How metrics that measure accessibility in terms of conformance relate to accessibility in use?"

<shadi> +1 to joshue

<sharper> GB: Important part is understanding when a metric is valid and when not

<sharper> GB: need to make it more concrete / operational - 1 idea is to create a test collection

<Joshue> Yes, and how can automated tools do this without explicitly reference some set of conformance criteria such as WCAG?

<sharper> GB: with pages witch known qualities to test these metrics - then expect metrics to perform in specific ways

<vconway> it would be great to develop some type of scoring system

simon: agrees, and refers to NLP and data mining

shadi: carryingout a test is not a job of this wg
... we should elaborate questions, refine them, ask opinions

<markel> thanks

shadi: the two notions of acc should not be diverging

<Joshue> yes, thanks markel :-)

<markel> :-)

shadi: are these two ways to measure acc? or can we come up with a combined method?

<markel> please could you guys turn the mic off?

<vconway> I am working on trying to set up some kind of scoring system which sounds like what you're asking

<sharper> GB: We should be like a steering committee making suggestions about where to steer this research

<shadi> +1 that our goal is steering researchers

<sharper> GB: we don't do it , but make proposals about the properties it should have

<sharper> +1 that our goal is steering researchers

<sharper> GB: would distinguish the acc in use and acc conf

<sharper> GB: discusses the possibilities of measuring accessibility in use

simon: agrees.
... we should discuss on test collections and also on the 2 notions of acc.

<vconway> I certainly agree that we need both

simon: there are no frequent references in literatutre to these two defs.

<Joshue> sorry, was that to me?

vconway: needs a scoring system to compare monthly but needs to weigh differently different aspects

say again which objects are you analyzing?

vconway: in AU they need to ensure that a site is conformant monhly
... there are different acc policies and there are different site genres

ok, thanks a lot.

mv: raises another question to vivianne

vivianne: needs a tool for measuring the progress over time

mv: asks vivianne if she used other published metrics

vivianne: has not found a scoring system coupled to wcag2

<markel> thanks vivienne

<markel> Giorgio do you remember the AIR reports?


<vconway> ibm with the adesigner are doing a bit of scoring

<Zakim> shadi, you wanted to talk about "scoring" term

shadi: in use vs conformance
... vivienne case is an interesting one (ie monitoring over time)

<vconway> It would be nice to say 'we have improved from 35% to 80%'

<Joshue> +q

<vconway> I lost the call

<vconway> Just got the call back, so probably missed something

joshue: likes the %approach; acc is not a binary thing

<markel> +q

shadi: questions whether AIR/knowbility is or not conformance

<vconway> I am having a user group also look at my 100 websites to see how the user groups relates to the %score

<Joshue> how about a traffic light system?

<vconway> traffic light?

shadi: if we could come with a scoring system that is objective and that mirors the level of acc it would be fine.


mv: encourages to read the references in thewiki page

<Joshue> yeah, like orange for 'in the middle, not great but not terrible', 'Green for a high level of conformance' and well "red' speaks for itself..

<Joshue> just an idea..

<vconway> good point Markel

<vconway> Josh, the traffic light sounds good

mv: suggsts reading those proposals of metrics and identify weak pointsand AVOID inventing the wheel again (giorgio's emphasis)

<Joshue> thanks vivienne, other abstractions could be expored. I don't know if the driving metaphor is suitable.

mv: what is objectivity? completion time is pretty objective.

<vconway> I think we need both

<Zakim> shadi, you wanted to disagree with "As guidelines conformance does not necessarily entail accessibility in use"

shadi: responds to markel's question of whether acc in use /acc forconfomance is more important
... if we don't comply with guidelines people with disabilities will have pbms in accssing the page

<vconway> I have been comparing my results with that of a user group with good success

<Joshue> shadi, it could be a subset of WCAG?

<markel> right

shadi: we might be looking to elicit suggstions of how a perfect scoring system could work

<Joshue> or it we could identify the most important 'must haves' for a11y

<Joshue> Structured content, well formedness etc

shadi: there might be several aspects, propeorties of metrics to consider; eg cost
... sugegsts a webminar for call of proposals
... maybe setting up a reviewpanel

<shadi> http://www.w3.org/WAI/RD/planning#seminar

<Joshue> Yes, thanks I'll read the supporting materials on the wiki

ok, thank you


giorgio: volunteers to be a co-editor

mv: will love helping me!

joshue: would like to help in asupporting role

<Joshue> I can do peer review etc

<vconway> I'm happy to help in some way, bt I'm a bit of a baby at this

<shadi> http://www.w3.org/WAI/RD/process#credits

vivienne: would also like (and we'd like to have babies!)

simon: sees editors as program chairs
... we got 2 co-editors MV+GB

shadi: let's not forget rui and yeliz though

vivienne: is available for any help she can

shadi: anybody who would lik to volunteer shouldcontact shadi or simon

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.136 (CVS log)
$Date: 2011/08/05 12:22:20 $