<scribe> scribe: MarkMccarthy
jamesn: only one new issue logged, looked at it last time, should be good
jamesn: 2 new PRs
https://github.com/w3c/aria/pull/1292
jamesn: this needs reviewing, we
talked about this last week, re-adding haspopup on links
... jcraig are you reviewing this too?
jcraig: yes, just haven't looked at it yet
jamesn: i'll merge once you do
https://github.com/w3c/aria/pull/1291
jamesn: this is 1.3, 3 approving reviews, i'll merge shortly
pkra: quick question for jcraig on that PR - jcraig approved but made a suggestion
jcraig: if you can't update it that's fine with me
pkra: i'll think on it and see
what i can do
... no worries on merging
jcraig: +1
pkra: i won't forget, i'll make a note
jamesn: reminder, no meeting next week. holiday week in US and chairs are busy
<jamesn> https://github.com/w3c/aria/issues?q=is%3Aopen+is%3Aissue+label%3Adeep-dive
jamesn: only 2 issues that have
deep dive tags, put off because we wouldn't be able to spend
enough time on them
... if folx could tag some issues or send me an email, we can
log an issue and see about chatting about it
jcraig: proposal - custom sliders
in mobile with touch events
... some good ideas around AOM events but privacy concerns,
other problems with other examples.
... sina's reminder this week prompted some more conversation,
there are some old issues that might be helpful
... handful of things that are in APG that are window's
specific, some things that don't have AT actions/gestures, but
generally, reps from major browsers have decided this is a good
idea
... alice put up a patch for webkit, they're proof of concept,
but still worth discussing
mck: that's a GREAT topic
sina: WORLDS of yes
jcraig: just have to think about LTR or RTL
mck: we have aria-orientation for /that?/
jcraig: let's discuss next deep dive
jamesn: fun fact, they worked okay on Windows phones!
jcraig: yeah...
jamesn: i'll send out a note once it's closer, still need topics for july
<jamesn> https://github.com/w3c/aria/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22ARIA+1.2%22
jamesn: only a couple things left
open, one we already talked about and have a PR for
... the other is the aria-owns one, but i think there's many
new issues that need to be logged based on it
... ready to merge and 1.2 is closed out (woohoo!) but we do
need to log follow-up issues
... i'm not super familiar with it, but if someone is more
familiar with this and can help out Alex, then we can put this
to bed
jongund: seems like the major concern was aria-owns to aria-controls?
jamesn: matt has a few issues
too, maybe aria-busy should be 1.3 and idk if that's been
logged. i don't want to lose any issues that came up from the
comments, that's all
... if you don't feel your comment has been resolved and you
would like to, PLEASE log an issue, even if it links back to
this
... otherwise, once everyone is happy, i'll close and merge
this
... i myself am not familiar enough with this, but i'll try and
see if i can log a meta-issue
... aaronlev if you can check through that'd be great
mck: for context: the way i like
to talk about the goal of this is to think about AT as UI
rendering tech
... for instance, we'll compare a screen reader to visual
displays
... for vis displays, we have a stack of tech that helps
engineers be confident that the pixels appear on the glass in a
way they can anticipate -- it's deterministic
... with screen readers, you can write a bunch of ARIA, but the
syllables spoken or dots raised are less deterministic
... we agreed not to tell UI designers how to design, and we
intend to respect that. there's room for creativity and
innovation in the AT space
... at the same time, engineers need to be sure that designs
rendering a screen reader or AT experience render something
faithful to the design
... what we haven't done yet is say what aspects of the
semantics have to be rendered, and what does meaningful
rendering look like?
... so first we have to agree on what is spoken is sufficient,
and what is equivalent to the pixels on glass
... what does support -actually- mean, in a given
circumstance?
... second, how do we create a tech stack that scales with
these expectations across ATs?
... frankly, it's not fair that folx using vis displays have
tech to make sure they get a reliable, robust experience. but
there isn't a way to assure that when releasing a new version
of SR or browser
... not a way -yet-
... providing a means to say "here's how well X AT is
supporting ARIA etc." will help with that. ideally in the
future, this can be automated and easy to run
... i'm partnering with folx in Beaucoup, a community group
supporting/guiding, but the people doing lots of engineering
work are people behind WPT.FYI
... need to make sure this scales over time.
... questions?
[silence]
jamesn: please continue Matt!
mck: at aria-at.w3.org -- right
now it's a little basic for the default user
... have to be a member of one of the aria-at teams in github
(admin or user). if you're not, all you get is a homepage and
how to sign up
... there's a test platform, which i won't dive too deep into,
but right now all testing is manual.
... we envision running a cycle of tests, certain number of
patterns with certain AT/browser combos
... noting versions of all software and tests themselves.
... once all that is defined it creates a queue.
... starting, though, with test reports. so we've run one pilot
test cycle with checkbox and menubar patterns using JAWS, NVDA,
and VoiceOver with Chrome, Safari, and Firefox
... and published one report from that (checkbox with Safari
and VO). This is all with desktop for now.
... Within the tests themselves, we have required and optional
tests. Sometimes there are optional tests depending on the
overall architecture of the example.
... tests were organized around user tasks, so when the user
does X, does the expected behavior result?
... in the example, there's 9 means to navigate to an unchecked
checkbox.
... Overall we see that 93% of required behaviors were
observed, and only 54%.
... One thing about optional behaviors, they are -optional-.
Percents are purely qualitative .
... We're trying to focus on basics first, but there's lots of
behaviors worth tracking and testing, and what that means to
users might need more discussion.
... We also track unexpected behaviors. So things that
interfere with testing or positive results. So maybe a screen
reader said what it was supposed to, but then a lot more.
... We track what commands we used, then capture the specific
output from the screen reader, with breakdowns of role, name,
and state (at least in checkbox example).
... It's the job of the tester to determine if the output
captured meets the need.
... "Conveyed" was carefully chosen to be AT agnostic. "Spoken"
doesn't work for Braille displays, or a SR may not "speak," but
it still conveys its meaning to the user.
jcraig: for instance, VO speaks roles, but you could alternatively choose a sound icon to reduce speech.
mck: that brings up another
important part of testing, in that they require assertions are
met with -default- settings.
... we plan in the future to also look into custom settings
etc., to make sure semantics are still conveyed or
rendered.
... we still want to measure SR support for them.
... The test queue is organized from tester's perspective,
including what combo they're testing with.
[technical demo difficulties]
mck: As an admin, I can open the
test plan and view it as the person who tested it, and a couple
other options.
... because there's interpretation involved, our protocol says
two people run each test. for each test, the system makes a
diff and tries to reconcile an agreement. if they don't agree,
there's an option for tester to raise an issue.
... they can then discuss, get resolved on public Github. Once
there's concensus from two testers across all test plans, then
admin has the option to mark the results as "Draft
Results."
... The system will explain exactly where the differences are,
can copy to the clipboard, and provide quick means to file an
issue.
... The only way to edit results is to start over, the form
can't yet be rehydrated.
... Intention is that, after "Draft Results" stage, we review
with the AT vendor and see if they agree, then we publish the
report.
... Goal is to go through the entire APG. First round is to go
through desktop screen readers.
... This year's goal is to go through 10 examples, and I'd love
to have one person dedicated to writing tests, though that's an
uphill battle.
... Questions?
Greg: two questions. based on initial context, it seems primary focus is on testing that the AT doesn't regress based on browser changes. Is that right?
mck: eventually, yes. at this
point, we're trying to -establish- a baseline.
... AT is trying its best, but sometimes it's goofy. Hasn't
been enough discussion of what it means to render ARIA, so
that's the overarching goal.
... But yes, eventually we hope to get to regression
testing.
Greg: I think that long term,
it'd be great to automate this. We did this at MS with the UIA
tree a few years ago.
... It'd be worth having discussion about a planning doc, maybe
a report for cost of a headless implementation of various
AT?
... One of the main reasons we did so is that someone would
change something in Windows that would break Edge and make lots
of bugs.
... Now, doesn't matter where you are in Windows org, but it
tests to make sure that thing doesn't regress.
... Should be able to run with Selenium to check on pressing a
key command, did AT report this back?
... Would of course have to talk to AT vendors about cost
etc.
mck: Absolutely! Manual testing
now is just to establish expectations, level of support, etc.
baselines. But figuring out automation is key.
... And I hope to start that by end of the year.
... Seems like there IS a need for AT protocol, not unlike web
driver.
Greg: https://github.com/MicrosoftEdge/A11y
... An AT Driver would be interesting to extract stuff, get an
answer.
... this is a really cool project!
MichaelC: you mentioned Beaucoup were involved in web platform tests - is there an idea this can be part of those tests and central QA?
mck: ultimately yes, we'd love to get to a place like that
Greg: wanted to tie it back to
open UI and the overlap - it's very similar to the tests that
we do. ultimately, when i talk to component libraries, there's
desire for that to be part of the test suites.
... everytime they have a build or commit, they want to run
these suites to make sure they're not regressing
... and solving end to end
mck: kind of orthoganol though, as the patterns we choose to test we did partly because ARIA semantics are what we're focused on
jcraig: to MichaelC's comments,
one of things is that computed label and role has been added to
web driver; the group added a web platform test.
... even now, could apply some WPT tests that access this web
driver, but may be tedious.
... on the other hand, something all the way on the end of an
AT driver, it might not -fully- happen, but there might be an
intermediary way to handle that
... intermediary might be easier to standardize, or what's in
browsers/exposed to platform APIs might be easier.
sina: for automation piece, i
played with this a while ago. you can extract NVDA output
before synthesis, so that's something
... to do so consistently to capture audio and run ASR. if you
do that and figure out the most accurate voice settings, you
-could- make an automated tester bot that could at least report
what's happening. could be fun
... there were some accessibility issues during test running
phase, it'd be good to improve some of that so we can really
walk the walk/talk the talk
mck: absolutely!!
jamesn: thanks Matt for an excellent demo!
<pkra> bye everyone.
<jongund> James have a good vacation, you deserve a break
This is scribe.perl Revision of Date Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) Succeeded: s/that//that?/ Succeeded: s/"speak,/"speak," but it still conveys its meaning to the user./ Succeeded: s/Results"/Results."/ Succeeded: s/dedicated to this/dedicated to writing tests/ Succeeded: s/focused/focused on/ Present: StefanS jamesn pkra MichaelC Scott_O MarkMccarthy jongund CurtBellew BGaraventa harris carmacleod Jemma Sina Joanmarie_Diggs Found Scribe: MarkMccarthy Inferring ScribeNick: MarkMccarthy WARNING: No meeting title found! You should specify the meeting title like this: <dbooth> Meeting: Weekly Baking Club Meeting WARNING: No meeting chair found! You should specify the meeting chair like this: <dbooth> Chair: dbooth WARNING: No date found! Assuming today. (Hint: Specify the W3C IRC log URL, and the date will be determined from that.) Or specify the date like this: <dbooth> Date: 12 Sep 2002 People with action items: WARNING: IRC log location not specified! (You can ignore this warning if you do not want the generated minutes to contain a link to the original IRC log.)[End of scribe.perl diagnostic output]