WCAG 2.0 Evaluation Methodology Task Force Teleconference

27 Feb 2014

See also: IRC log


Vivienne, Mike, Kathy, Alistair, Liz, Detlev, Martijn, Sarah, MoeKraft, Mary_Jo, Tim



Eric has sent the link to the survey and we should all get this answered

Question 1: does the survey have accessibility problems? We can make some changes.

Kathy: have access to accessibility people at Qualtrics so can get things fixed

MH: we used the accessibility guidelines and the accessible question guides, but we need to know if there are any other problems
... please have a look and send accessibility issues to Eric

DF: had a look at the options for the success criteria - checkboxes or radio buttons. Probably not clear how to separate the different comments on the different pages - can paste in the URL. problem is that if we don't separate the pages it isn't quite clear whether people make mistakes because we have to put all the comments for all of the pages together. If would have been better to have a

tool that could separate the pages.

Df: it's not necessary to ask people to make comments on areas where things were not present - that could be dropped
... regarding radio buttons or checkboxes a test that goes across a number of pages, checkboxes may be the better option
... should just have 1 comment box for pass/fail

MH: wanted to ask if you liked checkboxes or radio buttons better

Sarah: looks like it's been changed to checkboxes except for 1.1.1 and 1.2.2 but everything else was a checkbox

MH: told Eric to change 1 to a radio button so we can compare

Sarah: prefer checkboxes
... we want to know which pages people thought passed, and which pages people thought failed

MH: you can have pages that fail and pages that pass - pro for checkbox approach
... if you have comments later, put it on mailing list or email Eric
... leaning toward checkboxes

DF: maybe its good to give a general short instruction before you start step 4 explaining to people how you explain input - e.g. checkboxes that you can check both so you're not restricted to pass or fail, but could pick both and copy URL and extend the appropriate comments for that URL consistently. If we provide guidance we will get better answers.

MH: what about the length of the questionnaire - 10 pages is quite a lot. We do have a timeout and you can log off and get back to it
... any other opinions?
... timing - when should it be launched - comments for last version is up tomorrow. Ideally we should start tomorrow to get the survey done - keep it open for a month.
... any objections to starting soon?

<Detlev> fine with me

SS: not sure what the issue is about when we do the testing. How long is the testing period for the group?

MH: thinking about keeping it open for the whole of March - is that too long or too short?

would be a bit short for me

<Sarah_Swierenga> +1 on a month

<Detlev> Is it just one site we will be testing?

MH: just 1 site
... would be good to finish in April and process the results in May

<Detlev> why just one site? Will that not a bit one-sided?

<Mike_Elledge> +1 for a month

I will be away for last 2 weeks in March and then the first couple of weeks in April for WWW 2014

DF: not sure why just 1 site. 2 sites might be better
... if we had different sites, we might get more coverage e.g. dynamic content, applications etc. Many issues can only be appreciated when you test more than 1 site

MH: mostly a time issue. We have 2 volunteers for sites. We're asking a lot of people to do a full site.

ME: could consider doing them sequentially. Do 1 site first and get the feed-back and then follow this with a more complicated or different elements

DF: wonder whether we could extend the group of testers beyond the group here. Is there anythng against having some kind of open request for outside help? More people might be willing to have a go at it
... it might be good to have 3 types of sites and people can pick which one they'd like to do

MH: it was my idea that this would be open to the public so everyone could have a chance to use it and feed the results back to us
... if there are more people involved, it wouldn't be so much of a problem

SS: I have a question about opening it wide up for the general public. In the earlier discussions we had talked about keeping the site and the results anonymous.

<Detlev> well, site nominations given in spite of that openness would be fine, wouldn't they?

SS: reach out to colleagues to people who would want to help but respect the anonymity of the website.

MH: this might be necessary for W3C also

Alistair: we need to maintain control - the issue is that you wouldn't be able to understand if the public understood the requirements etc. We need to have good people doing precise checking

MH: maybe not public, but people we know who would be able to do good testing

The university website would not be okay with it being open to the public

MH: we would have to talk with W3C about it as well. Would mean limited people answering the survey. Maybe not do too many websites in that case

DF: if a site owner agrees to offering the site for testing by the general public, then there shouldn't be a problem. We wouldn't mind to have that openly tested for the site we've offered. Let people know that we're testing the methodology and not the site. We might be able to get propositions for sites.
... we can keep it open then and let anyone test it. The methodology is a public document and aims to help more people learn to test websites. This would also tell us how people with less experience would test and whether they would get meaningful results.

MH: we could combine with public of some sites and more private for others. Will check with Eric to see if it was meant to be public or internal.

SS: ask Eric and Shadi what they think

Could get the website owner some negative publicity

Alistair: there is no problem in running the methodology on anyone's website as it's public. The issue is who would say whether the EM was run correctly. Think it should be a limited number of websites and keep it within a controlled group and then put it out for the general public also later.

DF: is there any harm in publishng an open call explaining we're testing the methodology if you're a website owner and you may not mind - are you willing to propose your site and we can see if we get sites that nominate themselves for testing. If they offer themselves, there is no harm in people test it even if they're not experienced - tells us something about the methodology
... you can's constrain it too much because we don't have a way of tracking how they did the testing. They may accept minor flaws or they may not. That will be something we can't keep a tab on it. I don't think we'll have much control anyway - better to open it up to get more exposure - more testers. Would let us know where the problems are.

MH: some good ideas. Have to check whether W3C will allow us to do public testing.

DF: how could they not? It's an open document and should be exposed as widely as possible to find the problems

Alistair: the point is to have a controlled group so that you can compare the results. Without that you can't compare the results. If you open this up, who would validate whether the results for that website is reasonable.

DF: you would probably see a pattern. People would have some common misunderstandings, a pattern could appear. We could see whether people misunderstand instructions. Test can be used as a heuristic to guide you to things that you want to change

Alistair: I can't see how you can do this with no control group

DF: there would be obvious errors

MH: question about the number of sites - small number of sites gives more opportunity to compare. We want to include the general public as much as possible.

DF: about control group - would mean that we would need a group that have the same understanding - unlikely to happen even within this group of experts.

Alistair: that is the purpose of the methodology. Pick 1 or 2 sites. Should be a controlled experiment to some extent. We could then have another group besides the control to open to the public and that way we've got something to compare.

MH: let's move this subject to the list

Topic 3: choice of website

MH: 2 websites have been suggested and they score about the same

<Detlev> www.incobs.de

DF: described the website - a modern site, not huge. Some changes to DOM scripting with the navigation. One case of a modern clean site which tries to be accessible and meets WCAG 2.0 AA.

MH: if we ask the public for a site, we would have to make sure that we can have consent.

SS: yesterday I sent an email to Eric & Shadi -happy to forward the note. Got a receptive web master in charge of all of the medical related websites at the university. He suggested we take a part of 1 of the websites.
... He would be happy to get feedback

<Sarah_Swierenga> http://msue.anr.msu.edu/topic/info/4_h


SS: this part of the website would be more than big enough

some parts of the library website are quite difficult to make compliant because of third-party product such as databases, logins, interactive campus maps etc.

DF: no harm in looking at our websites, it's all good information

MH: that gives us 3 good candidates for testing the methodology. We need to talk to Eric and Shadi about whether it is public or not and whether we have consent.

<Mike_Elledge> +1

MH: topic: face to face meeting in San Diego. Need to register asap. How many are attending?

<Liz> Tim at NIST will be attending.

<Sarah_Swierenga> +1 for the conference, but not until Tuesday afternoon

<MaryJo> I will be attending.

Eric & Shadi will be there. Shadi has got us a room to meet. Arrange a skype session for an hour or so

I would like that

<MoeKraft> +1

<MartijnHoutepen> +1

<Detlev> maybe..

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2014/03/06 08:43:15 $