See also: IRC log
Eric has sent the link to the survey and we should all get this answered
Question 1: does the survey have accessibility problems? We can make some changes.
Kathy: have access to accessibility people at Qualtrics so can get things fixed
MH: we used the accessibility guidelines and the
accessible question guides, but we need to know if there are any other
problems
... please have a look and send accessibility issues to Eric
DF: had a look at the options for the success criteria - checkboxes or radio buttons. Probably not clear how to separate the different comments on the different pages - can paste in the URL. problem is that if we don't separate the pages it isn't quite clear whether people make mistakes because we have to put all the comments for all of the pages together. If would have been better to have a
tool that could separate the pages.
Df: it's not necessary to ask people to make
comments on areas where things were not present - that could be dropped
... regarding radio buttons or checkboxes a test that goes across a number of
pages, checkboxes may be the better option
... should just have 1 comment box for pass/fail
MH: wanted to ask if you liked checkboxes or radio buttons better
Sarah: looks like it's been changed to checkboxes except for 1.1.1 and 1.2.2 but everything else was a checkbox
MH: told Eric to change 1 to a radio button so we can compare
Sarah: prefer checkboxes
... we want to know which pages people thought passed, and which pages people
thought failed
MH: you can have pages that fail and pages that
pass - pro for checkbox approach
... if you have comments later, put it on mailing list or email Eric
... leaning toward checkboxes
DF: maybe its good to give a general short instruction before you start step 4 explaining to people how you explain input - e.g. checkboxes that you can check both so you're not restricted to pass or fail, but could pick both and copy URL and extend the appropriate comments for that URL consistently. If we provide guidance we will get better answers.
MH: what about the length of the questionnaire -
10 pages is quite a lot. We do have a timeout and you can log off and get back
to it
... any other opinions?
... timing - when should it be launched - comments for last version is up
tomorrow. Ideally we should start tomorrow to get the survey done - keep it
open for a month.
... any objections to starting soon?
<Detlev> fine with me
SS: not sure what the issue is about when we do the testing. How long is the testing period for the group?
MH: thinking about keeping it open for the whole of March - is that too long or too short?
would be a bit short for me
<Sarah_Swierenga> +1 on a month
<Detlev> Is it just one site we will be testing?
MH: just 1 site
... would be good to finish in April and process the results in May
<Detlev> why just one site? Will that not a bit one-sided?
<Mike_Elledge> +1 for a month
I will be away for last 2 weeks in March and then the first couple of weeks in April for WWW 2014
DF: not sure why just 1 site. 2 sites might be
better
... if we had different sites, we might get more coverage e.g. dynamic
content, applications etc. Many issues can only be appreciated when you test
more than 1 site
MH: mostly a time issue. We have 2 volunteers for sites. We're asking a lot of people to do a full site.
ME: could consider doing them sequentially. Do 1 site first and get the feed-back and then follow this with a more complicated or different elements
DF: wonder whether we could extend the group of
testers beyond the group here. Is there anythng against having some kind of
open request for outside help? More people might be willing to have a go at
it
... it might be good to have 3 types of sites and people can pick which one
they'd like to do
MH: it was my idea that this would be open to the
public so everyone could have a chance to use it and feed the results back to
us
... if there are more people involved, it wouldn't be so much of a problem
SS: I have a question about opening it wide up for the general public. In the earlier discussions we had talked about keeping the site and the results anonymous.
<Detlev> well, site nominations given in spite of that openness would be fine, wouldn't they?
SS: reach out to colleagues to people who would want to help but respect the anonymity of the website.
MH: this might be necessary for W3C also
Alistair: we need to maintain control - the issue is that you wouldn't be able to understand if the public understood the requirements etc. We need to have good people doing precise checking
MH: maybe not public, but people we know who would be able to do good testing
The university website would not be okay with it being open to the public
MH: we would have to talk with W3C about it as well. Would mean limited people answering the survey. Maybe not do too many websites in that case
DF: if a site owner agrees to offering the site
for testing by the general public, then there shouldn't be a problem. We
wouldn't mind to have that openly tested for the site we've offered. Let people
know that we're testing the methodology and not the site. We might be able to
get propositions for sites.
... we can keep it open then and let anyone test it. The methodology is a
public document and aims to help more people learn to test websites. This would
also tell us how people with less experience would test and whether they would
get meaningful results.
MH: we could combine with public of some sites and more private for others. Will check with Eric to see if it was meant to be public or internal.
SS: ask Eric and Shadi what they think
Could get the website owner some negative publicity
Alistair: there is no problem in running the methodology on anyone's website as it's public. The issue is who would say whether the EM was run correctly. Think it should be a limited number of websites and keep it within a controlled group and then put it out for the general public also later.
DF: is there any harm in publishng an open call
explaining we're testing the methodology if you're a website owner and you may
not mind - are you willing to propose your site and we can see if we get sites
that nominate themselves for testing. If they offer themselves, there is no
harm in people test it even if they're not experienced - tells us something
about the methodology
... you can's constrain it too much because we don't have a way of tracking
how they did the testing. They may accept minor flaws or they may not. That
will be something we can't keep a tab on it. I don't think we'll have much
control anyway - better to open it up to get more exposure - more testers.
Would let us know where the problems are.
MH: some good ideas. Have to check whether W3C will allow us to do public testing.
DF: how could they not? It's an open document and should be exposed as widely as possible to find the problems
Alistair: the point is to have a controlled group so that you can compare the results. Without that you can't compare the results. If you open this up, who would validate whether the results for that website is reasonable.
DF: you would probably see a pattern. People would have some common misunderstandings, a pattern could appear. We could see whether people misunderstand instructions. Test can be used as a heuristic to guide you to things that you want to change
Alistair: I can't see how you can do this with no control group
DF: there would be obvious errors
MH: question about the number of sites - small number of sites gives more opportunity to compare. We want to include the general public as much as possible.
DF: about control group - would mean that we would need a group that have the same understanding - unlikely to happen even within this group of experts.
Alistair: that is the purpose of the methodology. Pick 1 or 2 sites. Should be a controlled experiment to some extent. We could then have another group besides the control to open to the public and that way we've got something to compare.
MH: let's move this subject to the list
Topic 3: choice of website
MH: 2 websites have been suggested and they score about the same
<Detlev> www.incobs.de
DF: described the website - a modern site, not huge. Some changes to DOM scripting with the navigation. One case of a modern clean site which tries to be accessible and meets WCAG 2.0 AA.
MH: if we ask the public for a site, we would have to make sure that we can have consent.
SS: yesterday I sent an email to Eric & Shadi
-happy to forward the note. Got a receptive web master in charge of all of the
medical related websites at the university. He suggested we take a part of 1 of
the websites.
... He would be happy to get feedback
<Sarah_Swierenga> http://msue.anr.msu.edu/topic/info/4_h
http://www.ecu.edu.au/centres/library-services/overview
SS: this part of the website would be more than big enough
some parts of the library website are quite difficult to make compliant because of third-party product such as databases, logins, interactive campus maps etc.
DF: no harm in looking at our websites, it's all good information
MH: that gives us 3 good candidates for testing the methodology. We need to talk to Eric and Shadi about whether it is public or not and whether we have consent.
<Mike_Elledge> +1
MH: topic: face to face meeting in San Diego. Need to register asap. How many are attending?
<Liz> Tim at NIST will be attending.
<Sarah_Swierenga> +1 for the conference, but not until Tuesday afternoon
<MaryJo> I will be attending.
Eric & Shadi will be there. Shadi has got us a room to meet. Arrange a skype session for an hour or so
I would like that
<MoeKraft> +1
<MartijnHoutepen> +1
<Detlev> maybe..