Re: ACTION-335 logotypes and ISSUE-96 discussion

Ok, I think I see where you're coming from. I agree that if something
doesn't work under the best of circumstances then we have a problem,
however I haven't convinced myself that we will be testing under the
best of circumstances. The best of circumstances to me implies that
the user is already familiar with their environment, and a short test
in which many things are new to them doesn't necessarily meet that
criteria. I do generally believe that the effect of something new (and
distracting, as opposed to subtle/unobtrusive) would probably be most
pronounced in the short test, I'm not sure how that applies in the
general case though, much less the case of the EV indicators.

Basically agree with the rest of your post, and you're right, we do
need a concrete proposal before we can start really getting into this,
I just wanted to raise the issue about adoption rates (and also length
of study) as one that we should consider, and I think I have done so.

On Nov 13, 2007 3:39 PM, Serge Egelman <egelman@cs.cmu.edu> wrote:
>
>
> Ian Fette wrote:
> > Disagree.
> >
> > While I don't agree with all of Phil's points, there was one that I
> > definitely agree with that Serge seems to have glossed over. That would
> > be the point about whether you're testing the user over a half-hour in a
> > lab, or a longer (30+ day) field-study in their natural environment.
> > Phil's point was that anything new and disruptive is likely to show a
> > strong effect in the short-term, but over the long-term the effect may
> > be drastically different (including causing people to stop using the
> > product). This is a very good point, and I think that if possible we
> > should aim to do a longer field-study as opposed to a 30m in-lab study.
> >
>
> This is something that can be examined later.  There are pros and cons
> for both types of study designs.  But then again, if your only argument
> is that the effect is stronger in the lab study, I'm not sure how that's
> a problem.  If the users can be easily fooled in the best case scenario
> (or they simply don't notice/trust the indicators), then it's likely
> this effect will only be stronger in the wild.  Of course playing arm
> chair quarterback here isn't going to do much good.  We need an actual
> study design before we can critique it.  To do that we need to first
> figure out which questions we want the study to answer.
>
> > As for "testing them in a perfect world" - I have no idea why this is a
> > good experiment to run, because we know that we will never be operating
> > in a perfect world. I'm not saying we should test in a world with zero
> > adoption, but rather I'm saying that we should try to figure out (guess)
> > what /reasonable/ adoption is, and test in that world. We already know
> > that there are some sites that are not adopting EV because of the cost
> > model. I'm sure someone is more knowledgeable about the specifics than
> > I, but my understanding is that, for instance, Google could not buy one
> > EV certificate for google.com <http://google.com> and use it across all
> > of our numerous servers, rather we would have to pay some increased
> > (large) fee based on number of servers. (Also, does EV support wildcard
> > certs?). Given that, you can come up with a list of companies for which
> > EV would be very expensive and likely not adopted (eBay?), and test with
> > the assumption that those sites won't adopt. What does that do to the
> > overall model?
>
> This is kind of a no-brainer.  The system will not work when there's
> very low adoption.  Users will go to a website, see no EV certificate
> indicator, and just assume that the site never had one.  That's because
> this is the norm.  Thus, if we assume everyone adopts EV, we can test
> the rate of success in the best case.  If the system is not successful
> in the best case scenario, then we know it won't be successful in
> conditions worse than that.  Likewise, if we test the system under
> "reasonable" circumstances (forgetting for a second that "reasonable" is
> completely subjective and we're unlikely to agree on that definition)
> and it's a failure or success, the validity of the study comes into
> question because someone will invariably ask, "well, what happens when
> everyone adopts EV?"
>
> If we're going to agree that a minority of sites will adopt EV, I'm not
> convinced we should be making recommendations about it to begin with.
> Users will be required to remember which websites are EV-enabled and
> which ones aren't, and that's a completely ridiculous assumption to
> make.  Even if (again, best case scenario) they can make this
> distinction, no doubt they'll still interact with websites that are not
> EV-enabled.  These websites will remain targets for attack.
>
> >
> > Finally, I'm extremely concerned about the attitude of "Well, it works
> > in lab studies, so let's mandate it, vendors be damned." I understand
> > the desire not to be seen as being beholden to the desires of browser
> > manufacturers, but on the other hand, I have a very real desire not to
> > be seen as floating around in la-la land, disconnected from reality. If
> > something is going to cause people not to adopt a product, a vendor is
> > not going to implement it, regardless of any mandates from W3C. There is
> > a very real risk of steering ourselves towards irrelevancy. Without
> > getting into too many politics, that's why WHATWG was formed, and
> > provides a good bit of background for the current HTML5 /realpolitik/. I
> > don't want to see us go the way of XForms 2.
>
> Who said if it works in lab studies we'll automatically recommend it?  I
> see the lab studies as a way for weeding things out.  If they perform
> well in lab studies, obviously we'll need to conduct further tests.  Of
> course, this doesn't work if only three of us are doing the lab studies
> on our own time...
>
> serge
>
> >
> > My $0.02 x 3 (== £0.03)
> >
> > On Nov 13, 2007 8:51 AM, Dan Schutzer < dan.schutzer@fstc.org
>
> > <mailto:dan.schutzer@fstc.org>> wrote:
> >
> >     agreed
> >
> >     -----Original Message-----
> >     From: public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org>
> >     [mailto:public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org>] On
> >     Behalf Of Serge Egelman
> >     Sent: Tuesday, November 13, 2007 11:23 AM
> >     To: Hallam-Baker, Phillip
> >     Cc: Ian Fette; W3C WSC Public
> >     Subject: Re: ACTION-335 logotypes and ISSUE-96 discussion
> >
> >
> >     This is irrelevant for our purposes.  If we test them and find that in a
> >     perfect world they don't work, then this is moot.  If we test them and
> >     find that they're effective, then we make a recommendation, and it's out
> >     of our hands.  At that point the application vendors aren't in
> >     compliance.
> >
> >     serge
> >
> >     Hallam-Baker, Phillip wrote:
> >      > I have never had the slightest difficulty selling the idea of
> >     logotypes
> >      > to customers. The problem is purely on the application side. The
> >     logos
> >      > have no value unless they are displayed.
> >      >
> >      > So we risk a chicken and egg situation where the application side
> >     people
> >      > refuse to do anything about implementation until they are assured
> >     that
> >      > there will be 100% adoption by the site owners which is not going to
> >      > happen until there are applications to present the logos.
> >      >
> >      > Someone has to make the first move, we cannot gate the scope of
> >     what we
> >      > will consider by requiring an assurance of total adoption by any
> >     market
> >      > participant.
> >      >
> >      >
> >     ------------------------------------------------------------------------
> >
> >      > *From:* public-wsc-wg-request@w3.org
> >     <mailto:public-wsc-wg-request@w3.org> on behalf of Ian Fette
>
> >      > *Sent:* Fri 09/11/2007 4:49 PM
> >      > *To:* W3C WSC Public
> >      > *Subject:* ACTION-335 logotypes and ISSUE-96 discussion
> >      >
> >      > This action (ACTION-335) was to provide discussion topics for
> >     ISSUE-96.
> >      > I only really have one point, and I will try to state it more clearly
> >      > than at the meeting.
> >      >
> >      > To me, the effectiveness of any of the logotype proposals (or the EV
> >      > proposals, for that matter) depends greatly upon the adoption of
> >     these
> >      > technologies by sites. We can do really cool flashy things when
> >     we get
> >      > an EV cert, or an EV-cert with a logo, but right now the only two
> >     sites
> >      > I can find using an EV cert are PayPal and VeriSign. Therefore, I
> >     wonder
> >      > how habituated people would become in practice, if they never (or
> >      > rarely) saw the EV/logotype interface stuff in use.
> >      >
> >      > My proposal is that any usability testing of the EV and/or logotype
> >      > things in the spec not only reflect how users would behave in a land
> >      > where everyone is using EV-certs and life is happy, but rather
> >     also test
> >      > a more realistic case. That is, look at what the adoption is
> >     presently
> >      > and/or what we can reasonably expect it to be at time of last
> >     call, and
> >      > do usability testing in an environment that reflects that
> >     adoption rate
> >      > - i.e. some percentage of sites using EV certs, some percentage also
> >      > using logos, and another percentage still using "normal" SSL
> >     certs. My
> >      > worry is that we may be thinking "EV certs will solve X,Y, and
> >     Z", but
> >      > that may only be the case if users are used to seeing them on the
> >      > majority of sites, and should that not end up being the case, we
> >     need to
> >      > look at the usability and benefit in that scenario as well.
> >      >
> >      > I think this is what the ACTION wanted, i.e. for me to state this
> >     point
> >      > more explicitly. I am going to therefore assume that my work on this
> >      > action is complete, unless I hear otherwise.
> >      >
> >      > -Ian
> >
> >     --
> >     /*
> >     PhD Candidate
> >     Vice President for External Affairs, Graduate Student Assembly
> >     Carnegie Mellon University
> >
> >     Legislative Concerns Chair
> >     National Association of Graduate-Professional Students
> >     */
> >
> >
> >
>
> --
>
> /*
> PhD Candidate
> Vice President for External Affairs, Graduate Student Assembly
> Carnegie Mellon University
>
> Legislative Concerns Chair
> National Association of Graduate-Professional Students
> */
>

Received on Wednesday, 14 November 2007 00:22:06 UTC