ACTION-330 Requirements for usability testing for conformace

ACTION-330 is about requiring usability tests for conformance, specifically
whether we can make any recommendations on how to conduct required usability
testing. My fear is that we are going to get into a situation like the
following:

We end up having a section of the recommendation saying "Do not show X in
the section of the chrome intended to convey trust information" or "Inform
the user of X", where usability testing is required to configure out whether
the user thought a particular part of chrome conveys trust decisions, or
user testing is done to figure out whether the user was actually informed.

The person doing the testing then has to design an experiment to test this
feature. The person doing the testing has an incentive to construct a test
where they will do well (to achieve conformance). You can imagine someone
therefore constructing an experiment in which the user is shown help pages
first, or given a manual and 1/2 hour to read it, or some other
non-realistic setting. This would likely produce a different result than an
experiment where the user simply dives right in to using the product.

You could also imagine less sinister ways to skew the results. For instance,
testing "whether the user was informed" - Someone could decide to sit a user
down for a half hour, have them go through a few sites (some of which
produce notifications), and then see that the user watched the notices.
Another person may say "Well, they notice the dialogs now because this is
the first time they're using the product, but after a while they might just
ignore them" and instead do a 30-day study, and see that the results on day
30 are very different than a 30-minute user study.

Hence, my main concern is that we are going to require usability testing for
conformance, and the way the test is constructed will be the primary factor
in whether an implementation appears "usable". As such, I think we would
have to lay out very clear guidelines on how the usability testing should be
done (basically specifying the experimental design), which seems fraught
with peril given how different implementations might be and might become
over time, or we would have to take a huge leap of faith. Personally, my
preference would be to avoid requiring [in the MUST sense] usability testing
for conformance in general, and instead come up with good guidelines for how
a usability test SHOULD be conducted to address these issues.

I believe this fulfills my requirements for ACTION-330.

-Ian