From Rotan Hanrahan <Rotan.Hanrahan@MobileAware.com>:

If validator gives boolean result, then "false" is so negative that you are not encouraged 
to try again. But if validator gives a ranking (e.g. percentage) then you are encouraged to 
improve your result. Eventually you get 100% and are conformant. [...]

At a minimum the errors should be classified, and a weighting assigned to each
category according to perceived severity and difficulty of correction. Add all
of the (weighted) errors in the page. Divide the sum by a factor proportional to
the log of the size of the document (so that larger documents do not have
excessively large error scores, since the correction of one error in a document
can easily be replicated for all such errors that occur in the document). When
your score is Zero, you are conformant. Of course, this formula is quite simplistic,
but would be easy to implement and, regardless of its actual interpretation, if
the scoring system is consistent and has the property of always decreasing whenever
a correction is made, then it should be useful. Anyway, the marketing people won't
care as long as their scores beat the competition :-)