Tests with difficult/confusing pass condition/wording

Hi,

the following lists a number of tests that were found to be confusing with  
regard to determining a test result.

(Authors mentioned: CSS1 Test Suite Contributors, Elika J. Etemad, Gérard  
Talbot, Hilbrand Edskes, Ian Hickson, James Hopkins, Microsoft, Richard  
Ishida)

==== CSS1 Test Suite Contributors, Ian Hickson ====

http://test.csswg.org/suites/css2.1/20100815/html4/c414-flt-ln-002.htm
* The description is too complicated, and it's difficult to tell easily if  
things are really level. Maybe this can be split up into multiple simpler  
tests?
* Says "unless there is not enough room for the big box to fit on the  
small box's line after the small box" - but in fact some of the big boxes  
are /left/-floated
* "big coloured boxes should be level with the top of..." (assuming it's  
meant that the top border edges of the two boxes should be level) - not  
necessarily true. The float's top may not be higher than the top of the  
span's linebox, so the TC seems to assume that 1.3em lineheight is exactly  
enough space for a medium border.

http://test.csswg.org/suites/css2.1/20100815/html4/c44-ln-box-003.htm
Wording can be confusing. "just the green outline of an empty box" might  
be slightly better

http://test.csswg.org/suites/css2.1/20100815/html4/c525-font-wt-000.htm
http://test.csswg.org/suites/css2.1/20100815/html4/c527-font-10.htm
This assumes that there is a "very light" weight available for the default  
font or that a tester knows which faces are available and that using the  
lightest one is fine

==== Gérard Talbot ====

http://test.csswg.org/suites/css2.1/20100815/html4/separated-borders-model-007.htm
Uses the term "separated borders", would perhaps be better to describe  
expected appearance more directly. Moreover, this test seems to depend on  
the HTML 'rules' attribute mapping to/interacting with CSS rules in a  
certain unspecified way (Opera currently applies the horizontal rules to  
the <tr>s for instance, haven't investigated what makes Fx/IE fail it;  
HTML5 attempts to specify the presentational hints but seems  
unfinished/broken)

==== Gérard Talbot, Hilbrand Edskes ====

http://test.csswg.org/suites/css2.1/20100815/html4/min-height-106.htm
It's not at all clear from the wording that the green square should be  
covered by the scrollbar (which basically means that it will look like a  
non-square rectangle)

==== Gérard Talbot, James Hopkins ====

http://test.csswg.org/suites/css2.1/20100815/html4/max-height-105.htm
(Probably correct per the current spec, but none of the browsers I tried  
show a square in this case so this might change, see thread starting at  
<http://lists.w3.org/Archives/Public/www-style/2010Aug/0541.html>)

==== Ian Hickson ====

http://test.csswg.org/suites/css2.1/20100815/html4/background-root-020.htm
There are no "links below". Also, the last two sentences should be  
rewritten to not require an understanding of the specification

http://test.csswg.org/suites/css2.1/20100815/html4/quotes-035.htm
http://test.csswg.org/suites/css2.1/20100815/html4/quotes-036.htm
These are missing a proper pass condition

http://test.csswg.org/suites/css2.1/20100815/html4/tables-001.htm
This test could use a description/an explicit pass condition

==== Microsoft ====

http://test.csswg.org/suites/css2.1/20100815/html4/table-visual-layout-024.htm
This kind of pass condition is confusing in general (when does the test  
not pass?). In this particular case the "may" is actually also incorrect;  
the assert refers to a note, but the note links to a normative section, so  
overflow should be expected. (Of course this assumes that the span's  
content is wider than 3em when rendered with the default font, probably a  
safe assumption though using Ahem or non-textual content seems better)

http://test.csswg.org/suites/css2.1/20100815/html4/min-height-080.htm
http://test.csswg.org/suites/css2.1/20100815/html4/max-height-080.htm
See also  
http://lists.w3.org/Archives/Public/public-css-testsuite/2009Sep/0014.html  
- to an observer not familiar with the spec's terminology, it is not  
obvious that a thick line counts as a "box"

http://test.csswg.org/suites/css2.1/20100815/html4/background-130.htm
The wording "on top of" is potentially confusing. Maybe use "above"  
instead (also, again the issue of "box" vs e.g. "line" or "stripe")

http://test.csswg.org/suites/css2.1/20100815/html4/font-matching-rule-008.htm
* "a box" usually implies a rectangular shape, which is not what's being  
expected here
* the "slightly shorter" "half" is not actually supposed to be a half...
* it's not clear in which direction the shortening is to take place
* synthesizing small-caps is just a "may" i.e. optional

http://test.csswg.org/suites/css2.1/20100815/html4/text-transform-003.htm
Bad description: "if the line of 'Filler Text' and all of the characters  
of the text" should probably be "if all of the characters below this  
sentence are in uppercase" or something similar

http://test.csswg.org/suites/css2.1/20100815/html4/caption-side-001.htm
http://test.csswg.org/suites/css2.1/20100815/html4/caption-side-002.htm
http://test.csswg.org/suites/css2.1/20100815/html4/caption-side-003.htm
Bad description, expects "Filler Text" even though "FillerText" is  
specified

http://test.csswg.org/suites/css2.1/20100815/html4/table-visual-layout-025.htm
Not having vertical-align:top here might be confusing because the "2"  
digit will be overlapped (and isn't in the sample renderings)

http://test.csswg.org/suites/css2.1/20100815/html4/outline-003.htm
As has already been mentioned  
(<http://lists.w3.org/Archives/Public/public-css-testsuite/2008Mar/0002.html>)  
the tests in this series could use some extra horizontal margins. This one  
in particular is confusing since it calls for a comparison of widths, and  
the full widths of the left and right parts of the outline are clipped

http://test.csswg.org/suites/css2.1/20100815/html4/empty-cells-applies-to-001.htm
The second X could end up partially overlapping the border (depending on  
e.g. font), can cause confusion on whether the result counts as a "box"

http://test.csswg.org/suites/css2.1/20100815/html4/table-background-edge-and-border-model-002.htm
The second instance of "below" is misleading, makes it sound like the  
second square is supposed to be below the first square

http://test.csswg.org/suites/css2.1/20100815/html4/separated-border-model-005.htm
Misleading, gives the impression that the widths of cells in column 2 are  
supposed to be shorter than the ones in column 2. Also makes it seem like  
the cells are supposed to be bigger when compared to the border widths

http://test.csswg.org/suites/css2.1/20100815/html4/text-align-applies-to-001.htm
Having just one instance of "Filler Text" in the box would be less unclear  
(there is a space to the left and right of every instance except one,  
after all). Also, apparently the space outside the border doesn't count,  
so a browser that incorrectly right-aligns the text might not fail the  
test.

==== Richard Ishida, Elika J. Etemad ====
http://test.csswg.org/suites/css2.1/20100815/html4/list-style-type-armenian-002.htm
An improvement over previous version, but still not quite right (trying to  
use two different formulations at once). Using the word "numbering" might  
confuse some people who would expect to see Latin numerals


-- 
Řyvind Stenhaug
Core Norway, Opera Software ASA

Received on Wednesday, 8 September 2010 14:13:54 UTC