Test ids do need to be unique across all test suites, this is by design allowing the same tests to be used in different test suites. 

Results are stored per test and will be reported in all suites that use the same version of the test.

If the tests are truly unique, then they need to have unique names. If they are different versions of the same test, then the proper version information needs to be present in the manifest when the tests are imported into the framework.

Thanks for clarifying that we need to avoid potential clashes when using tests in other test suites. I'll revisit the i18n naming policy to try to ensure that this is the case.

I guess I'm not fully clear about what constitutes a 'unique' test, and what the best course of action is for the tests in question. I can see a couple of possibilities.

The tests in question test exactly the same assertion, it's just that the test itself is served up in different syntax, depending on whether this is the html4, html5, xhtml1.0(html), xhtml1.0(xml), xhtml5 or xhtml1.1 test suite.

(There are some tests that are not relevant in some formats, or are only relevant as exploratory tests. If a test is not relevant to a format, it is not included in a the test suite for that format. These conditions tend to occur in tests of character encoding declarations, language decls, etc. that use different markup depending on the format. For an example of how tests map to formats differently you can see http://www.w3.org/International/tests/html-css/list-encoding. My concern here is with tests that *are* relevant in more than one format.)

I could create ids so that each test suite only shows the results of tests in that particular format.

On the other hand, i could allow results to accumulate, as they currently do, if they are testing the same thing. In this case, however, it would be desirable to be able to tell which test relates to which format. I assume that this is dependent on the "proper version information" in the manifest. My question is, what version information is that, specifically?  Is it the format field? How do i distinguish between html4 vs html5 tests?

FWIW there is no way that child tests in javascript testsuites will have globally unique names. I am extremely reluctant to consider anything less than (test url, testcase name) to be a unique identifier for an individual test (or, in the case of reftests, (test url, ref url)). I haven't studied the test harness so I don't know if this is a problem or if I am misinterpreting the issue here.

If the tests are the same but only presented in a different format, then they
should have the same name. The framework still considers them to be the same
test but does record which format the result was reported against.

The summary report will merge results from different formats, but the details
report displays the format information.

The test suites are not meant to be separated by format, but to track the
different formats within a single suite.

You can also designate which formats should be excluded for specific tests by
setting the appropriate flags for those tests (and the corresponding filter
flags in the format information). 

This is sounding like a setup issue, and possibly a different expectation of
test results. 

Instead of having different test suites, you should only have one, with the
proper test urls set per format. It looks like you also need to get Mike to set
up the proper format information.

(In reply to comment #3)
> FWIW there is no way that child tests in javascript testsuites will have
> globally unique names. I am extremely reluctant to consider anything less than
> (test url, testcase name) to be a unique identifier for an individual test (or,
> in the case of reftests, (test url, ref url)). I haven't studied the test
> harness so I don't know if this is a problem or if I am misinterpreting the
> issue here.

The child tests won't need to have globally unique names, only the test files containing the child tests do.

Also, the term 'global' in this case means the set of tests stored within one instance of a framework (and one test source repository for the management system).

Using filenames as some kind of identifier worries me greatly. Not only do I think that requiring uniqueness is an undue burden, I am also concerned that it doesn't capture the full informtion required to load a test. In particular some tests will require specific query strings or fragment identifiers. Therefore the only way to correctly identify a test is by its URL and name (for javascript tests) or URL and reference URL (for reftests). We got this wrong in our internal TC management system and the impdedance mismatch between the parts that incorrectly assume tests are directly related to files on the filesystem and the parts that correctly assume tests are identified by the given, URL-containing, tuples is a frequent source of problems.

(In reply to comment #4)

While it is common to run CSS tests in the CSS test suite in more than one
format, those tests are usually checking an assertion about the same part of
the *CSS* spec.

That's not what we're talking about here. 

I'm talking about a test in html5 format that is checking something against the
html5 spec, and another test, that happens to be testing the same assertion,
but that is testing a point in a completely different spec, such as HTML4. 

If I'm interested in reviewing the results of tests related to the html5 spec,
do i really want to see the results of tests for all the other formats that
happen to test the same thing? Probably not, most of the time. Certainly not,
if I'm trying to ascertain whether we have the two passes that are needed for
CR entry criteria for the html5 spec.

(In reply to comment #7)
> (In reply to comment #4)
> 
> While it is common to run CSS tests in the CSS test suite in more than one
> format, those tests are usually checking an assertion about the same part of
> the *CSS* spec.
> 
> That's not what we're talking about here. 
> 
> I'm talking about a test in html5 format that is checking something against the
> html5 spec, and another test, that happens to be testing the same assertion,
> but that is testing a point in a completely different spec, such as HTML4. 
> 
> If I'm interested in reviewing the results of tests related to the html5 spec,
> do i really want to see the results of tests for all the other formats that
> happen to test the same thing? Probably not, most of the time. Certainly not,
> if I'm trying to ascertain whether we have the two passes that are needed for
> CR entry criteria for the html5 spec.

Ok, then the tests should still be linking to the relevant sections of the specs they're testing.

So you have one test 'foo' in HTML5 format, linked to a section of the HTML5 spec. Then you can have another test, also called 'foo' in HTML4 format, linked to potentially a different section of the HTML4 spec.

In both cases, the tests are linked to the specs that they are testing, and the results will show up in the relevant test suites.

So I do now see that there's an issue in the result reporting where tests in  formats not specified by the test suite may show up in results (and I'm taking the bug to fix that), however you're still not setting up your test suites properly.

First, you need to remove your section data, this is only for sections of specs, not of test suites. 

Second, you need Mike to set up the proper format data in the framework, there is currently only one HTML format: 'html', you need separate formats listed for each of your test formats, ie: HTML4, HTML5, etc. 

Third, ensure the proper spec section data is loaded for the specs you are testing: ie: HTML4, HTML5 etc. 

Then make sure your test suites are set up properly, ie: the i18n-HTML4 test suite lists only the tests in HTML4 format, linked to the relevant sections of the HTML4 spec and using the HTML4 format identifier; the i18n-HTML5 test suite lists only the tests in HTML5 format, linked to the HTML5 spec, using the HTML5 format identifier, etc.

The test suites you are creating are really sub-sections of the HTML4, HTML5, etc test suites that deal with i18n issues. The tests in those suites will be listed by the sections of the specs they are testing, and any results gathered will be shared with other test suites that use the same tests (like a full HTML5 test suite).

> First, you need to remove your section data, this is only for sections of
> specs, not of test suites. 
> 
> Second, you need Mike to set up the proper format data in the framework, there
> is currently only one HTML format: 'html', you need separate formats listed for
> each of your test formats, ie: HTML4, HTML5, etc. 

See https://www.w3.org/Bugs/Public/show_bug.cgi?id=15061

> 
> Third, ensure the proper spec section data is loaded for the specs you are
> testing: ie: HTML4, HTML5 etc. 

See https://www.w3.org/Bugs/Public/show_bug.cgi?id=15079

> Then make sure your test suites are set up properly, ie: the i18n-HTML4 test
> suite lists only the tests in HTML4 format, 

Already the case. 

>linked to the relevant sections of
> the HTML4 spec and using the HTML4 format identifier;

Waiting for action on the above two bugs before I can do that for HTML4. 
 
> the i18n-HTML5 test suite
> lists only the tests in HTML5 format, linked to the HTML5 spec, using the HTML5
> format identifier, etc.

This is now all done for the html5 tests. I had the mechanism in place, but was waiting to transition to it. The spec indexes were intended to be a temporary thing until I was confident that things should work the way they are intended, and to enable me to do some testing.

> The test suites you are creating are really sub-sections of the HTML4, HTML5,
> etc test suites that deal with i18n issues. The tests in those suites will be
> listed by the sections of the specs they are testing, and any results gathered
> will be shared with other test suites that use the same tests (like a full
> HTML5 test suite).

One thing that I was (and to an extent still am) unsure about is that the i18n tests are often exploratory or test things that are not required by the spec at all. For now, i have aligned those tests with sections that contain related information.  I assume that those tests will be ignored if the HTML5 (or other) WG decides that they want to import our tests to their test suite.