This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 16056 - Invalidate submitted results when the test changes
Summary: Invalidate submitted results when the test changes
Status: RESOLVED INVALID
Alias: None
Product: Testing
Classification: Unclassified
Component: Test Framework (show other bugs)
Version: unspecified
Hardware: All All
: P3 normal
Target Milestone: ---
Assignee: Michael[tm] Smith
QA Contact: This bug has no owner yet - up for the taking
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-21 11:19 UTC by Ms2ger
Modified: 2012-02-21 21:44 UTC (History)
2 users (show)

See Also:


Attachments

Description Ms2ger 2012-02-21 11:19:37 UTC
http://w3c-test.org/framework/details/dom4-Ms2ger-submissions/Node-cloneNode.html/ contains a lot of failures because the test used to be wrong. Now I fixed the test, these old results should be removed somehow, preferably automatically.
Comment 1 Michael[tm] Smith 2012-02-21 12:51:47 UTC
We don't yet have any way to automatically remove test results. It can be done manually by going into the DB and running some SQL commands but even then it requires touching two or three different tables and is error-prone and leaves some messiness behind. So having a clean way to do it from the Web UI would definitely be great.
Comment 2 Michael[tm] Smith 2012-02-21 13:27:24 UTC
Philippe points out to me that if you reset that test-suite data, but submitting a new manifest or even re-submitting the original manifest, the old results will disappear. I'm not sure they disappear from the DB, but they do at least disappear from the UI.
Comment 3 Peter Linss 2012-02-21 19:01:11 UTC
The proper way to handle this is to have revision information for the tests. I strongly recommend that you remove the manifest import options that don't have revision information and require revision IDs for all tests.

All results are stored per revision of the test, if the test changes (and gets a new revision ID) then old results won't be applied to the new revision.

The old result data is preserved for historical information.
Comment 4 Ms2ger 2012-02-21 21:05:51 UTC
They are in a VCS, so I believe it is pointless to store any revision data elsewhere.
Comment 5 Peter Linss 2012-02-21 21:44:57 UTC
ms2ger, it's not about storing revision information elsewhere, it's about making the revision information available to the test framework. The framework happily accepts (and is in fact designed to accept) mercurial changeset IDs for the revision info.

The test suite build code used by the CSS WG automatically generates the manifest files and populates the revision information with the most recent revision from the VCS.

Also note that getting the revision of the test file is generally insufficient unless the test is entirely comprised of a single file. The latest revision of the test, support files (stylesheets, images) or reference files is what's relevant here. Please see the docs for the framework.