Testing/CSS Tests on WPT

From W3C Wiki

Technical constraints have prevented the CSS WG from migrating its test suites to the main test repository. This has proven to make collecting and reviewing tests during TestTWF events particularly difficult.

Recently a proposal was made to allow the contribution of CSS tests directly to the main repository during TestTWF events.

This wiki page aims to gather the pros and cons of this proposal and those of maintaining the current status quo so as to help us make an informed decision.

Cons of the status quo

1. HG-Git setup is complicated and takes an extremely long time.

  • The initial setup requires 3-4 hour handshake between the 2 repos. If something goes wrong during this process, you have to start over.
  • There are strict compatibility issues with Python, HG, the HG-git extension, and the method by which you install this stuff (pip vs. easyinstall)

2. Merging individual pull requests takes much longer than merging directly into a github repo.

  • This is a multi step process, with added complexities when a user has multiple PRs with multiple branches. Just doing the manual steps takes 5ish minutes, and that's if you know what you're doing and have the steps well memorized. It's easy to screw up.

3. There is no clean easy way to merge just one PR from a user when they've submitted multiple PRs.

  • The hg (git) pull command pointed at a user's entire repo, not to a specific PR. When you do this, you get everything as branches and you must merge down to a single branch before pushing back to the main Mercurial repo.

4. Once the PR is merged into Mercurial, it takes 5-30 minutes to show up in the github mirror. The extra time happens when there are multiple PRs being merged in as there some queueing mechanism that happens. At TestTWF events, attendees and experts alike were confused when they didn't see their merged PRs appear in the tree.

5. We cannot use the issue tracker, which would be extremely useful during events (and beyond) for listing & assigning tests to be written.

6. Exactly 3 people in the CSSWG know how to do this, so the post-processing of these PRs is bottlenecked simply by who is able to merge them.

7. When PRs are merged, the notification is sent from 'W3C Sync Bot' - there is no way to know who merged it.

8. It's painful to have to document and support 2 separate repos. The current github101 doc is cluttered than it needs to be.

9. CSS test authors who want to use testharness.js have to pull the WPT repo anyway b/c testharness is a subrepo there.

10. It's way too much to ask of experts volunteering their time to help at these events to get trained on this to be able to merge themselves. Even if some were willing all of these other problems persist.

Pros of the status quo

Cons of the proposal to move to the main repository

See: http://lists.w3.org/Archives/Public/public-test-infra/2013OctDec/0014.html

Pros of the proposal to move to the main repository

1. No extra setup required

2. No extra training/knowledge (beyond basic Github PR knowledge)

3. Lots of people already know how to do Github PRs

4. All OWP tests are in one repo, so the docs can be simplified

5. PRs are merged into the main repo immediately

6. PR merge notifications are sent from person who did the merge

7. We can use the issue tracker

Alternate proposals

See: http://lists.w3.org/Archives/Public/public-test-infra/2013OctDec/0014.html

Retrieved from "https://www.w3.org/wiki/index.php?title=Testing/CSS_Tests_on_WPT&oldid=68902"