ACT Benchmark description

ACT Benchmark Method

The ACT Task Force will define a method for benchmarking ACT rules. This is intended to get hard numbers about the accuracy of rules. The following questions will need to be answered:

What pages should a rule be tested against?
How should the expected outcome be determined?
How many results are required to determine the rule's accuracy?

The ACT Benchmark Method will either be published as part of the ACT Framework, or as a separate note. Further discussion is needed to determine which is the preferable approach.

ACT Benchmark Tool

The ACT Benchmark Tool will be developed by Auto-WCAG, and should serve both as proof of the method, and as tool to be used in further development of the ACT Rule Suite (deliverable 3). The exact details of this product can be determined once the benchmark method is decided upon.

This tool will be used to test that rules don't cause (unexpected) false positives. This will be done by applying an implementation of ACT Rules, and either having those results manually verified by an accessibility expert, or comparing the result to the findings from a manual audit of that same webpage. Doing so a number of times will give an indication of the rule's accuracy.