FYI: Using EARL for Validator Web Service Format

 
FYI, from the Unicorn project [1] mailing list.
IMO there are several interesting questions for discussion, can't
imagine why ERT is not included in the cross posting :o/

[1] - [http://www.w3.org/QA/Tools/Unicorn/]

Regards, 
 CI.


> -----Mensaje original-----
> De: public-unicorn-request@w3.org 
> [mailto:public-unicorn-request@w3.org] En nombre de olivier Thereaux
> Enviado el: viernes, 02 de marzo de 2007 7:07
> Para: www-validator Community
> Asunto: Using EARL for Validator Web Service Format 
> 
> 
> Hi all.
> 
> [ This message is cross-posted to the Markup Validator, CSS 
> Validator, Unicorn and qa-dev lists, with a reply-to set to 
> the list with the largest number of participants. Apologies 
> for the cross- post, I hope that was the best way to include 
> all parties involved.  
> Please respond in the www-validator list only. Thank you. ]
> 
> In the context of discussions around "a simpler web service 
> response format" for our validators and for unicorn [0], we 
> recently had a look at EARL [1], which has been evolving a 
> lot recently, and will soon get in last call.
> 
> [0] Thread starting at:
> http://lists.w3.org/Archives/Public/www-validator/2006Dec/0053.html
> continued at:
> http://lists.w3.org/Archives/Public/www-validator/2007Jan/0004.html
> 
> [1] http://www.w3.org/WAI/ER/
> 
> The validators are an obvious customer for EARL. Actually, 
> EARL identifies from the introduction of the spec the example 
> of a validator output using EARL (See [2], Example 2) as one 
> of their basic use cases.
> 
> [2] http://www.w3.org/WAI/ER/EARL10/WD-EARL10-
> Schema-20070226#testresults
> 
> At this point, only one of the validators produced by W3C has 
> an output in EARL [3], and it's a very old version of EARL. 
> Updating that is on the radar, and I'm thinking it could very 
> well replace the format we had tentatively designed for 
> unicorn, if it allows a better, simpler (and standardized) 
> format to be used.
> 
> [3] http://validator.w3.org/docs/users.html#Output
> 
> 
> The basic model of EARL takes an assertor, checking a test 
> case against a test subject and making an assertion on the 
> test result. So far, so good.
> 
> A few of the questions we asked ourselves while starting to 
> learn about EARL were:
> 
> Q1) how could we use earl and have a way to show, in the 
> results, the list of messages (errors, warnings) that usually 
> come with validation reports? Would we have to extend it with 
> another namespace?
> 
> earl:info could do the job.
> [[
>      Additional information beyond the description of the 
> result. For example warnings or other informative messages 
> that may help a reader better understand the result. It is 
> recommended to use Literal values for such additional messages.
> ]]
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testresult
> 
> I think the recommendation to parse as Literal is new. Which 
> would make more sense, if we want to list errors, warnings 
> and info messages, possibly along with a count info? Use 
> divs, ol, li in a Literal? Or use something else, in another 
> namespace? What would be simpler, what would be more flexible?
> 
> 
> Q2) In the case of the CSS validator, when checking an html 
> page for example, the results are given for that page and the 
> CSS files it links to. What, then, is the "test subject"?
> 
> Still unsure. I think it would be best to keep the page given 
> by the user to be checked as the test subject, but it would 
> also be useful to identify the fact that there are sub-subjects.
> 
> Maybe the CSS validator could make assertions on the single 
> CSS documents, and have information on the fact that they are 
> "part" of a wider test subject with dct:hasPart or dct:isPartOf:
> [[
> dct:hasPart
>      Relationship to other subjects that are part of this 
> subject dct:isPartOf
>      Relationship to other subjects of which this subject is 
> a part of ]] 
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testsubject
> 
> 
> Q3) What are our Test Criteria?
> 
> I suppose we'd have to have test criterion URIs for each of 
> the validations.
> And consider the "tasks" of Unicorn (general conformance, XHTML+CSS 
> +...) as earl:TestRequirement
> [[
>      A higher-level requirement that is tested by executing 
> one or more sub-tests. For example WCAG 1.0 Checkpoint 1.1 
> which is tested by executing several sub-tests and combining 
> the results.
> ]]
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testable
> 
> 
> Q4) What does Unicorn do? Collect and present assertions, or 
> is it an assertor too?
> 
> Would it be interesting to consider it as a compound assertor?
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-
> Schema-20070226#compoundassertor
> but that's only interesting if Unicorn itself has an earl 
> OUTput, while for now I think the interesting part is to have 
> the various observers have EARL outputs that are used by 
> unicorn as INput to be merged and processed.
> 
> Q5) Does the Outcome hard-coded pre-defined values cover all 
> our needs?
> 
> Would be nice asking the ER WG to add some, if not.
> http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#outcome
> At least, EARL would be more precise than what the Unicorn 
> format currently has, because the Unicorn format says 
> pass/fail without describing what it passes or fails. It is 
> unclear whether it means "did not pass validation" or "could 
> not validate". EARL has values for both.
> 
> Q6) Misc Notes
> 
> * earl:sourceCopy could be interesting for "direct input", 
> although I'm unsure whether it would be useful to copy 
> massive chunks of text around like that. Could we just use a 
> hash as earl:subject? We need to find ways to identify the 
> subject for direct input and file upload, in any case.
> 
> * Our assertors are earl:Software acting in earl:automatic mode.
> 
> * We can probably make the output formats short by using URIs 
> refering to full description for the assertors and tests, 
> instead of having the full description and classes each time.
> 
> * ... I had more notes, but I need to find the deadwood I had 
> written them on.
> 
> 
> Hope this gives us an interesting material for discussion, for now.
> 
> olivier
> --
> olivier Thereaux - W3C - http://www.w3.org/People/olivier/ 
> W3C Open Source Software: http://www.w3.org/Status
> 
> 
> 
> 
>

Received on Friday, 2 March 2007 08:39:38 UTC