Web Annotation Working Group Teleconference

16 Sep 2016


See also: IRC log


Shane McCarron, Tim Cole, TB Dinesh, Dan Whaley, Takeshi Kanai, Paolo Ciccarese
Rob, Benjamin



<TimCole> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/09-annotation-minutes.html


RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/09-annotation-minutes.html


<TimCole> See https://lists.w3.org/Archives/Public/public-annotation/2016Sep/0058.html

Charter extended through February.

Issue Updates

<TimCole> None


Testing Updates


There is an issue with the link into w3c-test.org with a deep path doesn't seem to work right

TimCole: there might be an issue in the future when there are other test collections

ShaneM: it works for me as well

dwhly: I will try to sort it out

TimCole: Janina mentioned that refreshing the window index.html sometimes didnt work.
... Nick reported the timing of the text box means pasting before the page populates will cause the informaiton int eh window to get cleared out.

ShaneM: That doesnt surprise me

TimCole: We did consolidate the tests. My impression from everyone is that is better.

ShaneM: Dont change the anmes of any tests in the future.

TimCole: I think it is order sensitive too. I will not change names nor order.

<tbdinesh> update: test links seem to work ok now. both. maybe its to do with server load?

ivan: At the end when we are finished we will need to reconcile things. We have different columns for the same implementation now at different times.

tbdinesh: oh, that's possible.

ShaneM: The various columns represent tests run with different input.

TimCole: If you have an implementation that generates different kinds of annotations... for example Janina's can link images, annotation, or transcribe... you end up with different annotations that use different features of the model.
... so in order to see the complete set of features, you need to test multiple annotations and then collapse the columns together to show the various features supported.
... I still think that we should collapse the columns together.

ivan: I didnt know that. I am not really sure how we do that at the end.

TimCole: I think that means if there are any from the same implementation, you OR them together.
... you may not want to do that with the mandatory tests.
... you might want to eliminate columns with failed mandatory tests.

ivan: from the CR point of view... which tests the spec. If an implementation doesn't pass a required feature, I would expect essentially the three columns to be identical for a correct implementation.

TimCole: It might be that an implementation hasn't implemented a new kind of selector. We have an assertion that checks if you have SpecificResource in your annotation and it has a selector it is one of the six of seven types defined in the model.
... if I define a new sleector type, one of my annotations may not pass that test.
... have I failed from a requirements standpoint? That's something we would need to interpret. If we leave it a MUST test we need to decide if we can ignore that kind of failure or not.

ivan: if it is an extension then it cannot be a must

TimCole: We have a section that defines how you extend the model.

ivan: if it is not a normative section then it is not a must
... so in this case the extension is not something we want to test. It feels esoteric.
... for MUST tests all the columns should have pass and fail. So we can throw out the aberrant results if some fail a mandatory test as long as some tests PASS the mandatory tests for the same implementation.

TimCole: We should talk to Rob about this. We we try to make the sections about extensions normative then we need to update the SHOULDS to MUSTS

(some discussion about extending the vocabulary...)

TimCole: let's not finish this here...

ivan: no. it is important. We need to be sure that we have not made editorial errors.
... for example I see that the JSON-LD Frame is normative and it should not be.
... the boilerplate for the spec says appendices are normative unless they explicitly say they are informative

TimCole: Discussion of optional "fails". Should we report them at all?

ivan: I am a little bit messed up... what happens right now?

TimCole: for example we have 36 checks related to agents.

agents is optional. You SHOULD have an annotationCreator but it is not required

scribe: if you don't have one, then there are a bunch of related tests that will fail
... if you don't report those you will get 8 yellow boxes. If you report the SHOULDs you will get some red and some yellow.

ivan: what are our options?

ShaneM: We don't really have any control. There is pass, or fail, or no data (yellow)

ivan: Since it is not a MUST it is not really a fail.

TimCole: So how are we looking at the may requirements? If no one implements text direction as a result of testing, does that mean we should mark it at risk?

ivan: that seems crazy to me.

TimCole: It is not a SHOULD nor a MUST. it is in because people think it might be useful.
... there might not be anyone who uses it.

ShaneM: can we actually mark things at risk now?

ivan: No, we cannot.
... If a SHOULD is not implemented, then it can be yellow... but I feel a little uneasy about it.
... it is not FAIL in that it doesn't meet the specification.

ShaneM: THe features need to be called out.

TimCole: The SHOULDs seem to map to features.
... MAYs are not really features.

<Zakim> ShaneM, you wanted to ask about what we do to recreate the results?

ivan: the problem is that the reporting mechanism is too strict.

ShaneM: we can change it... but how?

ivan: On the report... if there is a SHOULD or MAY and if an implementation doesn't do it then I would like to see that as a different entry that means not implemented. Something that indicates it is not an error.
... the FAILS should be only appearing for the MUST features.
... for optional features there should be a way to indicate that an implementation supports or does not support an optional feature.

ShaneM: What happens if an optional feature is supported but does not pass the tests?

ivan: then that is an error.

TimCole: We have that in the mandatory tests now.
... Reviews what are "features" in the data model
... no options are listed as features.

ivan: then they should never me listed as a fail.

TimCole: They only thing fuzzy one agent class as related to an annotation. With predicates creator and generator.
... they are currently listed as SHOULDs

ivan: I believe the agentClass is an option and has requirements

TimCole: In actuality the agents do not have this requirement

ivan: then those two entries do not mean anything
... hard to discuss without Rob, but there doesn't seem anothing relevant here to exit criteria

TimCole: so in terms of the tests, we have a couple options.
... we could delete the tests. We could NOT report them failing. But we could still gather the data (in case there is any debate later). Or we can leave them as they are and they show red.

ivan: we sort of left it open. Is there a logic we can follow?

<Zakim> ShaneM, you wanted to make a proposal

<TimCole> Proposal: Shane has a version of the test library that will not report when an optional assertion fails

<TimCole> ShaneM: If an assertion is a should or may and it fails, the test does not report a result

<TimCole> ... the effect is a yellow box in the final report

<TimCole> ... suppose there is an assertion that the sky is pink. No implementation passes, so no row will appear for this assertion

<TimCole> ... it's yellow for all others if at least one implementation uses and all others don't

TimCole: I have a reference implementation that supports every optional

ivan: I don't mind that being in the report. It is a real implementation.
... I dont like it to look like a FAIL on a SHOULD is the same as a FAIL on a MUST

ShaneM: Isn't it okay to have that indicated int he name

ivan: No - it isn't. People don't look closely.

TimCole: If we have the implemenmtation that ignores shoulds and mays, let's try it.
... For example in one implementation there are three real fails. I hope they choose to fix those before the end of CR. If fixed they would have green for everything mandatory and some optionals
... they would have yellow for everything else.

<Zakim> ivan, you wanted to comment on something else

ivan: Ralph and I were looking at the reports. A minor comment. We use in the text "Reference Implementation". We have not defined it. We should use a different term.

TimCole: I didn't realize that

ShaneM: I did that - I called it the RI.

TimCole: No meeting next week because of TPAC
... meeting the week after that. Need to prioritize how to reach out to OA implementors that illustrate changes to the current model.
... changes of key names and a few other things. We have not collated those all in one place.

ivan: wait... I thought that
... in the model document there is appendix F.3. It details the changes.

TimCole: You're right. Good!

ivan: Might not be anything in the vocab but I think this is done. We may need more explanation.

TimCole: I will talk to Rob about it for next week.

Next meeting in two weeks

Summary of Action Items

Summary of Resolutions

  1. Minutes of the previous call are approved: https://www.w3.org/2016/09/09-annotation-minutes.html
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/09/16 16:05:31 $