See also: IRC log
<TimCole> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/09-annotation-minutes.html
RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/09-annotation-minutes.html
<TimCole> See https://lists.w3.org/Archives/Public/public-annotation/2016Sep/0058.html
Charter extended through February.
<TimCole> None
*crickets*
There is an issue with the link into w3c-test.org with a deep path doesn't seem to work right
TimCole: there might be an issue in the future when there are other test collections
ShaneM: it works for me as well
dwhly: I will try to sort it out
TimCole: Janina mentioned that refreshing
the window index.html sometimes didnt work.
... Nick reported the timing of the text box means pasting before the
page populates will cause the informaiton int eh window to get cleared
out.
ShaneM: That doesnt surprise me
TimCole: We did consolidate the tests. My impression from everyone is that is better.
ShaneM: Dont change the anmes of any tests in the future.
TimCole: I think it is order sensitive too. I will not change names nor order.
<tbdinesh> update: test links seem to work ok now. both. maybe its to do with server load?
ivan: At the end when we are finished we will need to reconcile things. We have different columns for the same implementation now at different times.
tbdinesh: oh, that's possible.
ShaneM: The various columns represent tests run with different input.
TimCole: If you have an implementation that
generates different kinds of annotations... for example Janina's can
link images, annotation, or transcribe... you end up with different
annotations that use different features of the model.
... so in order to see the complete set of features, you need to test
multiple annotations and then collapse the columns together to show the
various features supported.
... I still think that we should collapse the columns together.
ivan: I didnt know that. I am not really sure how we do that at the end.
TimCole: I think that means if there are any
from the same implementation, you OR them together.
... you may not want to do that with the mandatory tests.
... you might want to eliminate columns with failed mandatory tests.
ivan: from the CR point of view... which tests the spec. If an implementation doesn't pass a required feature, I would expect essentially the three columns to be identical for a correct implementation.
TimCole: It might be that an implementation
hasn't implemented a new kind of selector. We have an assertion that
checks if you have SpecificResource in your annotation and it has a
selector it is one of the six of seven types defined in the model.
... if I define a new sleector type, one of my annotations may not pass
that test.
... have I failed from a requirements standpoint? That's something we
would need to interpret. If we leave it a MUST test we need to decide if
we can ignore that kind of failure or not.
ivan: if it is an extension then it cannot be a must
TimCole: We have a section that defines how you extend the model.
ivan: if it is not a normative section then
it is not a must
... so in this case the extension is not something we want to test. It
feels esoteric.
... for MUST tests all the columns should have pass and fail. So we can
throw out the aberrant results if some fail a mandatory test as long as
some tests PASS the mandatory tests for the same implementation.
TimCole: We should talk to Rob about this. We we try to make the sections about extensions normative then we need to update the SHOULDS to MUSTS
(some discussion about extending the vocabulary...)
TimCole: let's not finish this here...
ivan: no. it is important. We need to be
sure that we have not made editorial errors.
... for example I see that the JSON-LD Frame is normative and it should
not be.
... the boilerplate for the spec says appendices are normative unless
they explicitly say they are informative
TimCole: Discussion of optional "fails". Should we report them at all?
ivan: I am a little bit messed up... what happens right now?
TimCole: for example we have 36 checks related to agents.
agents is optional. You SHOULD have an annotationCreator but it is not required
scribe: if you don't have one, then there
are a bunch of related tests that will fail
... if you don't report those you will get 8 yellow boxes. If you report
the SHOULDs you will get some red and some yellow.
ivan: what are our options?
ShaneM: We don't really have any control. There is pass, or fail, or no data (yellow)
ivan: Since it is not a MUST it is not really a fail.
TimCole: So how are we looking at the may requirements? If no one implements text direction as a result of testing, does that mean we should mark it at risk?
ivan: that seems crazy to me.
TimCole: It is not a SHOULD nor a MUST. it
is in because people think it might be useful.
... there might not be anyone who uses it.
ShaneM: can we actually mark things at risk now?
ivan: No, we cannot.
... If a SHOULD is not implemented, then it can be yellow... but I feel
a little uneasy about it.
... it is not FAIL in that it doesn't meet the specification.
ShaneM: THe features need to be called out.
TimCole: The SHOULDs seem to map to
features.
... MAYs are not really features.
<Zakim> ShaneM, you wanted to ask about what we do to recreate the results?
ivan: the problem is that the reporting mechanism is too strict.
ShaneM: we can change it... but how?
ivan: On the report... if there is a SHOULD
or MAY and if an implementation doesn't do it then I would like to see
that as a different entry that means not implemented. Something that
indicates it is not an error.
... the FAILS should be only appearing for the MUST features.
... for optional features there should be a way to indicate that an
implementation supports or does not support an optional feature.
ShaneM: What happens if an optional feature is supported but does not pass the tests?
ivan: then that is an error.
TimCole: We have that in the mandatory tests
now.
... Reviews what are "features" in the data model
... no options are listed as features.
ivan: then they should never me listed as a fail.
TimCole: They only thing fuzzy one agent
class as related to an annotation. With predicates creator and
generator.
... they are currently listed as SHOULDs
ivan: I believe the agentClass is an option and has requirements
TimCole: In actuality the agents do not have this requirement
ivan: then those two entries do not mean
anything
... hard to discuss without Rob, but there doesn't seem anothing
relevant here to exit criteria
TimCole: so in terms of the tests, we have a
couple options.
... we could delete the tests. We could NOT report them failing. But we
could still gather the data (in case there is any debate later). Or we
can leave them as they are and they show red.
ivan: we sort of left it open. Is there a logic we can follow?
<Zakim> ShaneM, you wanted to make a proposal
<TimCole> Proposal: Shane has a version of the test library that will not report when an optional assertion fails
<TimCole> ShaneM: If an assertion is a should or may and it fails, the test does not report a result
<TimCole> ... the effect is a yellow box in the final report
<TimCole> ... suppose there is an assertion that the sky is pink. No implementation passes, so no row will appear for this assertion
<TimCole> ... it's yellow for all others if at least one implementation uses and all others don't
TimCole: I have a reference implementation that supports every optional
ivan: I don't mind that being in the report.
It is a real implementation.
... I dont like it to look like a FAIL on a SHOULD is the same as a FAIL
on a MUST
ShaneM: Isn't it okay to have that indicated int he name
ivan: No - it isn't. People don't look closely.
TimCole: If we have the implemenmtation that
ignores shoulds and mays, let's try it.
... For example in one implementation there are three real fails. I hope
they choose to fix those before the end of CR. If fixed they would have
green for everything mandatory and some optionals
... they would have yellow for everything else.
<Zakim> ivan, you wanted to comment on something else
ivan: Ralph and I were looking at the reports. A minor comment. We use in the text "Reference Implementation". We have not defined it. We should use a different term.
TimCole: I didn't realize that
ShaneM: I did that - I called it the RI.
TimCole: No meeting next week because of
TPAC
... meeting the week after that. Need to prioritize how to reach out to
OA implementors that illustrate changes to the current model.
... changes of key names and a few other things. We have not collated
those all in one place.
ivan: wait... I thought that
... in the model document there is appendix F.3. It details the changes.
TimCole: You're right. Good!
ivan: Might not be anything in the vocab but I think this is done. We may need more explanation.
TimCole: I will talk to Rob about it for next week.
Next meeting in two weeks