Web Annotation Working Group Teleconference

<TimCole> Agenda: https://lists.w3.org/Archives/Public/public-annotation/2016Mar/0104.html

<TimCole> scribenick: bjdmeest

Acceptance of Minutes: https://www.w3.org/2016/03/18-annotation-minutes.html

<TimCole> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/03/18-annotation-minutes.html

TimCole: any concerns?

RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/03/18-annotation-minutes.html

Results of the CFC

RESOLUTION: CFC was sent out last week
... 13 plus ones since a couple of minutes ago
... no 0 or -1s
... so CFC is accepted
... any concerns on the call?

<TimCole> https://github.com/w3c/web-annotation/issues/186

RESOLUTION: the CFC was under the assumption that the minor editorial issues would be addressed before publishing
... everything done except for one

ivan: that's fine now, remaining is for the future
... if we decide to make a FPWD, there are two (minor) consequences
... first: history of the splitting up is lost
... second: patent policy would require to start from scratch for that document
... so at least 6 months needed between FPWD and REC
... that's not ideal
... I discussed
... result:
... Vocab doc is published as FPWD, and previous version is the Model doc
... so consequences are resolved
... practical consequence is a small editorial change

<TimCole> Proposed Resolution: publish all 3 as Working Draft, with previous draft for Vocabulary being the earlier Data Model draft

RESOLUTION: publish all 3 as Working Draft, with previous draft for Vocabulary being the earlier Data Model draft

ivan: question is: are the documents as they are today final and ready to be published?

bigbluehat: yes

ivan: also: have they been checked by link checker and html checker etc.?

paolociccarese: We can do it again

ivan: I will have to check it to be safe, but if you guys could do that by Monday, I can do the rest of the admin on Monday.
... PaoloCiccarese, also change the previous version of the Vocab doc to the Model doc, as discussed
... I will pick it up from there

ShaneM: Ivan, should SoTD be updated to say this is a split?

ivan: Paolo can do it as well, but yes!
... in the status of the Vocab document, there should be an extra sentence that this is a split from the Model doc

Welcome ShaneM

ShaneM: I've been with W3C since '97
... I;m with spec ops, doing standards related work
... Shepazu contacted me about testing
... I have a ton of questions
... I've been doing standards work since '85, and testing since '87

ivan: Shane is modest. He was one of the main editors for the RDFa spec
... which might be useful
... he also co-maintains respec

TimCole: other anouncements?

<ShaneM> It is the section entitled "Status of this document"

Testing

<TimCole> https://www.w3.org/2016/03/18-annotation-minutes.html#test

TimCole: There are some notes in last week's minutes
... we have to look into our documents, find the features that are described, and provide for a strategy to test these features
... and make sure they are unambiguous and implementable
... I welcome help
... Identifying the features is important, implementing them also
... particularly the selector approach might not be implemented in every system
... we have to have at least two implementations for each feature
... how do we go on with this?

ivan: the real question (for the model) is: what is really what we want to test?
... it is a vocabulary for a specific usage, we have to identify what we want to test
... the current direction is that we define specific scenarios
... an implementation should show that these scenarios can be mapped on the correct annotation structures
... and maybe also the other way around: annotation structures should be understood by implementations and in some way tell us what they would do with these annotations
... questions are: does that provide a reasonable way of testing the spec, and can this be translated into proper tools?
... we have to be very careful that we are not testing implementations, but we are testing the spec

<Zakim> ShaneM, you wanted to disagree a little with ivan about what must be tested...

TimCole: about the scenarios: a suggestion was to have sample resources to be tested, to illustrate the features

ShaneM: we're not testing implementations was said...
... but each feature must be implemented
... so we test the implementations implementing the features

ivan: if my goal would be to test the implementation, I would use automatic tests
... in this case, we test the specifications, we can ask implementers to send a report, without knowing how they tested their implementation

ShaneM: In the Web annotation spec, there are requirements on a lot of different actors
... data structure requirements
... client requirements
... interpretation requirements
... server requirements
... I assume you want to test all of them?

TimCole: I think so
... question was: what does an implementation have to do with an existing annotation to make sure it interprets it correctly?

ShaneM: you don't have behaviour requirements
... or UI requirements
... so that makes your testing burden lower

,,, here, you just have to ensure that consumers of the data format receive the data format intact

scribe: you can ignore SHOULDs for testing purposes
... I would focus on the MUSTs and MUST NOTs

<tbdinesh> +1 for Tim

scribe: Talking about MUSTs
... there's some data structural requirements, which come for free via JSON-LD
... so testing conforming output is probably kind of manual
... e.g. these N selection annotations needs to be tested
... you don't want to test if the region makes sense or the CSS selector is correct etc.

ivan: Let's say that we describe a scenario: here's an SVG file, we want to put an annotation on this circle on the upperhand corner
... the resulting annotation structure should correspond with an annotation put on that corner
... in the output, we assume an SVGSelector to the correct corner
... so we need to check for correct JSON-LD, and correct to our spec (i.e., that it's an SVGSelector)
... but we don't have to check that the SVGSelector actually selects the correct target?

ShaneM: you go into that depth, but I'm not sure it's required
... because there are a lot of ways of specifying that region
... suppose you have a server, he's going to analyze that annotation, but it's hard to analyze every detail
... you would need an SVGrenderer
... you could do that manually, but that very consuming

ivan: I was considering human testing, but that's very time consuming

ShaneM: I always look at this as: what can we automate?

ivan: the only advantage is that the model is relatively simple
... we don't have a huge number things to test

ShaneM: there are some combinatorial things that I have noted

TimeCole: Manual can be very expensive, we thought about: I have a specific scenario: this is the image, this is the exact circle to annotate, and that should limit the number of ways to do it
... e.g., textQuoteSelectors doesn't have that many ways

ShaneM: depends on what kind of constraints you want to put on the client
... talking about textRangeSelection
... you allow for a number of ways to express
... not all clients will implement all ways
... And I assume the client decides which expression will be the right way
... depending on the context
... do you want to require for test X that the client gives a CSS selector, and test Y gives a textRangeSelector

ivan: that doesn't sound right

ShaneM: another way would be: here's a sample document, select these 5 words
... the server-side should check: which expression does it use, and is that correct?
... that way, you simplify the test matrix, without testing every possible combination
... you can say: the textselector works

ivan: it can happen that one of the selectors is never used
... because another combination of selectors always works for the given scenarios
... does that mean that that selector shouldn't be in the spec

ShaneM: you could say it becomes a feature at risk
... or, it could be part of the testing cycle
... e.g., are there ways to require the feature

TimCole: one extra thing before the end of the call
... who on the WG could help to build the list of features?
... someone (pref. not the editors) should identify the features in the docs
... certainly the MUSTs, maybe the SHOULDs

ivan: we have to be careful to minimize people's time
... we have to know in what format to make these scenarios

<Zakim> ShaneM, you wanted to say that you should also identify if requirements are on a repository vs. a generator vs. a consumer

ShaneM: it's useful, I've been doing it, I think one of the critical pieces are checking whether MUSTs and SHOULDs are really MUSTs and SHOULDs
... and also, what parts of the system these features are one
... we need to compartimentalize by
... repository vs. a generator vs. a consumer
... in terms of interoperability, we should make sure any generator can talk to any repository

ivan: you have implementations not bound to a specific server
... there are 2 directions of testing:
... scenario-based
... and from annotation-structure to interpretation of the implementation

TimCole: I don't know whether the compartiments are already named very well

PaoloCiccarese: I don't have specific testing-ideas
... I didn't have to test for different servers or clients
... we changed the specs over time, in my case, we used SPARQL validators for the model
... and adapted over time

TimCole: The Annotation Community Group identified 55(?) MUSTs and SHOULDs, and validated semantically using a set of SPARQL queries
... two for every must and should
... one to check whether a feature applied, a second to validate
... but the spec has changed since then
... and not only semantic anymore
... there must be three components, but there not defined yet

<PaoloCiccarese> +1

ShaneM: I agree, and I would structure the tests that way

ivan: I would be very helpful if Paolo could give a small description of how his implementation was tested
... it would help me, but maybe also Shane
... maybe I could ask Takeshi to do something similar

takeshi: I have some, but I was thinking about modifying the testing files to test for internationalitation

<ShaneM> Every little bit helps

ivan: having a good feeling of what is currently being done would be very helpful

ShaneM: every little piece will come together in the whole
... tests should be as discrete as possible
... big (i.e., integration of system) tests exist
... but the small tests show where something goes down
... e.g., the same scenario needs to be tested for 11 different variables

PaoloCiccarese: critical point would be the republishing of annotations
... I'm not sure we have the solution to that
... it will be interesting to test
... it will be cross-system testing
... test system a, then system b, then send from a to b, and have certain expectations
... it's one of the most delicate points
... duplications of annotations will make things go out of control
... but it's the web, so it will happen
... about my current testing: I mostly do code testing, very tailored, lots of RDF
... so testing is many roundtrips to the triple store

TimCole: There's a need to talk about what we think are the compartiments (generator, consumer, repository)
... then, we need to talk about scenarios and features
... to make some progress before F2F, for next week we might also talk about this testing topic

ivan: email discussion is fine at the moment

ShaneM: I'll put issues or send questions to the mailing list

ivan: will you join the WG?

ShaneM: I'll ask, I know the drill

TimCole: [adjourn]
... next week: F2F, conformance
... bye!

<ivan> trackbot, end telcon