Web Annotation Working Group Teleconference

12 Aug 2016


See also: IRC log


Rob Sanderson (azaroth), Tim Cole, Nick Stenning (nickstenn), Jacob Jett, Ivan Herman, Dan Whaley, Ben De Meester, Benjamin Young (bigbluehat), Shane McCarron
Rob Sanderson, Tim Cole
Tim_Cole, nickstenn, azaroth


Agenda review

<TimCole> azaroth: minutes, announcements, internationalization issues (brief), testing

<TimCole> ... any other topics for today?

Minutes Approval

<azaroth> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/08/05-annotation-minutes.html

<ivan> +1

<azaroth> +1

<TimCole> +1

<Jacob> +1

RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/08/05-annotation-minutes.html


<TimCole> None


azaroth: The discussion around I18N has continued. We now have six (6) open issues around this topic.
... To summarise:
... #335: WONTFIX -- long thread with social web WG
... last state was I18N folks were discussing with Activity Streams folks and would get back to us
... #342: A suggestion from Sergiu to add a note to say if dc:language is specified and processingLanguage is not, the latter should be assumed to be the former
... seems like a good editorial note
... Similarly with #343.
... which is about whether processingLanguage should be required to be a language in dc:language
... Haven't had much of a chance to look at #345, also about processingLanguage
... #341 also about processingLanguage for multilingual resources, and we've decided to postpone

<ivan> Issue #345 is an attempt from Richard to close an issue and discussion with gsergiu via an editorial change proposal

azaroth: suggest we spend the time on the call discussing testing rather than going into the details on some of these I18N issues
... any problems?

Model Testing

TimCole: regarding the creation of the underlying schemas for the tests -- we've captured everything 3.1, 3.2, and 3.3 excepting agents, and most of section 4
... these schemas are in the "definitions" folder
... and are referenced by the schemas we intend to use for assertions

<TimCole> http://testdev.spec-ops.io:8000/tools/runner/index.html

TimCole: I've just started working on the test scripts. Have been hashing out with ShaneM what those look like.
... This [^] is a test environment ShaneM has set up.
... You can use this with "Run tests under path" using the following path:

<ShaneM> http://testdev.spec-ops.io:8000/tools/runner/index.html?path=/annotation-model should work too

TimCole: "/annotation-model"
... you can paste JSON[-LD] in and run through through the test suite
... that's all working.
... Having some small issues for SHOULD requirements, where we're not necessarily expecting you to pass the test. Currently if you *do* pass the requirement, the test fails.

<ShaneM> try going to this URI now:

<ShaneM> http://testdev.spec-ops.io:8000/annotation-model/annotations/3.1-model-musts-v3-manual.html

<azaroth> Example annotations to play with: https://github.com/w3c/web-annotation/tree/gh-pages/model/wd2/examples/correct

TimCole: ShaneM: [walking us through how to use the test tool at the link above to test annotations]

<ShaneM> Errors: data should have required property '@context'; expected true got false

ShaneM: Question: do we want to suppress the output above ^

<Zakim> azaroth, you wanted to ask about display:none in HTML ?

azaroth: Would it be possible to have the AJV stack trace be in a display:none; area with a button to reveal it or similar?

ShaneM: I don't think so. We just report data back to the test harness, which is responsible for the display.

azaroth: the "data should have required property @context" is particularly useful to understand what's going on

ShaneM: I'll leave it there, then.

ivan: currently can't rerun with updated JSON

ShaneM: I'll see if we can fix that

<TimCole> http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/3.2-model-manual.html

TimCole: if you put in a test annotation that *passes* the SHOULD requirements, the output is a little harder to interpret
... not sure quite how to improve the output in that case

<azaroth> I used: https://github.com/w3c/web-annotation/blob/gh-pages/model/wd2/examples/correct/anno41.json

ivan: [proceeds to find a bug in one of the testing schemas while on the call]

TimCole: [paraphrasing heavily] currently for the SHOULD assertions, we expect non-conformance, which means that if it actually is conformant, the test fails

<azaroth> Pass if it's not used, with a success message that it SHOULD be there?

ShaneM: no way of doing "warning" in the framework
... but we might be able to use "testType" to distinguish between MUST and SHOULD assertion types

<azaroth> Result: Pass Message: WARNING: Format SHOULD be included for bodies, if known

<ShaneM> use assertionType of must, may, or shold

TimCole: we have other scenarios where we say "SHOULD have 1, MAY have more than 1 X"

<ShaneM> testType has to do with automation.

TimCole: there are other cases where you MUST NOT have more than 1
... so in these cases we can have multiple assertions for the different cardinalities

<ShaneM> http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/3.2-model-manual.html

ShaneM: If you look at the top of [^] you'll notice that the page fills in as the page loads.
... In the description at the top of the page, there's a list of things the test will check
... do we want to include SHOULD/MUST/MAY information in these descriptions?

[noises of general agreement]

TimCole: going to spend the next little while cleaning this up with a view to sharing it more widely
... how are the test outputs recorded?

ShaneM: anyone with an implementation can record their JSON test output and add it to a git repository which contains all the results

TimCole: how do people want to break up the various test for bodies/targets/optional keys/etc.?

ShaneM: speaking as "not an implementer" -- the smallest number of manual tests that get us the information we need is probably a reasonable guideline

TimCole: probably a discussion for the mailing list

<azaroth> https://github.com/w3c/web-annotation/blob/gh-pages/model/wd2/examples/correct/anno41.json

azaroth: example 41 [^] is a completely contrived example at the end of the spec
... it seems unlikely (in the short term at least) that any client would generate such an annotation
... but perhaps not outside the realms of possibility
... putting in the 3.2 set of tests, it passes 8 but fails 5
... wondering what those are: problems with the test harness, the SHOULD problem, or problems with the data?

TimCole: would need to have a look, but it's probably the SHOULD issue with multiple formats

<Zakim> azaroth, you wanted to ask about fails for example 41

azaroth: we should probably spend some time talking about protocol testing

Protocol Testing

bigbluehat: have mostly passed the work I've done onto ShaneM

ShaneM: the server tests bigbluehat are awesome, but let's talk about client tests for a second
... the server runs in the WPT environment

<azaroth> ( issue https://github.com/w3c/web-annotation/issues/344 )

ShaneM: the way annotations work is that an annotation collection lives at a IRI, and thus the server needs to serve at some named route within WPT

<bigbluehat> PUT to overwrite

ShaneM: but we need to work out how create/update/destroy operations work in the test server
... in particular because we don't want ephemeral data created on the test server
... so we're going to be arranging things such that data created by clients is destroyed as soon as it is read back from the server

ivan: how is that going to work if you want to prepare a bunch of data and then run a load of tests against that?

ShaneM: in those cases the client will access some static collection of annotations rather than data they created

<azaroth> scribenick: azaroth

nickstenn: Question about how this is going to work -- the protocol spec doesn't say what the server is supposed to do with the data that you give it, even reasonable things
... for example, in a distributed annotation system, you POST to create it, but you might not be able to read it back again straight away
... want us to be careful that we're not testing that you can read something straight away

bigbluehat: The protocol says that it comes back with the full representation

ivan: Comes back with an id

nickstenn: That's a different point though. Could return it straight away, but the server doesn't necessarily have state beyond that
... intuitively reasonable assumptions are fine, but that's not in the spec
... need to be careful to not write tests on our understanding of the spec, but what the spec actually says

bigbluehat: e.g. there's no guarantee that you'll be able to get the annotation back after you create it

ShaneM: Wondering if the server tests you wrote rely on creating and then immediately retrieving it

bigbluehat: would need to go through the tests to see if it requests the same ones later, I don't think so
... it puts stuff in but I don't think it checks again later

ShaneM: If that's the case, then we're good

<bigbluehat> this is the thing ShaneM's been mentioning btw: https://github.com/BigBlueHat/web-annotation-protocol-tester

ShaneM: in our last 2m, let's just agree on what we think is going to happen over the next week
... I have a couple of next actions from this conversation and will get on those straight away
... I have another to ensure people know how to run tests and upload results
... also working on getting the protocol stuff implemented

TimCole: going to fill in a few more schemas to cover sections 3 and 4
... if people could help make some invalid annotations to help test the failure cases, that would be helpful

azaroth: I can do some of that

ivan: When do you folks think we can begin to pester implementers to provide reports?

TimCole: I think we need to do a reality check next Friday before we start inviting implementers -- maybe the week after that?


Summary of Action Items

Summary of Resolutions

  1. Minutes of the previous call are approved: https://www.w3.org/2016/08/05-annotation-minutes.html
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/08/12 16:07:50 $