See also: IRC log
<TimCole> azaroth: minutes, announcements, internationalization issues (brief), testing
<TimCole> ... any other topics for today?
<azaroth> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/08/05-annotation-minutes.html
<ivan> +1
<azaroth> +1
<TimCole> +1
<Jacob> +1
RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/08/05-annotation-minutes.html
<TimCole> None
azaroth: The discussion around I18N has
continued. We now have six (6) open issues around this topic.
... To summarise:
... #335: WONTFIX -- long thread with social web WG
... last state was I18N folks were discussing with Activity Streams
folks and would get back to us
... #342: A suggestion from Sergiu to add a note to say if dc:language
is specified and processingLanguage is not, the latter should be assumed
to be the former
... seems like a good editorial note
... Similarly with #343.
... which is about whether processingLanguage should be required to be a
language in dc:language
... Haven't had much of a chance to look at #345, also about
processingLanguage
... #341 also about processingLanguage for multilingual resources, and
we've decided to postpone
<ivan> Issue #345 is an attempt from Richard to close an issue and discussion with gsergiu via an editorial change proposal
azaroth: suggest we spend the time on the
call discussing testing rather than going into the details on some of
these I18N issues
... any problems?
TimCole: regarding the creation of the
underlying schemas for the tests -- we've captured everything 3.1, 3.2,
and 3.3 excepting agents, and most of section 4
... these schemas are in the "definitions" folder
... and are referenced by the schemas we intend to use for assertions
<TimCole> http://testdev.spec-ops.io:8000/tools/runner/index.html
TimCole: I've just started working on the
test scripts. Have been hashing out with ShaneM what those look like.
... This [^] is a test environment ShaneM has set up.
... You can use this with "Run tests under path" using the following
path:
<ShaneM> http://testdev.spec-ops.io:8000/tools/runner/index.html?path=/annotation-model should work too
TimCole: "/annotation-model"
... you can paste JSON[-LD] in and run through through the test suite
... that's all working.
... Having some small issues for SHOULD requirements, where we're not
necessarily expecting you to pass the test. Currently if you *do* pass
the requirement, the test fails.
<ShaneM> try going to this URI now:
<ShaneM> http://testdev.spec-ops.io:8000/annotation-model/annotations/3.1-model-musts-v3-manual.html
<azaroth> Example annotations to play with: https://github.com/w3c/web-annotation/tree/gh-pages/model/wd2/examples/correct
TimCole: ShaneM: [walking us through how to use the test tool at the link above to test annotations]
<ShaneM> Errors: data should have required property '@context'; expected true got false
ShaneM: Question: do we want to suppress the output above ^
<Zakim> azaroth, you wanted to ask about display:none in HTML ?
azaroth: Would it be possible to have the AJV stack trace be in a display:none; area with a button to reveal it or similar?
ShaneM: I don't think so. We just report data back to the test harness, which is responsible for the display.
azaroth: the "data should have required property @context" is particularly useful to understand what's going on
ShaneM: I'll leave it there, then.
ivan: currently can't rerun with updated JSON
ShaneM: I'll see if we can fix that
<TimCole> http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/3.2-model-manual.html
TimCole: if you put in a test annotation
that *passes* the SHOULD requirements, the output is a little harder to
interpret
... not sure quite how to improve the output in that case
<azaroth> I used: https://github.com/w3c/web-annotation/blob/gh-pages/model/wd2/examples/correct/anno41.json
ivan: [proceeds to find a bug in one of the testing schemas while on the call]
TimCole: [paraphrasing heavily] currently for the SHOULD assertions, we expect non-conformance, which means that if it actually is conformant, the test fails
<azaroth> Pass if it's not used, with a success message that it SHOULD be there?
ShaneM: no way of doing "warning" in the
framework
... but we might be able to use "testType" to distinguish between MUST
and SHOULD assertion types
<azaroth> Result: Pass Message: WARNING: Format SHOULD be included for bodies, if known
<ShaneM> use assertionType of must, may, or shold
TimCole: we have other scenarios where we say "SHOULD have 1, MAY have more than 1 X"
<ShaneM> testType has to do with automation.
TimCole: there are other cases where you
MUST NOT have more than 1
... so in these cases we can have multiple assertions for the different
cardinalities
<ShaneM> http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/3.2-model-manual.html
ShaneM: If you look at the top of [^] you'll
notice that the page fills in as the page loads.
... In the description at the top of the page, there's a list of things
the test will check
... do we want to include SHOULD/MUST/MAY information in these
descriptions?
[noises of general agreement]
TimCole: going to spend the next little
while cleaning this up with a view to sharing it more widely
... how are the test outputs recorded?
ShaneM: anyone with an implementation can record their JSON test output and add it to a git repository which contains all the results
TimCole: how do people want to break up the various test for bodies/targets/optional keys/etc.?
ShaneM: speaking as "not an implementer" -- the smallest number of manual tests that get us the information we need is probably a reasonable guideline
TimCole: probably a discussion for the mailing list
<azaroth> https://github.com/w3c/web-annotation/blob/gh-pages/model/wd2/examples/correct/anno41.json
azaroth: example 41 [^] is a completely
contrived example at the end of the spec
... it seems unlikely (in the short term at least) that any client would
generate such an annotation
... but perhaps not outside the realms of possibility
... putting in the 3.2 set of tests, it passes 8 but fails 5
... wondering what those are: problems with the test harness, the SHOULD
problem, or problems with the data?
TimCole: would need to have a look, but it's probably the SHOULD issue with multiple formats
<Zakim> azaroth, you wanted to ask about fails for example 41
azaroth: we should probably spend some time talking about protocol testing
bigbluehat: have mostly passed the work I've done onto ShaneM
ShaneM: the server tests bigbluehat are
awesome, but let's talk about client tests for a second
... the server runs in the WPT environment
<azaroth> ( issue https://github.com/w3c/web-annotation/issues/344 )
ShaneM: the way annotations work is that an annotation collection lives at a IRI, and thus the server needs to serve at some named route within WPT
<bigbluehat> PUT to overwrite
ShaneM: but we need to work out how
create/update/destroy operations work in the test server
... in particular because we don't want ephemeral data created on the
test server
... so we're going to be arranging things such that data created by
clients is destroyed as soon as it is read back from the server
ivan: how is that going to work if you want to prepare a bunch of data and then run a load of tests against that?
ShaneM: in those cases the client will access some static collection of annotations rather than data they created
<azaroth> scribenick: azaroth
nickstenn: Question about how this is going
to work -- the protocol spec doesn't say what the server is supposed to
do with the data that you give it, even reasonable things
... for example, in a distributed annotation system, you POST to create
it, but you might not be able to read it back again straight away
... want us to be careful that we're not testing that you can read
something straight away
bigbluehat: The protocol says that it comes back with the full representation
ivan: Comes back with an id
nickstenn: That's a different point though.
Could return it straight away, but the server doesn't necessarily have
state beyond that
... intuitively reasonable assumptions are fine, but that's not in the
spec
... need to be careful to not write tests on our understanding of the
spec, but what the spec actually says
bigbluehat: e.g. there's no guarantee that you'll be able to get the annotation back after you create it
ShaneM: Wondering if the server tests you wrote rely on creating and then immediately retrieving it
bigbluehat: would need to go through the
tests to see if it requests the same ones later, I don't think so
... it puts stuff in but I don't think it checks again later
ShaneM: If that's the case, then we're good
<bigbluehat> this is the thing ShaneM's been mentioning btw: https://github.com/BigBlueHat/web-annotation-protocol-tester
ShaneM: in our last 2m, let's just agree on
what we think is going to happen over the next week
... I have a couple of next actions from this conversation and will get
on those straight away
... I have another to ensure people know how to run tests and upload
results
... also working on getting the protocol stuff implemented
TimCole: going to fill in a few more schemas
to cover sections 3 and 4
... if people could help make some invalid annotations to help test the
failure cases, that would be helpful
azaroth: I can do some of that
ivan: When do you folks think we can begin to pester implementers to provide reports?
TimCole: I think we need to do a reality check next Friday before we start inviting implementers -- maybe the week after that?