Web Annotation Working Group Teleconference

<scribe> scribenick: azaroth

Minutes Approval

PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/02-annotation-minutes.html

+1

<TimCole> +1

<bjdmeest> +1

<ivan> +1

RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/02-annotation-minutes.html

Announcements

TimCole: We have successfully republished the three documents in CR
... TPAC is fast approaching. We won't have any meetings there, but need to think about schedule for the calls

ivan: I'll be out
... I leave on Sunday so here next week, but out the following

ShaneM: I'll be tied up too
... 4pm in Lisbon?
... I have a meeting at 4pm on Friday.

Ivan: I'll be fried :)

TimCole: Cancel Sept 23rd meeting

azaroth: My regrets for next week.

TimCole: Okay, cancel 23rd but will meet next week
... Will meet on the 30th. Any other announcements or questions?

Issue updates?

TimCole: There was an issue closed

Ivan: Sarven said that he was okay to close it, so I did

TimCole: What about the i18n issues?

ivan: No idea :( Anyone on the social web group might know more?

TimCole: Would it be okay to reach out to Richard?

Ivan: I can do that.

bigbluehat: Social picked a very different solution for i18n issue. DPUB on the other hand did the same as us
... they went with no text direction stated. Web manifests and ourselves have it as an explicit property
... Social are trusting the bidi character will be recognised and implemented

TimCole: That's okay, we'll see what happens at the end of CR
... Any editorial progress?

azaroth: None, have been waiting for i18n issue to resolve before making any further changes

TimCole: Any other issue related topics?

Testing

TimCole: Posted a short note about the state of the model testing. A couple of issues have come up
... Haven't gotten new reports from implementers.
... For sections 1-4 of the model, we now have 173 assertions, organized into 10 tests
... broken down along validation of MUST, or whether a feature is implemented
... 10 tests means that if you run the suite, you paste your annotation in 10 times
... the advantage of it is that if you want to test only the requirements or a particular feature you can do so by running only a subset of tests
... impressed by how quickly they run
... could probably reduce it however to fewer tests
... Shane, do you see any issue with the test runner software having one test with 173 assertions?

ShaneM: No, the system doesn't care
... we moved the text box. It might not be merged yet though
... I don't have merge capability. The code codes reviewed and hopefully someone merges it.

<bigbluehat> TimCole: which server are you looking at? w3c-test.org? or testdev.spec-ops.io?

ShaneM: The people who have merge capability don't seem to pay attention to our requests

ivan: You should try to talk to Philippe about this
... He's the one responsible for getting WGs to do things properly. We went down this route for testing, but we shouldn't be thwarted by issues like this
... at the end of the day, it's his responsibility

ShaneM: I can ping him now?

ivan: It may need a longer discussion.

ShaneM: There just needs to be someone on staff who takes care of this sort of stuff

ivan: I don't have a practical proposal, but the current way doesn't work
... it has created barriers for us many times already, which isn't acceptable
... we're bound to deadlines, and accountable to W3M, but that means we need to be able to do what we have to do

ShaneM: From our perspective, I should have stayed on top of the PRs...

ivan: No, don't take it on you, you shouldn't have to chase people, there should be a process

ShaneM: There's 173 outstanding requests, so it's not just us

ivan: I discussed this with Ralph as well. There's something fundamentally wrong. We can come back to this in Lisbon

<bigbluehat> https://github.com/w3c/web-platform-tests/pulls?q=is%3Aopen+is%3Apr+label%3Awg-annotation

TimCole: One question is whether we should go down to a few tests

<ShaneM> bigbluehat: https://github.com/w3c/web-platform-tests/pull/3634 r?

TimCole: easy to make that change now

<bigbluehat> yeah. working on that one now

TimCole: would just require all the assertions in one big test

azaroth: Would it be possible to copy the annotation from the first test to the second test, and so forth? then it would populate the text box with that annotation

ShaneM: If it's the same annotation, that might be possible. If it's a different annotation, it wouldn't make sense

TimCole: Still need to deal with section 5 of the model, as it's not annotations
... you'd paste in a collection or a page, which are fundamentally different
... as many /clients/ won't implement that, it'll be a bit strange

azaroth: We'd probably want annotation server implementers to do the collections and pages

TimCole: Should this be a fourth set of tests?

azaroth: Easier at the end of CR if it was part of the model testing, just a different section
... (explains issue)

ivan: We have entries in the model that are relevant to the server, so closer to the protocol
... how does it affect the reporting?

TimCole: if there's one series of tests that starts with annotations and then the collection, the client developers won't know what to do with the collection and page tests

azaroth: and the server implementers won't necessarily have anything useful for the first part

ShaneM: What if those tests were added to the protocol suite?

<Zakim> ShaneM, you wanted to ask if we should be pulling ajv into the protocol tests to see what features of the data model are used there? OR add a protocol exerciser to the model tests

ShaneM: or add server tests to the model tests to hit a server and get the content?

ivan: My reaction was the same as your first option -- those tests should be performed by the protocol testing procedure

<ShaneM> protocol !== data model though

ivan: that's where they come into the picture
... so the server/protocol implementers will test some of the model, which is fine
... the reporting is tricky

TimCole: one thing we could do is name the tests for the collection and page carefully, and give different URLs to the testers
... so the client people would only get annotations, and the server people would only get collections and pages

+1 if that's possible

scribe: we could do that with regexps or folder names perhaps

ShaneM: Have a PR on the regexps

TimCole: The problem would be that the results would be incomplete -- it would only have collection or annotation tests. Would that merge okay?

ShaneM: I don't know the answer

azaroth: We really need that to work, as a client might not do all the tests and we want to know what they did do

TimCole: I'd prefer that, as it keeps everything together in the right place

ShaneM: THe test description needs to clearly lay out the expectations for what to put in the text box

TimCole: We may need to move all the tests into a different child folder. Everything in this folder is for the annotation, and everything in this folder is for the collections
... number of tests... since we can't currently retain the annotation between tests, might be better to reduce the number of tests and put lots of assertions
... one for all the MUSTS and one for all the optional SHOULDs and MAYs
... would expect most of them to fail, but should get some for each

+1

ShaneM: I'm fine with that

TimCole: Even clicking the button 10 times if the annotation is copied would be a pain
... Last big issue for model testing is the process for uploading results
... Single client implementation will create multiple types of annotations.
... First type might exercise specific resources, second might do text bodies
... but might not have one annotation that does both
... so want client implementers to run multiple annotations through both the tests
... affects some of the counts and how to name the files
... need to not count the same implementation twice
... instructions get a bit complicated for what should be uploaded
... e.g. that the readme needs to be updated
... Illinois developer reported that the instructions were long and cumbersome

ShaneM: Don't have a problem with people having to read instructions. Issue is that we're all learning about this way of testing
... don't have to upload the annotations if you don't want to

TimCole: Could create codes for people that we invite
... but they'd need to fill out the other information
... and can't do that for everyone up front

ivan: Try to invite people, but hard to invite folks not in the WG
... have to actively bring people in
... if they come they'll have questions and someone will have to help them get over the hurdles
... don't remember how difficult for the RDFa testing, for example
... might have been a bit more automatic, but there were instructions about what to set up
... there's always something like that, there's always some help and intervention needed

TimCole: Would shorten things if we can set up the base line as a template
... then don't need to describe it... just say to copy the template
... can set up the folders for them to make it easier

ShaneM: If you don't understand github, there's no way you'll be able to do this
... would be nice if there /was/ a way to upload of course

TimCole: Have to rename file from the results, then fork the repo, create a PR, add the results, annotations and update the readme file
... so not terribly complicated but harder with no examples
... who can help with instructions and Readme?

azaroth: Can try to take a look early next week

TimCole: We'll make some edits before Monday
... anyone else?

<tbdinesh_> I will ask someone in my team to do that

TimCole: Shared some of this with Randall and Nick but don't know if they've had a look at it yet
... Probably they'll say it's too long, Benjamin gave the same feedback
... Rob do you have annotations from a client

azaroth: Yep, I can do that

TimCole: Thanks Dinesh

<Zakim> ShaneM, you wanted to talk aout updating process

ShaneM: in terms of update process, just do a single PR
... I've already corrected the files that weren't named correctly for her

TimCole: By the end of the day we should have the right set of reports for 1 implementation with 3 annotations
... going to rearrange the tests

ShaneM: I'll withdraw the PR to put in the new tests then

ivan: Looking at the 3 result files. Protocol looks pretty good
... no information about the two implementations

ShaneM: The implementation details are in the readme
... no way to have the implementation details in the response file

ivan: Can the results file have a link to where to go for where to find the things?
... Management will have no idea what to do with the file
... no facility in WPT reports for how to do it, so would have to do it by hand
... Ahh, don't do it by hand. Just needs to be clear that there's a readme file

<ShaneM> https://github.com/w3c/test-results/tree/gh-pages/annotation-protocol

ivan: in the HTML file, through github.io, I'd just like a static link to say go here to understand these

TimCole: We could put that link at the top of the file?

ShaneM: If that satisfies Ivan's requirement?

ivan: If that link is on the report, that's fine
... can it not say fail for optional things?

ShaneM: We've talked about that :( It has to say fail at the moment
... we just need to say it's optional

ivan: If I look at the model, there's lots of fails
... the impression is that it looks bad
... don't know what to do though

TimCole: For any given annotation, for optional features, it might have only very few features
... so just need two greens across the row
... even if there's 100 fails

ivan: As a reporting issue, it could be a problem
... at the transition call there'll be a long discussion as to what is going on

<ShaneM> https://w3c.github.io/test-results/annotation-model/less-than-2.html

TimCole: We can reduce down what we think of as a feature

<ShaneM> oops...

<ShaneM> Take a look at that

TimCole: we have a test for 'is this a selector' and then 7 tests for what sort of selector
... will see more green for general selectors than for specific ones
... need to talk about this before the end of the month
... or put them into skipped

ivan: it's more the reporting

<ShaneM> "what is skipped

ivan: from high up, if what's optional is put somewhere else in the report, it gives a different view

TimCole: THere'll be the top part with lots of greens, the MUSTS, and then 100 something mostly reds for the optionals

ivan: Psychologically speaking I'd try to find a diffrent color than red
... but at least we need to separate them

TimCole: a skip would not show up in red?

ShaneM: what's a skip?

TimCole: In the test format, there's an option to skip?

ShaneM: ahh, that's just flow control on assertions

TimCole: If I skip an assertion, they wouldn't show up?
... there'd be a blank box?

ShaneM: Probably?

ivan: You understand the problem though, it will create issues

ShaneM: The fundamental problem is that the Web Platform (in general) does not envision the notion of optional
... but we have it. I don't know how to represent that except by naming the assertion clearly
... That's not reflected in the report, but we could

ivan: Is it possible?

ShaneM: It is. The WPT people hate that we've made our subtest names complicated but whatever
... putting mandatory or optional at the beginning will make them more complicated

azaroth: Could we leave failed optional as a blank box

ivan: that would be better

TimCole: I'll play with flow control this afternoon

ivan: Gregg has done full vocab testing?

azaroth: I have as well

ivan: The report is empty though
...
...
...

*tumbleweed*

scribe: ...

ShaneM: I'm doing my best

<Zakim> ShaneM, you wanted to talk about protocol testing...

ShaneM: I can do it by hand

ivan: Once we've done it, it's a small report we can do by hand

TimCole: Sorry for not talking about HTML serialization
... or protocol testing

ShaneM: Lets work on the sequencing thing and see if we can figure it out

TimCole: No reason to test selectors if you don't have a specific resource
... the more consolidated tests will make it easier
... Worried that the report will correctly have blank cells

ShaneM: Suggest reorganizing tests first

TimCole: Thanks all

Bye!

<ivan> trackbot, end telcon