14:28:58 <RRSAgent> RRSAgent has joined #annotation
14:28:58 <RRSAgent> logging to http://www.w3.org/2016/09/09-annotation-irc
14:29:00 <trackbot> RRSAgent, make logs public
14:29:00 <Zakim> Zakim has joined #annotation
14:29:02 <trackbot> Zakim, this will be 2666
14:29:02 <Zakim> ok, trackbot
14:29:03 <trackbot> Meeting: Web Annotation Working Group Teleconference
14:29:03 <trackbot> Date: 09 September 2016
14:29:30 <ivan> Agenda: http://www.w3.org/mid/092901d20a41$502a57f0$f07f07d0$@illinois.edu
14:29:44 <ivan> ivan has changed the topic to: Agenda for 2016-09-09: http://www.w3.org/mid/092901d20a41$502a57f0$f07f07d0$@illinois.edu
14:30:00 <ivan> Chair: Rob, Tim
14:48:55 <azaroth> azaroth has joined #annotation
14:55:23 <TimCole> TimCole has joined #annotation
15:00:32 <ShaneM> present+ ShaneM
15:00:47 <TimCole> present+ Tim_Cole
15:01:08 <azaroth> Present+ Rob_Sanderson
15:01:45 <bjdmeest> bjdmeest has joined #annotation
15:02:04 <tilgovi> tilgovi has joined #annotation
15:02:10 <bjdmeest> Present+ Ben_De_Meester
15:03:18 <tbdinesh> tbdinesh has joined #annotation
15:03:28 <tilgovi_> tilgovi_ has joined #annotation
15:03:42 <azaroth> scribenick: azaroth
15:03:57 <bigbluehat> Present+ Benjamin_Young
15:04:09 <TimCole> Topic: Minutes Approval
15:04:10 <tbdinesh> Present+ TB_Dinesh
15:04:10 <azaroth> TOPIC: Minutes approval
15:04:19 <tilgovi_> Present+ Randall_Leeds
15:04:26 <azaroth> PROPOSED RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/02-annotation-minutes.html
15:04:29 <azaroth> +1
15:04:33 <TimCole> +1
15:04:35 <bjdmeest> +1
15:04:35 <ivan> +1
15:04:45 <azaroth> RESOLUTION: Minutes of the previous call are approved: https://www.w3.org/2016/09/02-annotation-minutes.html
15:04:51 <takeshi> takeshi has joined #annotation
15:04:53 <azaroth> TOPIC: Announcements
15:05:29 <azaroth> TimCole: We have successfully republished the three documents in CR
15:05:47 <azaroth> ... TPAC is fast approaching. We won't have any meetings there, but need to think about schedule for the calls
15:05:51 <azaroth> ivan: I'll be out
15:06:01 <azaroth> ... I leave on Sunday so here next week, but out the following
15:06:08 <azaroth> ShaneM: I'll be tied up too
15:06:20 <azaroth> ... 4pm in Lisbon?
15:06:32 <azaroth> ... I have a meeting at 4pm on Friday.
15:06:37 <azaroth> Ivan: I'll be fried :)
15:06:49 <azaroth> TimCole: Cancel Sept 23rd meeting
15:07:07 <azaroth> azaroth: My regrets for next week.
15:07:27 <azaroth> TimCole: Okay, cancel 23rd but will meet next week
15:08:29 <azaroth> TimCole: Will meet on the 30th.   Any other announcements or questions?
15:08:34 <TimCole> topic: Issue updates?
15:08:58 <azaroth> TimCole: There was an issue closed
15:09:18 <azaroth> Ivan: Sarven said that he was okay to close it, so I did
15:09:25 <azaroth> TimCole: What about the i18n issues?
15:09:38 <azaroth> ivan: No idea :( Anyone on the social web group might know more?
15:09:48 <azaroth> TimCole: Would it be okay to reach out to Richard?
15:09:58 <azaroth> Ivan: I can do that.
15:11:04 <azaroth> bigbluehat: Social picked a very different solution for i18n issue. DPUB on the other hand did the same as us
15:11:25 <azaroth> ... they went with no text direction stated. Web manifests and ourselves have it as an explicit property
15:11:41 <azaroth> ... Social are trusting the bidi character will be recognised and implemented
15:11:54 <azaroth> TimCole: That's okay, we'll see what happens at the end of CR
15:12:32 <azaroth> ... Any editorial progress?
15:12:45 <azaroth> azaroth: None, have been waiting for i18n issue to resolve before making any further changes
15:12:52 <azaroth> TimCole: Any other issue related topics?
15:12:54 <azaroth> TOPIC:  Testing
15:13:21 <azaroth> TimCole: Posted a short note about the state of the model testing.  A couple of issues have come up
15:13:34 <azaroth> ... Haven't gotten new reports from implementers.
15:13:55 <azaroth> ... For sections 1-4 of the model, we now have 173 assertions, organized into 10 tests
15:14:09 <azaroth> ... broken down along validation of MUST, or whether a feature is implemented
15:14:30 <azaroth> ... 10 tests means that if you run the suite, you paste your annotation in 10 times
15:14:59 <ShaneM> q+ to ask if it is the same annotation in every form for most implementors?
15:15:02 <azaroth> ... the advantage of it is that if you want to test only the requirements or a particular feature you can do so by running only a subset of tests
15:15:14 <azaroth> ... impressed by how quickly they run
15:15:26 <azaroth> ... could probably reduce it however to fewer tests
15:15:53 <azaroth> ... Shane, do you see any issue with the test runner software having one test with 173 assertions?
15:15:57 <azaroth> ShaneM: No, the system doesn't care
15:16:17 <azaroth> ... we moved the text box. It might not be merged yet though
15:16:41 <azaroth> ... I don't have merge capability. The code codes reviewed and hopefully someone merges it.
15:17:01 <bigbluehat> TimCole: which server are you looking at? w3c-test.org? or testdev.spec-ops.io?
15:17:07 <azaroth> ... The people who have merge capability don't seem to pay attention to our requests
15:17:18 <azaroth> ivan: You should try to talk to Philippe about this
15:17:48 <azaroth> ... He's the one responsible for getting WGs to do things properly.  We went down this route for testing, but we shouldn't be thwarted by issues like this
15:17:55 <azaroth> ... at the end of the day, it's his responsibility
15:18:01 <azaroth> ShaneM: I can ping him now?
15:18:21 <azaroth> ivan: It may need a longer discussion.
15:18:42 <azaroth> ShaneM: There just needs to be someone on staff who takes care of this sort of stuff
15:18:53 <azaroth> ivan: I don't have a practical proposal, but the current way doesn't work
15:19:04 <azaroth> ... it has created barriers for us many times already, which isn't acceptable
15:19:25 <azaroth> ... we're bound to deadlines, and accountable to W3M, but that means we need to be able to do what we have to do
15:19:37 <azaroth> ShaneM: From our perspective, I should have stayed on top of the PRs...
15:19:50 <azaroth> ivan: No, don't take it on you, you shouldn't have to chase people, there should be a process
15:20:10 <azaroth> ShaneM: There's 173 outstanding requests, so it's not just us
15:20:31 <TimCole> q?
15:20:33 <azaroth> ivan: I discussed this with Ralph as well. There's something fundamentally wrong.  We can come back to this in Lisbon
15:20:34 <bigbluehat> https://github.com/w3c/web-platform-tests/pulls?q=is%3Aopen+is%3Apr+label%3Awg-annotation
15:20:37 <ShaneM> q-
15:20:41 <TimCole> ack ShaneM
15:21:09 <azaroth> TimCole: One question is whether we should go down to a few tests
15:21:12 <ShaneM> bigbluehat: https://github.com/w3c/web-platform-tests/pull/3634 r?
15:21:13 <azaroth> ... easy to make that change now
15:21:24 <bigbluehat> yeah. working on that one now
15:21:29 <azaroth> ... would just require all the assertions in one big test
15:21:32 <azaroth> q+
15:21:40 <TimCole> ack azaroth
15:21:41 <azaroth> ack azaroth
15:22:47 <azaroth> azaroth: Would it be possible to copy the annotation from the first test to the second test, and so forth? then it would populate the text box with that annotation
15:23:08 <azaroth> ShaneM: If it's the same annotation, that might be possible. If it's a different annotation, it wouldn't make sense
15:23:29 <azaroth> TimCole: Still need to deal with section 5 of the model, as it's not annotations
15:23:44 <azaroth> ... you'd paste in a collection or a page, which are fundamentally different
15:24:03 <azaroth> ... as many /clients/ won't implement that, it'll be a bit strange
15:24:58 <azaroth> azaroth: We'd probably want annotation server implementers to do the collections and pages
15:25:12 <azaroth> TimCole: Should this be a fourth set of tests?
15:26:41 <azaroth> azaroth: Easier at the end of CR if it was part of the model testing, just a different section
15:26:54 <uskudarli> uskudarli has joined #annotation
15:27:35 <uskudarli> uskudarli has left #annotation
15:27:46 <uskudarli> uskudarli has joined #annotation
15:27:59 <azaroth> azaroth:  (explains issue)
15:28:13 <azaroth> ivan: We have entries in the model that are relevant to the server, so closer to the protocol
15:28:20 <azaroth> ... how does it affect the reporting?
15:29:06 <azaroth> TimCole: if there's one series of tests that starts with annotations and then the collection, the client developers won't know what to do with the collection and page tests
15:29:24 <ShaneM> q+ to ask if we should be pulling ajv into the protocol tests to see what features of the data model are used there?  OR add a protocol exerciser to the model tests
15:29:34 <azaroth> azaroth: and the server implementers won't necessarily have anything useful for the first part
15:29:44 <azaroth> ShaneM: What if those tests were added to the protocol suite?
15:29:44 <TimCole> ack ShaneM
15:29:44 <Zakim> ShaneM, you wanted to ask if we should be pulling ajv into the protocol tests to see what features of the data model are used there?  OR add a protocol exerciser to the model tests
15:29:58 <tbdinesh_> tbdinesh_ has joined #annotation
15:29:59 <azaroth> ... or add server tests to the model tests to hit a server and get the content?
15:30:19 <azaroth> ivan: My reaction was the same as your first option -- those tests should be performed by the protocol testing procedure
15:30:22 <ShaneM> protocol !== data model thoughj
15:30:25 <azaroth> ... that's where they come into the picture
15:30:32 <ShaneM> s/thoughj/though/
15:30:36 <azaroth> ... so the server/protocol implementers will test some of the model, which is fine
15:30:39 <TimCole> q+
15:30:40 <azaroth> ... the reporting is tricky
15:30:48 <azaroth> ack TimCole
15:30:55 <TimCole> ack TimCole
15:31:22 <azaroth> TimCole: one thing we could do is name the tests for the collection and page carefully, and give different URLs to the testers
15:31:37 <azaroth> ... so the client people would only get annotations, and the server people would only get collections and pages
15:31:43 <azaroth> +1 if that's possible
15:31:52 <azaroth> ... we could do that with regexps or folder names perhaps
15:32:09 <azaroth> ShaneM: Have a PR on the regexps
15:32:36 <azaroth> TimCole: The problem would be that the results would be incomplete -- it would only have collection or annotation tests.  Would that merge okay?
15:32:48 <azaroth> ShaneM: I don't know the answer
15:33:15 <azaroth> azaroth: We really need that to work, as a client might not do all the tests and we want to know what they did do
15:33:27 <azaroth> TimCole: I'd prefer that, as it keeps everything together in the right place
15:33:41 <azaroth> ShaneM: THe test description needs to clearly lay out the expectations for what to put in the text box
15:34:14 <azaroth> TimCole: We may need to move all the tests into a different child folder. Everything in this folder is for the annotation, and everything in this folder is for the collections
15:35:00 <azaroth> ... number of tests... since we can't currently retain the annotation between tests, might be better to reduce the number of tests and put lots of assertions
15:35:13 <azaroth> ... one for all the MUSTS and one for all the optional SHOULDs and MAYs
15:35:26 <azaroth> ... would expect most of them to fail, but should get some for each
15:35:29 <azaroth> +1
15:35:49 <azaroth> ShaneM: I'm fine with that
15:36:20 <azaroth> TimCole: Even clicking the button 10 times if the annotation is copied would be a pain
15:36:52 <azaroth> ... Last big issue for model testing is the process for uploading results
15:37:13 <azaroth> ... Single client implementation will create multiple types of annotations.
15:37:31 <azaroth> ... First type might exercise specific resources, second might do text bodies
15:37:39 <azaroth> ... but might not have one annotation that does both
15:37:54 <azaroth> ... so want client implementers to run multiple annotations through both the tests
15:38:05 <azaroth> ... affects some of the counts and how to name the files
15:38:29 <azaroth> ... need to not count the same implementation twice
15:38:51 <azaroth> ... instructions get a bit complicated for what should be uploaded
15:38:58 <azaroth> ... e.g. that the readme needs to be updated
15:39:56 <azaroth> ... Illinois developer reported that the instructions were long and cumbersome
15:40:31 <azaroth> ShaneM: Don't have a problem with people having to read instructions.  Issue is that we're all learning about this way of testing
15:40:41 <azaroth> ... don't have to upload the annotations if you don't want to
15:41:24 <azaroth> TimCole: Could create codes for people that we invite
15:41:34 <azaroth> ... but they'd need to fill out the other information
15:41:49 <azaroth> ... and can't do that for everyone up front
15:42:13 <azaroth> ivan: Try to invite people, but hard to invite folks not in the WG
15:42:17 <azaroth> ... have to actively bring people in
15:42:37 <azaroth> ... if they come they'll have questions and someone will have to help them get over the hurdles
15:42:47 <azaroth> ... don't remember how difficult for the RDFa testing, for example
15:42:59 <azaroth> ... might have been a bit more automatic, but there were instructions about what to set up
15:43:17 <azaroth> ... there's always something like that, there's always some help and intervention needed
15:43:39 <azaroth> TimCole: Would shorten things if we can set up the base line as a template
15:43:48 <azaroth> ... then don't need to describe it... just say to copy the template
15:44:04 <azaroth> ... can set up the folders for them to make it easier
15:44:31 <azaroth> ShaneM: If you don't understand github, there's no way you'll be able to do this
15:44:39 <azaroth> ... would be nice if there /was/ a way to upload of course
15:45:17 <azaroth> TimCole: Have to rename file from the results, then fork the repo, create a PR, add the results, annotations and update the readme file
15:45:30 <azaroth> ... so not terribly complicated but harder with no examples
15:45:43 <azaroth> ... who can help with instructions and Readme?
15:46:16 <azaroth> azaroth: Can try to take a look early next week
15:46:36 <azaroth> TimCole: We'll make some edits before Monday
15:46:50 <azaroth> ... anyone else?
15:47:05 <tbdinesh_> I will ask someone in my team to do that
15:47:05 <azaroth> ... Shared some of this with Randall and Nick but don't know if they've had a look at it yet
15:47:35 <azaroth> ... Probably they'll say it's too long, Benjamin gave the same feedback
15:47:42 <azaroth> TimCole: Rob do you have annotations from a client
15:47:46 <azaroth> azaroth: Yep, I can do that
15:47:51 <ShaneM> q+ to talk aout updating process
15:47:52 <azaroth> TimCole: Thanks Dinesh
15:48:03 <TimCole> ack ShaneM
15:48:03 <Zakim> ShaneM, you wanted to talk aout updating process
15:48:20 <azaroth> ShaneM: in terms of update process, just do a single PR
15:48:37 <azaroth> ... I've already corrected the files that weren't named correctly for her
15:48:58 <azaroth> TimCole: By the end of the day we should have the right set of reports for 1 implementation with 3 annotations
15:49:02 <ivan> q+
15:49:08 <ShaneM> q+ to talk about protocol testing...
15:49:19 <azaroth> ... going to rearrange the tests
15:49:46 <azaroth> ShaneM: I'll withdraw the PR to put in the new tests then
15:49:50 <azaroth> q?
15:49:51 <azaroth> ack ivan
15:49:55 <TimCole> ack ivan
15:50:13 <azaroth> ivan: Looking at the 3 result files. Protocol looks pretty good
15:50:19 <azaroth> ... no information about the two implementations
15:50:36 <azaroth> ShaneM: The implementation details are in the readme
15:50:47 <azaroth> ... no way to have the implementation details in the response file
15:51:02 <azaroth> ivan: Can the results file have a link to where to go for where to find the things?
15:51:11 <azaroth> ... Management will have no idea what to do with the file
15:51:26 <azaroth> ... no facility in WPT reports for how to do it, so would have to do it by hand
15:51:43 <azaroth> ivan: Ahh, don't do it by hand. Just needs to be clear that there's a readme file
15:52:21 <ShaneM> https://github.com/w3c/test-results/tree/gh-pages/annotation-protocol
15:52:23 <azaroth> ... in the HTML file, through github.io, I'd just like a static link to say go here to understand these
15:53:07 <azaroth> TimCole: We could put that link at the top of the file?
15:53:14 <azaroth> ShaneM: If that satisfies Ivan's requirement?
15:53:20 <azaroth> ivan: If that link is on the report, that's fine
15:55:13 <azaroth> Ivan: can it not say fail for optional things?
15:55:21 <tbdinesh> tbdinesh has joined #annotation
15:55:27 <azaroth> ShaneM: We've talked about that :( It has to say fail at the moment
15:55:33 <azaroth> ... we just need to say it's optional
15:55:43 <azaroth> ivan: If I look at the model, there's lots of fails
15:55:53 <azaroth> ... the impression is that it looks bad
15:55:58 <azaroth> ... don't know what to do though
15:56:14 <azaroth> TimCole: For any given annotation, for optional features, it might have only very few features
15:56:23 <azaroth> ... so just need two greens across the row
15:56:28 <azaroth> ... even if there's 100 fails
15:56:37 <azaroth> ivan: As a reporting issue, it could be a problem
15:56:47 <azaroth> ... at the transition call there'll be a long discussion as to what is going on
15:56:56 <ShaneM> https://w3c.github.io/test-results/annotation-model/less-than-2.html
15:56:58 <azaroth> TimCole: We can reduce down what we think of as a feature
15:57:00 <ShaneM> oops...
15:57:06 <ShaneM> Take a look at that
15:57:14 <azaroth> ... we have a test for 'is this a selector' and then 7 tests for what sort of selector
15:57:33 <azaroth> ... will see more green for general selectors than for specific ones
15:57:37 <azaroth> q+
15:57:45 <azaroth> ... need to talk about this before the end of the month
15:57:50 <azaroth> ... or put them into skipped
15:58:03 <azaroth> ivan: it's more the reporting
15:58:06 <ShaneM> "what is skipped
15:58:26 <azaroth> ... from high up, if what's optional is put somewhere else in the report, it gives a different view
15:58:46 <azaroth> TimCole: THere'll be the top part with lots of greens, the MUSTS, and then 100 something mostly reds for the optionals
15:58:59 <azaroth> ivan: Psychologically speaking I'd try to find a diffrent color than red
15:59:05 <azaroth> ... but at least we need to separate them
15:59:26 <azaroth> TimCole: a skip would not show up in red?
15:59:30 <azaroth> ShaneM: what's a skip?
15:59:31 <shepazu> shepazu has joined #annotation
15:59:43 <azaroth> TimCole: In the test format, there's an option to skip?
15:59:49 <azaroth> ShaneM: ahh, that's just flow control on assertions
15:59:58 <azaroth> TimCole: If I skip an assertion, they wouldn't show up?
16:00:05 <azaroth> ... there'd be a blank box?
16:00:10 <azaroth> ShaneM: Probably?
16:00:23 <azaroth> ivan: You understand the problem though, it will create issues
16:00:44 <azaroth> ShaneM: The fundamental problem is that the Web Platform (in general) does not envision the notion of optional
16:01:02 <azaroth> ... but we have it. I don't know how to represent that except by naming the assertion clearly
16:01:20 <azaroth> ... That's not reflected in the report, but we could
16:01:26 <azaroth> ivan: Is it possible?
16:01:43 <azaroth> ShaneM: It is.  The WPT people hate that we've made our subtest names complicated but whatever
16:01:53 <TimCole> ack azaroth
16:01:53 <azaroth> ... putting mandatory or optional at the beginning will make them more complicated
16:02:19 <azaroth> azaroth: Could we leave failed optional as a blank box
16:02:24 <azaroth> ivan: that would be better
16:02:31 <azaroth> TimCole: I'll play with flow control this afternoon
16:02:53 <azaroth> ivan: Gregg has done full vocab testing?
16:02:59 <azaroth> azaroth: I have as well
16:03:06 <azaroth> ivan: The report is empty though
16:03:11 <azaroth> ...
16:03:12 <azaroth> ...
16:03:13 <azaroth> ...
16:03:18 <azaroth> *tumbleweed*
16:03:22 <azaroth> ...
16:03:25 <azaroth> ...
16:03:32 <azaroth> ShaneM: I'm doing my best
16:03:40 <TimCole> ack ShaneM
16:03:40 <Zakim> ShaneM, you wanted to talk about protocol testing...
16:03:42 <azaroth> ... I can do it by hand
16:04:06 <azaroth> ivan: Once we've done it, it's a small report we can do by hand
16:04:11 <azaroth> q?
16:04:25 <azaroth> TimCole: Sorry for not talking about HTML serialization
16:04:29 <azaroth> ... or protocol testing
16:04:43 <azaroth> ShaneM: Lets work on the sequencing thing and see if we can figure it out
16:04:52 <azaroth> TimCole: No reason to test selectors if you don't have a specific resource
16:05:00 <azaroth> ... the more consolidated tests will make it easier
16:05:11 <azaroth> ... Worried that the report will correctly have blank cells
16:05:26 <azaroth> ShaneM: Suggest reorganizing tests first
16:05:32 <azaroth> TimCole: Thanks all
16:05:34 <azaroth> Bye!
16:07:14 <ivan> rrsagent, draft minutes
16:07:14 <RRSAgent> I have made the request to generate http://www.w3.org/2016/09/09-annotation-minutes.html ivan
16:07:39 <ivan> trackbot, end telcon
16:07:39 <trackbot> Zakim, list attendees
16:07:39 <Zakim> As of this point the attendees have been ShaneM, Tim_Cole, Rob_Sanderson, Ben_De_Meester, Benjamin_Young, TB_Dinesh, Randall_Leeds
16:07:47 <trackbot> RRSAgent, please draft minutes
16:07:47 <RRSAgent> I have made the request to generate http://www.w3.org/2016/09/09-annotation-minutes.html trackbot
16:07:48 <trackbot> RRSAgent, bye
16:07:48 <RRSAgent> I see no action items