13:14:19 <RRSAgent> RRSAgent has joined #epub
13:14:19 <RRSAgent> logging to https://www.w3.org/2021/09/24-epub-irc
13:14:21 <Zakim> RRSAgent, make logs Public
13:14:22 <Zakim> please title this meeting ("meeting: ..."), ivan
13:14:32 <ivan> ivan has changed the topic to: Meeting Agenda 2021-09-24: https://lists.w3.org/Archives/Public/public-epub-wg/2021Sep/0035.html
13:14:33 <ivan> Chair: dauwhe
13:14:33 <ivan> Date: 2021-09-24
13:14:33 <ivan> Agenda: https://lists.w3.org/Archives/Public/public-epub-wg/2021Sep/0035.html
13:14:33 <ivan> Meeting: EPUB 3 Working Group Telco
13:14:33 <ivan> Regrets+ matthew, toshiakikoike
13:48:59 <MURATA> MURATA has joined #epub
13:52:30 <wendyreid> wendyreid has joined #epub
13:53:11 <dauwhe> dauwhe has joined #epub
13:56:06 <BenSchroeter> BenSchroeter has joined #epub
13:59:15 <MasakazuKitahara> MasakazuKitahara has joined #epub
13:59:22 <MasakazuKitahara> present+
13:59:37 <ivan> present+
13:59:52 <ivan> regrets- matthew
14:00:01 <Aimee> Aimee has joined #epub
14:00:04 <ivan> present+ wendyreid
14:00:10 <ivan> present+ dauwhe
14:00:12 <Aimee> present+
14:00:18 <ivan> present+ Murata
14:00:23 <MattChan_> MattChan_ has joined #epub
14:00:23 <gpellegrino> gpellegrino has joined #epub
14:00:26 <ivan> present+ gregorio
14:00:34 <ivan> present+ BenSchroeter
14:00:37 <MattChan_> present+
14:00:40 <gpellegrino> present+
14:00:45 <BenSchroeter> present+
14:01:08 <LauraB__> LauraB__ has joined #epub
14:01:15 <LauraB__> Present+
14:01:48 <ivan> present+ george
14:01:50 <George> George has joined #epub
14:01:54 <MattChan_> scribe+
14:01:55 <duga> duga has joined #epub
14:01:59 <ivan> present+ duga
14:02:02 <duga> present+
14:02:14 <ivan> present+ dlazin
14:02:24 <George> George has joined #epub
14:02:28 <ivan> present+ charles
14:02:31 <MURATA> present+
14:02:42 <George> present+
14:02:43 <George> present+
14:03:35 <MattChan_> dauwhe: we have a meeting devoted to testing. This will be one of most important things we do on path to REC.
14:03:46 <MattChan_> TOPIC: Testing Update
14:04:22 <dlazin> dlazin has joined #epub
14:04:24 <dlazin> dlazin+
14:04:26 <dlazin> https://github.com/w3c/epub-tests
14:04:29 <MattChan_> dlazin: we've made some progress. Updates in repo.
14:04:34 <MattChan_> ... above is link to epub tests repo
14:04:36 <CharlesL> CharlesL has joined #epub
14:04:37 <MattChan_> ... a number of cool things
14:04:49 <MattChan_> ... readme on main page gives you step by step on how to write a test
14:04:50 <George> George has joined #epub
14:04:51 <CharlesL> present+
14:05:00 <MattChan_> ... its quick to do, but you do need basic github knowledge
14:05:13 <MattChan_> ... there's templates to work from, fairly straightforward
14:05:23 <George> George has joined #epub
14:05:42 <MattChan_> ... 2nd, Ivan has built report generation tool, uses github actions to generate HTML pages that lists all tests we have, and implementation status of RS against those tests
14:05:55 <MattChan_> ... everything in there so far are just examples
14:05:58 <ivan> -> https://w3c.github.io/epub-tests/results/ example reports with fake implementation reports
14:06:04 <MattChan_> ... are the reports working in the real repo now?
14:06:10 <MattChan_> ivan: yes!
14:06:30 <MattChan_> dlazin: the first page (linked to by Ivan) are the implementation results
14:06:40 <dlazin> https://w3c.github.io/epub-tests/results/tests.html
14:06:43 <MattChan_> ... the one we're working on now is linked as a separate document (here)
14:06:49 <George> George has joined #epub
14:06:50 <MattChan_> ... this lists all the tests that are currently marked up
14:07:04 <MattChan_> ... we have several dozen tests right now, but only a handful of them have the proper markup
14:07:29 <MattChan_> ... we need to add tags to them so they can be picked up in the reports
14:08:11 <MattChan_> ... the specs column links to where the normative statement is in the spec that is being tested, the ref links back to the original report that tells you implementation status across RS
14:08:19 <MattChan_> ... now wendyreid can demonstrate how to write a test
14:08:38 <MattChan_> wendyreid: disclaimer: this portion of the meeting will be recorded for future reference
14:08:49 <George> George has joined #epub
14:08:52 <MattChan_> ... feel free to mute yourself or step out of the call now
14:09:08 <MattChan_> ivan: recording now
14:09:26 <MattChan_> wendyreid shares screen
14:09:40 <MattChan_> wendyreid: i'm just going to follow the step by step instructions
14:09:52 <MattChan_> ... #1 find untested or undertested normative statement
14:10:25 <MattChan_> ... we're going to test fxl, RS MUST product 1 page per spine item
14:10:43 <dauwhe> q?
14:10:49 <MattChan_> ... for people who are unfamiliar with github, I use a tool called github desktop
14:10:49 <George> George has joined #epub
14:11:01 <MattChan_> ... i've created a branch called fxl-properties2
14:11:11 <MattChan_> ... i'm going to publish my branch
14:11:24 <MattChan_> ... try to give them descriptive names
14:11:27 <George> George has joined #epub
14:11:44 <MattChan_> ... in my branch, i'm going to us VSCode to open the repo
14:12:01 <MattChan_> ... make a copy of the template
14:12:07 <George> George has joined #epub
14:12:26 <MattChan_> ... rename the templated file with a descriptive name
14:12:36 <MattChan_> ... now modify the templated file to meet the needs of my test
14:13:06 <MattChan_> ... fill out package.opf (use the metadata guide if you need help)
14:13:59 <MattChan_> ... fill out dc:coverage, dc:creator, dc:date, dc:description (describe the test)
14:14:12 <MattChan_> dlazin: this is a good reason to do a group of related tests at once
14:14:28 <MattChan_> wendyreid: spec-anchor is the location in the spec that you are testing
14:14:40 <MattChan_> ... and it needs to be the TR one
14:14:49 <George> George has joined #epub
14:15:07 <MattChan_> dlazin: i recommend inspecting the spec document, and taking the anchor that is closest to the normative statement you are testing
14:15:25 <MattChan_> ... and if there isn't one, then you should create one and name it according to your new test
14:15:45 <MattChan_> wendyreid: important to use TR version of the URL, not the github version of URL
14:16:42 <MattChan_> ivan: dc:identifier should be the file name of your test
14:16:49 <George> George has joined #epub
14:16:55 <MattChan_> dlazin: we're still debating the precise naming of tests
14:17:25 <George> George has joined #epub
14:17:26 <MattChan_> ... testing WG will sort it out and clarify documentation
14:17:47 <George> George has joined #epub
14:17:53 <MattChan_> wendyreid: so that is basically the metadata done
14:18:04 <MattChan_> ... for this test i don't need to make any other changes to the opf
14:18:07 <avneeshsingh> avneeshsingh has joined #epub
14:18:23 <avneeshsingh> present+
14:18:49 <George> George has joined #epub
14:19:27 <MattChan_> ... but what i do have to do is update the actual XHTML content
14:19:44 <MattChan_> ivan: i went through this process this AM with a specific RS going through a test
14:19:52 <MattChan_> ... implementers will run one test after another
14:20:04 <MattChan_> ... and it must be extremely clear to the implementer when a test passes, and when it fails
14:20:25 <MattChan_> ... in this case it is straightforward, but there could be some tests where you see something on the screen, and its not clear whether the result is a pass or fail
14:20:36 <MattChan_> ... testers must be able to tell quickly what the result is
14:20:49 <George> George has joined #epub
14:21:00 <CharlesL> q+
14:21:23 <MattChan_> wendyreid: so in my XHTML I edit the content to make clear that if the tester sees the <p> without a break, then the test is a pass
14:21:49 <George> George has joined #epub
14:22:00 <MattChan_> ... repeat for each XHTML page
14:22:26 <George> George has joined #epub
14:22:57 <MattChan_> ... now its time to make it into an epub
14:23:14 <MattChan_> ... you can use eCanCrusher or however else you make an epub
14:24:03 <MattChan_> ... there's a cmdline way to do it in the documentation if you're on mac
14:24:37 <MattChan_> ... move the epub file into the test folder of the repo
14:24:49 <George> George has joined #epub
14:25:30 <MattChan_> ... i'm going to quickly open it to check it. And yes, it works.
14:25:50 <MattChan_> ... there are instructions in the documentation for how to open it in various RS
14:26:14 <MattChan_> ... now i'm going to commit my changes to my branch
14:26:22 <MattChan_> ... give a descriptive commit msg
14:26:49 <George> George has joined #epub
14:26:57 <MattChan_> ... now my changes are up in my branch, and I'm going to create my PR
14:27:07 <duga> q+ for after presentation
14:27:23 <George> George has joined #epub
14:27:26 <MattChan_> ... and dlazin or ivan will review my test
14:28:08 <MattChan_> ... when you do the PR, you don't have to worry about potentially breaking the test repo. Your work will be reviewed, and you will get feedback if necessary.
14:28:36 <MattChan_> ... only once all the potential issues with your test are fixed will it get merged
14:28:49 <George> George has joined #epub
14:29:17 <MattChan_> ... once PR has been merged, you are going to fork the repo for the spec (e.g. EPUB 33 or EPUB RS 33)
14:30:16 <MattChan_> ... i go into version of spec in my repo, and add a data-tests property, with value equal to the name of the test I just created
14:30:30 <MattChan_> dlazin: let's put the data-tests property on the same element as the id
14:30:49 <George> George has joined #epub
14:30:57 <MattChan_> ... this has to be a separate PR because the spec is stored separately from the tests
14:31:28 <MattChan_> ivan: if you go to the editors draft of the spec, then respec will show you the number of tests that are avail for a given normative statement, and give you a link back to the tests
14:31:42 <MattChan_> ... the tests and specs are going to be cross-referrenced in this way
14:31:49 <George> George has joined #epub
14:32:21 <MattChan_> wendyreid: once your change to the spec is done, then you will 1) commit it, and 2) do a PR for the spec, just like you did with your test
14:32:26 <MattChan_> ... the end
14:32:31 <MattChan_> ... any questions?
14:32:38 <wendyreid> q?
14:32:39 <duga> q?
14:32:39 <dauwhe> q?
14:32:44 <wendyreid> ack CharlesL
14:33:03 <MattChan_> CharlesL: one, shouldn't epubcheck should be part of the workflow to make sure your epub is valid?
14:33:16 <MattChan_> dauwhe: not necessarily, it depends on the test
14:33:32 <MattChan_> s/should be/be
14:33:57 <MattChan_> dlazin: i agree that it should be. Will add to documentation.
14:34:28 <dauwhe> ack du
14:34:28 <Zakim> duga, you wanted to discuss after presentation
14:34:30 <MattChan_> CharlesL: for this particular test, shouldn't there be one XHTML page with a page break? So we can test what happens there too?
14:34:49 <George> George has joined #epub
14:35:03 <MattChan_> duga: one of the things we have to do is ensure that test would fail if we removed the thing being tested
14:35:43 <MattChan_> ... e.g. if rendition-layout weren't set to pre-paginated, you would still get 4 pages
14:36:49 <George> George has joined #epub
14:36:52 <MattChan_> ... so in this case putting a pagebreak in might be the thing to do, but if the RS doesn't support pagebreak AND doesn't support fxl, then we still won't get an unambiguous result
14:37:09 <wendyreid> ack duga
14:37:18 <MattChan_> duga: i wasn't involved in the creation of the process, but what is there looks excellent. thank you all.
14:37:31 <MattChan_> ... first, how hard should we be making these tests?
14:38:02 <MattChan_> ... e.g. i could make a test that validates this normative statement, but would either pass or fail depending on the test
14:38:24 <MattChan_> ... e.g. on Google RS, a pure fxl would pass, but a mixed pagination would fail
14:38:32 <ivan> present+ tzviya
14:38:40 <MattChan_> dlazin: we have a lot of testing to do, and we should prioritize
14:38:49 <George> George has joined #epub
14:38:54 <MattChan_> ... there are a lot of places where there could be 6 different unit tests for a single assertion
14:39:04 <MattChan_> ... but lets try to get the basic stuff in first, and then do improvement passes later
14:39:23 <MattChan_> ... e.g. ivan was testing dir property earlier this week. We have 6 different tests for that.
14:39:43 <dauwhe> q+
14:39:49 <MattChan_> ... let's try to get the MUSTs and MUST NOTs, then move on to the SHOULDs and SHOULD NOTs etc.
14:40:23 <MattChan_> ... but if you have the appetite to do rigorous testing with multiple cases at once, then go ahead
14:40:27 <Hadrien> Hadrien has joined #epub
14:40:44 <ivan> present+ Hadrien
14:40:47 <MattChan_> duga: we should verify with RS developers how they want their RS tested
14:40:49 <George> George has joined #epub
14:41:13 <MattChan_> ... e.g. for Google Play, the doc points people to the user upload path rather than the publisher upload path
14:41:24 <MattChan_> ... a test might fail on one path, and pass on the other path
14:41:45 <MattChan_> ... and using the right path will be difficult for someone who isn't me or a publisher
14:41:49 <George> George has joined #epub
14:41:55 <MattChan_> dlazin: i think we want the RS devs to do their own testing
14:42:14 <MattChan_> ... and if we can't get RS implementors to run the test suites, then we'll do it for them
14:42:23 <MattChan_> ... but we prefer to them to do it themselves
14:42:25 <George> George has joined #epub
14:42:35 <Hadrien> q+
14:43:06 <MattChan_> ... in the case of Google, you might actually want to test user path and publisher path for each of web, ios, and android...
14:43:27 <MattChan_> duga: last, how are we claiming a test?
14:43:45 <MattChan_> ... if i want to start working on making a test for one statement, what do I do?
14:43:53 <MattChan_> dlazin: I think the doc says to claim it in the spreadsheet
14:44:30 <MattChan_> duga: maybe the better way to do this is to create a github issue, and put a link to the issue in the spreadsheet?
14:44:38 <ivan> +1 to brady
14:44:45 <MattChan_> dlazin: i like that for tracking purposes, but it adds more overhead
14:44:49 <George> George has joined #epub
14:45:21 <MattChan_> duga: the alternative is automatically generating a github issue for each record in the spreadsheet...
14:45:50 <MattChan_> dlazin: also, be realistic about which tests you will actually be able to test, what you have time to do quickly
14:46:19 <MattChan_> ... also reason why we track in the spreadsheet rather than in spec is because spec sections can still move around
14:46:46 <MattChan_> ... if we were to turn everything in the spreadsheet into issues, we might get false confidence about how much testing is left to do
14:46:49 <George> George has joined #epub
14:47:07 <wendyreid> ack dauwhe
14:47:11 <MattChan_> duga: okay, let's do JIT then
14:47:34 <ivan> q+
14:47:38 <MattChan_> duga: note about having RS do their own tests, i have a concern that test results should be repeatable by 3rd parties
14:47:45 <MattChan_> ... but point about multiple paths is a valid one
14:47:59 <MattChan_> ... re. how hard tests should be, a lot depends on where we think there might be problems in the spec
14:48:10 <MattChan_> ... if we think things in spec are weird, we should test more
14:48:25 <MattChan_> ... to try to understand better, find interop problems
14:48:33 <wendyreid> ack Hadrien
14:48:49 <George> George has joined #epub
14:49:00 <MattChan_> Hadrien: about RS testing and publisher path vs user path, in some cases, there is no user path
14:49:09 <MattChan_> ... problematic for this type of testing
14:49:21 <MattChan_> ... and you have RS where those two different paths will behave inconsistently
14:49:33 <MattChan_> ... and there are cases where testing happens in some separate app that is not really an RS
14:49:55 <MattChan_> ... given all of that, i think we should target RS where there is a user path, and where there will be some consistency
14:49:59 <duga> q+
14:50:18 <MattChan_> ... we can identify a number of RS that can provide the type of reproducibility that dauwhe was referring to
14:50:20 <ivan> q-
14:50:31 <MattChan_> ... this is important because not all RS is equal in that regard
14:50:44 <wendyreid> ack ivan
14:50:49 <George> George has joined #epub
14:50:56 <wendyreid> ack duga
14:51:14 <MattChan_> duga: fair points, but i know that e.g. Google's user's path is far more lenient
14:51:27 <MattChan_> ... because we see a lot more bad content from user path vs publisher path
14:51:33 <MattChan_> ... so it kind of plays to the weakest link
14:51:41 <ivan> q+
14:51:49 <George> George has joined #epub
14:51:53 <MattChan_> ... that's a reason why we shouldn't limit to user path
14:52:29 <MattChan_> ... from Google Play books perspective, I can publish all these books and make them available as a way for you to check my evaluation of results
14:52:30 <wendyreid> ack ivan
14:52:50 <MattChan_> ivan: we shouldn't forget that the goal of this exercise is to test the spec
14:53:04 <MattChan_> ... we're not providing an environment for measuring one implementation against another
14:53:29 <MattChan_> ... my approach is to trust the implementors. They aren't trying to fool us. We're trying to get the spec properly specified.
14:54:45 <MattChan_> wendyreid: new test writers shouldn't be afraid to try. There will be stumbling blocks. But there are people to provide support.
14:54:49 <George> George has joined #epub
14:55:02 <MattChan_> ivan: i'm stopping the recording now
14:55:23 <MattChan_> ... with Zoom it takes some time to generate the video, and then we will clean it up a little bit
14:55:37 <MattChan_> ... i will clean the minutes of the meeting as usual, and provide the link to the video as a follow-up
14:55:46 <avneeshsingh> q+ to comment on finalized time with TPAC APA meetings
14:55:57 <wendyreid> ack avneeshsingh
14:55:57 <Zakim> avneeshsingh, you wanted to comment on finalized time with TPAC APA meetings
14:56:15 <MattChan_> avneeshsingh: we've finalized time for TPAC APA meeting
14:56:15 <avneeshsingh> https://www.w3.org/WAI/APA/wiki/Meetings/TPAC_2021
14:56:25 <MattChan_> ... here's link to schedule
14:56:44 <MattChan_> ... we will discuss a11y horizontal review for epub 33
14:56:59 <MattChan_> ... second meeting about normative references to schema
14:57:09 <MattChan_> ... the two dates are in the link
14:57:34 <wendyreid> Thursday 28 October at 10:00 Boston / 1400Z (Confirmed)
14:57:37 <LauraB__> LauraB__ has left #epub
14:57:50 <MattChan_> ivan: should I create calendar entries for the group?
14:58:19 <MattChan_> avneeshsingh: we can discuss with Janina, as she maybe creating calendar entries, which we can just copy for our group
14:58:38 <MattChan_> TOPIC: AOB?
14:58:55 <MattChan_> duga: thank you again to dlazin, wendyreid, dauwhe, ivan for all of this
14:59:02 <MattChan_> wendyreid: most of that thanks to dlazin!
14:59:06 <CharlesL> CharlesL has left #epub
14:59:16 <MattChan_> wendyreid: everyone have a wonderful weekend, bye!
15:18:17 <ivan> rrsagent, draft minutes
15:18:17 <RRSAgent> I have made the request to generate https://www.w3.org/2021/09/24-epub-minutes.html ivan
15:21:22 <gpellegrino> gpellegrino has joined #epub
15:21:45 <gpellegrino> gpellegrino has joined #epub
15:28:07 <github-bot> github-bot has joined #epub
15:45:16 <dauwhe> dauwhe has joined #epub
15:54:13 <ivan> zakim, end meeting
15:54:13 <Zakim> As of this point the attendees have been MasakazuKitahara, ivan, wendyreid, dauwhe, Aimee, Murata, gregorio, BenSchroeter, MattChan_, gpellegrino, LauraB__, george, duga, dlazin,
15:54:16 <Zakim> ... charles, CharlesL, avneeshsingh, tzviya, Hadrien
15:54:16 <Zakim> RRSAgent, please draft minutes
15:54:16 <RRSAgent> I have made the request to generate https://www.w3.org/2021/09/24-epub-minutes.html Zakim
15:54:18 <ivan> rrsagent, bye
15:54:18 <RRSAgent> I see no action items
15:54:19 <Zakim> I am happy to have been of service, ivan; please remember to excuse RRSAgent.  Goodbye