EPUB 3 Working Group Telco — Minutes

Date: 2021-09-24

Attendees

Present: Masakazu Kitahara, Ivan Herman, Wendy Reid, Dave Cramer, Aimee Ubbink, Murata Makoto, gregorio, Ben Schroeter, Matthew Chan, Gregorio Pellegrino, Laura Brady, George Kerscher, Brady Duga, Dan Lazin, Charles LaPierre, Avneesh Singh, Tzviya Siegman, Hadrien Gardeur

Regrets: Toshiaki Koike

Guests:

Chair: Dave Cramer

Scribe(s): Matthew Chan

Dave Cramer: we have a meeting devoted to testing. This will be one of most important things we do on path to REC.

1. Testing Update

Dan Lazin: dlazin+

Dan Lazin: https://github.com/w3c/epub-tests

Dan Lazin: we’ve made some progress. Updates in repo.
… above is link to epub tests repo
… a number of cool things
… readme on main page gives you step by step on how to write a test
… its quick to do, but you do need basic github knowledge
… there’s templates to work from, fairly straightforward
… 2nd, Ivan has built report generation tool, uses github actions to generate HTML pages that lists all tests we have, and implementation status of RS against those tests
… everything in there so far are just examples

Ivan Herman: See example reports with fake implementation reports.

Dan Lazin: are the reports working in the real repo now?

Ivan Herman: yes!

Dan Lazin: the first page (linked to by Ivan) are the implementation results
… the one we’re working on now is linked as a separate document (here)

Dan Lazin: See list of the tests themselves.

Dan Lazin: this lists all the tests that are currently marked up
… we have several dozen tests right now, but only a handful of them have the proper markup
… we need to add tags to them so they can be picked up in the reports
… the specs column links to where the normative statement is in the spec that is being tested, the ref links back to the original report that tells you implementation status across RS
… now wendyreid can demonstrate how to write a test

Wendy Reid: disclaimer: this portion of the meeting will be recorded for future reference
… feel free to mute yourself or step out of the call now

Ivan Herman: recording now

Matthew Chan: wendyreid shares screen

Wendy Reid: i’m just going to follow the step by step instructions
… #1 find untested or undertested normative statement
… we’re going to test fxl, RS MUST product 1 page per spine item
… for people who are unfamiliar with github, I use a tool called github desktop
… i’ve created a branch called fxl-properties2
… i’m going to publish my branch
… try to give them descriptive names
… in my branch, i’m going to us VSCode to open the repo
… make a copy of the template
… rename the templated file with a descriptive name
… now modify the templated file to meet the needs of my test
… fill out package.opf (use the metadata guide if you need help)
… fill out dc:coverage, dc:creator, dc:date, dc:description (describe the test)

Dan Lazin: this is a good reason to do a group of related tests at once

Wendy Reid: spec-anchor is the location in the spec that you are testing
… and it needs to be the TR one

Dan Lazin: i recommend inspecting the spec document, and taking the anchor that is closest to the normative statement you are testing
… and if there isn’t one, then you should create one and name it according to your new test

Wendy Reid: important to use TR version of the URL, not the github version of URL

Ivan Herman: dc:identifier should be the file name of your test

Dan Lazin: we’re still debating the precise naming of tests
… testing WG will sort it out and clarify documentation

Wendy Reid: so that is basically the metadata done
… for this test i don’t need to make any other changes to the opf
… but what i do have to do is update the actual XHTML content

Ivan Herman: i went through this process this AM with a specific RS going through a test
… implementers will run one test after another
… and it must be extremely clear to the implementer when a test passes, and when it fails
… in this case it is straightforward, but there could be some tests where you see something on the screen, and its not clear whether the result is a pass or fail
… testers must be able to tell quickly what the result is

Wendy Reid: so in my XHTML I edit the content to make clear that if the tester sees the <p> without a break, then the test is a pass
… repeat for each XHTML page
… now its time to make it into an epub
… you can use eCanCrusher or however else you make an epub
… there’s a command line way to do it in the documentation if you’re on mac
… move the epub file into the test folder of the repo
… i’m going to quickly open it to check it. And yes, it works.
… there are instructions in the documentation for how to open it in various RS
… now i’m going to commit my changes to my branch
… give a descriptive commit msg
… now my changes are up in my branch, and I’m going to create my PR
… and dlazin or ivan will review my test
… when you do the PR, you don’t have to worry about potentially breaking the test repo. Your work will be reviewed, and you will get feedback if necessary.
… only once all the potential issues with your test are fixed will it get merged
… once PR has been merged, you are going to fork the repo for the spec (e.g. EPUB 33 or EPUB RS 33)
… i go into version of spec in my repo, and add a data-tests property, with value equal to the name of the test I just created

Dan Lazin: let’s put the data-tests property on the same element as the id
… this has to be a separate PR because the spec is stored separately from the tests

Ivan Herman: if you go to the editors draft of the spec, then respec will show you the number of tests that are available for a given normative statement, and give you a link back to the tests
… the tests and specs are going to be cross-referenced in this way

Wendy Reid: once your change to the spec is done, then you will 1) commit it, and 2) do a PR for the spec, just like you did with your test
… the end
… any questions?

Charles LaPierre: one, shouldn’t epubcheck be part of the workflow to make sure your epub is valid?

Dave Cramer: not necessarily, it depends on the test

Dan Lazin: i agree that it should be. Will add to documentation.

Charles LaPierre: for this particular test, shouldn’t there be one XHTML page with a page break? So we can test what happens there too?

Brady Duga: one of the things we have to do is ensure that test would fail if we removed the thing being tested
… e.g. if rendition-layout weren’t set to pre-paginated, you would still get 4 pages
… so in this case putting a page break in might be the thing to do, but if the RS doesn’t support page break AND doesn’t support fxl, then we still won’t get an unambiguous result
… i wasn’t involved in the creation of the process, but what is there looks excellent. thank you all.
… first, how hard should we be making these tests?
… e.g. i could make a test that validates this normative statement, but would either pass or fail depending on the test
… e.g. on Google RS, a pure fxl would pass, but a mixed pagination would fail

Dan Lazin: we have a lot of testing to do, and we should prioritize
… there are a lot of places where there could be 6 different unit tests for a single assertion
… but lets try to get the basic stuff in first, and then do improvement passes later
… e.g. ivan was testing dir property earlier this week. We have 6 different tests for that.
… let’s try to get the MUSTs and MUST NOTs, then move on to the SHOULDs and SHOULD NOTs etc.
… but if you have the appetite to do rigorous testing with multiple cases at once, then go ahead

Brady Duga: we should verify with RS developers how they want their RS tested
… e.g. for Google Play, the doc points people to the user upload path rather than the publisher upload path
… a test might fail on one path, and pass on the other path
… and using the right path will be difficult for someone who isn’t me or a publisher

Dan Lazin: i think we want the RS devs to do their own testing
… and if we can’t get RS implementors to run the test suites, then we’ll do it for them
… but we prefer to them to do it themselves
… in the case of Google, you might actually want to test user path and publisher path for each of web, ios, and android…

Brady Duga: last, how are we claiming a test?
… if i want to start working on making a test for one statement, what do I do?

Dan Lazin: I think the doc says to claim it in the spreadsheet

Brady Duga: maybe the better way to do this is to create a github issue, and put a link to the issue in the spreadsheet?

Ivan Herman: +1 to brady

Dan Lazin: i like that for tracking purposes, but it adds more overhead

Brady Duga: the alternative is automatically generating a github issue for each record in the spreadsheet…

Dan Lazin: also, be realistic about which tests you will actually be able to test, what you have time to do quickly
… also reason why we track in the spreadsheet rather than in spec is because spec sections can still move around
… if we were to turn everything in the spreadsheet into issues, we might get false confidence about how much testing is left to do

Brady Duga: okay, let’s do JIT then
… note about having RS do their own tests, i have a concern that test results should be repeatable by 3rd parties
… but point about multiple paths is a valid one
… re. how hard tests should be, a lot depends on where we think there might be problems in the spec
… if we think things in spec are weird, we should test more
… to try to understand better, find interop problems

Hadrien Gardeur: about RS testing and publisher path vs user path, in some cases, there is no user path
… problematic for this type of testing
… and you have RS where those two different paths will behave inconsistently
… and there are cases where testing happens in some separate app that is not really an RS
… given all of that, i think we should target RS where there is a user path, and where there will be some consistency
… we can identify a number of RS that can provide the type of reproducibility that dauwhe was referring to
… this is important because not all RS is equal in that regard

Brady Duga: fair points, but i know that e.g. Google’s user’s path is far more lenient
… because we see a lot more bad content from user path vs publisher path
… so it kind of plays to the weakest link
… that’s a reason why we shouldn’t limit to user path
… from Google Play books perspective, I can publish all these books and make them available as a way for you to check my evaluation of results

Ivan Herman: we shouldn’t forget that the goal of this exercise is to test the spec
… we’re not providing an environment for measuring one implementation against another
… my approach is to trust the implementors. They aren’t trying to fool us. We’re trying to get the spec properly specified.

Wendy Reid: new test writers shouldn’t be afraid to try. There will be stumbling blocks. But there are people to provide support.

Ivan Herman: i’m stopping the recording now
… with Zoom it takes some time to generate the video, and then we will clean it up a little bit
… i will clean the minutes of the meeting as usual, and provide the link to the video as a follow-up

2. Meeting with APA

Avneesh Singh: we’ve finalized time for TPAC APA meeting

Avneesh Singh: https://www.w3.org/WAI/APA/wiki/Meetings/TPAC_2021

Avneesh Singh: here’s link to schedule
… we will discuss a11y horizontal review for epub 33
… second meeting about normative references to schema
… the two dates are in the link

Wendy Reid: Thursday 28 October at 10:00 Boston / 1400Z (Confirmed)

Ivan Herman: should I create calendar entries for the group?

Avneesh Singh: we can discuss with Janina, as she maybe creating calendar entries, which we can just copy for our group

3. AOB?

Brady Duga: thank you again to dlazin, wendyreid, dauwhe, ivan for all of this

Wendy Reid: most of that thanks to dlazin!
… everyone have a wonderful weekend, bye!