EPUB 3 Working Group Telco — Minutes

Date: 2022-04-22

Attendees

Present: Masakazu Kitahara, Ivan Herman, Toshiaki Koike, Avneesh Singh, Ben Schroeter, Brady Duga, Wendy Reid, Gregorio Pellegrino, Matthew Chan, Dan Lazin, George Kerscher, GeorgeK, Charles LaPierre, Jen Goulden

Regrets: Dave Cramer, Tzviya Siegman, Matt Garrish

Guests:

Chair: Wendy Reid

Scribe(s): Matthew Chan

Wendy Reid: the topic for today is how to run tests.
… if everything goes well, we’ll be in CR soon, so we’ll want to start running tests.
… what will that actually look like?.
… if everyone is okay with recording the part of the meeting where dlazin explains this, we’ll go ahead with that.

Ivan Herman: can we add an agenda item?

1. WebViews CG

Ivan Herman: I have sent a mail to the group a few days ago, but i wanted it recorded here too.
… there is a CG on WebViews now.
… i was there for the opening call, but I don’t have the technical expertise to participate.
… yet, the epub RS community is one of the major users of WebViews.
… any work in that CG might affect RS developers in the future.
… there is a list of issues that has been raised already.

Ivan Herman: See WebViews issues.

Ivan Herman: the goal of the group is to write a use-case and requirements document, to be done shortly after TPAC.
… they will try to identify where WebViews implementations might converge, additional features, etc..
… i could be the go-between, but probably better for RS implementors with the necessary background to participate in that CG to make sure that what is around in the long term will make your lives easier.

Brady Duga: I joined the WebViews CG, but it would be great to have some non-Google people representing there.

2. How to run tests?

Wendy Reid: okay, please comment now if you have an issue with me recording?

Dan Lazin: hello everyone, let’s first re-familiarize you with where the tests are at, and how RS can determine whether they are in compliance with the spec/whether we have 2 independent implementors of each feature.

Dan Lazin: See test repository on github.

Dan Lazin: here is the test repo for our spec.
… i will attempt to share screen now.
… most of what we talked about previously are the tests themselves, in /tests repo.
… for each test there is a folder of uncompressed stuff, and further down the zipped epub test itself.
… today we talk about running the tests themselves.
… we store the results as json, which gets used to populate the tests results page.
… there are two main documents, the contribution guidelines and the implementation report.
… the implementation reports page shows the test results for each RS that supplies implementation results.
… we have tables of consolidated results, as well as implementation results.

Dan Lazin: See page on contributing.

Dan Lazin: using Wysebee as example, it has 4 separate implementations. Google Play would similarly record its implementation specific results in different columns.

Dan Lazin: See page on test results.

Dan Lazin: we’re going to do an example assuming we are Apple Books.
… opening github app, and switching to repo for epub-tests, fetch to make sure I am up-to-date.
… in /epub-tests/reports you can see submitted test results.
… the xx-template.json contains a rest results template with everything set to false.
… create a duplicate of this for each implementation.
… update name.
… each test result is currently a boolean value set to false, I would comment out/remove all tests I have not yet run to distinguish a test failure from ‘test not yet run’.
… get a test epub from the repo, and open it in your RS.
… view the epub to determine whether your RS has passed the test.
… if you need more context, you can view the unzipped test epub.
… update the test result value for the test you just performed (true/false).
… create a branch on your fork, and submit a PR with your test result.
… that’s the idea – go through the tests and run as many as you can.
… note that the list of tests is still being added to, so keep track of which ones you’ve done/which you still need to run.
… in the github version of the spec, respec will show an inline tests notice when there are tests for a given assertion.
… if you contributing to writing tests, you can use this to determine where more tests are needed.

Brady Duga: from a work standpoint, can we use null to signify when a test has not been run on a given implementation yet?.
… when we complete new tests, will we populate the existing test result json files? Or will testers have to go back and compare their test result json again the updated template?.

Dan Lazin: for now we aren’t going back and updating existing test result files.
… yes to changing template test result file to use null instead of false for tests yet to be run.

Ivan Herman: re. updating test results template and updating existing test results files with new tests, i feel uneasy about changing RS dev’s test results files.

Brady Duga: could we make it so that testers have an option to run a script to say ‘yes, add new tests to my test results file’?.

Ivan Herman: no.

Brady Duga: okay, so testers will just add new tests to their test results file manually for now.
… re. visibility of tests, one of the easiest ways for me to test is to make the tests public, can we do that?.

Dan Lazin: all the tests currently contain copyright information, and if you are happy to have the tests in your ebook store, we are happy for you to do that.

Ivan Herman: i think the current copyright statement is a cc with attribution to w3c.

Dan Lazin: no, we don’t use cc, we use w3c license.

Ivan Herman: oh, right, yes, we checked with with w3c Wendy Seltzer (in legal) and it was okay.

Wendy Reid: any other questions?.
… alright, so, we still need tests to be written, and we want tests to be performed.
… don’t feel the need to start testing right away, as we are not yet in CR.
… we are waiting for feedback from i18n and ping, as the Director has asked us to check with them.
… once we are in CR we’ll start testing in earnest.
… if you can contribute tests, please do. If you have questions about contributing tests, you can come to me or dlazin.

3. “structural” tests.

Ivan Herman: another big area of testing that these tests will not cover is the content documents.
… i.e. the MUSTs in the Core that are not executable.
… we had a long discussion about using epubcheck as a basis for testing.
… the epubcheck team is busy producing a version for epub 3.3, as part of which they will be creating tests for themselves.
… not all will be relevant for us (i.e. where they test HTML features).
… we will extract the tests that are relevant for us, and I will script something that will use those tests to create a sort of test results table.
… most likely what we will have is an inline pulldown menu at the section level of the spec, listing the tests performed by epubcheck.
… so things are evolving on that side of testing.

4. AOB?.

Wendy Reid: i sent an email to the group on behalf of locators group, re. assistive tech and locators.
… please reply to the email if you have input, some people already have.
… okay, thanks all, bye!.