EPUB 3 Working Group Telco — Minutes

Date: 2020-11-20

See also the Agenda and the IRC Log

Attendees

Present: Ivan Herman, Matthew Chan, Avneesh Singh, Toshiaki Koike, Masakazu Kitahara, Tzviya Siegman, Ben Schroeter, Wendy Reid, Brady Duga, Bill Kasdorf, Charles LaPierre, George Kerscher, Hadrien Gardeur, Matt Garrish, Zheng Xu (徐征), Gregorio Pellegrino

Regrets: Daihei Shiohama (塩濱大平)

Guests: Dave Cramer

Chair: Wendy Reid

Scribe(s): Matthew Chan

Content:

Wendy Reid: welcome. We’re mainly going to be talking about testing this week

1. testing

Dave Cramer: there’s been progress since the f2f, ivan created a repo for tests

Dave Cramer: https://github.com/w3c/epub-tests/

Dave Cramer: let’s go over some of the big questions about testing

Dave Cramer: https://github.com/w3c/epub-tests/wiki/People-interested-in-testing

Dave Cramer: e.g. who is doing what work?
… there’s a wiki on the repo about this topic
… check the repo if you volunteered to make sure you’re on the list
… we still need an official test lead to manage and drive the testing effort
… if you are that person, please let us know!
… Q2 what do we test?
… i.e. What is testable in the spec, What is the scope of testing? What doesn’t need to be tested? What will satisfy W3 management?
… How do we write and organize test?
… I’ve started a PR in the test repo, a folder structure based on how the spec is organized
… folder for core spec, folder for RS conformance
… there are also a couple tests already in there that I’ve written in the past
… e.g. if epub version = 0
… yes, go ahead and merge the PR Ivan
… Q. How do we deal with all the overhead of our tests?
… Most of these tests will be an HTML file that shows an issue, or a package file that shows an issue. But for it to be an epub it will still have to have all the other elements that make a complete epub
… could each test be that bare minimum html or package file + a script that builds it into a complete epub?
… i’ve experimented with that approach a little
… or should we just have a generic epub that we modify to have whatever specific assertion that we are testing?
… not sure which way is better
… maybe we should just pick a small piece of the spec and write a handful of tests for that, and discover best-practice based on that

Tzviya Siegman: agree with dave about experimenting to see what will work
… did Richard say that he had already started experimenting with something like this, or a parallel effort?
… we should figure out what he has that we can build on

Ivan Herman: If we had a script that could make an epub from a test html, it should be possible to have github automatically turn it into a epub

Wendy Reid: that’s github actions
… does the w3c have access?

Ivan Herman: i wouldn’t worry about that

George Kerscher: at epubtest.org, each test book has a unique id for each test, along with the success criteria for that individual test
… when we go to the website and query a RS that we are testing against, we then mark that test as fail/pass, and the grid gets updated
… the books are built though github scripts that we maintain
… the tests are manually carried out
… would something like this be helpful to this group?
… also, would the existing books that we have for epubtest be used for our testing in this group?
… should I share my screen and do a demo?

Wendy Reid: let’s go through the queue first

Matt Garrish: i think this could get difficult, esp. with package files that contain special settings
… may wish to consult roman

Brady Duga: i would like as little in the github as possible, to keep it clear what is being tested in each case
… prefer not to have dummy files that clutter up the repo
… but i also want something easy to test, i.e. a packaged up epub that is ready to be tested
… some sort of scripting mechanism will probably be necessary, but trying to do it cleanly is going to be interesting

Bill Kasdorf: when you start from a master and make variants from it, we have to be careful not to have variants of variants
… quickly gets out of hand with duplicates

Dave Cramer: maybe some of this can be handled with careful file naming conventions

Wendy Reid: yes, we’ll have to have a robust naming convention for different versions of files

Dave Cramer: one thing that happens in the css test is that the html files have a prescribed structure and use meta elements in the html to describe things about the test
… e.g. the a meta element contains the test assertions, a meta element that contains a link to the section of the spec being tested

Brady Duga: looking at the version = 0 test, I wouldn’t know what this test was meant to show
… that’s why that test comes with an html file that describes the test
… if you had a script that could generate a package based on a number of named parameters, we could include a desc of the test in those parameters

Dave Cramer: web developers would probably do this using a json data structure that goes into the script

Brady Duga: well, most of this is just boilerplate, so the json structure could just say version = 0, that would work

Wendy Reid: another thing to consider is that the tests should have very clear pass/fail states
… e.g. in some of the tests I’ve run in the past, there were partial-passes, which made things difficult for testers, complicates efforts to use automation
… at Kobo we’ve had automated rendering tests that just use a compare, if you run it enough its very easy to tell if anything has changed
… took testing time down from ~day to ~15 minutes

Bill Kasdorf: +1 to Wendy on the tests should be pass/fail and automatable; automation also reduces the variations you get from various people doing the testing

Dave Cramer: this is a difficult subject to hash out with out being in person, with a whiteboard…

Wendy Reid: w3c has etherpad… but there are also other similar tools

Dave Cramer: or what about more information collaboration? IRC outside of official meeting hours
… I’m going to keep filling the testing repo, and welcome people to comment

Brady Duga: {package: {version: 0} result: {text: "This test checks if a reading system will open an EPUB with package version attribute less than 3"}}

Dave Cramer: what about when tests start involving interactions between files? Could the json be modified to describe these interactions?

Brady Duga: the json could be modified to include a reference to a file name, yeah

Dave Cramer: okay, we have lots of ideas, so let’s go and experiment
… tell us what you’ve done and what the results are in the repo using issues

Wendy Reid: we also need to pull out a new list of assertions

Dave Cramer: how should we time that with the FPWD?
… maybe wait until FPWD?

George Kerscher: I like the idea of developing a unique ID for each assertion, or each test of each assertion, maybe pointing to the specific section of the spec being testing

2. FPWD

Wendy Reid: mgarrish has made a number of changes to a bunch of the documents
… it feels like now is a good time to publish the FPWD
… the group just has to agree to publish
… there are a number of open PRs right now, ~7

Matt Garrish: yes, some are for the a11y spec though
… i’m going away for a while, so you guys can review and merge during that interim period

Ivan Herman: mgarrish, how long will you be gone for?

Matt Garrish: probably from next wed thru to ~Dec 20 I’ll be on the road
… i’ll still be contactable though

Ivan Herman: we won’t be able to publish by next Wed
… i would prefer to wait until you are back
… even if we agree today to go to FPWD, I think we should agree not to pub until January. But we can get the process of approvals for FPWD started now

Avneesh Singh: I think Jan is fine. Our deadline for FPWD isn’t until Feb
… regarding a11y, we’ve just started. I’d like to have some substance in this section of the spec before we publish
… I’d prefer if we stick to the timeline

Ivan Herman: maybe we can make a formal decision about publishing FPWD, and I can start the publication process then?

Avneesh Singh: fine with me

Bill Kasdorf: I’m not sure 2020 will concede.

Proposed resolution: Publish the EPUB Core and Reading System documents as FPWD in January 2021 (Wendy Reid)

Matt Garrish: +1

Gregorio Pellegrino: +1

Tzviya Siegman: +1

Ben Schroeter: +1

Matthew Chan: +1

Ivan Herman: +1

Brady Duga: +1

Wendy Reid: +1

Avneesh Singh: +1

Charles LaPierre: +1

Bill Kasdorf: +1

George Kerscher: +1

Masakazu Kitahara: +1

Zheng Xu (徐征): +1

Resolution #1: Publish the EPUB Core and Reading System documents as FPWD in January 2021

Proposed resolution: Use the ECHIDNA publishing system (Wendy Reid)

Tzviya Siegman: +1

Matt Garrish: +1

Bill Kasdorf: +1

Matthew Chan: +1

Avneesh Singh: +1

Ivan Herman: +1

Charles LaPierre: +1

Brady Duga: +1

Wendy Reid: +1

Ben Schroeter: +1

George Kerscher: +1

Resolution #2: Use the ECHIDNA publishing system

3. AOB

Wendy Reid: is there AOB?

Dave Cramer: What about the patent policy?

Wendy Reid: I submitted the survey. Ivan, will this be processed in the beginning of Dec?

Ivan Herman: yes

Wendy Reid: so everyone, look out for an email about the policy/re-joining

Avneesh Singh: Did this group have a resolution about patent policy?

Ivan Herman: Yes, two weeks ago it was discussed and agreed, a formal resolution happened during a Japan time meeting, but we should not separate the groups. A resolution for one applies to the other as well

Avneesh Singh: is that true?

Wendy Reid: Yes… but when a resolution is raised, we won’t act on it until a few days later, to give the rest of the members time to comment via email mailing list, to contact the chairs, etc.
… but we won’t re-do the resolution, no
… Note: we are not going to be meeting next week
… happy Thanksgiving to those celebrating next week!

4. Resolutions