EPUB 3 Working Group Telco — Minutes

Date: 2022-12-02

See also the Agenda and the IRC Log

Attendees

Present: Ivan Herman, Masakazu Kitahara, Tzviya Siegman, Wendy Reid, Matthew Chan, John Foliot, David Hall, Charles LaPierre, Avneesh Singh, Brady Duga, Laurent Le Meur

Regrets: Bill Kasdorf

Guests:

Chair: Dave Cramer

Scribe(s): Matthew Chan

Content:

• 1. Testing Updates.

1. Testing Updates.

Wendy Reid: https://w3c.github.io/epub-tests/results.

Wendy Reid: we’re going to be talking about test results.
… link to results.
… we have results from 17 sources.
… not a complete list.
… still more results expected to come in.
… we’ve results from Apple, Google, Thorium, VitalSource, Wysebee.
… the consolidated results show consolidation of results for all platforms (any single pass counts as a consolidated pass).

Ivan Herman: how up to date is this?.
… Wysebee seems unchanged for past few months….
… we know that Kobo is not final, not sure if Thorium is final yet.

David Hall: i believe Apple is final, no changes made since we did testing.

Ivan Herman: for those who are not familiar, if you click link to one of the test, you get description of what the test entails.

Charles LaPierre: i got stalled on testing of Thorium on mac, I still plan on doing some more in the next couple weeks before I go on vacation.

Ivan Herman: good, thank you!.
… what do we do with those tests where we have 1 or fewer pass flags?.

Wendy Reid: section 3.1 are the consolidated results… so if 2 variants of the same test pass, does that count as a pass overall?.
… e.g. two kobo implementations.

Ivan Herman: are the epub engines underlying the implementations the same? if not, then they should be submitted as separate implementations.
… same for apple.

Tzviya Siegman: having sat on formal objection councils, a lot comes down to what it means to have more than one implementation.
… agree that kobo on ios and kobo on android count as separate, but this table doesn’t really show that well.
… might want to show tests based on platform (e.g. android, iOS, etc.).

Ivan Herman: the detailed implementation result (sec 3.2) shows, for example, Thorium tests on Windows, MacOS, etc..
… but because they use the same engine, the results are merged in sec 3.1 consolidation.
… for the Director, its the sec 3.1 consolidation results that count.
… i.e. the engine for thorium reader (instantiated on different OSes) pass certain tests.

Wendy Reid: what i think might bridge this game is if we add to the consolidated results how many different platforms each implementation covers.
… e.g. Apple Books (2) etc..
… we could even list how many passed, and how many failed.

Ivan Herman: i think this might be more confusing for the reviewer of the results.

Tzviya Siegman: agree it might be confusing.

John Foliot: having gone down this path with other specs, as a reviewer, i’d be looking at each row of the consolidated table and i want to see two green squares.
… splitting it into platforms is interesting, but its not the type of information the review panel is looking for.

Avneesh Singh: +1 JF.

Ivan Herman: yes, my intention exactly.

Tzviya Siegman: so maybe we just point them to the detailed results (sec 3.2) instead.

Avneesh Singh: +1 Ivan, detailed result opens more can of worms.

Ivan Herman: but the detailed results may raise questions about why certain results should be counted twice.
… when some of them are the same engine.
… could this be misleading?.

Charles LaPierre: on the consolidated view, we could also include the number of passes for a given test.
… also, in the detailed view, it would be good to know which implementations are different engines.

Ivan Herman: that information is the responsibility of the testers.
… e.g. dhall has the right to decide that the two Apple implementations are separate engines, not up to us.
… if the two are merged in consolidated (3.1) that’s because he said they are the same.

David Hall: they are very close to the same engine.
… in some of our tests you can see that we pass on one, but not the other.
… if we really have 2 unique engines in test matrix, then they should present as unique results in consol matrix.

Wendy Reid: I thought that’s what variant was meant to do, because Kobo has 4 engines in market right now, 2 platforms share the same engine.
… i’m not testing iOS and web reader, as they use the same engine.
… I’m happy to clarify with testers whether variant means ‘different engine’ or ‘different platform/OS’?.
… we’re not distinguishing between different webkits here, right? that’s out of the control of the implementors.
… so to clarify, consolidated is okay as is, but the detailed results should be the results for different engines.
… and if the detailed results show two passes, then we should be okay.

John Foliot: if the purpose of the exercise is to have a detailed matrix to see where we have gaps, the detailed table is helpful. If the purpose is to show support for spec, then maybe we care about platforms (Windows, etc.).
… point is to show interoperability, and support in the marketplace.
… review panel will be interested in the consolidated report, while engineers interested in the detailed report.
… maybe make the detailed report an appendix of the consolidated report.

Ivan Herman: or make them two separate high level sections?.

John Foliot: for someone who just cares about seeing 2 separate implementation, it may help to just move the detailed report somewhere else (e.g. an appendix).
… we submit consolidated report to the review panel, and anyone else who needs more detail can go to the detailed one.
… i think that’s where the confusion really lies.

Charles LaPierre: i thought we only needed two independent implementations, not two for each platform.

Wendy Reid: so, what is considered independent?.

Ivan Herman: that’s up to the submitters.

Ivan Herman: q.

Brady Duga: it depends per feature what is independent, and what isn’t. If a feature depends on backend processing, that will work the same. But if the feature depends on something client side, the feature might work or not depending on the platform.
… and whether the reviewer will see it that way is an entirely different story.

Tzviya Siegman: we might be overthinking this.
… this needs to be useful for engineers and AC review.
… platform or engine, whichever way you choose to do it, we just need to present it in a way that is digestable to reviewers.

Avneesh Singh: instead of thinking of council point of view, think about where the objections might arise.

Laurent Le Meur: we will provide more Thorium results next week.
… in the detailed implementation results, Thorium has different columns for Mac and Windows. Doesn’t make sense because its the same code, same engine.
… let’s merge those.
… also, some tests have been modified, which makes it hard to finalize. Can we make sure there are stable periods in which to do testing without having to monitor whether tests have changed?.

Ivan Herman: having discussed with gautier about similar thing before, the list of tests can be ordered based on latest tests to show recent test changes.
… for the time being i have no intention of adding new tests, unless new features get added (no intention of doing that either).
… re the multiple Thorium columns, I propose gautier remove one of the two columns in a PR.
… two things. One is that i can restructure to separate consolidated report and detail report more..
… to make it more readable.

Wendy Reid: https://w3c.github.io/epub-tests/results#introduction.

Ivan Herman: I also tried to come up with introductory text in report to explain the two tables. I would welcome review of that text to make more clear.
… please review and provide feedback.
… Second, the agreement we have with Director is that if a feature doesn’t get 2 green passes, then we will not remove that feature because of our charter, but we will editorially label each of those as underimplemented.
… there is one area that i’m worried about, which is fallbacks.
… this permeates the spec. The spec is described in terms of fallbacks.
… there two possibilities for dealing with this.
… we can say, well, its too much to change. Or mgarrish and I could review the spec and try to simply.
… I am in favor of doing the surgery, but its up to the WG.

John Foliot: FWIW, +!to option 2 “surgery”.

Wendy Reid: I will review intro section. Its clear that even with multiple engines, it would look better if it were multiple implementors (e.g. Kobo and Google, etc.).
… I do think its still important to have the variants.
… even if Windows and Mac run on same engine, seeing two results will still be valuable to industry.
… but maybe we can de-prioritize.
… no comments on fallback. I’d wait to see more test results before we decide anything.

John Foliot: +1 as a former AC rep I think what Wendy is proposing is spot-on.

Wendy Reid: many more results need to be collected before we decide.

Charles LaPierre: re. Thorium and the 2 columns in detailed report. Gregorio and I did this over TPAC..
… one of us was on Mac, and one was on PC.

Laurent Le Meur: for me, its the exact same implementation, same code.
… what about screenreader, which are not in the tables right now?.
… gautier has reviewed most of the tests within the last month, let’s see what he uploads on Monday.
… maybe we can discuss with Daniel and other developers later about fallbacks.
… from what gautier will upload, if someone else wants to review the tests, it can be discussed, and maybe can make different tests on different devices.

John Foliot: i would suggest at this point not to introduce screenreaders to the text matrix.
… it would be a test of the screen reader instead of the spec anyway.
… too many different screen readers in the market.

Ivan Herman: we deliberately did not test things that are not defined by our spec.
… e.g. not testing CSS, HTML, etc..

Avneesh Singh: Laurent brought screen readers, but the context was different, not for testing.

Wendy Reid: this was something we started talking about at TPAC, but it was completely separate from this.
… so, we’re still waiting on additional test results, hopefully within the next week.
… we will delay the decision about fallbacks until after that, maybe meet about it next week.
… we need to initiate transition in January to meet charter deadline.
… we would need to request extension if we don’t think we can make it.

Ivan Herman: we are fine for now, but review is 4-6 weeks away.
… charter runs until February, I think.
… i am almost sure we will have to ask for extension, but we can decide in Jan.
… do we want to talk about meeting blackout days?.

Charles LaPierre: few weeks ago I gave presentation on epub 3.3. It was well attended. Thanks to wendyreid and Dave for slide deck.

Wendy Reid: we can meet next week, but week after is unclear. I know Dave is unavailable after 12th.

Ivan Herman: i propose last meeting is next week. Reconvene on 6th Jan.

Wendy Reid: agreed.
… for anyone who has not submitted test results, please do.
… the tests are as final as they will get. One PR still to be closed today. But that’s it..
… next week hopefully we will be in better place to talk about fallbacks and features that don’t have necessary passes.
… and TAG advisory board elections are open.

Ivan Herman: including wendyreid!.

Wendy Reid: okay, that’s it for today. See you all next week!.