14:08:09 <RRSAgent> RRSAgent has joined #epub
14:08:09 <RRSAgent> logging to https://www.w3.org/2022/12/02-epub-irc
14:08:11 <Zakim> RRSAgent, make logs Public
14:08:12 <Zakim> please title this meeting ("meeting: ..."), ivan
14:08:46 <ivan> ivan has changed the topic to: Meeting Agenda 2022-12-02: https://lists.w3.org/Archives/Public/public-epub-wg/2022Nov/0021.html
14:08:47 <ivan> Chair: dauwhe
14:08:47 <ivan> Date: 2022-12-02
14:08:47 <ivan> Agenda: https://lists.w3.org/Archives/Public/public-epub-wg/2022Nov/0021.html
14:08:47 <ivan> Meeting: EPUB 3 Working Group Telco
14:08:47 <ivan> Regrets+ billk
14:18:30 <tzviya> tzviya has joined #epub
14:59:09 <MattChan> MattChan has joined #epub
14:59:16 <ivan> present+
14:59:35 <MasakazuKitahara> MasakazuKitahara has joined #epub
14:59:45 <MasakazuKitahara> present+
15:00:34 <ivan> present+ tzviya
15:00:35 <wendyreid> present+
15:00:38 <MattChan> present+
15:00:46 <ivan> present+ MattChan
15:00:51 <ivan> present+ jf
15:01:00 <ivan> present+ MattChan
15:01:08 <ivan> present+ MasakazuKitahara
15:01:46 <dhall> dhall has joined #epub
15:02:13 <JF> JF has joined #epub
15:02:19 <JF> Present+
15:02:23 <JF> agenda?
15:02:28 <tzviya> present+
15:02:45 <JF> zakim, who is here?
15:02:45 <Zakim> Present: ivan, MasakazuKitahara, tzviya, wendyreid, MattChan, jf
15:02:46 <CharlesL> CharlesL has joined #epub
15:02:48 <Zakim> On IRC I see JF, dhall, MasakazuKitahara, MattChan, tzviya, RRSAgent, Zakim, ivan, wendyreid, github-bot, npd, jcraig
15:02:54 <dhall> Present+
15:03:00 <CharlesL> present+
15:03:04 <MattChan> scribe+
15:03:09 <ivan> present+ dhall
15:03:14 <ivan> present+ charles
15:03:52 <MattChan> TOPIC: Testing Updates
15:04:04 <wendyreid> https://w3c.github.io/epub-tests/results
15:04:04 <MattChan> wendyreid: we're going to be talking about test results
15:04:23 <MattChan> ... link to results
15:04:29 <MattChan> ... we have results from 17 sources
15:04:34 <MattChan> ... not a complete list
15:04:47 <MattChan> ... still more results expected to come in
15:04:52 <ivan> q+
15:05:10 <MattChan> ... we've results from Apple, Google, Thorium, VitalSource, Wysebee
15:05:39 <wendyreid> ack ivan
15:05:40 <MattChan> ... the consolidated results show consolidation of results for all platforms (any single pass counts as a consolidated pass)
15:05:49 <MattChan> ivan: how up to date is this?
15:06:03 <MattChan> ... Wysebee seems unchanged for past few months...
15:06:20 <MattChan> ... we know that Kobo is not final, not sure if Thorium is final yet
15:06:24 <dhall> q+
15:06:28 <wendyreid> ack dhall
15:06:37 <CharlesL> q+
15:06:43 <MattChan> dhall: i believe Apple is final, no changes made since we did testing
15:06:46 <wendyreid> ack CharlesL
15:07:16 <MattChan> ivan: for those who are not familiar, if you click link to one of the test, you get description of what the test entails
15:07:45 <MattChan> CharlesL: i got stalled on testing of Thorium on mac, I still plan on doing some more in the next couple weeks before I go on vacation
15:07:54 <MattChan> ivan: good, thank you!
15:08:11 <MattChan> ... what do we do with those tests where we have 1 or fewer pass flags?
15:09:41 <ivan> q+
15:09:43 <tzviya> q+
15:09:47 <MattChan> wendyreid: section 3.1 are the consolidated results... so if 2 variants of the same test pass, does that count as a pass overall?
15:09:49 <wendyreid> ack ivan
15:10:22 <MattChan> ... e.g. two kobo implementations
15:11:01 <MattChan> ivan: are the epub engines underlying the implementations the same? if not, then they should be submitted as separate implementations
15:11:01 <wendyreid> ack tzviya
15:11:05 <MattChan> ... same for apple
15:11:32 <MattChan> tzviya: having sat on formal objection councils, a lot comes down to what it means to have more than one implementation
15:11:58 <MattChan> ... agree that kobo on ios and kobo on android count as separate, but this table doesn't really show that well
15:12:12 <ivan> q+
15:12:13 <wendyreid> q+
15:12:19 <wendyreid> ack ivan
15:12:24 <MattChan> ... might want to show tests based on platform (e.g. android, iOS, etc.)
15:12:59 <MattChan> ivan: the detailed implementation result (sec 3.2) shows, for example, Thorium tests on Windows, MacOS, etc.
15:13:16 <MattChan> ... but because they use the same engine, the results are merged in sec 3.1 consolidation
15:13:28 <MattChan> ... for the Director, its the sec 3.1 consolidation results that count
15:13:55 <MattChan> ... i.e. the engine for thorium reader (instantiated on different OSes) pass certain tests
15:13:58 <wendyreid> ack wendyreid
15:14:08 <AvneeshSingh> AvneeshSingh has joined #epub
15:14:22 <AvneeshSingh> present+
15:14:53 <MattChan> wendyreid: what i think might bridge this game is if we add to the consolidated results how many different platforms each implementation covers
15:15:00 <laurent_> laurent_ has joined #epub
15:15:08 <MattChan> ... e.g. Apple Books (2) etc.
15:15:21 <MattChan> ... we could even list how many passed, and how many failed
15:15:36 <MattChan> ivan: i think this might be more confusing for the reviewer of the results
15:16:16 <JF> Q+
15:16:31 <wendyreid> ack JF
15:16:32 <MattChan> tzviya: agree it might be confusing
15:17:00 <MattChan> JF: having gone down this path with other specs, as a reviewer, i'd be looking at each row of the consolidated table and i want to see two green squares
15:17:07 <CharlesL> q+
15:17:24 <MattChan> ... splitting it into platforms is interesting, but its not the type of information the review panel is looking for
15:17:24 <AvneeshSingh> +1 JF
15:17:30 <MattChan> ivan: yes, my intention exactly
15:17:48 <MattChan> tzviya: so maybe we just point them to the detailed results (sec 3.2) instead
15:18:04 <AvneeshSingh> +1 Ivan, detailed result opens more can of worms
15:18:08 <MattChan> ivan: but the detailed results may raise questions about why certain results should be counted twice
15:18:18 <AvneeshSingh> q+
15:18:20 <MattChan> ... when some of them are the same engine
15:18:27 <dhall> q+
15:18:33 <wendyreid> ack CharlesL
15:18:34 <MattChan> ... could this be misleading?
15:18:37 <wendyreid> q+
15:19:04 <JF> Q+
15:19:05 <MattChan> CharlesL: on the consolidated view, we could also include the number of passes for a given test
15:19:43 <AvneeshSingh> q-
15:19:47 <MattChan> ... also, in the detailed view, it would be good to know which implementations are different engines
15:20:06 <MattChan> ivan: that information is the responsibility of the testers
15:20:41 <MattChan> ... e.g. dhall has the right to decide that the two Apple implementations are separate engines, not up to us
15:20:53 <wendyreid> ack dhall
15:20:56 <MattChan> ... if the two are merged in consolidated (3.1) that's because he said they are the same
15:21:03 <MattChan> dhall: they are very close to the same engine
15:21:17 <MattChan> ... in some of our tests you can see that we pass on one, but not the other
15:21:34 <MattChan> ... if we really have 2 unique engines in test matrix, then they should present as unique results in consol matrix
15:22:04 <MattChan> wendyreid: I thought that's what variant was meant to do, because Kobo has 4 engines in market right now, 2 platforms share the same engine
15:22:19 <MattChan> ... i'm not testing iOS and web reader, as they use the same engine
15:23:01 <MattChan> ... I'm happy to clarify with testers whether variant means 'different engine' or 'different platform/OS'?
15:23:25 <MattChan> ... we're not distinguishing between different webkits here, right? that's out of the control of the implementors
15:23:56 <MattChan> ... so to clarify, consolidated is okay as is, but the detailed results should be the results for different engines
15:24:06 <MattChan> ... and if the detailed results show two passes, then we should be okay
15:24:29 <wendyreid> ack JF
15:24:34 <wendyreid> ack wen
15:24:54 <ivan> q+
15:25:16 <MattChan> JF: if the purpose of the exercise is to have a detailed matrix to see where we have gaps, the detailed table is helpful. If the purpose is to show support for spec, then maybe we care about platforms (Windows, etc.)
15:25:30 <MattChan> ... point is to show interoperability, and support in the marketplace
15:25:32 <CharlesL> q+
15:25:39 <wendyreid> ack ivan
15:25:56 <MattChan> ... review panel will be interested in the consol report, while engineers interested in the detailed report
15:26:07 <MattChan> ... maybe make the detailed report an appendix of the consol report
15:26:27 <MattChan> ivan: or make them two separate high level sections?
15:26:56 <MattChan> JF: for someone who just cares about seeing 2 separate implementation, it may help to just move the detailed report somewhere else (e.g. an appendix)
15:27:20 <MattChan> ... we submit consol report to the review panel, and anyone else who needs more detail can go to the detailed one
15:27:30 <MattChan> ... i think that's where the confusion really lies
15:27:38 <wendyreid> ack CharlesL
15:27:48 <duga> duga has joined #epub
15:27:54 <MattChan> CharlesL: i thought we only needed two independent implementations, not two for each platform
15:27:58 <duga> present+
15:28:18 <MattChan> wendyreid: so, what is considered independent?
15:28:42 <duga> q+
15:28:44 <tzviya> q+
15:28:54 <tzviya> q+ laurent
15:28:55 <MattChan> ivan: that's up to the submitters
15:29:00 <wendyreid> ack duga
15:30:06 <ivan> q
15:30:13 <ivan> q+
15:30:26 <MattChan> duga: it depends per feature what is independent, and what isn't. If a feature depends on backend processing, that will work the same. But if the feature depends on something client side, the feature might work or not depending on the platform
15:30:38 <wendyreid> ack tzviya
15:30:46 <MattChan> ... and whether the reviewer will see it that way is an entirely different story
15:30:56 <MattChan> tzviya: we might be overthinking this
15:31:05 <MattChan> ... this needs to be useful for engineers and AC review
15:32:14 <MattChan> ... platform or engine, whichever way you choose to do it, we just need to present it in a way that is digestable to reviewers
15:32:47 <ivan> present+ avneesh
15:32:55 <wendyreid> ack laurent
15:32:56 <ivan> present+ laurent_
15:32:58 <wendyreid> q+
15:33:04 <MattChan> AvneeshSingh: instead of thinking of council point of view, think about where the objections might arise
15:33:25 <MattChan> laurent_: we will provide more Thorium results next week
15:33:27 <CharlesL> q+
15:34:02 <MattChan> ... in the detailed implementation results, Thorium has different columns for Mac and Windows. Doesn't make sense because its the same code, same engine
15:34:12 <MattChan> ... let's merger those
15:34:18 <MattChan> s/merger/merge
15:34:53 <wendyreid> ack ivan
15:34:59 <MattChan> ... also, some tests have been modified, which makes it hard to finalize. Can we make sure there are stable periods in which to do testing without having to monitor whether tests have changed?
15:35:31 <MattChan> ivan: having discussed with gautier about similar thing before, the list of tests can be ordered based on latest tests to show recent test changes
15:35:59 <MattChan> ... for the time being i have no intention of adding new tests, unless new features get added (no intention of doing that either)
15:36:24 <MattChan> ... re the multiple Thorium columns, I propose gautier remove one of the two columns in a PR
15:37:05 <MattChan> ... two things. One is that i can restructure to separate consol report and detail report more.
15:37:17 <MattChan> ... to make it more readable
15:37:32 <wendyreid> https://w3c.github.io/epub-tests/results#introduction
15:37:40 <MattChan> ... I also tried to come up with introductory text in report to explain the two tables. I would welcome review of that text to make more clear
15:37:49 <MattChan> ... please review and provide feedback
15:38:45 <MattChan> ... Second, the agreement we have with Director is that if a feature doesn't get 2 green passes, then we will not remove that feature because of our charter, but we will editorially label each of those as underimplemented
15:39:17 <MattChan> ... there is one area that i'm worried about, which is fallbacks
15:39:42 <MattChan> ... this permeates the spec. The spec is described in terms of fallbacks
15:39:59 <MattChan> ... there two possibilities for dealing with this
15:40:30 <MattChan> ... we can say, well, its too much to change. Or mgarrish and I could review the spec and try to simply
15:40:38 <wendyreid> ack wendyreid
15:40:43 <MattChan> ... I am in favor of doing the surgery, but its up to the WG
15:40:52 <JF> FWIW, +!to option 2 "surgery"
15:41:58 <MattChan> wendyreid: I will review intro section. Its clear that even with multiple engines, it would look better if it were multiple implementors (e.g. Kobo and Google, etc.)
15:42:25 <MattChan> ... I do think its still important to have the variants
15:42:54 <MattChan> ... even if Windows and Mac run on same engine, seeing two results will still be valuable to industry
15:43:07 <MattChan> ... but maybe we can de-prioritize
15:43:56 <MattChan> ... no comments on fallback. I'd wait to see more test results before we decide anything
15:43:58 <JF> +1  as a former AC rep I think what Wendy is proposing is spot-on
15:44:02 <wendyreid> ack CharlesL
15:44:04 <MattChan> ... many more results need to be collected before we decide
15:44:37 <MattChan> CharlesL: re. Thorium and the 2 columns in detailed report. Gregorio and I did this over TPAC.
15:44:57 <MattChan> ... one of us was on Mac, and one was on PC
15:45:13 <MattChan> laurent_: for me, its the exact same implementation, same code
15:45:19 <JF> Q+
15:45:36 <MattChan> ... what about screenreader, which are not in the tables right now?
15:45:59 <MattChan> ... gautier has reviewed most of the tests within the last month, let's see what he uploads on Monday
15:46:17 <MattChan> ... maybe we can discuss with danielle and other developers later about fallbacks
15:46:41 <wendyreid> s/danielle/Daniel/
15:46:54 <MattChan> ... from what gautier will upload, if someone else wants to review the tests, it can be discussed, and maybe can make different tests on different devices
15:47:11 <wendyreid> ack JF
15:47:35 <MattChan> JF: i would suggest at this point not to introduce screenreaders to the text matrix
15:47:42 <ivan> q+
15:47:47 <MattChan> ... it would be a test of the screen reader instead of the spec anyway
15:48:24 <wendyreid> ack ivan
15:48:26 <wendyreid> q+
15:48:34 <MattChan> ... too many different screen readers in the market
15:48:53 <MattChan> ivan: we deliberately did not test things that are not defined by our spec
15:49:00 <MattChan> ... e.g. not testing CSS, HTML, etc.
15:49:47 <AvneeshSingh> Laurent brought screen readers, but the context was different, not for testing
15:50:06 <wendyreid> ack wendyreid
15:50:19 <MattChan> wendyreid: this was something we started talking about at TPAC, but it was completely separate from this
15:50:19 <ivan> q+
15:50:35 <MattChan> ... so, we're still waiting on additional test results, hopefully within the next week
15:50:48 <MattChan> ... we will delay the decision about fallbacks until after that, maybe meet about it next week
15:51:05 <MattChan> ... we need to initiate transition in January to meet charter deadline
15:51:11 <wendyreid> ack ivan
15:51:18 <MattChan> ... we would need to request extension if we don't think we can make it
15:51:37 <CharlesL> q+
15:51:38 <MattChan> ivan: we are fine for now, but review is 4-6 weeks away
15:51:56 <MattChan> ... charter runs until February, I think
15:52:18 <MattChan> ... i am almost sure we will have to ask for extension, but we can decide in Jan
15:52:24 <wendyreid> ack CharlesL
15:52:33 <MattChan> ... do we want to talk about meeting blackout days?
15:53:13 <MattChan> CharlesL: few weeks ago I gave presentation on epub 3.3. It was well attended. Thanks to wendyreid and Dave for slide deck
15:53:38 <MattChan> wendyreid: we can meet next week, but week after is unclear. I know Dave is unavail after 12th
15:53:52 <MattChan> ivan: i propose last meeting is next week. Reconvene on 6th Jan
15:54:04 <MattChan> wendyreid: agreed
15:54:17 <MattChan> ... for anyone who has not submitted test results, please do
15:54:47 <MattChan> ... the tests are as final as they will get. One PR still to be closed today. But that's it.
15:55:19 <MattChan> ... next week hopefully we will be in better place to talk about fallbacks and features that don't have necessary passes
15:55:35 <MattChan> ... and TAG advisory board elections are open
15:55:42 <MattChan> ivan: including wendyreid!
15:56:39 <ivan> rrsagent, draft minutes
15:56:39 <RRSAgent> I have made the request to generate https://www.w3.org/2022/12/02-epub-minutes.html ivan
15:57:25 <MattChan> wendyreid: okay, that's it for today. See you all next week!
15:57:39 <CharlesL> CharlesL has left #epub
15:57:41 <ivan> rrsagent, draft minutes
15:57:41 <RRSAgent> I have made the request to generate https://www.w3.org/2022/12/02-epub-minutes.html ivan
15:58:38 <ivan> zakim, end meeting
15:58:38 <Zakim> As of this point the attendees have been ivan, MasakazuKitahara, tzviya, wendyreid, MattChan, jf, dhall, CharlesL, AvneeshSingh, duga, laurent_
15:58:40 <Zakim> RRSAgent, please draft minutes
15:58:40 <RRSAgent> I have made the request to generate https://www.w3.org/2022/12/02-epub-minutes.html Zakim
15:58:43 <Zakim> I am happy to have been of service, ivan; please remember to excuse RRSAgent.  Goodbye
15:58:47 <Zakim> Zakim has left #epub
15:58:48 <ivan> rrsagent, bye
15:58:48 <RRSAgent> I see no action items