Meeting minutes
<ivan> Date: 2021-09-24
dauwhe: we have a meeting devoted to testing. This will be one of most important things we do on path to REC.
Testing Update
<dlazin> dlazin+
<dlazin> https://
dlazin: we've made some progress. Updates in repo.
… above is link to epub tests repo
… a number of cool things
… readme on main page gives you step by step on how to write a test
… its quick to do, but you do need basic github knowledge
… there's templates to work from, fairly straightforward
… 2nd, Ivan has built report generation tool, uses github actions to generate HTML pages that lists all tests we have, and implementation status of RS against those tests
… everything in there so far are just examples
<ivan> example reports with fake implementation reports
dlazin: are the reports working in the real repo now?
ivan: yes!
dlazin: the first page (linked to by Ivan) are the implementation results
<dlazin> https://
dlazin: the one we're working on now is linked as a separate document (here)
… this lists all the tests that are currently marked up
… we have several dozen tests right now, but only a handful of them have the proper markup
… we need to add tags to them so they can be picked up in the reports
… the specs column links to where the normative statement is in the spec that is being tested, the ref links back to the original report that tells you implementation status across RS
… now wendyreid can demonstrate how to write a test
wendyreid: disclaimer: this portion of the meeting will be recorded for future reference
… feel free to mute yourself or step out of the call now
ivan: recording now
wendyreid shares screen
wendyreid: i'm just going to follow the step by step instructions
… #1 find untested or undertested normative statement
… we're going to test fxl, RS MUST product 1 page per spine item
… for people who are unfamiliar with github, I use a tool called github desktop
… i've created a branch called fxl-properties2
… i'm going to publish my branch
… try to give them descriptive names
… in my branch, i'm going to us VSCode to open the repo
… make a copy of the template
… rename the templated file with a descriptive name
… now modify the templated file to meet the needs of my test
… fill out package.opf (use the metadata guide if you need help)
… fill out dc:coverage, dc:creator, dc:date, dc:description (describe the test)
dlazin: this is a good reason to do a group of related tests at once
wendyreid: spec-anchor is the location in the spec that you are testing
… and it needs to be the TR one
dlazin: i recommend inspecting the spec document, and taking the anchor that is closest to the normative statement you are testing
… and if there isn't one, then you should create one and name it according to your new test
wendyreid: important to use TR version of the URL, not the github version of URL
ivan: dc:identifier should be the file name of your test
dlazin: we're still debating the precise naming of tests
… testing WG will sort it out and clarify documentation
wendyreid: so that is basically the metadata done
… for this test i don't need to make any other changes to the opf
… but what i do have to do is update the actual XHTML content
ivan: i went through this process this AM with a specific RS going through a test
… implementers will run one test after another
… and it must be extremely clear to the implementer when a test passes, and when it fails
… in this case it is straightforward, but there could be some tests where you see something on the screen, and its not clear whether the result is a pass or fail
… testers must be able to tell quickly what the result is
wendyreid: so in my XHTML I edit the content to make clear that if the tester sees the <p> without a break, then the test is a pass
… repeat for each XHTML page
… now its time to make it into an epub
… you can use eCanCrusher or however else you make an epub
… there's a cmdline way to do it in the documentation if you're on mac
… move the epub file into the test folder of the repo
… i'm going to quickly open it to check it. And yes, it works.
… there are instructions in the documentation for how to open it in various RS
… now i'm going to commit my changes to my branch
… give a descriptive commit msg
… now my changes are up in my branch, and I'm going to create my PR
… and dlazin or ivan will review my test
… when you do the PR, you don't have to worry about potentially breaking the test repo. Your work will be reviewed, and you will get feedback if necessary.
… only once all the potential issues with your test are fixed will it get merged
… once PR has been merged, you are going to fork the repo for the spec (e.g. EPUB 33 or EPUB RS 33)
… i go into version of spec in my repo, and add a data-tests property, with value equal to the name of the test I just created
dlazin: let's put the data-tests property on the same element as the id
… this has to be a separate PR because the spec is stored separately from the tests
ivan: if you go to the editors draft of the spec, then respec will show you the number of tests that are avail for a given normative statement, and give you a link back to the tests
… the tests and specs are going to be cross-referrenced in this way
wendyreid: once your change to the spec is done, then you will 1) commit it, and 2) do a PR for the spec, just like you did with your test
… the end
… any questions?
CharlesL: one, shouldn't epubcheck be part of the workflow to make sure your epub is valid?
dauwhe: not necessarily, it depends on the test
dlazin: i agree that it should be. Will add to documentation.
<Zakim> duga, you wanted to discuss after presentation
CharlesL: for this particular test, shouldn't there be one XHTML page with a page break? So we can test what happens there too?
duga: one of the things we have to do is ensure that test would fail if we removed the thing being tested
… e.g. if rendition-layout weren't set to pre-paginated, you would still get 4 pages
… so in this case putting a pagebreak in might be the thing to do, but if the RS doesn't support pagebreak AND doesn't support fxl, then we still won't get an unambiguous result
duga: i wasn't involved in the creation of the process, but what is there looks excellent. thank you all.
… first, how hard should we be making these tests?
… e.g. i could make a test that validates this normative statement, but would either pass or fail depending on the test
… e.g. on Google RS, a pure fxl would pass, but a mixed pagination would fail
dlazin: we have a lot of testing to do, and we should prioritize
… there are a lot of places where there could be 6 different unit tests for a single assertion
… but lets try to get the basic stuff in first, and then do improvement passes later
… e.g. ivan was testing dir property earlier this week. We have 6 different tests for that.
… let's try to get the MUSTs and MUST NOTs, then move on to the SHOULDs and SHOULD NOTs etc.
… but if you have the appetite to do rigorous testing with multiple cases at once, then go ahead
duga: we should verify with RS developers how they want their RS tested
… e.g. for Google Play, the doc points people to the user upload path rather than the publisher upload path
… a test might fail on one path, and pass on the other path
… and using the right path will be difficult for someone who isn't me or a publisher
dlazin: i think we want the RS devs to do their own testing
… and if we can't get RS implementors to run the test suites, then we'll do it for them
… but we prefer to them to do it themselves
… in the case of Google, you might actually want to test user path and publisher path for each of web, ios, and android...
duga: last, how are we claiming a test?
… if i want to start working on making a test for one statement, what do I do?
dlazin: I think the doc says to claim it in the spreadsheet
duga: maybe the better way to do this is to create a github issue, and put a link to the issue in the spreadsheet?
<ivan> +1 to brady
dlazin: i like that for tracking purposes, but it adds more overhead
duga: the alternative is automatically generating a github issue for each record in the spreadsheet...
dlazin: also, be realistic about which tests you will actually be able to test, what you have time to do quickly
… also reason why we track in the spreadsheet rather than in spec is because spec sections can still move around
… if we were to turn everything in the spreadsheet into issues, we might get false confidence about how much testing is left to do
duga: okay, let's do JIT then
duga: note about having RS do their own tests, i have a concern that test results should be repeatable by 3rd parties
… but point about multiple paths is a valid one
… re. how hard tests should be, a lot depends on where we think there might be problems in the spec
… if we think things in spec are weird, we should test more
… to try to understand better, find interop problems
Hadrien: about RS testing and publisher path vs user path, in some cases, there is no user path
… problematic for this type of testing
… and you have RS where those two different paths will behave inconsistently
… and there are cases where testing happens in some separate app that is not really an RS
… given all of that, i think we should target RS where there is a user path, and where there will be some consistency
… we can identify a number of RS that can provide the type of reproducibility that dauwhe was referring to
… this is important because not all RS is equal in that regard
duga: fair points, but i know that e.g. Google's user's path is far more lenient
… because we see a lot more bad content from user path vs publisher path
… so it kind of plays to the weakest link
… that's a reason why we shouldn't limit to user path
… from Google Play books perspective, I can publish all these books and make them available as a way for you to check my evaluation of results
ivan: we shouldn't forget that the goal of this exercise is to test the spec
… we're not providing an environment for measuring one implementation against another
… my approach is to trust the implementors. They aren't trying to fool us. We're trying to get the spec properly specified.
wendyreid: new test writers shouldn't be afraid to try. There will be stumbling blocks. But there are people to provide support.
ivan: i'm stopping the recording now
… with Zoom it takes some time to generate the video, and then we will clean it up a little bit
… i will clean the minutes of the meeting as usual, and provide the link to the video as a follow-up
<Zakim> avneeshsingh, you wanted to comment on finalized time with TPAC APA meetings
avneeshsingh: we've finalized time for TPAC APA meeting
<avneeshsingh> https://
avneeshsingh: here's link to schedule
… we will discuss a11y horizontal review for epub 33
… second meeting about normative references to schema
… the two dates are in the link
<wendyreid> Thursday 28 October at 10:00 Boston / 1400Z (Confirmed)
ivan: should I create calendar entries for the group?
avneeshsingh: we can discuss with Janina, as she maybe creating calendar entries, which we can just copy for our group
AOB?
duga: thank you again to dlazin, wendyreid, dauwhe, ivan for all of this
wendyreid: most of that thanks to dlazin!
wendyreid: everyone have a wonderful weekend, bye!