Publishing Working Group Telco — Minutes

Date: 2019-12-09

See also the Agenda and the IRC Log

Attendees

Present: Wendy Reid, Dave Cramer, Marisa DeMeglio, Teenya Franklin, Nick Ruffilo, Romain Deltour, Ivan Herman, Benjamin Young, George Kerscher, Laurent Le Meur, Charles LaPierre, Avneesh Singh, Franco Alvarado, Matt Garrish, Bill Kasdorf, Brady Duga, Garth Conboy, Ben Schroeter, Ric Wright, Tzviya Siegman, Nellie McKesson

Regrets: Luc Audrain, Mateus Teixeira

Guests:

Chair: Wendy Reid

Scribe(s): Nick Ruffilo, Wendy Reid

Content:

• 1. Testing
• 2. Resolutions

Wendy Reid: https://www.w3.org/publishing/groups/publ-wg/Meetings/Minutes/2019/2019-12-02-pwg

Wendy Reid: First task - minutes, approval?
… approved.

Ivan Herman: +1

Resolution #1: last week’s minutes approved

1. Testing

Wendy Reid: https://github.com/w3c/publ-tests

Wendy Reid: The topic of the day is about testing. (link posted above). As discussed last week, we’ve created a separate repository for the tests. Because we’ve named it broadly, it can be the testing repo for all publishing activities.
… there are separate folders for audiobooks and manifests, reports… Everything will be located here. Matt and Ivan did a ton of work over the last couple days to ensure this is ready for view. Ivan - have I missed anything?

Ivan Herman: I think you got everything. If you go one step deeper, such as the publication manifest repository - it already has 2 subdirectories for various types of testing. Manifest processing and TOC processing.
… it’s something Matt has started. It is about testing the algorithm to extract TOC from an HTML file. There may be other testing for audiobooks, there is only Manifest processing for now.

Wendy Reid: I made a few but put them in the wrong area. I have a few things I need to clean up and potentially make them submittable. In the same format the other tests are in.

Ivan Herman: You did something that ….

Wendy Reid: They are just in text format right now.

Ivan Herman: https://github.com/w3c/publ-tests/tree/master/audiobooks/manifest_processing/tests

Ivan Herman: If I go to (pasted) directory, that’s meant only for tests related to the manifest. If you want user agent behavior test, it should be a separate folder

Wendy Reid: That’s correct, I just haven’t had a chance to do that yet

Ivan Herman: the manifest is relatively rich. It’s fairly easy to submit a new test. Finally there is a test report directory which we (pasted) have here. The intention is that the various types of testing should have their own subdirectory

Ivan Herman: https://github.com/w3c/publ-tests/tree/master/test_reports

Ivan Herman: https://github.com/w3c/publ-tests/blob/master/test_reports/manifest_processing/index.html

Ivan Herman: https://w3c.github.io/publ-tests/test_reports/manifest_processing/

Ivan Herman: Ideally every test section, whether TOC or the user agent behavior or whatever else, should have a report file like the one provided

Wendy Reid: Matt - anything to add?

Matt Garrish: Nope. Like Ivan said, I’ve started with the TOC tests. I’m having some issues making the tests atomic - as there’s alot that has to go into loading a TOC. In general, it’s in progress. Just still work to do

Charles LaPierre: Just a question - in the results directory, there is an index.html -> is that where the visual representation of the results will end up? Should we have a readme that points to the github IO?

Ivan Herman: Which file - can you paste a link?

Charles LaPierre: https://github.com/w3c/publ-tests/tree/master/test_reports/manifest_processing

Charles LaPierre: In that directory, there is an index.html

Ivan Herman: that’s the result of the test on Manifest processing

Charles LaPierre: can we have a readme that points to the github io about how this workS?

Ivan Herman: Ultimately, the whole repository can be looked at from the IO.

Charles LaPierre: and if you have a main dashboard that pulls in the main index files - i’m not sure if we want to do that…

Ivan Herman: we will have to do that at some point. We have to provide a report at the end. For the time being, that’s the only report.

Wendy Reid: I think we’re going to have to bump up the readme file on the main page to explain where to look/contribute. I think we’ve burried that in the subfolder for now.

Ivan Herman: there is a readme, it just needs to be improved

Wendy Reid: agreed

Charles LaPierre: https://w3c.github.io/publ-tests/test_reports/manifest_processing/index.html

Charles LaPierre: I found the github IO for the index (^) if you wanted to see what those tests look like, you can see that a whole bunch of tests are passing with a few fails.

Wendy Reid: In terms of test coverage, aside from the TOC test, I believe we have pretty much the manifest processing tests are almost complete. TOC are in progress. The user agent behavior tests are written but need to be made more usable.
… I don’t know if anyone has attempted to run the tests in any way. I know Ivan, Matt and I have given a chance. But if others could try and provide some comments on that…
… Lars is very busy with Colibrio right now, so we wont’ get an answer from him for a little bit

Ivan Herman: My question is - what kind of other tests do we need to put there for this round of CR? I have heard about the TOC, the user behavior testing… Are there any categories for this sort that we need additionally?

George Kerscher: I don’t know if this is relevant, but if audio is encoded in variable bit rate, is there a place to note a file is problematic. Do we want to include tests for that?

Wendy Reid: I think it’s out of scope, and it sounds like the user agent would want to run against their implementation. You would want to test that bitrates seek in a specific format.

George Kerscher: for variable bitrate, it could change…

Wendy Reid: I think that falls into the user agents - hopefully their standard testing would check that

Ivan Herman: Do we want to have tests about LPF and the way it handles files. Maybe we want, maybe we don’t want. There are some statements in the manifest processing about the duration values being consistent - such as the duration overall should be the sum of the parts. But other specifications around media specific items. Anything we require in the spec?
… there might be some features - like what George referred to, but more generally. What we test is the specifics of our specifications - so what items require testing of certain features.
… we do refer to sync media - but again, a question.

Wendy Reid: the LPF - there’s requirements within LPF, such as not zipping audio files. Laurent - would you consider writing tests around LPF or do you think it needs testing?

Laurent Le Meur: I think it’s a bit premature. We should first finalize the testing the publication manifest, then we can think about the DF. There would only be verification of names of the files and the deflates… Would be very quick
… we do have time for that

Wendy Reid: those all sound like valid tests.
… I’m already getting LPF questions / packaging. It’s worth having a handful of tests to make sure that things are named properly.

Tzviya Siegman: You pretty much said what I was going to say. Especially since Laurent said it’s straightforward, I think it’s worth doing even though it’s just a note.

Wendy Reid: would you consider writing some tests Laurent?

Laurent Le Meur: Sure.

Wendy Reid: just a few - such as naming and deflate mechanism.
… the other question is about sync media. I haven’t gotten many questions about it, and it might be too early to test and it might be complicated. I think it’s a bit early, but it could include something in the User Agent behavior.
… such as “if present, display, if not, don’t”

Marisa DeMeglio: I think it is premature, but that basic case is reasonable.

Wendy Reid: because it’s a peripheral feature, we don’t have to do it just yet, but worth reviewing

Ivan Herman: there was another work started by Matt and something else which is a very different kind of testing. Whether the terms we define are really useful and used by the community. That’s also important. Reminds me of the DPUB ARIA testing we had to do.

Matt Garrish: https://github.com/w3c/pub-manifest/wiki/Publication-Manifest-to-EPUB-Mapping

Ivan Herman: I think there was work on mapping the epub terms to our terms, indirectly proving that those terms are used by the community. This is important to be properly documented and would be in the testing directory

Matt Garrish: I just pasted in a link to the document, which is pretty complete at this point. I’ve gone through all of the packages spec for epub and cross-referenced to pub manifest. If we want to move it over or find a new home in the testing repository, that’s easy to do.

Ivan Herman: I think it would be better to have everything in one place.

Charles LaPierre: I was just looking that over and specifically under accessibility - I’m wondering about the 2 we don’t use often, the accessibility API and accessibility control.

Matt Garrish: I’m not sure if we still recommend them. They are the general rule - that you can express both because they are in schema.org…

Charles LaPierre: we only tell publishers to put that in if they’ve added in special controls such as special ARIA - so we could see it happen, but it’s usually not there.

Matt Garrish: It can go in there if it makes things more complete - it’s not a recommendation, just a mapping.

Ivan Herman: I don’t believe they are listed in the manifest.

Matt Garrish: we didn’t list them in the manifest. You could list them because they are schema.org, but they aren’t specifically named

Ivan Herman: to be selfish, our goal is to provide the testing for candidate recommendations. If a feature like this is not mentioned in the CR, we are in no obligation to have information about it. We can, but what we have is 5 accessibility terms that are in the document and those are important.

Matt Garrish: I agree, just pointing out.

Wendy Reid: Back to Ivan’s point, are there any tests we’re missing? Or categories?
… The goal is to have the tests completed by the end of the year. The only thing we need to finish are the User Agent Behavior, the LPF, and the rest of the manifests. We have one more meeting next week, and it’s the last for the year. I don’t want to put the same pressure on everyone else
… So we’ll round out the testing by next week.

Ric Wright: One question: I am looking at the tests that are posted, they look very impressive, but it reminds me of the epub test suite. You test each aspect or feature. What happens sometimes is that if you have several features that are all brought together, thats when you see problems.
… is the goal to just verify that each individual feature is there, or is this intended to help the reading systems to regression test?

Ivan Herman: The goal - since I’m not a reading system implementor, I didn’t have the complex situations, but implementors might run into these problems and they would be kind enough to submit to the test suite. The test suite came out of Matt and I testing the algorithm.
… because we didn’t have to do such global ones, and the same way we produce those tests, others can create them and submit back.

Wendy Reid: That’s a great question, I’m having that exact issue with epub this week. I feel like we have to think about this. I have to think about how things integrate - so that might be the challenge for the week.
… I’ll see if there is a way to test relational issues.

George Kerscher: Are we saying that we want to stress test a reading system? Complex TOC and monster audio files?

Wendy Reid: this comes down to more like - I’ll use my epub example - right now we’re looking at rendition properties. Some conflict with user properties. So - have we created a situation where certian properties conflict or cause issue

Matt Garrish: Just going to say - hopefully - we don’t have the same degree of problems when it comes to manifest and what epub has - but I do agree with Rick that it’s hard to think up the problems that might cause issues. With TOC there’s likely odd issues that could pop up, but it often takes people creating content to find the issues.
… as Ivan says, it’s something that hopefully people can provide info back.

Avneesh Singh: Regarding George’s notes. The scope of this testing can impact improvement in the specification. What George noted about stress testing - the scope is anything that leads to improving the specification.

Ivan Herman: +1 to Avneesh

Ric Wright: I agree with Avneesh. These sets of tests, these atomic tests, are for testing the specification. The complicated ones would more be on the user agents. Should they be somewhere else to ensure people don’t get confused?

Nick Ruffilo: I agree with everything said, but I think there’s a line where some are spec related and some are RS related
… if we have 2 conflicting properties, or cause things to blow up, but are mutually exclusive, we will see that in testing
… it’s all going to come in the testing process
… we’ll see as it comes

Tzviya Siegman: I’m wondering if someone wants to create some sample content for us to test?
… Someone may have already done that???

Wendy Reid: It’s just like in epub, you can make a spec perfect epub, but publisher X creates on and breaks everything because we didn’t think of something specific. If people are interested and they have content they want to test - then run it through the tests and see what happens.
… if someone wants to help create a bank of test content for implementors to use - that would be helpful

Matt Garrish: One thing that worries me is - we’ve had this over and over in epub. If we put metadata in, but don’t put in priority, when it comes to things like author - if author 3 is the one that displays, is that something we should test for? Should we note some of those issues?

Nellie McKesson: I could potentially create an output to create files for testing. My only concern is that I have alot going on and I’m not sure if I can do it now or in 2 weeks.

Ivan Herman: +1 to Nellie

Wendy Reid: I would say that would be super amazing, and it doesn’t need to be done in the next 2 weeks.
… The tests I’d like done in 2 weeks, but sample content and other resources, the only deadline there is end of March.

Wendy Reid: +1 to Nellie!

Nellie McKesson: OK, in a couple of weeks I should be able to have something for people to use.

Wendy Reid: On that note… We need User Agent Behavior, Manifest, and LPF. Last meeting is next week - everyone have a great week. Talk next week!

2. Resolutions

• Resolution #1: last week’s minutes approved