The new publication system: how is that going to help me? -- 04 Jun 2015

<koalie> tripu, my regrets for your prez; I wasn't able to reschedule my meeting in 5 minutes :/

<scribe> scribe: denis

Presentation: The new publication workflow

tripu: I want to focus on how the system's going to help team contacts and editors

How publishing happens today [slide 2]

tripu: on page 2, we have an overview of the general process
... your groups might be doing things a different way but it's more or less how it's happening
... the important thing to note here is the back and forth between the different roles
... the process of fixing issues can happen several times and there are humans behind that so it might be slow sometimes

<inserted> [Slide 3]

tripu: some of the groups and staff contacts are uploading documents to temporary locations
... the whole purpose is to check with pubrules
... it would be good to get rid of these temporary locations

Problems [slide 4]

<inserted> [Slide 4]

tripu: on slide 4, we have listed a few problems
... some people are kept busy for tasks that can be automated
... it's also difficult to publish very often
... validation of the specs can be cumbersome as well. Might need other people involved to help understanding where are the issues
... and the publication requires people to be awake

Solution (Automation) [slide 5]

<inserted> [Slide 5]

tripu: so let the robots steal our job

<inserted> [Slide 6]

<inserted> [Slide 7]

<MikeSmith> tripu, quit saying everything twice please

tripu: the slides 6 and 7 describe the ideal situation
... people could publish after each commit or trigger the publication by clicking on a button
... slide 6 shows how it could be integrated with other system like Github
... the system should then infer some data, run validators, etc
... and send feedback to the user

<r007a> tripu, which slide are we on?

<inserted> [Slide 8]

<r007a> occasional choppiness, tripu, but not often

tripu: that slide lists some advantages of the system
... it will be easier to publish more frequently, publish on your own
... it will save a lot of messages between the different people

Current Status [slide 9]

<inserted> [Slide 9]

tripu: Slide 9 lists features we have already
... email notifications being sent to a public mailing list
... when there's an error, the message contains the error
... so you know why your publication failed

<Zakim> liam, you wanted to ask if this means dated copies go away (having missed the start, sorry!)

<ted> [we plan on targeting those messages to specific groups instead of catch-all alias as it is at present]

liam: are you saying the dated spec goes away?

tripu: I found that we have some specs under /TR but not linked to anywhere

liam: some specs do get uploaded under /TR but not being published later

<phila> I admit to being responsible for a couple of TR orphans :-(

ivan: does it mean all the notifications go the same mailing list?

<tripu> Notifications will be better targeted, as Ted hinted.

ivan: it means I have to subscribe to the mailing list and will get a huge amount of unnecessary emails

<Zakim> r007a, you wanted to ask, at the appropriate point (maybe later), why docs not going to Rec are not covered and can we not bring them into the system

tripu: it should be much smarter. So far we don't have a lot of traffic but it makes sense to have it split between different mailing lists

richard: my WG is eager to use that system but we produce a lot of NOTEs
... when will we be able to use it?

tripu: at the moment, we only process WDs
... people have been asking for that on spec-prod@w3.org
... as much as we would like to expand the type of specs we can handle, we still have a few things to fix for WDs
... it's difficult to give a timeframe

<phila> Notes often start out as WDs

<Zakim> jeremie, you wanted to answer ivan

tripu: specberus can already check more types of documents

jeremie: the reason why we publish to a mailing list because it was the easiest way to do it
... instead of targeting the requester
... I recommend you don't subscribe to the mailing list but just look at the archive

<tripu> Notifications archive

jeremie: so you don't get too many emails

<inserted> [Slide 10]

tripu: slide 10 lists features we don't have yet
... for now, we don't have a web UI
... we handle only WDs

Key remaining Challenges [slide 11]

<inserted> [Slide 11]

tripu:

richard: if we have a document that is going to be a NOTE but is current a WD, can we use the system?

tripu: yes

richard: fantastic

phila: it doesn't handle FPWD, right?

tripu: no FPWD for now, only ordinary WD

<Ralph> [I note "Transitions" is mentioned on slide 11]

phila: no other standards bodies allow editors to publish like that
... it's great but the downside is nobody is going to read a version that can be obsoleted so quickly

tripu:

<jeremie> Issue: Extract more metadata: flag 'is informative'

jeremie: I'm sorry to Richard but we are not able to publish WD that are going to be NOTEs
... the details discussed on github #143
... we are planning to do so, but not possible for now

ivan: NOTEs are important for IG
... regarding the exceptions you listed on slide 11, it often happen that a team contact has a friendly discussion with the webmaster

<ted> "friendly"

ivan: when there's an error reported by pubrules
... sometimes, you get the approval from the webmaster

<ted> [webmaster and this system is tasked with enforcing w3c's process]

ivan: is there a way for the system to remember that exception?

tripu: the exceptions are a headache to handle

<jeanne> +1 -- I regularly link to outside a11y documentation with invalid, but working links

ivan: if there's a draft that is not 100% perfect and still pass the system, it would be great

tripu: we are thinking about adding a list of steps to skip on a specification basis

ivan: we have team contacts for each WG but we need to have a human filter that can approve these errors
... team contacts can be that filter

<ivan> +1 to richard

richard: it's great to be able to publish often on /TR because we would prefer to have people review the dated spec instead of the editor's draft

<Ralph> +1 to the workflow r12a describes; publish often with some identifier that makes it practical to tie comments to identified versions

ted: handling exceptions is pretty complex. We are thinking about having a general exceptions system

<r007a> i would note that we, as a horizontal activity, are constantly reviewing specs and want to be able to review dated versions too

ted: when you go to CR, you want your document to be clean and not have the exceptions

tripu: the rules exist for a reason

ivan: I agree we have to be strict when we go to CR, PR or REC
... and I wonder if we want to keep the current system for these 3

<koalie> [Coralie arrives]

ivan: for WDs, it should be fine

<ted> [phrased another way - if you want to use it, conform. if you want exceptions, go through manual appeal process]

ivan: I'm a bit worried with an automatic system
... it has a tendency that only the automatic way will be used

<r007a> i wonder whether there are identifiable things that must be correct, eg. patent policy or copyright statemnt, but other things that require less rigourous conformance

Benefits of these tools [slide 12]

<inserted> [Slide 12]

tripu: I hope the advantages of such system are clear
... now I want to show how the system works for you
... only the first time a specification is going to be published, you will need someone in the the team to request a token
... to request a token, you need the shortlink and the source of the spec (where it comes from)

timbl: do you really need those url? can't you get one from the other?

tripu: I think we could

<jeremie> timbl, we are not extracting data from the document at this point

timbl: it should be possible if you follow the spec (on GH for instance)

ivan: if you use respec, the information is already there

darobin: if you change the shortname in the source, you could publish everywhere on /TR

<Zakim> darobin, you wanted to answer that from Tim

tripu: in this GH example, you could publish everything under the source url

How to publish [slide 13]

<inserted> [Slide 13]

<inserted> [Slide 14]

tripu: then you make sure your document pass pubrules

<inserted> [Slide 16]

tripu: slide 16 shows how you can request a publication
... you can use your browser directly or CLI

<darobin> [and of course, in addition to browser and CLI you can build tools that call out to that HTTP API]

tripu: you'll get the id of the request

<inserted> [Slide 17]

tripu: and that's it, you just have to wait a bit and get the status of the request
... right now, we are dumping the result

<Ralph> [where is the documentation for the request parameters (both required and any optional params)?]

<darobin> [example of code that autopublishes from a build tool https://github.com/w3c/spork/blob/master/cherry-picker.js#L75]

tripu: the result is not user-friendly and shows the different steps

<tidoust> [The decision is the decision to go with the automated process, right? That is, the group does not need to agree to publish each new version, or does it?]

ivan: I try to combine what you are showing and the way we are using respec in my WGs
... we generate snapshots (dated spec) that are sent to the webmaster
... how can we combine it with that system?
... does it do all the respec management automagically?

<darobin> https://github.com/w3c/spec-generator

darobin: the good news is you don't need to do anything
... respec is supported by the system. It'll call another system called spec-generator

<tripu> Let's go to page 18, where the manifest is explained.

darobin: you will actually only need one line in your manifest

<darobin> https://github.com/w3c/echidna/wiki/How-to-use-Echidna-with-ReSpec-and-GitHub

<darobin> https://github.com/w3c/echidna/wiki/Setting-up-Echidna-as-a-GitHub-hook

<Ralph> [I suppose the documentation to which darobin refers in his answer to my question is https://github.com/w3c/echidna/wiki/How-to-use-Echidna ]

ivan: great, it does what we do manually

tripu: the next slides 18/19 show what robin was detailling

<inserted> [Slide 18]

<inserted> [Slide 20]

tripu: on slide 20, here's how you can use respec with the automatic publishing system

<darobin> [but it support ReSpec "more" in that you don't need to pre-generate]

tripu: echidna doesn't care whether you are using respec, bikeshed etc
... so far we have limited support for spec generators

<Zakim> darobin, you wanted to reply to ivan

richard: we have to create that manifest and make sure it references every images we have in the spec
... the system is not able to find these?

darobin: you could in theory write a tool that could generate the manifest from a spec
... I suspect echidna requires a manifest instead of parsing the html

ivan: I did something similar when I was converting TR spec to epub
... and I was relying on that kind of manifest

<r007a> http://www.w3.org/TR/2012/NOTE-jlreq-20120403/

ivan: and it does simplify things

richard: when you have documents like jlreq, maintaining the list of images, it becomes quickly a pain in the neck

darobin: I understand what you are saying. Please file an issue to modern tooling and I'll look into it
... it's going to be difficult to have it in echidna
... but I'm happy to look at a tool producing these manifest

jeremie: it will be extremely complex for have that kind of tools

<ted> [echidna is platform agnostic, not eg github specific. some platform repos sense of relative path might be different that others for us to retrieve over the web]

<jeremie> Elements of kihon-hanmen. (Example in vertical writing mode.) in Requirements for Japanese Text Layout

<jeremie> Small kana and the position of their letter face in the character frame. in Requirements for Japanese Text Layout

<jeremie> http://www.w3.org/TR/2012/NOTE-jlreq-20120403/Images-en/img1_[1-7].png

<Bert> (I'm using an install script for /TR that seems to find images automatically rather well.)

jeremie: maybe we could use special format to have a list of file with sequences (numbers, letters, ...)

<darobin> [basically, you load the document into PhantomJS, and intercept loaded resources as in https://github.com/w3c/spork/blob/master/spork.js#L103]

richard: with many images, the manifest is likely to have errors

<darobin> [then you need some ad hoc code to differentiate the resources you care about from those you don't as in https://github.com/w3c/spork/blob/master/profiles/html.js#L26]

<darobin> [it's not technically hard, but it's work :)]

tripu: robin agreed to have that tool ready by the end of the tech talk

Keep Echidna in the loop [slide 19]

<inserted> [Slide 19]

tripu: Slide 19 shows how you can integrate your spec with echidna using a continuous integration tool like travis

<Ralph> Travis CI

<jeremie> https://en.wikipedia.org/wiki/Travis_CI

tripu: CI tools make it easy to run tasks after each commit

<r007a> ivan, we often have more images in the directory that are used in the doc, and sometimes different directories (eg. for zh version and en version), so care still needs to be taken there

tripu: that file .travis.yml is basically all you need to integrate your spec with echidna
... it's getting closer to the long term goal of having everything automatic

<inserted> [Slide 21]

tripu: we are planning to improve the errors messages, UI, etc

ivan: when WDs are published, they are announced on the homepage
... but it was not mentioned during that talk
... should I get in touch with comm team?

tripu: at the very least, people in charge of that should get the notification and it shouldn't be too hard to plug it to another system

<darobin> [you need comments if you want me to finish the manifest tool before the end of the call]

Applause

<plh> tripu++

<timbl> +1

<plh> denis++

jeff: great job. This is something I've heard from a lot of editors. They are very excited about it

<Ralph> +1

<plh> jeremie++

<veronica> +1

<liam> +1

<tripu> team++

<koalie> team++

<ivan> +1

<guillaume> jeremie++

<guillaume> team++

<guillaume> tripu++

<guillaume> denis++

ivan: changing the operations for an existing WG is not that easy I guess. Might not suit a WG that is about to close

tripu: if some of you have questions about how your WG can migrate to that system, ask us. It shouldn't be too hard, at least in a manual way

<MikeSmith> thanks tripu

<darobin> FWIW https://github.com/darobin/echidna-manifester :)

- DRAFT -

The new publication system: how is that going to help me?

04 Jun 2015

Attendees

Contents