Philippe Le Hegaret (he/him): Go ahead.
Denis Ah-Kang: Okay, thank you so welcome everyone, so we are going to talk about the automatic publishing system today.
So, first let me introduce myself, I am Denis Ah-Kang and I'm the W3C webmaster and I'm also part of the systeam.
So, can you guys see my screen?
correctly?
yeah?
Philippe Le Hegaret (he/him): Yes, we can.
Denis Ah-Kang: All right, so.
I believe the slides have been shared on IRC.
So first.
What are the goals of the automatic publishing system?
So the main goal of enabling the automatic publication system in your repository is to help maintain /TR up to date.
So /TR should represent what the Web should be and so it's very important that this pagelinks to the specifications that are close to the editors draft.
So slide two.
So, in the past, the only way to publish the document to /TR was to submit a request to the webmaster and expect publication on Tuesdays and Thursdays, not even talking about the publication moratoria happening during TPAC and AC where our publications were not possible.
So, this would create delays, back and forth between the editors and the webmaster to make sure the document was ready on time for the publication.
So, given how heavy the process was, the specifications were not published very often, resulting in /TR being outdated easily.
So to fix that issue we built Echidna, a system to check if the documents meet all the requirements and publish them to /TR without having to wait for the for the feedback from the webmaster or specific day to publish.
So that system basically checks: if all the resources can be downloaded; the document submitted passes pubrules; ff the transition was approved by the director if it's a CR snapshot; And if there are no external resources we cannot ensure availability.
Now the system currently has some limitations and you will not be able to publish a spec from first public working draft all the way to Recommendation just using the automatic publishing system.
So in this graph the parts in red indicate the type of documents Echdina is able to process which basically are Working Drafts, Candidate Recommendations Snapshots and Candidate Recommendation Drafts with a couple of limitations, like documents will be rejected if it's published under a different Working Group than the previous version, Echidna cannot handle document with shortname changes; And you also cannot publish a Working Draft if the previous version was a CR.
So these cases don't happen very often, but these are some of the limitations of Echidna.
So next slide.
So the next slide shows the requirements you need to get started with the automatic publishing system.
The first, you will need your specifications to be on github which would be the case for most if not all groups these days.
Then you will need a record of the consensus in your group to publish with with Echidna.
This is discussed within the Working Group and that you need a record of that consensus.
You will also need a token for each specification.
That can be generated by your team contact or the Chair of the Group and this token should be kept private and not shared, as it will be, as it will be used to authenticate the request to Echdina.
And finally, you will need admin rights to the github repository to add the token as a repository secret.
We will give more details about that step a bit later.
So slide five.
So the case of CR snapshot is a bit different as the publication requires the director's approval and a draft announcement sent to the communication team.
To get the directors approval, the transition request should be raised as an issue in the repository w3c/transitions, so that is linked from the slide.
And the templates for the issues can be found in the readme.
So you can simply click on the link in the readme and you will have a pre-filled issue that you need to complete.
So when Echidna will receive a request to publish a CR snapshot, it will look for an issue with the title of the issue, ending with the specification shortname.
And if it finds one, it will check if the requirements are met by looking at for specific comments from the director and the communication team.
So there are linked two examples at the bottom of the slide, if you want to take a look, You can check these up.
So slide six.
So this slide introduces the preferred and easiest way to configure your repository for automatic TR udpates.
So a couple of months ago Sid Vishnoi - I believe he's on the call right now, yes he's in the call - So he developed a github action to automate the task with respec and bikeshed documents.
So for those who don't know what github actions are, it's basically a simple way to automate workflows - so it can help build test and deploy applications we've just a YAML file.
So what can spec-prod do exactly?
So it can generate snapshots from respec or bikeshed documents; it can detect markup… markup, CSS errors or broken links.
And it can publish the generated snapshots to GitHub pages or to /TR via Echidna.
If the document is a compound document, like, for example, it has images or style sheets or scripts, spec-prod will even download all the resources and publish them automatically.
So the next slides focus more on what you need to do to publish to echidna, but if that you can check spec the spec-prod repository if you have specific needs, for example, to publish to github pages.
So slide seven: so first thing you need to do before even configuring the github action is to add the token generated by the team contact or the chair.
The token should be added to the repository as a repository secret.
This can be done in the settings of your repository where there's a secret settings.
And you can click on the button new repository secret, here.
So then, on the next slide, from there, you can give a name to the token - that name will be used in the github action and copy the token value.
You can see, these are like environment environment variables, but github will encrypt them.
So, if everything goes well, you should see your token listed in the settings, the secret settings.
So you are now - so slide nine - so you're now ready to create the github action workflow, as I mentioned, this is a simple YAML file.
All the github actions are located in the folder .github/workflows of the repository.
So in this example, we created the pr-push.yml file in the github workflows directory.
But it can be anything, the name doesn't matter here.
So I will try to describe the YAML, what it's doing.
So what what we are seeing here is that we want the workflow to run on any request, and on push to the main branch only.
We then list the jobs and the actions to be used in the in the workflow.
The actions are here.
So here we pass parameters to the spec-prod action, which are the token we just configured as a secret.
So the way to use secrets variables is you prefix the variable with “secrets” and the name of the variable.
So as you remember, we named the token W3C_TR_TOKEN but if you choose a different name, you can just update the YAML file.
And we also have the link to the Working Group decision to use Echidna, so this is the URL to the record to the consensus.
So we have this workflow spec-prod will generate the snapshot from the source document, by default, it will look for index.html or index.bs source document.
And validate the document in the pull request.
And if it's a push on the main branch, it will also send the document to Echidna, because we are using these these properties, these parameters.
So slide 10.
How to customize the workflow?
So the slide lists a couple of parameters you can use with spec-prod; like TOOLCHAIN, if you want to specify the… to help spec-prod detect the processor to be used; SOURCE if the main file of your spec is noted index.html or index.bs; VALIDATE_LINKS and VALIDATE_MARKUP are here just to enable or disable the validations, and you can also pass some processor parameter using the the parameter W3C_BUILD_OVERRIDE.
And if you want to get… if you want Echidna to send you the result of the of the request, you can specify your email address with W3C_NOTIFICATIONS_CC.
So more options are described the spec-prod repository.
There's a link to a few options, you can check to get more details.
So slide 11 is an example of the customized workflow with the variables we've just discussed.
So here we specifie the TOOLCHAIN bikeshed, the different SOURCE document, here, foobar.bs.
So i'm missing - the attribute is wrong here, it's W3C_NOTIFICATIONS_CC- that we are sending the results at echdina@example.org; There is the working group decision and we pass other parameters for bikeshed to generation of the snapshot.
As I mentioned, I'm mainly focusing on the automatic update of /TR, but spec-prod can also help publish to GitHub pages, so there are more examples linked from the repository again.
Now, how can you check the result of the request?
Cecause a request to echidna doesn't necessarily mean your document is going to be published on /TR.
So you can check the results in the Actions tab of your repository.
That tab will list all the workflows that are, that were executed, and the results.
Here, you have the PR-push workflow And from here, you can get more details on the job status, so by clicking here, on the build-validate and deploy job.
So next slide.
So in the screenshot you can see that spec-prod sends a request to echidna via a curl command so I don't know if it's… if it's visible enough.
But basically what the action will do is to generate the curl command, and will send the generated snapshot to… to echidna.
Then spec-prod will check a few times, if echidna is done processing the request.
If the request has been processed, it will return the final result - here we have a success, which means the document was published on /TR.
But if it's still pending, the workflow completes and you will have to look for the report sent to the mailing list public-tr-notifications.
And if you added your email address to the parameter W3C_NOTIFICATIONS_CC, you will also get the report.
So the slide 15 gives an overview of what is done and the different steps of the automatic publication system.
So I mentioned… as I mentioned, the token creation can be done by the Chair or the W3C Team contact.
If there's, if it's a Candidate Recommendation Snapshot, the request should be sent to the repository w3c/transitions.
And then there's the github repository management, which is done by the chair, the staff contact and the editors.
So spec-prod is part of the repository, the action you are configuring in the repository, and that helps generate the snapshot from respec and bikeshed, run the validations which are optional, and send the request to echidna.
In return, echidna will validate the token, make sure that the token used is the right one, make sure the document passes pubrules, check the transition approval, check the external resources, and if everything goes well, publish to /TR.
So that's the overview of the whole system.
Okay, so we just discussed about the github action spec-prod which, so I mentioned, spec-prod was built a few months ago.
So before that, we had more complicated ways to use to send the request to echidna.
I will focus, I'm focusing more on spec-prod, because it's the easiest way to get started with echidna, but if you want to know more about the the old ways to use echidna, you can check the wiki that is linked here.
So if you have any problem, using echidna or spec-prod, or just you're having a hard time setting up your repo, your repository to enable the automatic publishing system, you can try the IRC channel #pub, or you can check the echdina and spec-prod repositories, create issues and we'll try to answer as soon as possible.
And also you can send the emails to webreq@w3.org.
So what's next?
So, in the future, we are, we want to check what the limitations are and how we can remove them.
So some of the limitations are related to the old Process Document, but with the new Process Document 2020, there are things we can we can improve.
We will also look at how we can facilitate the Recommendations revision per Process Document 2020.
And if we add new features, or we make major changes to Echdina, we will announce it to spec-prod before deploying.
So I'm done with the talk, if you have any questions, now is the moment.