Meeting Agenda· 2001-11-12· 2001-11-13·
Participants· Agenda· Agenda Discussion· Taxonomy Document· QA Glossary· QA Framework· Infoset Talk· OASIS Conformance Requirements·
KD: ACTION ITEM make the list of all people participating.
Olivier: ACTION: Ask Olivier, start looking for a unified glossary of W3C Terms, not QA related.
DD: Action to complete the QA glossary - KD and DD.
Kirill: Action to give all pointers to "Serialized Infosets in test output Comparison" to Karl for the resources page
Scribe: Karl Dubost
Hold in Bruxelles at CEN. Weather was foggy. The Argus Hotel is very close to CEN. Room Marie-Curie (5th Floor).
Welcome message from Daniel Dardailler. 2 minutes around the table.
DD: - There's no real agenda but will try to move forward the activity. I have started with the Framework Document. There are smaller documents
DD: Do you want to add something to the agenda?
LH: Practical things: Communications, Websites, Mailing-List, etc...
Framework of Certification which can be interesting for OASIS TC too.
MS: How to track issues list?
RL: There's a plan for Tracking system tools.
GO: Bugzilla... but sometimes it's complicated
KD: A simpler tool
RL: A common tool for every WG
DD: I guess we should not do the QA for all WGs
RL: We should be the Guinea Pigs for all WGs
MF: Bugzilla was too difficult and so it's not used the systeam didn't succeded to make it usable.
GO: Email not solved in the day goes in Bugzilla
LR: Conformance requirements, how to be sure that the same things is not done separately. Coordination with other organizations. Within OASIS, there's a TC for Conformance which is in charge of addressing all conformance isssues.
KD: Inter-WG communications, Strategy to invite WG to do QA when they have already been chartered
GO: Future notes with the same style.
DD: OT wants to work on this kind of materials.
KD: We received very good comments from the community.
RL: How far we go in QA review of specifications. CRec must have two interoperable implementations.
KG: is it a part of documents guidelines?
DD: We need to review some specifications
KG: In order to be productive we need to carefully scope and prioritize our work. There are lot of specifications coming out, need to be careful in taking a work item to review them all.
DD: Same experience in the WAI with the XML Access Guidelines. (XAG). Comm interWG is very important.
IJ: Framework for building TS. How to start a TS is the document from Lofton? I want to know how to organize the Test itself when I want really to do it.
LR: It really depends on the SPEC, if it's an API spec, a document format.
IJ: It's very difficult sometimes to identify what's testable.
KG: Need to carefully scope and prioritize our work.
KG: Examples of high priority in-scope deliverables to my mind are: framework document, TS process document, technical document for best practices, for example how to verify tests output - output comparison).
KG: Identify and discuss all deliverables of the WG.
KG: Agenda item: Use of Serialized infosets for XML testing (20min presentation
KH: What's the plan for WAI ER?
DD: EARL go to QA and others in other WAI WGs
MF: I want to see a document "QA is good for you"
KD: Max had a lot of pressure. Help Team contact to support the pressure
DD: QA on W3C budget doesn't scale. So we need that Members commit QA resources to WGs. Arnaud Le Hors (IBM) asked to members to commit more QA resources to WGs at the last AC Meeting.
MF: Who is participating?
KD: ACTION ITEM make the list of all people participating.
DD: WG should comply to their charter. So WGs must commit to QA they have chartered. having priority in QA
The Matrix: What should be inside.
The framework document.
Improved pubrules document: Pubrules + Style Guide (Susan Lesch) + Conformance extended (Lynne Rosenthal). How many documents we need.
QA report on specifications: Role of the WG reviewing specifications. <- task assignement tables for specifications.
How-to plug QA in the process document
Taxonomy Docs
Do we publish Note or Rec?
DD: Reading the charter
LR: E&O -> Conferences?
DD: Maintenance of an Agenda of Conferences where it should be good to have QA people.
LH: We need to prioritize our deliverables resources
Coffee
DD: The document was first published at the QA Workshop in April @@link@@. Any of those validators produce an output with a machine readable format. For example the HTML Validator. With a common format, you can compare with other tools. A report format will be good to have a common basis.
KD: Not only HTML validator, but syntax checker like in BBedit (html authoring tools.
IJ: Theere's no output format for tests cases.
DD: I hope EARL could be the output format for the test cases too.
KG: From my experience with the XML standards testing, serialized infosets are useful for output format/comparison.
DD: 1.1 Syntactic (validator). Do people agree?
DD: 1.2 Semantic (Evaluator). Do people agree? Checklinks is an example for human decision.
LR: Validation/Validator must be clearly defined, because for us validation is how to apply the conformance process.
IJ: in UAG we call validation (NIST) as an evaluation. Help! :) There's a need for definition
DD: 2. content-driven (test suite)
DD: 2.3 guidelines-driven For examples in UAG
DD and IJ: For examples, you can test on content as an example for an image with alt -> Is ALt accessible in the UI, but you have also the case where you have the HTML authoring must ask you to put an ALT Text.
KG: 2.3 Interface Scenario instead of guidelines.
DD: everything is a TS.
IJ: I would distinguish if it's automatable
LR: We (Nist) don't distinguish
KG: In Microsoft we use following categories: Content validation tests, UI tests, API tests, content driven (parser/compiler) tests.
LR: goals of Taxonomy is to categorize is the most thing that can occured, butyou can still have exceptions. For example, W3C do not do Certification but you can have a stamp from CSS validator.
IJ: Css branding
MS: There's a difference between branding and certification
LH: It's easier to put a conformance icon on content more than on a tool, or a software. To claim that a piece of content conforms 100% to the finite list of content conformance requirements in a specification is achievable, but to say that a tool (e.g., UA) conforms to a specification implies testing it against all (infinite) valid content, which is usually not possible even in theory.
RL: W3C logo is a self certification for CSS and HTML. We pass this test suite for a user-agent is still possible.
MS: but you can't claim conformance to the spec.
RL: agreed
DD: use the marc skall document. I'm sure NIST has already one. We should try to use common terms in organizations. Do you have a recent document?
MS: we should use the slides from the QA WS which has been produced.
DD: We should define the Conformance, Test Suite, etc. Katie, what's your experience with the WAI glossary you were in charge of.
KH: You come up with existing definitions, and word by word you're trying to build a common definition.
DD: What's a difference between a conformance clause and a Test Assertion? Conformance Clause: So it's a section in the spec which defines the conformance. The conformance clause should reference all the Test Assertions?
MS: It refers to the list of TA, which are requirements.
DD: Some WGs have produced Requirements documents at the begining, before WD. It's requirements for the deliverables, requirements on functionnalities which must be in the spec. in UAG, checkpoints are requirements
IJ: We have requirements on the claims itself. Because a Tool can't implement everything.
LR: Does the conformance make clear what kind of claims you can do.
KD: Glossary for all specifications?
KH: we can point in a single to all definitions in differents specifications.
DD: Susan has a style guide but for spelling.
KD: We need a matrix of terms.
DD: We need a kind of Unified glossary which falls in the Olivier's bucket. WAI has been doing that already for a part. So we can learn. ACTION: Ask Olivier, start looking for a unified glossary of W3C Terms, not QA related.
DD: Action to complete the QA glossary - KD and DD.
DD: Framework document should be the main topic of this afternoon. After the break, review the matrix and list of deliverables.
The QA Framework document has been designed by Lofton Henderson initially. Lofton is presenting the document.
LH: (Presenting the different sections of the document). I break the rules of the initial document. Do we need to break the document in two parts? The Framework has a process part and a technical part. High level question is what should be the document as a whole? What sould be the approach of the document? Should it be written as a normative prose where people can refer to?
DD: This document should it be the entry for the family of QA documents.
KG: Two requirements - A process document is needed to start the QA process and the other one is a technical framework.
KD: Could we bind documents? As document can't go forward without the other.
LH: Good Point. Organisational process is easier to done, technical framework is much more difficult to write and will have a lot of comments because there are a lot of different cases of doing it.
KG: In order to prioritize documents, I would propose the process/operational part to be done first (asap) because several WGs are waiting for it with ready TSs.
IJ: ???
LH: I have written it in two main parts. For the non technical part, it's quite easy to have a draft in 2/4 weeks after this meeting.
DD: I'm in favor of pulling it out too. (the technical part). In WAI we had the same kind of experience. Can we use the Process word instead of organizationnal.
LH: So the point 2. should be Process and the third is Operational
LR: When you read it now, it made sense to me because I know what you're talking about me.
IJ: I would be able to have a view on XSLT diagram for Conformance to have abig picture of it. It helps to understand.
LH: Explain the diagram @@LINK TO THE DIAGRAM@@ Submission Guidelines: What the contributors have to do. Review Guidelines: What the rewieves must do before a test come part of the TS. Instructions to test Labs: Explain how to use the test case. Where are the scripts which permits to run this kind of test.
KG: We need process/operational document to explain questions for Licenses, appeal process for erroneous tests, etc.
LH: XSLT done a really good job in organizing the process. They have anticipated all the technical details.
IJ: Explaining at the board the process. @@PICTURE@@
LR, IJ, LH: Discussion at the board about the architecture of QA.
DD: We need to have a QA Roadmap to explain what's going on QA with deadline.
DD: Are we agreeing on the fact that the QA Framework will be an operation guide, explaining the difference step.
LR: Agreed.
MS: Framework is a loosy name for normative goals
IJ: It could be a modular document with a "kind of primer" to explain how all fit together.
DD: Primer is reserved for a Case By example. So we can use guidelines which contain requirements as in WAI.
IJ: Guidelines in a WAI have practical requirements to achieve the statement.
Discussion about organization of guidelines, requirements, checkpoints.
DD: Going back to the QA document. Can we group the section 2 and 3.
RL: We want it asap, we should take care of organization wrt timeline
KD: Does it scale for Rec after final stage? We should take care of it. It should be still usable after it.
IJ: Advisory board is looking of a deprecating mechanism for specifications.
KD: Example: someone can a create a TS after WG life, and the work is good how to deal with it. How to accept or refuse, or recognize the work from someone else.
LR: Good point. I would be able to be recognized by W3C. W3C Recs
LH: I'm not sure if W3C wants to recognize the external TS. If the proformat is good, but the content a crap.
DD: So a review has to be done. What happen for example for PNG. There's no more WG. Someone from the Graphics activity should look at the work.
Discussion between LR, MS, and LH. We agree on that someone should have to review the work. Who is the question.
IJ: Guidelines are a small sentence sufficiently expressive to be understandable.
DD: No section and guidelines and checkpoint at a deeper level.
IJ: Drawing an outline of the document.
What's an Harness. Details explained by LR taking DOM as an example.
IJ: As a WG I would like to have a view for a WG. For example, first, set a page with the name of your WG and /Test. Do we do a specific chapter for W3C or do we spread W3C bits in all chapters.
Coffee
Presentation by KG Agreed to add this to technical framework document. Action: Kirill to give all pointers to this work to Karl for the resources page
by LR
LR: The purpose of this document is to provide some guidance about how to write a conformance clause and what you need to do it. Any suggestions to help in writing this document would be great, but it would be also to use it as an input for deliverables of this WG. I'd like to get comments and suggestions. This is a very early draft on the topic. It has been written with the help of Marc, Lofton, etc and others. It's a try to compile what's in the common sense of different organizations. What should be considered when you're writing conformance clause in a specification. Make distinction between content rendering tools, authoring tools, etc. IETF and ISO use others words. ISO refuses to use MUST. It tries to define keywords. Conforming to the rendering will be quite different from conforming to the authoring side. So specify the type of products you are conforming is important. Discussion on profiles, and subset of a specification. When allowing extension, how-to define the extensions
Discussion: There's a need for a common way of denominating specifications. CSS Level 1, CSS Level 2, etc... but you have SMIL 1.0, SMIL 2.0
Discussion: Profiles are good and bad for implementations, easier to claim conformance to a piece for example Basic, and you can add some features piece by piece until you reach the next level.
Discussion: differences between discretionnary items and extensions. Sometimes, you can specifiy a behavior as a choice, because there's an infinite number of cases possible.
Discussion: We don't promote at W3C centralized version of things. So If we do extension to a thing it doesn't necessary have to be on a central repository.
Discussion: You can have different ways to be conformant with requirements.
Last modified $Date: 2011/12/16 02:56:41 $ by $Author: gerald $