Workshop on Quality Assurance at W3C,  3-4 April 2001.

Position Paper

The Open Group

Authors: Andrew Thackrah, James de Raeve; Testing and Certification, The Open Group.

The Open Group
Apex Plaza
Forbury Road
Berkshire RG1 1AX
  The Open Group

Our Experience

The Open Group exists to support the promotion and adoption of open systems and standards. We have worked for many years in support of these goals and have a great deal of experience in supporting companies, forums and consortia in seeing systems developed from standards through testing to certification and market adoption.

The Open Group manages programs in a range of industry sectors (Wireless, mobile, security, directory) to promote open standards in these emerging technologies. The company provides Testing and Certification services to members, and others in support of these programs.

We have experience of online conformance testing of XML-based user agents.

The VSWAP test suite is used by the WAP Forum in its client certifcation program which is run, on its behalf by The Open Group. The test suite covers all aspects of WAP-enabled user agent behaviour - micro-browser conformance.

We have developed tests based on W3C specifications.

The VSWAP and VSNETCOM test suites include tests for HTTP 1.0/1.1 and HTML 3.2. We also have test suites using IETF specifications such as VSLDAP - an LDAP directory server test suite that has been designed to support the certification program of the Directory Interoperability Forum, managed by The Open Group.

We  have established conformance testing methodologies.

For web-based systems, Unix/POSIX and real time platforms, for API testing (ADL) and we provide products and services to support standalone and distributed testing.

We are a leading provider of certification and product branding services.

We own and actively manage the Unix and other brands and we provide certification services to groups such as the WAP Forum and the Directory Interoperability Forum.

Company Needs

We wish to extend our conformance testing and certification services to meet the challenge of the emerging XML-based systems (Mobile, wireless, B2B, protocols, &c)

We wish to discuss the issues of testing XML-based systems (particularly at the semantic level) with like-minded groups.

Expectations of the Workshop

The big challenge for QA of XML-based systems is not in the validation of DTD/schema but in the characterisation of conformance requirements at the semantic level.  XML is being used for file formats, object encapsulation and in protocol exchanges (both as payload and envelope).

It is on the ability of XML systems to work correctly at the organisation-to-organisation level that the reputation of XML rests and so it is vital that we understand the conformance issues. We hope to further discussion and awareness of these topics.


Our Contribution

By discussing our existing strategies, products and services we can help to provide a view of the `state of the art' in conformance, certification and interoperability testing. In particular we can foster awareness of the differences between these three types of testing as this is essential to successful QA.
  • Conformance testing: An arbitrary checklist, agreed by consensus. Typically standalone tests, ussualy at syntactic level.
  • Interoperability testing: Proof that an element of a system performs its role correctly. Typically involves [business] scenarios, usually at semantic level. Can we characterise common semantics?
  • Certification testing: Testing in support of a certification (AKA branding) program. Requires an additional layer of bureaucracy to ensure that the correct sets of tests have been run under appropriate conditions.

Our work with mobile client testing is leading us towards the development of tests for modularized XHTML. We now have considerable experience in managing testing and certification programs in this field.

The Open Group is planning a new generation of tools to test XML-based systems at the semantic level. We see XHTML as a suitable system on which to focus our development efforts.
We wish to explore ways in which we can avoid duplication of effort in common work on emerging W3C standards (which are likely to incorporated into many future systems).

We would like promote discussion on the importance of semantic (scenario) testing to the future of XML.
XML is being used in various ways, for example...

Each type of system has its own testing needs which should be characterised.

Valid HTML 4.01!