From: Anthony Wiles ETSI Position Statement for W3C Quality Assurance Workshop Washington April 3-4, 2001 Anthony Wiles ETSI PTCC Sophia Antipolis France anthony.wiles@etsi.fr 1. About ETSI ETSI, the European Telecommunications Standards Institute develops a wide range of standards and other technical documents as Europe's contribution to world-wide standardization of telecommunications and associated domains. Based in the Sophia Antipolis science park in southern France, ETSI brings together well-over 600 companies and some 4000 technical experts from around the world. 2. ETSI and Testing ETSI members have long recognised the important role that validation and testing play in the development of ETSI standards, and of the products based on those standards. Even in a climate where time-to-market is paramount, manufacturers are prepared to put time, effort and valuable expertise into testing in order to ensure interoperability. ETSI has two permanent entities involved in testing activities. The PTCC (Protocol and Testing Competence Centre) and the Bake-off Service. 2.2 The PTCC The PTCC provides support and services to ETSI Technical Bodies (TBs) on the application of modern techniques for specifying protocols and test specifications. The PTCC is also responsible for the technical management of the ETSI Specialist Task Forces (STFs) which develop conformance test specifications for ETSI standards. In the past 10 years test suites have been produced for many leading ETSI technologies, including 3G UMTS, GSM, DECT, TETRA, Hiperlan/2, TIPHON, INAP, B-ISDN etc. While certain areas quite rightly require regulatory testing the policy today is to keep this to a minimum. As a consequence, there has been a subtle but fundamental change in the development and application of ETSI test suites over the last few years. ETSI test specifications now concentrate on maximising the chances of interoperability, for example by increased focus on critical functionality or error recovery behaviour. The test suites are not part of a bureaucratic testing program but are increasingly requested for by fora and manufacturers as an integral step in product development processes. NOTE: The PTCC does not perform the actual testing but does have close contact with the companies and organisations that do. Neither is ETSI involved in certification. 2.2 The Bake-off Service The ETSI Bake-off service organises bake-offs or interoperability events for ETSI members and non-members alike. It provides the logistical and organisational support (often at ETSI premises) for such events. Recent events include SynchFest, IPv6 InterOp, IMTC SuperOp and Bluetooth UnplugFest#3. The intention with bake-offs is to validate (debug) both the standards and early products or prototypes of those standards as they are developed. This activity is considered to be one of validation, rather than testing. 2.3 Testing for Interoperability The fact that ETSI develops conformance test suites and provides bake-off services to its members shows that both approaches are considered valuable by the ETSI membership. In some cases only conformance testing is considered adequate, in others the reverse is true. It has been clearly demonstrated in GSM and 3G, for example, that conformance testing of mobile terminals provides a very high degree of confidence that handsets from different manufacturers will interoperate. So much so that with the reduction in regulatory testing for GSM in Europe, operators and manufacturers are co-operating to put in place a voluntary testing scheme for GSM terminals. In the IP world, where the culture of bake-offs is well-established, it is obvious that this is an excellent way to develop and validate the base standards. The trend, however, indicates that the industry realises that it is often not a case of either-or but rather a combination of the two. A focussed set of conformance tests can provide an excellent complement to bake-offs and/or interoperability testing. This is a view that is shared by other, non-ETSI bodies such as Bluetooth. 4. Conformance Testing Methodology Generally, ETSI follows the testing methodology defined in ISO/IEC 9646. The methodology defines precise test methods, architectures etc. The test specifications themselves comprise two parts: Test Purposes written in prose and detailed tests, or Test Cases, written in the test language TTCN. All ETSI conformance test specifications are developed by groups of experts, better known as a Specialist Task Force or STF, recruited from the ETSI membership. Experts from the relevant Technical Bodies, manufacturers, test system developers and other interested parties are also closely involved in the development of test specifications. In many cases, this activity gives valuable feedback to the base standards. At ETSI the test specifiers make a special effort to ensure that their experiences are fed-back to the relevant TB. However, if this input is to be effective it is essential that the development of the base standard and the test specifications is done in a co-ordinated and timely manner. An important spin-off from conformance testing, and one that is widely used even in cases where no test specifications are developed, is the ICS (Implementation Conformance Statement). This document, which is essentially a checklist of the capabilities supported by a given system, provides an overview of the features and options that are actually implemented in any given product. A comparison of the ICS for two (or more) different products can give an early indication of potential interoperability problems. 5. Use of TTCN-3 Nearly all ETSI conformance test specifications are written in the standardised test language TTCN. The latest version of this language, known as TTCN-3, has been developed by ETSI Technical Committee MTS (Methods for Testing and Specification). Typical areas of application of TTCN-3 are protocol testing including mobile and Internet protocols (including text-based protocols), supplementary service testing, module testing, testing of CORBA based platforms, testing of APIs etc. TTCN-3 is no longer restricted to conformance testing. It can be used for many types of test specification including interoperability, robustness (torture), regression, system and integration testing. Use of a standardised test language, such as TTCN benefits both ETSI (who produce the tests) and ETSI members (who use the tests). For example: - education and training costs are rationalised and reduced; - maintenance of test suites (and products) is easier; - off the shelf tools and TTCN-based test systems are readily available; - universally understood syntax and operational semantics; - tests concentrate on the purpose of the test (not the test system); - allows application of a common methodology and style. - constant maintenance and development of the language. Most importantly, though, the language incorporates well-proven testing-specific capabilities which as a whole are not present in any other programming language. For example - dynamic concurrent testing configurations; - synchronous and asynchronous communication mechanisms; - encoding information and other attributes (including user extensibility); - data and signature templates with powerful matching mechanisms; - optional presentation formats (e.g., tabular format, MSC format); - assignment and handling of test verdicts; - test suite parameterisation and test case selection mechanisms; - combined use of TTCN-3 with ASN.1 (and other languages such as IDL); - well-defined syntax, interchange format and static semantics; - precise execution algorithm (operational semantics). - type and value parameterisation. 6. Conclusion ETSI has long been committed to improving the quality of its standards and related products through validation and testing. To this end ETSI actively contributes to the development and standardisation of modern testing techniques (such as TTCN-3). Now that TTCN-3 and some early tools are available EP TIPHON has started the production of conformance test specifications for the Session Initiation Protocol (SIP) as specified in RFC 2543. Work will begin shortly on the test specifications for the TIPHON XML-based Open Settlement Protocol (OSP). This work is expected to continue throughout 2001, the goal being "Conformance testing for interoperability". The near future is expected to see an increased use by ETSI Members of the combined services of the ETSI Bake-off Service and the PTCC.