Outline

• Conformance
• Conformance Testing
  • Testing Methodology
• Validation
• Certification
• Testing Ingredients

Terminology

• Specification ---------------> Conformance Clause
• Conformance Testing --> test suite, test tools
• Validation ------------------> process, reporting
• Certification ---------------> certificate, brand

Terminology - Conformance

• CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2)
  • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification
  
  
• CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance.

Terminology - Conformance Testing

• CONFORMANCE TESTING - a way to verify implementations of a specifictaion to determine whether or not deviations from the specification exists.
  • tests capture the technical requirements within the specification
  • testing is bound in scope by the specification
  • internal structure of product-under-test not accessible (black box testing)
  
  
• Necessary, but not sufficient, for interoperability

Note: referred to as conformity assessment in ISO Guide 2

Testing Methodology

• Falsification Testing
  • find errors by means of experimentation
  • show presence of errors not their absence
  • prove non-conformance; can never prove conformance
  
  
• Tests Based on:
  • required functionality
  • logical errors by misunderstanding requirements
  • errors from boundary conditions and divergence
  • common programming errors

Terminology - Testing Processes

• VALIDATION
  • ensures that testing can be repeatable and reproducible
  • ensures that conclusions are consistent with facts presented in the evaluation
  • testing typically performed by a test laboratory
  • test report produced
  
  
• CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

Testing Ingredients