TestCasesOverview

From Media Fragments Working Group Wiki
Jump to: navigation, search

Overview on the Media Fragments Test Cases (MF-TC)

This page introduces the Media Fragments Test Cases (MF-TC), explains and motivates them and describes their relation to the Media Fragments Implementation Report (MF-IR).

Why ?

The Media Fragments Test Cases (MF-TC) fulfill a dual role:

  1. we need the MF-TC to verify the semantics of our specification, and
  2. the MF-TC will be used to check potential implementations and report on their coverage in the Media Fragments Implementation Report (MF-IR).

Verification of the Media Fragments Semantics

The Media Fragment Syntax defines the structure of a valid media fragment, however doesn't tell what the semantics are. The semantics can only be defined by humans, and rather than exhaustively listing all possible cases (which would not only be time-consuming but also quite confusing to understand), we will establish a generic processing model. We note that there are likely three parts of that processing model:

  • The client processing, captured in the User Agent Media Fragment Resolution and Processing;
  • The network communication, captured in HTTP and RTSP interaction model;
  • The server processing (Note: we don't have it yet - is this maybe part of the network communication or do we need a separate handling, here?)

So, in our upcoming Syntax and Processing WD, we will define a generic processing model that defines what happens when a UA++MF receives a MF, which steps are to be taken (validation, network communication, rendering, etc.).

This generic processing model is essentially a set of abstract rules, such as, for example, found in many legal systems. The idea there is to define abstract, general valid rules and then, given a certain case at hand apply the (abstract) rules. For example (very plain, please forgive me, I ain't no legal expert ;) let us assume a simple rule in the legal system:

It is forbidden by law to steal. If someone steals something, the degree of penalty is 1 year.

It is obvious that we can't apply this general rule straight-forward onto every single case (otherwise we wouldn't need judges, right? ;). One has to know the circumstances, the concrete conditions, the intention, etc. of the person acting, then one is able to tell if the above rule applies to that very case.

The point here is: the rules are abstract and generic. The cases at hand are concrete and differ (maybe only in tiny details). How can one tell if a certain rule applies?

Here our Test Cases (MF-TC) come into play. They are a sort of case based law. Together with the generic rules of the Syntax and Processing WD they form the bases of all implementations. One may work bottom-up to implement it (for example check step by step if the implementation conforms to each single MF-TC) or one may choose to work top-down, that is implementing the generic processing model and then check if the MF-TC are passed.

In any case, there may be edge-cases, where the specification is ambiguous or not specific enough, leaving room for interpretation.

We note that it is very likely there will be a back and forth between the generic processing model and the MF-TC. For example, as the semantics of a certain MF changes, all respective TC need to be updated. Equally, when approving a new TC we might find that the generic processing model is not precise enough, yielding an update, there.

Media Fragments Implementation Report (MF-IR)

As of our charter, two success criteria are:

  1. Test suites for each deliverable with conformance criteria
  2. Availability of at least two implementations of each deliverable with conformance criteria, as demonstrated by an implementation report (summarizing implementation status against the relevant test suite) for each testable class of product, including user agents

Our Working Draft (WD) document will (in late 2009) have a Last Call Announcement (LC). Then, we will want to advance from Candidate Recommendation (CR) (see also Call for Implementations) to Proposed Recommendation (PR) (see Call for Review of a Proposed Recommendation). One of the entrance criteria for PR is the availability of the MF-IR, that is (also enforced in our charter) reporting on implementations that demonstrate that the entire specification can be implemented.

Note: we will have to rehash how precisely we read our charter; Michael reads it the way the W3C process document requires us to, chairs and others think that we have higher requirements.

How?

As discussed in the 3rd F2F meeting in Barcelona we will create test cases for two different purposes:

  • MF-TC for testing the media fragments semantics itself
  • MF-TC for testing the end-to-end communication (protocol dependent?)

Each TC will in the future be formally described using a test case schema, a TC vocabulary in RDF. Additionally, we may introduce MF-specific terms in our own namespace, if necessary.

We agreed at the Barcelona F2F that we will start to collect and review the test cases on the Wiki; see the Test Cases Approval page for the current MF-TC. Note that this Wiki page will be used until we have set up a complete Test Harness that allows automated testing and a consistent tracking of the TC status.

Note, that to ensure an easy transition to the RDF-based, formal description of the MF-TC that is able to drive out future Test Harness, the respective fields (such as title, input, output, etc.) are already aligned with the TC vocabulary.

MFTC categorisation

The MFTC categories are defined at http://www.w3.org/2008/WebVideo/Fragments/TC/mftc.rdf

A human-readable, tabular rendering is as well available (based on an XSLT).