SWS Testbed Incubator Group Charter

The mission of the SWS Testbed Incubator Group, part of the Incubator Activity, is to develop a standard methodology for evaluating semantic web services based upon a standard set of problems and develop a public repository of such problems.

The approach is to analyze the process, infrastructure, and results of the Semantic Web Services Challenge workshops. In particular, we hope to establish a set of standard problems related to the use of Semantic Web Service technologies. Part of each problem description will also be a set of Web Service Interface Definitions (in WSDL). Along with the set of problems we develop a methodology for automatically verifying the messages that are exchanged in order to solve a particular problem. A reference implementation of the infrastructure can then automatically verify a problem solution by analysis of web service message exchanges. Additionally a peer review-based methodology for determining the software engineering value of the technologies used to solve the problems shall be established.

The rationale for the incubator is to standardize the evaluation of emerging semantic web service technologies. There are many claims for such technologies in academic workshops and conferences. However, there is no scientific method of comparing the actual functionalities claimed. More important, there is no way for industry to evaluate the robustness and applicability of these technologies. Progress in scientific development and in industrial adoption is thereby hindered. The SWS Challenge Workshops provides a major first step toward filling these lacunae.

The workshops are an ongoing experiment in developing a methodology for evaluating the functionality (versus performance) of semantic web service technologies. Based upon our first year's experience with a small group, we feel that we can now involve a larger community in this development, with a aim to standardizing the methodology and infrastructure. It should be noted that we carefully refrain from recommending technologies or providing solutions to the semantic web service problems. We are only trying to standardize the evaluation methodology.

Finally, we intend to continue the Challenge workshops as ongoing experiments in execution of the methodology. The incubator group is to observe these workshops and decide what is working and how best to refine a standardized methodology that can be widely applied.

Join the SWS Testbed Incubator Group.

End date 4 March 2008
Confidentiality Proceedings are public
Initial Chairs Charles Petrie
Amit Sheth
Initiating Members


The primary task will be to agree upon

Success Criteria

The incubator will have succeeded when we have agreement upon not only the set of problems, but also how to add new ones in a collaborative environment and how to review the code and claims of participants so that there is a consistent evaluation.

Out of Scope

Providing a reference implementation of the infrastructure technology and functions will not be addressed by this incubator, but is expected to be addressed by a volunteer provider.

Solutions to the problems are out of scope for this incubator and will be addressed by workshop participants.


This incubator will deliver a precise set of problems as well as a procedure by which the community will add to them. The incubator will also deliver a set of guidelines by which code and claims can be evaluated at workshops. The existing set of problems, as well as code, is now available at http://sws-challenge.org/wiki/index.php/Scenarios.


W3C Groups

It will be important to coordinate with the groups below but since this incubator will not recommend technical solutions, most of our work will be orthogonal to that of these groups.

External Groups


This group will communicate by occasional face-to-face meetings at at least two workshops per year (the next one in June in Innsbruck) and also all steering committee and participants can recieve announcements and comment by email. We already have an public Wiki for all information.


This group primarily conducts its work on the public mailing list public-xg-swsc@w3.org (archive) . The group's Member-only list is member-xg-swsc@w3.org (archive)

Information about the group (deliverables, participants, face-to-face meetings, teleconferences, etc.) is available from the SWSTestbed Incubator Group home page.

Decision Policy

As explained in the Process Document (section 3.3), this group will seek to make decisions when there is consensus. When the Chair puts a question and observes dissent, after due consideration of different opinions, the Chair should record a decision (possibly after a formal vote) and any objections, and move on.

Patent Policy

This Incubator Group provides an opportunity to share perspectives on the topic addressed by this charter. W3C reminds Incubator Group participants of their obligation to comply with patent disclosure obligations as set out in on a Royalty Free basis, as defined in the Section 6 of the W3C Patent Policy. While the Incubator Group does not produce Recommendation-track documents, when Incubator Group participants review Recommendation-track specifications from Working Groups, the patent disclosure obligations do apply.

Incubator Groups have as a goal to produce work that can be implemented on a Royalty Free basis, as defined in the W3C Patent Policy.

For more information about disclosure obligations for this group, please see the W3C Patent Policy Implementation.

About this Charter

This charter for the SWS Testbed Incubator Group has been created according to the Incubator Group Procedures documentation. In the event of a conflict between this document or the provisions of any charter and the W3C Process, the W3C Process shall take precedence.

Charter authors