SWS Testbed Incubator Group Charter

The mission of the SWS Testbed Incubator Group, part of the Incubator Activity, is to develop a standard methodology for evaluating semantic web services based upon a standard set of problems and develop a public repository of such problems.

The approach is to analyze the process, infrastructure, and results of the Semantic Web Services Challenge workshops. In particular, we hope to establish a set of standard problems related to the use of Semantic Web Service technologies. Part of each problem description will also be a set of Web Service Interface Definitions (in WSDL). Along with the set of problems we develop a methodology for automatically verifying the messages that are exchanged in order to solve a particular problem. A reference implementation of the infrastructure can then automatically verify a problem solution by analysis of web service message exchanges. Additionally a peer review-based methodology for determining the software engineering value of the technologies used to solve the problems shall be established.

The rationale for the incubator is to standardize the evaluation of emerging semantic web service technologies. There are many claims for such technologies in academic workshops and conferences. However, there is no scientific method of comparing the actual functionalities claimed. More important, there is no way for industry to evaluate the robustness and applicability of these technologies. Progress in scientific development and in industrial adoption is thereby hindered. The SWS Challenge Workshops provides a major first step toward filling these lacunae.

The workshops are an ongoing experiment in developing a methodology for evaluating the functionality (versus performance) of semantic web service technologies. Based upon our first year's experience with a small group, we feel that we can now involve a larger community in this development, with a aim to standardizing the methodology and infrastructure. It should be noted that we carefully refrain from recommending technologies or providing solutions to the semantic web service problems. We are only trying to standardize the evaluation methodology.

Finally, we intend to continue the Challenge workshops as ongoing experiments in execution of the methodology. The incubator group is to observe these workshops and decide what is working and how best to refine a standardized methodology that can be widely applied.

Join the SWS Testbed Incubator Group.

End date	4 March 2008
Confidentiality	Proceedings are public
Initial Chairs	Charles Petrie Amit Sheth
Initiating Members	Amit Sheth, Wright State University Deborah L. McGuinness, Stanford University Michal Zaremba, DERI University of Innsbruck Tomas Vitvar, National University of Galway, Ireland

Scope

The primary task will be to agree upon

The set of standard problems and their levels,
support of scalable public collaborative development of new problem scenarios and associated services, and
the standard methodology for peer-review of solutions.

Success Criteria

The incubator will have succeeded when we have agreement upon not only the set of problems, but also how to add new ones in a collaborative environment and how to review the code and claims of participants so that there is a consistent evaluation.

Out of Scope

Providing a reference implementation of the infrastructure technology and functions will not be addressed by this incubator, but is expected to be addressed by a volunteer provider.

Solutions to the problems are out of scope for this incubator and will be addressed by workshop participants.

Deliverables

This incubator will deliver a precise set of problems as well as a procedure by which the community will add to them. The incubator will also deliver a set of guidelines by which code and claims can be evaluated at workshops. The existing set of problems, as well as code, is now available at http://sws-challenge.org/wiki/index.php/Scenarios.

Coordination

W3C Groups

It will be important to coordinate with the groups below but since this incubator will not recommend technical solutions, most of our work will be orthogonal to that of these groups.

Semantic Web Services Interest Group
Semantic Annotations for Web Services Description Language Working Group

External Groups

The Semantic Web Services Testbed Incubator will benefit from interactions with technical experts in the SWS Challenge Workshop, OASIS Semantic Execution Environment (SEE) Technical Committee, DERI Innsbruck, and the Stanford Logic Group
In particular, the Semantic Web Services Testbed Incubator will coordinate their intermediate developments with the Steering Committee of the SWS Challenge Workshop and current participant teams.

Participation

This group will communicate by occasional face-to-face meetings at at least two workshops per year (the next one in June in Innsbruck) and also all steering committee and participants can recieve announcements and comment by email. We already have an public Wiki for all information.

Communication

This group primarily conducts its work on the public mailing list public-xg-swsc@w3.org (archive) . The group's Member-only list is member-xg-swsc@w3.org (archive)

Information about the group (deliverables, participants, face-to-face meetings, teleconferences, etc.) is available from the SWSTestbed Incubator Group home page.

Decision Policy

As explained in the Process Document (section 3.3), this group will seek to make decisions when there is consensus. When the Chair puts a question and observes dissent, after due consideration of different opinions, the Chair should record a decision (possibly after a formal vote) and any objections, and move on.

When deciding a substantive technical issue, the Chair may put a question before the group. The Chair must only do so during a group meeting, and at least two-thirds of participants in Good Standing must be in attendance. When the Chair conducts a formal vote to reach a decision on a substantive technical issue, eligible voters may vote on a proposal one of three ways: for a proposal, against a proposal, or abstain. For the proposal to pass there must be more votes for the proposal than against. In case of a tie, the Chair will decide the outcome of the proposal.
This charter is written in accordance with Section 3.4, Votes of the W3C Process Document and includes no voting procedures beyond what the Process Document requires.

Patent Policy

This Incubator Group provides an opportunity to share perspectives on the topic addressed by this charter. W3C reminds Incubator Group participants of their obligation to comply with patent disclosure obligations as set out in on a Royalty Free basis, as defined in the Section 6 of the W3C Patent Policy. While the Incubator Group does not produce Recommendation-track documents, when Incubator Group participants review Recommendation-track specifications from Working Groups, the patent disclosure obligations do apply.

Incubator Groups have as a goal to produce work that can be implemented on a Royalty Free basis, as defined in the W3C Patent Policy.

For more information about disclosure obligations for this group, please see the W3C Patent Policy Implementation.

About this Charter

This charter for the SWS Testbed Incubator Group has been created according to the Incubator Group Procedures documentation. In the event of a conflict between this document or the provisions of any charter and the W3C Process, the W3C Process shall take precedence.

Charter authors

Holger Lausen, DERI University of Innsbruck
Charles Petrie, Stanford University
Michal Zaremba, DERI University of Innsbruck