Re: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]

Hi,

I also think the issue of an SCXML test suite is very important, and I
would like to contribute two comments.

First, I think that many existing open source SCXML implementations have
written their own test suites. Would it be useful to issue an "RFTC"
(Request for Test Cases?) so that these existing implementations can
contribute their tests to the W3C for inclusion as part of an official,
standard test suite? This would also help to illustrate differences in
semantics between existing implementations.

Second, regarding testing infrastructure, an important part of the work I
did on SCION [0] was to create a language-independent test harness
[1]. This test harness could potentially allow SCXML implementations to be
tested in a uniform way, thus making it easier to ensure compliance.

To provide some background: the SCION engine is written in portable
ECMAScript, and so it was possible to write bindings to it for various
high-level languages and environments, including the JVM, .NET CLR,
CPython, as well as various existing JavaScript shell environments such as
Rhino and Node.js. There was a need to test each of these language
bindings, and so I created an HTTP-based client-server protocol for testing
SCXML implementations. This meant that in order to test each language
implementation, one simply had to write an HTTP server in the language in
which the SCXML engine API is exposed, and implement the server side of the
protocol. This made it possible to test each binding uniformly, independent
of language or environment, with minimal effort.

This testing harness is relevant to the W3C SCXML standardization effort,
because it would allow the W3C to publish a standard test suite, which
could then be easily run against any language implementation. It would save
specification implementers the trouble of determining how to run W3C's
tests and of rolling their own custom test harness for that test suite. I
think providing or recommending a testing harness would be an improvement
over providing test cases only.

I look forward to your comments. Thanks,

Jake

[0] https://github.com/jbeard4/SCION
[1] https://github.com/jbeard4/scxml-test-framework

On Tue, Jan 22, 2013 at 9:21 AM, Jim Barnett <Jim.Barnett@genesyslab.com>wrote:

> Stefan,
>   On the issue of the conformance test, a comprehensive set of tests will
> be provided as part of the Implementation Report Test Plan when we publish
> the Candidate Recommendation of SCXML.  Strictly speaking, these are not
> "conformance tests" since their purpose is not to validate a specific
> implementation but to prove that the specification can be implemented.
>  However, as a practical matter, they're a pretty good imitation of a
> conformance test.  We would be  delighted if you would  run them and submit
> a report.
>
> I'm not sure when the tests will be published.  If we receive comments
> that lead us to make substantive changes to the specification, we will have
> to publish another Last Call working draft.  Once all comments have been
> dealt with and we are confident that no further changes are necessary, we
> will publish the Candidate Recommendation, along with the tests.
>
> We will respond to your other comments once we have had a chance to study
> them.  If you will included "ISSUE-825" in the subject line of any
> subsequent correspondence, it will help us keep track of thread
> automatically.
>
> Best,
> Jim
>
> -----Original Message-----
> From: Voice Browser Working Group Issue Tracker [mailto:
> sysbot+tracker@w3.org]
> Sent: Tuesday, January 22, 2013 9:13 AM
> To: w3c-voice-wg@w3.org; www-voice@w3.org
> Subject: ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]
>
> ISSUE-825: comments from Darmstadt [SCXML-LC-Comments]
>
> https://www.w3.org/Voice/Group/track/issues/825
>
> Raised by: James Barnett
> On product: SCXML-LC-Comments
>
> from Stefan Radomski [radomski@tk.informatik.tu-darmstadt.de]
>
> Hey there,
>
> [....]
>
> So if you are using or have used SCXML, please take the time to send us
> your comments.  The more comments we get, the more confident we can be that
> we are producing a sound and useful standard.
>
> we implemented the SCXML draft (or large parts of it) in C++[1]. We do
> feature an ecmascript (v8) and an experimental prolog (SWI) data model. The
> overall state of the implementation is still rather crude and very much
> undocumented but it builds fine on MacOSX, Linux and every now and then on
> Windows.
>
> I guess what we missed the most is a set of conformance tests as some
> ..scxml files with sequences of configurations the interpreter is supposed
> to go through and internal events to raise. Especially the logic around
> "filterPreempted" is still somewhat fuzzy to us.
>
> Apart from that, there are some areas in the draft where we could imagine
> some improvements. Just off the top of my head:
> - When you use the "Null" datamodel, you cannot use the <param> element to
> pass even literal strings to e.g. an invoker as both "expr" and "location"
> are subject to evaluation by the datamodel. Maybe the obvious "value"
> attribute with a literal string could be included.
> - Sending events via the basichttp ioprocessor should allow for the other
> party to send events in the response.
> - Is there a reason scopes in the datamodel were disallowed so explicitly?
> - When evaluating foreach with "item" already defined in the datamodel, is
> it to be reset after foreach ends?
> - When embedding an interpreter, #_parent could be specified to send
> events to the embedding application.
>
> I am sure we will run into many more problems once interoperability with
> other interpreters becomes an issue.
>
> Best regards
> Stefan
>
> [1] https://github.com/tklab-tud/uscxml
>
> ---
> FB20 Telecooperation | Darmstadt University of Technology Hochschulstr. 10
> | D-64289 Darmstadt Germany | Room S2|02 / A108 Tel +49 (6151) 16-6670 |
> Fax +49 (6151) 16-3052
>
>
>
>
>