Cwm Development Agenda and Release Roadmap

See also:

• Changes already made. Things move from this page to that.
• Outstanding bugs. Bugs (and some small requests for enhancement) are tracked in RDF, and available as a TODO list in an online calendar.
• semwebcentral project entry

by Release 1.1 2005/11/?? (Requires Python 2.3)

This contains performance work which would have been been in 0.9 except for unexpected medical situation. This will use Python sets where appropriate (instead of homebrew sets or sequences) and will therefore require python 2.3.

Release candidate rc1 was made 2005/08

• done with cwm.py 1.180: Need for deep unification in nested formulae. This was a point where proof checking of a more complex scario broke down: the proof used different variable names in rules, and cwm doesn't match nested fromulae with different variable names. Formula canonicalization;
• RDFlib fork joined with pychinko. Mindswap and rdflib sites are cross-linked, right?

CVS version 2005-10-24:

• --why works with a greater variety of files, see test/reason/detailed.tests. Beacuse proofs don't handle bnodes, the --n3=B option is used with proofs.
• --n3=B flag allows blank nnodes to be explicitly named on input

To do:

• Distribution mechanisms
  • dpkg debian packaging Done once setup.py -> RPM -alien-> debian.
• Speed benchmark
• allow alternative parsers; in particular,
  • redland (RFE)
• Add crypto testing to regression tests.  This will require developers to have pycrypto installed.
• Fix bugs in unification

Longer Term

There is no timescale for these enhahcements, as they are not subject to existing funded development or volunteer commitment. Bother are of course welcome at any time!

Code Cleanup

Philosophy: To be able to integrate with other projects such as redfoot, pychinko, redland, etc, make the base classes for N3 terms as light as possible.  Move functionality such as indexing for query, parsing, pretty printing, serializing, etc into separate modules.

• cut all code to 80 column
• make API tests, including very simple API for user wanting to use just RDF level.
• clean up serialier/output notation interface renaming functions and adding start/end list and set
• n3xml namespace?
•  make new ultrlight term class structure
• tetss for other programs like toIcal, 
• Make regression test manifest format less fussy about directory in which test is run or make it run in the same directory as the file etc
• List other apps on home page
• Clean interface to indexed formula so that different versions with different indexing can be used
• Abstract interface to stored formula so that persistent store can be used (rdlib, 3store, etc)
• Reification: write it up - in tutorial and some reference form (ontology comments? DesignIssues? )
• N3 formal grammar (RFE 10 Jul) Not included in test as of 2004-11-10.
  • 20041105: predictiveParser.py does all the positive parser tests yosi has come up with. . The tests are only run pre-release (to check consistency of tests and grammar) not in post release. TODO: Include the grammar code in the tarball. Negative test generation and inclusion of Turtle negative tests wait for 1.1
• RDF/XML generation does not use collection syntax still.

Performance

Performance: Make test/perform test suite with standard tests so as to plot perfromance changes version to version.

Develop specific benchmarks for specific problems, such as pipelineable cases, large store/small rules, large rules/small data, etc.. Include feedback from users to characterize different types of way in whic cwm is used.

Clones

Cwm demonstrates the power of a particular langauge and a particular general-purpose tool. Encourage and liaise with compatible systems, and share test data. Work to define subsets of the rule langauges which work with different types of engine.

• Rete rule-based system
• Backward-chaining systems, eg Euler.
• Compiled systems

Syntax

N3 grammar close the loop: Connect N3 grammar to syntax tests by generating parser directly from the grammar. . Closing the loop on the syntax. We have a definition of the N3 syntax in N3. This is only a form of documentation until we have automatically generated a parser and/or a test suite from that syntax definition. A straightforward job - We have a syntax validator based on the BNF in RDF already. The interesting bit is the ontology for annotating the BNF grammar with RDF production rules, because it could then be reused to annotating foreign and legacy grammars to enable automatic extraction of RDF from them. (Apart from the niceness of having something defined it itself of course.) (July 2004 this is done without the grammar annotation necessary to actually generate triples in the store)

RDF Output - Use Collections for lists.

N3 syntax - test set syntax (,) (<a>,) (<a>, <b>)

Extended RDF output syntax for formulae (how? have RDF Core locked all the exits? Prevents RDF2 not superset of RDF/XML1)

Query/Inference

Built-ins for lists. in (needs muli value function, below); append, prepend, second..seventh, last, concat.

Sets, plus builtins, see syntax in N3. Information is lost, and optimizations are lost, when sets are actually represented as ordered lists because that is the only syntax easy to use in RDF. owl:oneOf provides the constructor for sets.

Build cwm on python sets, requiring python 2.3, to gain speed.

Builtins extracting closed world data: s=Q(F, y, G) is set of x such that formula F.substite((y, x)) log:includes G.

Symbols used as variables as separate python class - save some time checking occurrences

Paramodulation - smushing built into store. Reuse redirection used for lists?

Builtin operations:

• Return many bindings from builtin eg list:in, os:argv
• Fairly complete XQ functions and operators implementations (use same URIs?)
• Extract datatype, language and string from a literal.

Remove all vestiges of the old pair represetation of symbols.

Implement backward chainer (like Euler) on the same store. Allow forward and backward chainers to interact, share built-ins.

Store

Implement versions of the store based on Redland, Jena, etc.

Implement rete method. Rules with state as objects. <= student project?

Net

Implement persistent cache - make an ontology (based very stongly on HTTP headers) for recording the results of an HTTP (etc) get. Store metadata about fetched ontologies. Build an index into the persistent cache for offline working.

Functions: find the closure of these documents under eg --closure=ptr and make sure than they are all in my persistent cache. Allow environment variable to specify persistent cache mode (default: ontologies only, write to cache when online and expiry check fails, read from cache when offline). Provide pcache.urlopen() as a transparent extension of urlopen()

Persistent Cache management and software installation become the same thing.

Remote Operation

Support DAWG remote query protocol as client, as currently done using SQL.

DAWG server.

Variations of DAWG using N3 where DAWG is not exactly N3.

IRC bot (sWBot) -- Your friendly N3-happy robotic chat room companion. See sandbox security below.

Security

Sandboxed version of cwm, which is immune to attacks from rules in data, builtin functions in untrusted rules, confientiality violations by indirect web access, and so on. While you can build a secure trusted system without needing a sandbox at all (it isn't really the way to build one) sandboxing is handy for things like demo servers and IRC bots.