Warning:
This wiki has been archived and is now read-only.

Submitting Test Results

From RDB2RDF

(Redirected from Submitting Implementation Reports)

Jump to: navigation, search

1 Overview
2 Phase 1. Download the Test Suite
3 Phase 2. Run the engine over the TS
- 3.1 R2RML
- 3.2 DM
4 Phase 3. Download the Test Harness software
5 Phase 4. Run the TH software against your Test Results
6 Phase 5. Upload the Test Report
7 Phase 6. Generate the Implementation Report
8 Notes
9 Annex
- 9.1 BASH script to compare tests

Overview

The Working Group is developing a Test Suite (TS). The test cases that comprise the Test Suite are described here

In this page we describe the process of creating the RDB2RDF R2RML/DM Implementation Reports.

This process consists of six phases.

The following figure depicts the whole process.

Error creating thumbnail: Unable to save thumbnail to destination

Phase 1. Download the Test Suite

Implementors should to download the zip file that contains the TS.

Alternatively, implementors should clone the mercurial repository that contains the TS.

The test cases are organized by database. Each database has its own folder. The folder's name includes a code (e.g., D001) and description of the database (e.g., D000-1table0rows). Within each folder we have:

a manifest.ttl file that contains a description of the database, direct graph, r2rml mappings, and outputs.
a create.sql file that contains the sql script for the creation of the database.
a directGraph.nt that contains the direct mapping for the given database.
one or more r2rml.ttl files that include the r2rml mappings for the database.
one or more mapped.nq files that contain the resultant graph for the corresponding r2rml.ttl file.

Phase 2. Run the engine over the TS

Once the implementors have the TS, it is time to run the engine over the test cases.

R2RML

R2RML implementors should scan all the directories included in the TS and do the following for each one of them:

load the sql script (create.sql) in the given DBMS.
load the r2rml mapping (r2rml.ttl) file.
execute the r2rml mapping over the database loaded.
generate the output RDF dataset and include the content in N-quads format in the mapped-<yourtoolname>.nq, for example, mapped-d2r.nq

DM

DM implementors should do the following:

load the sql script (create.sql) in the given DBMS
generate the direct mapping for the database and include the output in TURTLE format in directGraph-<your-tool-name>.ttl, for example directGraph-morph.ttl

Phase 3. Download the Test Harness software

The Working Group is developing a Test Harness (TH) that allows comparison of test results from a given implementation against the official test results. The TH software is available here.

Phase 4. Run the TH software against your Test Results

After you have downloaded the TH software, you have to do the following

Unzip it to the local file system.

Modify the ts.ttl file, including the description of your tool, developer , the dbms used in the tests, and the location where the Test Suite with test results are stored, for example

<ts> a rdb2rdftest:TestSuite;
	dcterms:title "Test Suite for Direct Mapping & R2RML" ;
	dcterms:identifier "TS-01" ;
	rdb2rdftest:workingDirectory "./local" .
	

<myProject>
        a doap:Project ;
        doap:name "morph" ;
        doap:homepage <https://github.com/boricles/morph>  ;
        doap:programming-language "Java"  ;
        doap:developer <http://boris.villazon.terrazas.name/me> ;        
        rdb2rdftest:dbms r2rml:Oracle ;
        rdb2rdftest:implementsDirectMapping false ;
        rdb2rdftest:implementsR2RML true .
        
<http://boris.villazon.terrazas.name/me>
		a foaf:Person ;
		foaf:mbox <mailto:boris.villazon@terrazas.name> ;
		foaf:name "Boris Villazon Terrazas" .

For the dbms you should use the URIs available here

Run the rdb2rdf-th script with the manifest file as the argument, for example

./rdb2rdf-th ts.ttl

After the execution the script generates the earl-toolname-dm.ttl/earl-toolname-r2rml.ttl file that contains the description of the test results. The content of the file looks like this

<http://mappingpedia.org/rdb2rdf/myTestHarness/morph>
      a       <http://www.w3.org/ns/earl#Software> .

[]    a       <http://www.w3.org/ns/earl#Assertion> ;
      <http://purl.org/NET/rdb2rdf-test#dbms>
              <http://www.w3.org/ns/r2rml#Oracle> ;
      <http://www.w3.org/ns/earl#assertedBy>
              <http://mappingpedia.org/rdb2rdf/myTestHarness/morph> ;
      <http://www.w3.org/ns/earl#result>
              [ a       <http://www.w3.org/ns/earl#TestResult> ;
                <http://purl.org/dc/elements/1.1/date>
                        "2012-04-03 14:45:59"^^<http://www.w3.org/2001/XMLSchema#date> ;
                <http://www.w3.org/ns/earl#outcome>
                        <http://www.w3.org/ns/earl#pass>
              ] ;
      <http://www.w3.org/ns/earl#subject>
              <http://mappingpedia.org/rdb2rdf/myProject/morph> ;
      <http://www.w3.org/ns/earl#test>
              <http://www.w3.org/2001/sw/rdb2rdf/test-cases/#DirectGraphTC0005> .

Phase 5. Upload the Test Report

Within this step, implementors should provide a link for the Test Report, e.g., earl-toolname-dm.ttl/earl-toolname-r2rml.ttl file, to the rdb2rdf mailing list.

Phase 6. Generate the Implementation Report

The working group will collect the test reports of all the available implementors of R2RML/DM and generate an implementation report similar to this.

Notes

We are creating a matrix that shows the coverage of the R2RML TC and DM TC.

Annex

BASH script to compare tests

What this script does:

for each directory in the test suite:
- load the SQL script into the database (tested with PostgreSQL and MySQL)
- query the manifest.ttl graph to extract info regarding each test-case
  - run each test-case using XSPARQL
  - compare the result of the XSPARQL query to the expected output using JENA's 'rdfcompare'

#!/bin/bash

# XSPARQL config
if [ -z $XSPARQLHOME ]; then
    XSPARQLHOME=`pwd`
fi
export JENAROOT=/Users/nl/work/deri/sw/Jena-2.6.4/
RDFCOMP=$JENAROOT/bin/rdfcompare

CLASSPATH="$CLASSPATH:$XSPARQLHOME/cli/target/cli-0.3-SNAPSHOT-bin/libs/*"
OPTS=

EXAMPLESDIR=rdb2rdf/rdb2rdf-tests/
PSQL="psql91"
MYSQL="mysql5"

function usage() {
    echo "USAGE: $0 [-mysql|-psql]";
    exit 1;
}

function read_config () {
# read configuration from properties file
    MKTEMP="mktemp -t tmp.XXXXXXXXXX"
    TEMPFILE=$($MKTEMP)
    cat $XSPARQLHOME/$DBCONFIG |\
sed -e 's/"/"/'g|sed -e 's/=\(.*\)/="\1"/g'>$TEMPFILE
    source $TEMPFILE
    rm -f $TEMPFILE

    dbName=rdb2rdf_tests
}

# ============== command line processing

GREP="."
XSPARQLOPTS=""

# check if there is an argument from the command line
if [ $# -gt 0 ]; then
    if [ $1 == '-mysql' ]; then
        DBCONFIG=mysql.properties
        read_config 
        DB="$MYSQL -u $dbUser"
        if [ "${dbPasswd-x}" != "x" ] ; then
            DB="$DB -p$dbPasswd"
        fi
        DEFAULTDB=mysql
        XSPARQLOPTS="-mysql"
    fi
    if [ $1 == '-psql' ]; then
        DBCONFIG=psql.properties
        read_config
        DB="$PSQL -U $dbUser"
        DEFAULTDB=postgres
        XSPARQLOPTS="-psql"
    fi

    shift 1

    if [ -n $1 ]; then
        GREP="$1"
        shift 1
    fi
else
    usage
fi

XSPARQL="java $OPTS -cp $CLASSPATH org.deri.xsparql.Main "
XSPARQLRDB="$XSPARQL -dbConfig $XSPARQLHOME/$DBCONFIG $XSPARQLOPTS" 

MKTEMP="mktemp -t tmp.XXXXXXXXXX"
TMPFILE=$($MKTEMP)

failed=0
ntests=0

function compare_result() {
    # function arguments
    RES=$1
    MAP=$2/$3

    # compare the datasets
    $RDFCOMP $TMPFILE $MAP N3 N3 &> /dev/null
    COMP=$?

    # no result expected
    if [ -z $3 ] ; then
        if [ $COMP -ne 0 ]  &> /dev/null
        then 
	    echo -e " PASS (negative test)"
        else
	    echo -e " FAIL (negative test)"
	    let failed++
        fi
    else
        if [ $RES -eq 0 ] && [ $COMP -eq 0 ]  &> /dev/null
        then
	    echo -e " PASS"
        else
            if [ $COMP -eq 255 ]; 
            then
	        echo -e " NQUAD"
            else
	        echo -e " FAIL"
	        let failed++
            fi
            
            echo -e ">>>> $MAP"
            cat $MAP
            echo "<<<<"
            echo ">>>> XSPARQL"
            cat $TMPFILE
            echo -e "<<<<\n\n\n"
        fi
    fi
}


function test_r2rml() {
    $XSPARQL <<EOF > $TMPFILE
prefix rdb2rdftest: <http://purl.org/NET/rdb2rdf-test#> 
prefix dcterms: <http://purl.org/dc/elements/1.1/> 

let \$res := <res>{
for * from <file:$1>
where { \$a a rdb2rdftest:R2RML;	
            dcterms:identifier \$id; 
            rdb2rdftest:mappingDocument \$map .
        optional { \$a rdb2rdftest:output \$output }
}
order by \$map
return <row><output>{fn:concat("""", \$output, """")}</output><ident>{fn:concat("""", \$id, """")}</ident><map>{fn:concat("""", \$map, """")}</map></row>
}</res>
return 
 fn:concat("local OUTPUTS=(",fn:string-join(\$res//output, " "), ")

local IDENTS=(", fn:string-join(\$res//ident, " "), ")

local MAPS=(", fn:string-join(\$res//map, " "),")
")
EOF
    source $TMPFILE

    SIZE=${#MAPS[@]}
    for (( i=0; i<$SIZE; i++ )); do
        local MAP=${MAPS[$i]};
        local IDENT=${IDENTS[$i]};
        local OUTPUT=${OUTPUTS[$i]};

        echo -ne "        $IDENT: `dirname $1`/$MAP .... "
        let ntests++
        $XSPARQLRDB -dbName $dbName -r2rml `dirname $1`/$MAP > $TMPFILE 2>/dev/null
        RES=$?
        
        sed 's/>/>/g' < $TMPFILE >$TMPFILE.2
        sed 's/</</g' < $TMPFILE.2 >$TMPFILE
        rm $TMPFILE.2

        compare_result $RES `dirname $1` $OUTPUT

    done


}

function test_direct_mapping() {
    
    # check direct mapping
    $XSPARQL <<EOF > $TMPFILE
prefix rdb2rdftest: <http://purl.org/NET/rdb2rdf-test#> 
prefix dcterms: <http://purl.org/dc/elements/1.1/> 

for * from <file:$1>
where {\$a a rdb2rdftest:DirectMapping;	rdb2rdftest:output \$output; dcterms:identifier \$id}
return fn:concat("local OUTPUT=""",fn:data(\$output), """
local IDENT=""", \$id,"""
")
EOF
    source $TMPFILE

    if [ -z "$IDENT" ]; then
        return 0;
    fi

    echo -en "        $IDENT ... "
    let ntests++
    $XSPARQLRDB -dbName $dbName -dm "" > $TMPFILE 2>/dev/null
    RES=$?

    sed 's/>/>/g' < $TMPFILE >$TMPFILE.2
    sed 's/</</g' < $TMPFILE.2 >$TMPFILE
    rm $TMPFILE.2

    compare_result $RES `dirname $1` $OUTPUT
  
}

function test_file () {
    # create the database
    $DB $DEFAULTDB <<EOF &> /dev/null
DROP DATABASE IF EXISTS $dbName ;
CREATE DATABASE $dbName;
EOF

    # read SQL script file
    SQL=$($XSPARQL <<EOF 
prefix rdb2rdftest: <http://purl.org/NET/rdb2rdf-test#> 

for * from <file:$1>
where {\$a rdb2rdftest:sqlScriptFile \$c}
return fn:data(\$c)
EOF
)
    if [ -z "$SQL" ]; then
        return 0;
    fi

    # create database contents from SQL script file
    $DB $dbName < `dirname $1`/$SQL &> /dev/null

    # echo "  => Direct Mapping"
    # test_direct_mapping $1

    echo "  => R2RML"
    test_r2rml $1

    
     # delete the database
     $DB $DEFAULTDB <<EOF &> /dev/null
 DROP DATABASE $dbName;
 EOF

}

function test_files() {
    FILES=$(find $EXAMPLESDIR -iname manifest.ttl | grep  "$GREP" );
    for FILE in $FILES
    do
	echo -e "\n===> Manifest: $FILE"
	test_file $FILE
    done
}


# test manifest files
test_files

# cleanup
rm -f $TMPFILE

echo ========== XSPARQL tests completed ==========

echo Tested $ntests queries
echo $failed failed

echo =============================================

exit $failed

Retrieved from "https://www.w3.org/2001/sw/rdb2rdf/wiki/index.php?title=Submitting_Test_Results&oldid=1852"

Submitting Test Results

Contents

Overview

Phase 1. Download the Test Suite

Phase 2. Run the engine over the TS

R2RML

DM

Phase 3. Download the Test Harness software

Phase 4. Run the TH software against your Test Results

Phase 5. Upload the Test Report

Phase 6. Generate the Implementation Report

Notes

Annex

BASH script to compare tests

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Navigation

Tools