RSP Benchmarking

At the F2F meeting in Portoroz [1], we discussed some metrics and RSP system features that should be taken into account at the time of evaluating an RSP engine. This is ongoing work and we invite members of the group to add their suggestions to the table.

RSP System metrics

Tests

• Soak testing addresses the system performance under the expected production load over a continuous period of time.
• Stress testing checks the response of the system under heavy loads.

Metrics

• Memory consumption
• Query execution time
• Query throughput
• Data throughput
• CPU usage
• Correctness of results
• Size of knowledge base
• Reasoning
• Caching

Input / Parameters

Query

• Number of joins
• Type of join
• Implies reasoning
• Number of streams
• Aggregation functions
• Selectivity
• Window size/slide

Data

• Variety of data (structure and values)

Stream

• Number of triples / graph
• Input rate

Background data

• Location (local vs remote)
• Size of the data:
  • storable in primary memory
  • storable in secondary memory

Existing works

RSP

• LSBench [2] - Paper [3]
• SRBench [4] - Paper [5]
• CSRBench [6] - Paper [7]
• YABench [8] - Paper [9]
• CityBench
• SP Bench

DSMS/CEP

• Linear road [10] - Paper [11]
• SenseBench [12]

Open points/Additional requirements

• Historical data