File:Monitoring.docx

Title

Open Data Monitoring and Benchmarking

Short Description

Open data initiatives should be measurable in order to be enhanced. Indicators and methodology should be transparent and universal, allowing external assessment, and enabling comparison among initiatives.

In order to evolve and enhance Open Data initiatives we should be able to assess its quality. Thus we can carry out actions and make assessments based on availability, access, quality, reuse, impact and so on. Tools like the Open Knowledge Global Open Data Index, Open Data City Censuses the PSI Scoreboard and Open Data Monitor and Open Data Barometer can be used as an indicator of the current state of play or the development of open data initiatives over time (positive or negative). These tools rank countries, and cities offering a reference to be able to compare the good and bad aspects of initiatives. Publishers should also be able to view these benchmarking tools and use the information to improve their open data publishing practices.

It may be interesting to rank countries, cities and then to also rank regions, or according to thematics such as transport and pollutants to illuminate where the best open data practices can serve as exemplars for others.

The Global Open Data Index measures the openness of initiatives depending on civil society or citizen perception of level of publication of specific and basic datasets from official sources (transport, registry of business, etc.). Getting a higher rank in the table is a motivation for governmental representatives to keep on doing the work.

Why?

Sometimes it is possible to measure reuse of open data, but most of the time, it is not obvious if applications of services use public information, so you cannot quantify this way. If you cannot quantify it, you cannot measure the potential enhancements.

Benchmarking tools are effective ways of encouraging a healthy competition between countries, cities and regions to improve their ranking year by year. But further work can be done to ensure that we make methodologies more transparent, to share common assessment frameworks, to share the definition of open according to the open definition from Open Knowledge and perhaps to develop these measurements further by cities and by regions and by dataset thematics.

Intended Outcome

Every open data initiative should be measured and evaluated using common, universal indicators where it is sensible to do so. Establishing some baseline indicators for each assessment would make these assessments more comparable (where data can be shared between assessments).

Relationship to PSI-Directive

6. A minimum harmonisation is required to determine what public data are available for re-use in the internal information market

27. Public sector information scoreboard

28. Review by an impartial review body

Possible Approach

If we do not have common or universal indicators, we are not able to compare between the different projects measuring the state of open data.

There is work on common assessment frameworks between some of these organisations measuring open data that will be presented at the Open Data Conference in Ottawa in May 2015.

How to Test

Ideally, in the future machines should be able to compare initiatives through the score of initiatives.

Methodology and indicators should be transparent and clear to human analysis. Where it has been possible to share common indicators across these open data measurement tools, this should also be made obvious when assessments are made against this best practice.

Evidence

Some indices to take into account:

* http://index.okfn.org

* http://www.epsiplatform.eu/content/european-psi-scoreboard

* http://opendatabarometer.org

* http://www.opendatamonitor.eu

Tags

index, scoreboard, ranking, score, indicator, feedback, measure, management,

Status

Draft

Intended Audience

data managers, senior officials, organisations measuring open data, strategy makers, policy makers

Related Best Practice

Contact

emma.beer@okfn.org