Use Case Metrics for Auto Assessment

Owner

Jeff Waters

Decision Aspects

• Primary: Measurement
• Secondary: Assessment

eXtreme Design Components

There is a method proposed specifically for reusing Content Ontology Design Patterns, called eXtreme Design (XD). See http://ceur-ws.org/Vol-516/pap21.pdf. In summary, the idea is to use an agile and iterative approach for developing ontologies through the reuse of component patterns. Also see the Ontology Design Pattern Tour. The Context, Story and Competency Questions below are designed to support this XD process.

Context

An organization or individual wants to specify the metrics used in making a decision and have those metrics support an initial automated assessment of the options.

Story: Metrics Support an Initial Automated Assessment

A user needs to make a decision, such as a computer purchase. There are several options which must be evaluated. The user wishes to specify the metrics (such as cost, hard disk space, warranty length, etc.) to be used in the assessment. The user wishes to specify these metrics in a manner which allows the user to specify the thresholds for what is allowed as an option (e.g. cost < $1500, hard disk space > 500GB, warranty > 1 year) and a weighting of the metrics based on user preference or other criteria (e.g. cost may be twice as important as hard disk space). The user wants to combine these metrics in a manner which allows the options to be rated on one overall scale (e.g. 1 (bad) to 100 (great)). In essence, the user wants to combine multiple measures (such as cost, quality, warranty) into one overall measure for purposes of rating the options and making a decision. The organization or individual would like the metrics to support an automated assessment of the options. In other words, each metric should be normalized in such a fashion that an automated assessment (scoring) of the options is possible resulting in a filtered and ordered set of options representing the best options for answering the decision question. The metrics should be easily configurable, reusable and sharable.

Competency questions (CQs) and contextual statements of decision information flow

  1.  What will be (are) (were) the metrics for this decision?
  2.  What are the weights given to the metrics?
  3.  What are the thresholds for determining which options to consider?
  4.  Is this metric to be considered better if higher (e.g hard disk space) or if lower (e.g. cost)?
  5.  How will these metrics be combined to generate an overall score for each option?
  

Background and Current Practice

This use case is derived from the need to make and document reasoned, rational decisions. Often a decision is made and there is no record of what metrics were used in the process and how they were weighted and what thresholds were considered. The result is that the decisions may appear arbitrary or ill-considered. Currently, decisions are either not represented in a computer format, or are represented in a proprietary format, or are represented in a format consisting largely of free text (such as e-mails, reports, or slide presentations). Most decisions and decision processes are poorly recorded with little attempt for machine understandability. The information which goes into a decision is usually not well documented. The ability to easily access and reuse decision components, such as metrics, is not well supported. The rationale behind a decision is often not apparent to those not involved in the decision process. Although complex decision-making cannot be boiled down into a simple formula, the attempt to define and quantify metrics, at least as a first cut approximation, is well worth the effort. In addition, more automated processes are being developed and a format for representing decision processes by computers are important to document, manage, archive and reuse in a standardized manner.

Goal

The goal is to ensure that any proposed decision standard format can effectively represent the metrics used to evaluate the options, including the weighting and thresholds given to those metrics by the decision-makers.

Use Case Scenarios

An individual would like to rate computers based on purchase price, speed, hard disk space, ram and warranty. A user would like to rate cities based on climate, location, population, quality of hospitals, and responsiveness of emergency services in order to decide where to retire. A user would like to rate earthquakes in terms of their significance for a weekly news report. A city would like to rate its record of disaster relief funding received over the last 10 years in comparison with cities of similar size with similar levels of natural disasters so it can decide whether to employ additional resources on a specific grant application.

Problems and Limitations

Objectifying subjective metrics, normalizing metrics to a common scale.

Existing Work

[1] A basic overview of metrics but in the context of product development, http://www.humiliationstudies.org/documents/Beaumontmetricsid.pdf.