The Pineapple and the Hare. A fable, a current news piece, and an annotation challenge

There’s a spectacular set of annotation challenges lurking in the documents underlying this N.Y. Times piece about standardized reading tests for U.S. schoolchildren. Such tests are very controversial among U.S. educators, and this piece is amusing for the silliness of a particular question. The setup, quoted in the piece, is an extraction from a story by a well-liked children’s author. Also quoted are the questions, along with multiple choice answers, about various aspects of the narrative, allegedly to test the children’s ability to make deductions both about the story and what might be deduced from it.

The Times piece is a fun read, but—and I am serious about this—a good set of competency questions for an annotation ontology could at least focus on something, informally, like this:

• Annotate the test such that for each of the test questions, and for each of the possible answers, what evidence is presented in the extracted narrative for or against that answer.

There are probably other opportunities for competency questions, for example about annotation of the Times piece itself. One of the most interesting challenges of that will arise when you get to the part that seems to be an explanation from test designers of how it is determined what in fact the correct answers are.