Important note: This Wiki page is edited by participants of the RDWG. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.


Paper Selection Criteria

From Research and Development Working Group Wiki
Jump to: navigation, search

This page defines some of the criteria that RDWG uses to select papers and contributions for the RDWG online symposia. It is intended as guidance for reviewers from RDWG. Please send comments and feedback to public-wai-rd@w3.org.

Generic instructions for reviewers

Papers should deal with important issues relating to practical application or implications of accessibility metrics. Papers must be technically sound in how they deal with these issues.

When reviewing, consider these points:

  • If the paper is mainly theoretical, it should be judged on the insightfulness of its ideas and the rigor of its arguments;
  • If it is mainly empirical, it should be judged on the validity of its results and the appropriateness of its techniques;
  • If it is mainly a methodology paper, it should be judged on the usefulness of its proposed methodology;
  • If it is a design paper, it should be judged on the quality, novelty, and appropriateness of its solution to a problem; the rationale for the design decisions; the implementation; and the assessment of its success and weaknesses.

Consider reading T. Roscoe's paper Writing Reviews for Systems Conferences for more information.

Rating schema

Use the following rating scheme with respect to the question "How much do you agree that this paper ranks high according to that particular criterion":

  1. disagree
  2. mild disagreement
  3. neutral
  4. mild agreement
  5. agreement

Make sure to explain your judgment.

Relevance/Appropriateness

  • Is the paper relevant?
  • Does it address a topic listed in the CFP?
  • Does this paper fit in the CFP?

For more information, please see the CFP.

Contribution/Significance

  • Is the work described a significant contribution?
  • Is the CFP target audience likely to learn from it?

Clarity/Presentation

  • Relative to the subject material, is the paper understandable without requiring too much effort on the part of the reader?
  • For the reasonably well-prepared reader, is it clear what was done and why? Is the paper well-written and well-structured?

The paper should have a clear and logical presentation, so as to be readable by the whole interdisciplinary accessibility audience and not just by specialists in some sub-field. Any interested accessibility reader should be able to learn something from the paper.

If an article is poorly written due to grammatical errors, it may make it more difficult to understand the content.

Structure

Is the article clearly laid out? Are all the key elements (where relevant) present?

  • Title: Does it clearly describe the article?
  • Problem addressed: Does it describe what the author hoped to achieve accurately, and clearly state the problem being investigated? Normally, the introduction should summarize relevant research to provide context, and explain what other authors' findings, if any, are being challenged or extended. It should describe the experiment, the hypothesis(es) and the general experimental design or method.
  • Background: Does it present relevant work? Does it provide one or more perspectives against which to contrast this paper?
  • Strategy: Does it explain the problem being tackled?
  • Major difficulties: Does it describe the major obstacles found during the process?
  • Outcomes: Does it explain the main outcomes, lessons learned, and mistakes made?
  • Open research avenues: Does it explain the future perspectives that the work opens?
  • Conclusion/Discussion: Are the conclusions and discussions clearly laid out and explained?

Soundness/Correctness

  • Does the paper have enough substance?
  • Is the technical approach sound and well-chosen?
  • Can one trust the claims of the paper -- are they supported by proper background and proper evaluation (with the target users, when appropriate)?

Make sure you also look at the correctness of statistics and correctness of claims.

Correctness of statistics and data

Results: This is where the author/s should explain in words what he/she discovered in the research. It should be clearly laid out and in a logical sequence. You will need to consider if the appropriate analysis has been conducted. Are the statistics correct?

Correctness of claims

Are the analyses and conclusions a logical outcome of the data and discussion?

Conclusion/Discussion: Are the claims in this section supported by the results, do they seem reasonable? Have the authors indicated how the results relate to expectations and to earlier research? Does the article support or contradict previous theories? Does the conclusion explain how the research has moved the body of scientific knowledge forward?

Impact

The paper will be judged on its impact in relationship to the work of the RDWG and the editors of the document - and Seminar - related to the topic under discussion. This maybe based on the level of originality and innovation of the work reported, its impact on the field, community, or beneficiaries. However, the deciding factor will be its impact and importance to the RDWG topic itself.

Generality of results/claims

  • How general are the claims?
  • How widely can they be applied?
  • What are the practical implications?

Relation with other work

The paper must be scholarly and not insular, i.e., it must establish its relation to other work in the field. How do the results or the methods or the conclusions discussed in this paper relate to others' findings?

Overall Recommendation

Provide your recommendation.

  • Accept
  • Accept with minor revisions
  • Reject

Reviewer's Confidence

  • (expert)
  • (high)
  • (medium)
  • (low)

Detailed comments

Summarize strengths and weaknesses.

Comments for the Program Committee

(Not sent to authors) Private comments to the chairs.