A frequently encountered problem in decision making is the following review problem : review a large number of objects and select a small number of the best ones. An example is selecting conference papers from a large number of submissions. This problem involves two sub-problems: assigning reviewers to each object, and summarizing reviewers' scores into an overall score that supposedly reflects the quality of an object. In this paper, we address the score summarization sub-problem for the scenario where a small number of reviewers evaluate each object. Simply averaging the scores may not work as even a single reviewer could influence the average significantly. We recognize that reviewers are not necessarily on an equal ground and propose the notion of "leniency" to model this difference of reviewers. Two insights underpin our approach: (1) the "leniency" of a reviewer depends on how s/he evaluates objects as well as on how other reviewers evaluate the same set of objects, (2) the "leniency" of a reviewer and the "quality" of objects evaluated exhibit a mutual dependency relationship. These insights motivate us to develop a model that solves both "leniency" and "quality" simultaneously. We study the effectiveness of this model on a real-life dataset.

This paper presents the problem that it is difficult to calculate reviewer scores that accurately reflect the quality of the object (i.e. a product) being reviewed. The authors note that reviewer scores are subjected to biases and they introduce the concept of leniency (which captures the tendency of a reviewer to give a higher or lower review score) to model these biases. Both the leniency of the reviewer and the quality of the object being reviewed are mutually dependent and must be determine simultaneously. These concepts are mapped into what the authors have termed as the, Differential Model. Results from this model is then compared against simply averaging reviewer scores for objects in a simulated test case and for video reviews on the Epinions website. The results showed that the Differential Model was able to differentiate the quality of highly ranked objects and there were greater differences between the rank of high quality objects when compared to employing the simple averaging approach. I feel that the authors have developed a theoretically sound model and could possibly be implemented in a user contribution measurement model that assesses the quality of reviewed content / contributions.

• Leniency of reviewers / raters
• Law of large numbers
• Data-centric approach
• Network approach to leniency
• Mutual dependency of quality and leniency
• Differential Model
• Exact and ranked solution
• Epinions