Abstract
Evaluation is of paramount importance in data-driven research fields such as Natural Language Processing (NLP) and Computer Vision (CV). Current evaluation practice largely hinges on the existence of a single ground truth against which we can meaningfully compare the prediction of a model. However, this comparison is flawed because more than one answer may be correct and disagreement among annotators is ubiquitous. We argue that current methods of adjudication, agreement, and evaluation need serious reconsideration. Rather than minimizing disagreement, we should capture its sources to improve evaluation practice. We discuss disagreement arising from annotators, data, and context, and argue that the growing use of multiply annotated datasets and disagreement-aware modeling should extend to evaluation.