Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, Rion Snow, Brendan O'Connor, Daniel Jurafsky, Andrew Y. Ng, 2008Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP) - Examines the effectiveness of non-expert annotations for natural language processing tasks, offering practical insights into task design, annotator training, and quality assessment, including inter-annotator agreement.
A Coefficient of Agreement for Nominal Scales, Jacob Cohen, 1960Educational and Psychological Measurement, Vol. 20 (SAGE Publications)DOI: 10.1177/001316446002000104 - The seminal paper introducing Cohen's Kappa, a widely used statistical measure for inter-annotator agreement, essential for evaluating the reliability of human qualitative assessments.