Evaluation of training with an annotation schema for manual annotation of clinical conditions from emergency department reports

Abstract

Objective: Determine whether agreement among annotators improves after being trained to use an annotation schema that specifies: what types of clinical conditions to annotate, the linguistic form of the annotations, and which modifiers to include. Methods: Three physicians and 3 lay people individually annotated all clinical conditions in 23 emergency department reports. For annotations made using a Baseline Schema and annotations made after training on a detailed annotation schema, we compared: (1) variability of annotation length and number and (2) annotator agreement, using the F-measure. Results: Physicians showed higher agreement and lower variability after training on the detailed anno..