================================================
Abstract - Submission for CLIN 2003; 3.11.2003 
[Submission form on CLIN homepage]
================================================

A Framework for the Evaluation of Manual Annotations

Petra Saskia Bayerl, Giessen


Despite efforts to automatize annotations, manual annotations are still 
a major source of information in many areas of computational linguistics. 
The evaluation of their quality therefore is an important task for securing
the quality and reliability of resulting data. Nonetheless, a systematic framework 
for evaluations and analyses seems not yet established. For instance, methodological 
standards for calulating reliability or carrying out evaluation studies seem not 
yet to be thoroughly defined. Hence, in order to develop a more systematic approach to 
the task of evaluating manual annotations, it seems fruitful to start with a 
theoretical consideration of basic influencing factors and their impact on annotations. 
Furthermore, methodological requirements of evaluation studies, basing mainly 
on theoretical assumptions and findings grounded in social sciences, will be 
discussed.