Reliability of discourse structure annotation

To investigate the prosodic expression of text structure, a theoretically motivated coding system for text structure is needed. Rhetorical Structure Theory (Mann & Thompson, 1988) is a descriptive theory of the structural organization of texts. It characterizes the structure of a text in terms of relations that hold between parts of that text. One of the main criticisms towards RST is that it is a subjective method. A prerequisite for using RST to describe the structure of a text is that RST can be reliably applied. Therefore, this study investigates to what extent six RST experts agree with each other in applying RST. The agreement is measured by Kendall's W. The current report describes the analyses of one text only. The preliminary results revealed that the way human analysts used RST is reproducible, since the reliability of RST is quite high. At the global level of the organization of the text the analysts agreed strongly. They disagreed at the local level , but they did it in a systematic and traceableway. These cases of disagreement could be explained in terms of observable text properties.