Hostname: page-component-cd9895bd7-fscjk Total loading time: 0 Render date: 2024-12-25T01:42:17.346Z Has data issue: false hasContentIssue false
  • English
  • Français

On the subjectivity of human-authored summaries*

Published online by Cambridge University Press:  01 April 2009

BALAKRISHNA KOLLURU  and
YOSHIHIKO GOTOH
BALAKRISHNA KOLLURU
Affiliation:
Department of Computer Science, University of Sheffield, Sheffield S1 4DP, United Kingdom e-mail: b.kolluru@dcs.shef.ac.uk, y.gotoh@dcs.shef.ac.uk
YOSHIHIKO GOTOH
Affiliation:
Department of Computer Science, University of Sheffield, Sheffield S1 4DP, United Kingdom e-mail: b.kolluru@dcs.shef.ac.uk, y.gotoh@dcs.shef.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

Human-generated summaries are a blend of content and style, bound by the task restrictions, but are ‘subject to subjectiveness’ of the individuals summarising the documents. We study the impact of various facets that cause subjectivity such as brevity, information content and information coverage on human-authored summaries. The scale of subjectivity is quantitatively measured among various summaries using a question–answer-based cross-comprehension test. The test evaluates summaries for meaning rather than exact words based on questions, framed by the summary authors, derived from the summary. The number of questions that cannot be answered after reading the candidate summary reflects its subjectivity. The qualitative analysis of the outcome of the cross-comprehension test shows the relationship between the length of a summary, information content and nature of questions framed by the summary author.

Type
Papers
Information
Natural Language Engineering , Volume 15 , Issue 2 , April 2009 , pp. 193 - 213
Copyright
Copyright © Cambridge University Press 2008

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Boret, M. and Peyrot, J. 1969. Le Résumé de Texte: Épreuve d'une Pédagogie Nouvelle. Paris: Chotard et Associés.Google Scholar
Cieri, C., Graff, D., Liberman, M., Martey, N. and Strassel, S. 1999. The TDT-2 text and speech corpus. In Proceedings of DARPA Broadcast News Workshop, Herndon, VA, pp. 57–60.Google Scholar
Cohen, J. 1968. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37–46.Google Scholar
Cremmins, E. T. 1982. The Art of Abstracting. Philadelphia, PA: ISI Press.Google Scholar
Dang, H. and Lin, J. 2007. Different structures for evaluating answers to complex questions: pyramids won't topple, and neither will human assessors. In Proceedings of the ACL 2007, Prague, pp. 768–775.Google Scholar
Endres-Niggemeyer, B. and Neugebauer, E. 1998. Professional summarizing: no cognitive simulation without observation. Journal of the American Society for Information Science 49 (6): 486–506.3.0.CO;2-Y>CrossRefGoogle Scholar
Hirschmann, L., Burger, J., Palmer, D. and Robinson, P. 1999. Evaluating content extraction from audio source. In Proceedings of the ESCA Workshop: Accessing Information in Spoken Audio, Cambridge, UK, pp. 54–59.Google Scholar
Hovy, E., Lin, C.-Y., Zhou, L. and Fukumoto, J. 2006. Automatic summarization evaluation with basic elements. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), Genoa.Google Scholar
Jing, H. 2002. Using hidden Markov modeling to decompose human-written summaries. Computational Linguistics 28 (4): 527–543.Google Scholar
Kintsch, W. and van Dijk, T. A. 1978. Toward a model of text comprehension and production. Psychological Review 85 (5): 363–394.CrossRefGoogle Scholar
Kolluru, B., Christensen, H. and Gotoh, Y. 2005. Multi-stage compaction approach to broadcast news summarisation. In Proceedings of Interspeech 2005, Lisbon, pp. 69–72.Google Scholar
Kolluru, B. and Gotoh, Y. 2007. Relative evaluation of informativeness in machine generated summaries. In Proceedings of Interspeech 2007, Antwerp, pp. 1338–1341.Google Scholar
Lin, C.-Y. and Hovy, E. 2003. The potential and limitations of automatic sentence extraction for summarization. In Proceedings of the HLT-NAACL 2003 Workshop on Automatic Summarization, Edmonton, pp. 73–80.Google Scholar
Lin, J. and Demner-Fushman, D. 2006. Will Pyramids built of nuggets topple over? In Proceedings of HLT/NAACL 2006, New York, pp. 383–90.Google Scholar
Mani, I. 2001. Automatic Summarization. Jon Benjamins Publishing Company, Amsterdam.Google Scholar
Nenkova, A. and Passonneau, R. 2004. Evaluating content selection in summarization: the pyramid method. In Proceedings of the HLT-NAACL 2004, Boston, pp. 145–152.Google Scholar
Nenkova, A., Passonneau, R. and McKeown, K. 2007. The Pyramid method: incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing 4 (2), article 4.Google Scholar
PintoMolina, M. Molina, M. 1995. Documentary abstracting: toward a methodological model. Journal of the American Society for Information Science 46 (3): 225–234.Google Scholar
Radev, D. R., Jing, H., Styś, M. and Tam, D. 2004. Centroid-based summarization of multiple documents. Information Processing and Management 40, 919–938.CrossRefGoogle Scholar
Rowley, J. E. 1988. Abstracting and Indexing, 2nd ed.London: Bingley.Google Scholar
Teufel, S. and van Halteren, H. 2004. Evaluating information content by factoid analysis: human annotation and stability. In Proceedings of the EMNLP 2004, Barcelona, pp. 419–426.Google Scholar
Van Halteren, H. and Teufel, S. 2003. Examining the consensus between human summaries: initial experiments with factoid analysis. In Proceedings of the HLT-NAACL 2003 Workshop on Automatic Summarization, Edmonton, pp. 57–64.Google Scholar
Vanderwende, L., Banko, M. and Menezes, A. 2004. Event-centric summary generation. In Proceedings of DUC 2004, Boston, pp. 76–81.Google Scholar