Hostname: page-component-78c5997874-t5tsf Total loading time: 0 Render date: 2024-11-02T14:14:54.553Z Has data issue: false hasContentIssue false

The Talent system: TEXTRACT architecture and data model

Published online by Cambridge University Press:  11 October 2004

MARY S. NEFF
Affiliation:
IBM T.J. Watson Research Center, 19 Skyline Drive, Hawthorne, NY 10532, USA e-mail: maryneff@us.ibm.comroybyrd@us.ibm.combran@us.ibm.com
ROY J. BYRD
Affiliation:
IBM T.J. Watson Research Center, 19 Skyline Drive, Hawthorne, NY 10532, USA e-mail: maryneff@us.ibm.comroybyrd@us.ibm.combran@us.ibm.com
BRANIMIR K. BOGURAEV
Affiliation:
IBM T.J. Watson Research Center, 19 Skyline Drive, Hawthorne, NY 10532, USA e-mail: maryneff@us.ibm.comroybyrd@us.ibm.combran@us.ibm.com

Abstract

We present the architecture and data model for TEXTRACT, a robust, scalable and configurable document analysis framework. TEXTRACT has been engineered as a pipeline architecture, allowing for rapid prototyping and application development by freely mixing reusable, existing, language analysis plugins and custom, new, plugins with customizable functionality. We discuss design issues which arise from requirements of industrial strength efficiency and scalability, and which are further constrained by plugin interactions, both among themselves, and with a common data model comprising an annotation store, document vocabulary and a lexical cache. We exemplify some of these by focusing on a meta-plugin: an interpreter for annotation-based finite state transduction, through which many linguistic filters can be implemented as stand-alone plugins. The framework and component plugins have been extensively deployed in both research and industrial environments, for a broad range of text analysis and mining tasks.

Type
Papers
Copyright
© 2004 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)