2. Before corpora: rule-based systems

If we turn back the clock to 1994—25 years ago—and the start of this journal, we will find a new NLP technology being introduced to the wider world.

This announcement was based on a series of MUCs which had defined the task of IE and its evaluation. The conference series began in 1988 with invitations to a meeting (“MUC-1”) at NOSC (Naval Ocean Systems Command) to discuss how IE might be evaluated. In order to be able to compare systems, there was agreement on the need for a shared template capturing the most important information in a document. Systems would be judged on how accurately they filled these template slots. MUC-2 represented a trial run of such an evaluation; MUC-3 agreed on scoring using recall, precision, and F measure. (F measure, the harmonic mean of recall and precision, was suggested as the primary metric for assigning a rank to participating systems.)

MUC-1 and MUC-2 both used Navy exercise message traffic (“rainforms” and “opreps”) as the corpus. A typical message is as follows:

The template planned for MUC-2 is shown in Appendix A.

MUC-3 and 4 used news about terrorism in Latin America (Chinchor et al. Reference Chinchor, Hirschman and Lewis1993). A sample message is shown in Appendix B along with one of the filled templates generated from this message.

As the task got better defined, the number of participants grew. By MUC-5, in 1993 there were 16 participants, evenly divided between universities and companies (primarily defense contractors) (MUC 1993). The MUC tasks were getting larger in other respects as well. Participants in MUC-5 had a choice of two extraction topics (joint ventures or microelectronics) and two languages (English or Japanese). The templates were substantially more complex than in prior years.Footnote ^a Following MUC-5, the tasks were simplified to limit the effort required to participate. To emphasize faster development of IE systems for new domains, the time from the release of training material to the actual evaluation was reduced to one month. MUC-6 involved executive succession; MUC-7 involved rocket launches.

Participation in multiple MUCs had led to some convergence of extraction architecture, a long pipeline including some familiar names (Hobbs Reference Hobbs1993). It quickly became clear, for example, that a preprocessor to identify names was essential. But there were still basic areas of disagreement.

3. Supervised methods: ACE

4. Semi-supervised methods

ACE was a success in terms of producing annotated corpora and research results, but there were issues it did not address. In particular, it treated documents separately, whereas many realistic tasks involved large numbers of interrelated documents. Information about an individual may need to be pieced together from several documents. To address these questions, NIST (the US National Institute of Standards and Technology) organized the annual “Text Analysis Conference” and its central task, “Knowledge Base Population” (KBP) (Ji and Grishman Reference Ji and Grishman2011). Starting in 2009, the KBP task added additional components year by year. We will describe the “Cold Start” variant as of 2017, when the data sets were largely complete.

Participants were given a large collection of unannotated documents, a mix of newspaper articles and blogs, two to four million documents in each of English, Chinese, and Spanish. A small portion of these, 30,000 documents in each language, served as the test corpus; sites were expected to build a graph in which each node represented an individual, organization, GPE (Geo-Political Entity), location, or facility mentioned in the test collection. Associated with each type of node were a set of properties. whose value could be a number, a date, a string, or another node in the network. For example, a person node would have an age property whose value is an integer and whose city_of_birth was a GPE node.

In addition, sites had to link the entities to the arguments of events appearing in the test collection.

Compared to ACE, the test corpora were about two orders of magnitude larger. At this scale, complete manual annotation of the test corpus was not feasible. Scoring was done by sampling: NIST selected some names mentioned in the test corpus and checked whether (1) the system had created a node for this name and (2) the node had the desired property. Training documents for the various annotation tasks were minimal—small samples the first year a task was run, augmented in subsequent years by the annotations required for scoring.

The large volumes of unannotated data and the lack of annotated training encouraged experimentation with semi-supervised methods—learning from partially labeled data. Most direct was the generalization of the earlier work in MUC-4 to bootstrapping, an iterative strategy starting from a small labeled seed. Bootstrapping was successfully applied to scenario template (Yangarber et al. Reference Yangarber, Grishman, Tapanainen and Huttunen2000), named entities (Collins and Singer Reference Collins and Singer1999), and relations (Agichtein and Gravano Reference Agichtein and Gravano2000). However, success was not always assured; adding an incorrect element might lead the bootstrapping badly astray.

Participants were also provided with a large data base, BaseKB. This enabled researchers to explore an approach to training a relation classifier termed distant supervision (Mintz et al. Reference Mintz, Bills, Snow and Jurafsky2009). The basic idea of distant supervision is to convert an existing set of facts into an annotated corpus and then use the annotated corpus to train a classifier in the usual way. Suppose we have a database with a relation R consisting of pairs $\lt x_{1},y_{1} \gt, \lt x_{2}, y_{2} \gt, \ldots$, and that some of these pairs appear in the corpus separated by word sequences w _i. We will annotate every sequence w _i as expressing relation R.

The basic model makes strong assumptions which are not satisfied by realistic data. It assumes that if a pair $\gt x_{i}, y_{i} \gt$ matches a sentence in the corpus, then that sentence expresses relation R. Violations of this assumption lead to a noisy annotated corpus, with many false positives and false negatives. An alternative MIML (MultiInstance MultiLabel) model requires only that at least one instance of the pair represent a relation and that the pair may represent more than one relation label. This model leads on average to cleaner annotations (Surdeanu et al. Reference Surdeanu, Tibshirani, Nallapati and Manning2012). Further improvements can be made by combining the distant supervision with some manually annotated data (Pershina et al. Reference Pershina, Min, Xu and Grishman2014).

More radical approaches are also being tried, including few-shot methods and even zero-shot methods. These address the situation where you have an event extractor which can recognize N event types and now want to add the ability to recognize an N + first event type. In a few-shot method, a small amount of training data is provided; in a zero-shot method, no additional training data is provided. Huang et al. (Reference Huang, Ji, Cho, Dagan, Riedel and Voss2018) propose to ground the event types and event instances in a shared semantic space based on the arguments to the event and then, given a new event instance, assign it to the closest type. Levy et al. (Reference Levy, Seo, Choi and Zettlemoyer2017) convert a relation into a set of questions and then rely on a reading comprehension system to answer these questions.

Whether distant supervision can outperform hand-built patterns or supervised training depends on several factors. Preparing patterns by hand requires considerable skill and insight but may yield a relatively clean (high precision) system. The preparation of an annotated corpus may require less skill but more time. Distant supervision requires the least labor but may produce the noisiest model. Most likely the best method will involve some combination of these approaches.

5. Deep learning

Advances in deep learning (multilayer neural networks) have had a dramatic effect on all of NLP over the past few years; IE was no exception.

Neural networks provide a major advantage over the trainable models which preceded them (primarily maximum entropy models): given sufficient training data and time, they can capture arbitrary functions of their inputs. That means that they do not require manual feature engineering. On the other hand, the time factor may be significant; citing training times of one or two weeks is not unusual.

The most widespread change brought about by neural networks was the way in which words are represented. Although prior models had made some use of smoothing lexical dependencies, words were generally treated as discrete symbols. If a vector representation were required it would take the form of a sparse 1-hot vector. Practical neural networks, however, required a representation using continuous-valued, low-dimension vectors. In effect, each word is represented by a point in d-space, termed its word embedding. Several methods were developed which captured the semantic properties of the vocabulary, in particular that words which were semantically similar would appear close-by in d-space.

Wt note here some aspects of the deep learning IE models. The primary network types currently in use are CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks) using LSTMs (Long Short-Term Memories) (Yin et al. Reference Yin, Kann, Yu and Schütze2017).

5.0.0.1 Named entities. The best named entity performance is currently obtained by combining a dual token/character model with contextualized word embeddings (Akbik et al. Reference Akbik, Bergmann and Vollgraf2019) Performance on the standard test set (the Reuters newswire used by CONLL for the 2003 evaluation) has improved from an F measure of 89 in 2003 to an F of 93 (Li et al. Reference Li, Sun, Han and Li2018).

5.0.0.2 Relations. CNNs offer a particularly simple network structure but the convolution operates within a fixed window size, which may limit the ability to capture dependencies spanning the entire sentence. ACE relations are mostly realized at close range, with the entities separated by fewer than four words. This makes it reasonable to implement relation extraction using a CNN; Nguyen and Grishman (Reference Nguyen and Grishman2015) reported good results with windows of two, three, and four tokens.

5.0.0.3 Events. As noted above, event extraction can involve multiple interactions which may benefit from joint inference. In a neural network, these interactions can be captured directly through a set of “memory matrices” whose values are assigned as part of the network training and then used for event trigger and argument prediction (Nguyen et al. Reference Nguyen, Cho and Grishman2016).

Event extraction is in substantial part a matter of word sense disambiguation. But until recently each word was assigned a single word embedding and so did not capture sense distinctions. Contextualized word embeddings relax that constraint, making the embedding dependent on context. Using contextualized word embeddings on the ACE corpus improves event classification by about two points F-measure (Lu and Nguyen Reference Lu and Nguyen2018).

6. User-generated media

Another significant addition of the last few years was the processing of user-generated data. Twitter was founded in 2006; currently about 500,000,000 tweets are sent each day. Automatically monitored tweets provide a source of current activities second to none, so they have become a target for NLP developers (Panem et al. Reference Panem, Gupta and Varma2014). There is now a annual workshop on the analysis of such informal communication, WNUT (Workshop on Noisy User-Generated Text, web site http://noisy-text.github.io/).

But the tweets are quite different from the well-edited texts of newswires which had been the target of most NLP. The tweets may contain many variant spellings, little or no punctuation, and newly coined terms. In consequence, taggers which were trained on edited text performed poorly on tweets (e.g., a top-ranked named entity tagger which obtained an F score over 90% on the standard Reuters test corpus obtained an F score of about 40% on tweet corpora).

The WNUT workshops include an annual multi-site evaluation, but the performance of these tweet-optimized systems was not much better; top performance in the 2016 evaluation was F = 52% (Strauss et al. Reference Strauss, Toma, Ritter, de Marneffe and Xu2016). Generally the taggers used similar designs to those described above, principally CRFs and RNNs built using LSTMs. Because individual tweets provide much less context, tweet taggers must rely more on name lists (e.g., gazetters). Taking advantage of global consistency—a preference for assigning the same token the same tag in different tweets—is also important (Ritter et al. Reference Ritter, Clark, Mausam and Etzioni2011; Liu et al Reference Liu, Zhang, Wei and Zhou2011; Cherry and Guo Reference Cherry and Guo2015). (As we noted earlier, global consistency also plays a role, but a smaller one, in tagging edited text.)

7. Evaluation

At first glance, IE evaluation seems rather straightforward. We agreed already at MUC-3 to score using recall, precision, and F measure. We prepare a key and compare it to the IE system’s response.

\begin{equation*} recall = \frac{\mbox{number of slots correctly filled in system response}} {\mbox{number of slots filled in key}} \end{equation*}

\begin{equation*} precision = \frac{\mbox{number of slots correctly filled in system response}} {\mbox{number of slots filled in system response}} \end{equation*}

and then compute the F-measure using

\begin{equation*} 1 / F = \frac{1 / recall + 1 / precision}{2} \end{equation*}

It quickly became clear that things would not be so simple. Systems were supposed to generate one template per event; if a document reported two events, two templates should be filled. However, the system response did not explicitly specify how to pair up the templates in the key and response. To address this issue, possible alignments of key and response templates were generated and scored, and the maximum score was reported (MUC no date). A similar problem arose at a smaller scale if there were multiple participants in an event. In general this recall/precision model provided satifactory and intuitive scores when new tasks were added to MUC. The one exception was the coreference task. One scoring scheme was originally designed and an elegant alternative was proposed at the MUC conference, but neither seemed intuitive.Footnote ^d To this day there are disagreements regarding coreference scoring metrics (Luo Reference Luo2005).

When MUC was divided into four and later five tasks, each was given its own scoring metric, which made sense since each task might be used independently. The ACE evaluation, in contrast, was based on a set of parallel cost models for entities, relations, and events. Each model combined detection, classification, clustering (i.e., coreference), and additional features. The official score (“ACE value”) was based on all these factors, suitably weighted. A positive value is assigned to each element correctly recognized and a false alarm penalty is charged for each incorrect output. The score could be negative if the number of errors exceeded the number of correctly identified elements (Doddington et al. Reference Doddington, Mitchell, Przybocki, Ramshaw, Strassel and Weischedel2004). This is a standard ROC (Receiver Operating Characteristic) model but was not intuitive to the participants; in consequence, it was used for formal Government reports but little used in the published literature.Footnote ^e

In place of the cost model, most researchers report recall/precision scores for relations and events. These scores are highly dependent on the accuracy of entity extraction since only entities can serve as arguments of relations and events. To isolate improvements to relation and event extraction, most researchers assume that the relation or event extractor is provided with perfect information about entities. This has the benefit of producing higher (more optimistic) scores than running a real entity extractor.

With the shift to deep-learning taggers which are capable of representation learning, some researchers now assume the relation tagger has minimal information regarding entities—only their position in the sentence, not their semantic type. These shifts—reflecting changing research goals—must be taken into account when comparing tagger performance.

8. Looking ahead

We have briefly described the wide range of approaches that have been developed over the past 25 years for building IE systems, and the gradual rise in task performance which has accompanied the introduction of these approaches. The result is a growing set of applications in finance (Ding et al. Reference Ding, Zhang, Liu and Duan2015), medicine (Wang et al. Reference Wang, Wang, Rastegar-Mojarad, Moon, Shen, Afzal, Liu, Zeng, Mehrabi, Sohn and Liu2018), and science (Peters et al. Reference Peters, Zhang and Livny2014). Still, performance (F score) after more than 25 years of development has only advanced from the low 60s to the low 70s on standard event classification benchmarks, and there are serious obstacles to be faced in further improving the scores. What are our prospects?

1. (1) In some regards, the standard benchmarks (drawn from newswires and blogs) are particularly difficult because the range of topics is so broad, increasing the risk of event misclassification Most applications involve a narrower range of topics and so yield higher performance than the benchmarks.

2. (2) There will be errors and uncertainties in the human annotations which limit the score we can get. This applies even to texts carefully prepared using dual annotation and reconciliation, such as ACE corpora. Annotating relations requires identifying two endpoints which are easily missed. Relatively abstract categories will lead to uncertainties in classification for both relations and events (Min and Grishman Reference Min and Grishman2012). We should embrace this vagueness as part of the power of natural language and take account of it in our evaluations.

3. (3) There will be examples which require world knowledge and inference. For instance, the ACE events include a phone event (a subtype of contact). Given the sentence “Fred phoned Jim and he later returned the call.” the system must be able to infer that Jim later called Fred. Handling such cases properly may require a deeper modeling of the events. This is much more feasible in a narrow domain.

4. (4) Insufficient training data. We expect that we would get several percent improvement in event extraction just by doubling the amount of ACE training data. But “just” may not be an appropriate word when the data were a major government investment. Going forward, we could not afford similar investment for everyone who wants an IE system of their own. Here we may be saved by semi-supervised or unsupervised methods. At a minimum the unsupervised systems could provide cores of relation and event types, which then can be extended and adjusted for particular users, using some form of domain adaptation.

5. (5) Pipeline problems. IE remains a multi-stage process where earlier stages may introduce errors which are magnified by later stages. Joint inference strategies can reduce this effect.

And we should keep in mind that deep learning is still a young technology from which we can expect continuing improvements in machine learning just as the advent of Bidirectional Encoder Representations from Transformers (BERT) and contextualized embeddings has given many systems a boost of late (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2018). So our prospects for continued improvement seem pretty good.

As performance improves, the number of applications which become commercially viable will continue to grow. To maintain market share for their platforms, every one of the “tech giants” (along with multiple start-ups) is now counted on to provide an NLP API including all of the elements of the pipeline, and to update it steadily, bringing state-of-the-art NLP components much closer to IE applications. In this market-driven environment there may be less demand for the Government to guide research by funding fresh evaluations.