2.1 Comparing Twitter and newswire as news sources

Two similar works to this study are Petrovic et al. (Reference Petrovic, Osborne, McCreadie, Macdonald, Ounis and Shrimpton2013), Verma et al. (Reference Verma, Karimi, Lee, Gnawali and Shakery2019), which compare Twitter and newswire, as two main sources of events information, from different viewpoints. In these papers, the authors compare the events covered in Twitter and newswire and the timeliness of these two sources in news reporting. The type of events that are reported on Twitter before newswire is also investigated. The main similarity between these papers and our work is extracting events from two different sources published contemporaneously.

The main difference is that in Petrovic et al. (Reference Petrovic, Osborne, McCreadie, Macdonald, Ounis and Shrimpton2013), Verma et al. (Reference Verma, Karimi, Lee, Gnawali and Shakery2019), the authors compare the event-related content on Twitter and newswire while in this work, the joint information in both resources is utilized in a framework to improve the event detection performance.

Two other differences between Petrovic et al. (Reference Petrovic, Osborne, McCreadie, Macdonald, Ounis and Shrimpton2013), Verma et al. (Reference Verma, Karimi, Lee, Gnawali and Shakery2019) and the present work are (1) how the events are represented and (2) how the similarity between tweet events and newswire events is calculated. In Petrovic et al. (Reference Petrovic, Osborne, McCreadie, Macdonald, Ounis and Shrimpton2013), tweets and newswire events are represented by the sum of all document vectors that correspond to the cluster (event). The similarity values between tweet events and newswire events are computed using cosine similarity between their vectors. Also, the nearest neighbor method is used to find the events (clusters) that are reported on both Twitter and newswire. In Verma et al. (Reference Verma, Karimi, Lee, Gnawali and Shakery2019), each pair of tweet and news article is represented by a set of features and for finding similar pairs, a classification model is learned using a training set with matching labels.

In McCreadie, Macdonald, and Ounis (Reference McCreadie, Macdonald and Ounis2013), a user study is conducted on the use of news articles and user-generated content as two sources of event-related information in news retrieval and aggregating the two sources in the retrieval results to satisfy end-users.

Another study comparing news stream and tweets in reporting events is in Mele et al. (Reference Mele, Bahrainian and Crestani2019). The authors propose a discrete dynamic topic modeling and Hidden Markov Model for event detection. In their next step, the detected events are used for clustering news documents, and finally, the tweets’ and news articles’ timeliness in reporting events is analyzed.

Next, we review the relevant previous work on discovering the linkage between these two sources of online event-related content, that is tweets and news articles.

2.2 Tweet–news linkage discovery

One method that addresses the task of exploiting correlations between tweets and news is proposed in Guo et al. (Reference Guo, Li, Ji and Diab2013). The authors propose a graph-based latent variable model to model text-to-text correlations. For this goal, they employ the named entities of news articles, as news specific features, hashtags of tweets, as tweet specific features, and the temporal information in both genres to exploit correlations between texts.

In Shi et al. (Reference Shi, Ifrim and Hurley2014), the authors introduce a framework for connecting news articles to Twitter conversations. The method uses keyword similarity to separate tweets per article. Local cosine similarity, global cosine similarity, local frequency of hashtags, and global frequency of hashtags are the features extracted for each article–hashtag pair for classification. These features are used in a series of Weka classifiers including MultilayerPerceptron, regularised logistic regression, and K^* to retrieve all the hashtags relevant to each article.

In the study, Abel et al. (Reference Abel, Gao, Houben and Tao2011b), the task of linking tweets with news articles is investigated for constructing user profiles. Here, two sets of strategies to find relevant news articles to each tweet are proposed: URL-based strategies and content-based strategies. In content-based strategies, the similarities between hashtag-based, entity-based, and bag-of-word-based representations of tweets and news articles are computed to discover their relationships.

Next, we discuss work on employing tweets and news articles for two news-related tasks including controversy detection and news recommendation.

2.3 Controversy detection in news and events

Another use of newswire in event-related tasks is detecting controversial events from Twitter. In Popescu and Pennacchiotti (Reference Popescu and Pennacchiotti2010), authors propose regression models that use a rich feature set from Twitter content and news articles’ content. They show that the features they define and use from news and the Web are useful in events’ controversy detection and mention that “coupling Twitter information with traditional media helps validate and explain social media reactions.”

In Lourentzou et al. (Reference Lourentzou, Dyer, Sharma and Zhai2015), another example of using joint information in news articles and social media is proposed. Here, the controversial sentences in news articles are found by leveraging relevant comments on Twitter as well as comments on news websites to score the controversy of opinions about an issue mentioned in the news article. In this method, each sentence of news articles is used as a query and the relevant comments on Twitter and news websites are retrieved and investigated for controversy detection. This study employs tweets for finding controversial sentences in news articles, while in our paper, the news articles’ content is employed for event detection enhancement on tweets.

2.4 News recommendation systems

News recommendation has benefited from Twitter content. In Abel et al. (Reference Abel, Gao, Houben and Tao2011a), a new user modeling framework is introduced in the context of a personalized news recommendation system. In this framework, hashtags, named entities, and topics mentioned in tweets, as well as their related news articles, are employed to construct hashtag-based, entity-based, and topic-based profiles for a user of news recommender systems. They show that semantic enrichment using Twitter content improves the quality of the generated user profiles.

In Krestel et al. (Reference Krestel, Werkmeister, Wiradarma and Kasneci2015), the authors propose a method for recommending tweets for any given news article. For this, they use information retrieval, classification, and text similarity computation methods to find tweets similar for each news article, including language models, topic models, logistic regression, and boosting. In addition to tweets and news articles’ textual content, a few Twitter-specific features such as publication time, length, and follower count are also used to find the most relevant tweets to each news article.

The use of Twitter content for recommendation systems is not restricted to news recommenders. In Chakraborty (Reference Chakraborty2018), tweets are employed to supplement the recommendation process in a contextual point of interest recommendation system. “Contextual Point of Interest (POI) recommendation is of particular interest for travel and tourism, to suggest places to visit for a user traveling to a new city.” In that paper, people’s opinions about a POI on Twitter are used to enrich the document representation for POIs. Then, the enriched representations of POIs are used as documents in a retrieval model.

Now, we present the proposed methodology for event detection enhancement using news stream.

3.1 Step 1: Event detection

First, we need to define the term event in our paper. There are various definitions for event in the literature (Yang et al. Reference Yang, Carbonell, Brown, Pierce, Archibald and Liu1999; Chua, Razikin, and Goh Reference Chua, Razikin and Goh2011; Allan Reference Allan2012; Kalyanam et al. Reference Kalyanam, Quezada, Poblete and Lanckriet2016; Allan et al. Reference Allan, Carbonell, Doddington, Yamron and Yang2018). According to Chua et al. (Reference Chua, Razikin and Goh2011), “An event, in the context of social media, is an occurrence of interest in the real world which instigates a discussion on the event-associated topic by various users of social media, either soon after the occurrence or, sometimes, in anticipation of it.”

Based on the framework presented in Chua et al. (Reference Chua, Razikin and Goh2011), event detection approaches in the literature include techniques that are based on term interestingness, topic modeling, or incremental clustering. The clustering approach is employed to “group related tweets so that each group/cluster of tweets corresponds to a candidate event.”

In our paper, we leverage the clustering approach, a source of event-related simultaneously published textual data, and language modeling to detect the group of words that represent events that occurred in the studied time frame. In the first step of the proposed framework, an event detection method is applied on two sets of data: tweets and news articles that are both generated during the same time interval and thereby cover similar sets of real-world events (Figure 2).

Figure 2. Event detection step.

In this study, a short text clustering method called GSDMM which is a collapsed Gibbs Sampling algorithm for the Dirichlet Multinomial Mixture model (Yin and Wang Reference Yin and Wang2014) is employed as a tool for event detection. So, in the rest of this study, the terms “event” and “cluster” are used interchangeably.

The main idea of the method in Yin and Wang (Reference Yin and Wang2014) is to repeatedly compute the probability of document d being generated by cluster z based on the similarity between the document’s content and the cluster’s content through several iterations until the probabilities converge. With this idea, each document is assigned to a random cluster in the initialization step, then in each iteration, a cluster is re-assigned to each document d according to the following conditional probability distribution and the iterations are repeated until convergence. This probability is based on two rules: the completeness of the clusters which prioritize larger clusters (the first part of the formula) and homogeneity which assigns higher probabilities to clusters that have more similar content to the document in comparison with other clusters (the second part of the formula).

(1) \begin{equation}p(z_d = z | \vec{z}_{\neg{d}} , \vec{d}) \propto {\frac{m_{z,\neg{d}} + \alpha}{D - 1 + K \alpha}}{\frac{\prod_{w\in d} \prod^{N^w_d}_{j=1}(n_{z,\neg{d}}^w + \beta + j - 1)}{\prod^{N_d}_{i=1}(n_{z,\neg{d}} + V\beta + i - 1)}}\end{equation}

In this formula (Yin and Wang Reference Yin and Wang2014), $\vec{z}_{\neg{d}}$ represents the vector of clusters excluding the cluster label of document d, $m_{z,\neg{d}}$ represents the number of documents in cluster z without considering document d, D is the total number of documents in the corpus, K is the number of clusters, $\alpha$ and $\beta$ are Dirichlet priors for each of the two parts of the formulas, $n_z$ is the number of words in cluster z, $N_d$ is the number of words in document d, $N^w_d$ is the frequency of word w in document d, $n^w_{z,\neg{d}}$ represents the frequency of word w in cluster z without considering document d, $n_{z,\neg{d}}$ is the number of words in cluster z without considering document d and $|V|$ is the vocabulary size. GSDMM is used as the event detection method in this framework since it has been proposed for short text clustering and also provides language model representations of events, that is clusters, so it comports with our language model based framework.

In the first step, the events are detected from tweets and news articles using the GSDMM method. In the proposed framework, the language model representations of events are needed for the following steps. According to Yin and Wang (Reference Yin and Wang2014), the probability of word w in each cluster z, $\phi_{z,w}$ , can be computed using the following formula.

(2) \begin{equation}\phi_{z,w}=\frac{n^{w}_{z} + \beta}{\sum^V_{w=1} n^w_z + |V|\beta}\end{equation}

where $n^{w}_{z}$ represents the number of occurrences of word w in cluster z. Using this probability which also represents the importance of word w to cluster z, the language model of cluster z, $\theta_z$ , can be estimated as:

(3) \begin{equation}\theta_z : \{\phi_{z,w_i}\}^{|V|}_{i=1}\end{equation}

Therefore, at the end of the first step, the language models of all clusters of tweets, $\theta^T$ , and news articles, $\theta^N$ , are calculated.

(4) \begin{equation}\theta^T = \left\{\theta^T_{z}| {0} \leq {z} \leq {k} \right\}\end{equation}

(5) \begin{equation}\theta^N = \left\{\theta^N_{z}| {0} \leq {z} \leq {k^\prime}\right\}\end{equation}

where $\theta^N_{z}$ represents the language model of cluster z from news articles that has $k^\prime$ clusters in total.

3.2 Step 2: Expansion data computation

A simple way to employ news articles for expansion is to use their textual content directly. However, due to different characteristics of tweets and news articles including their different sizes, using the news articles’ content directly for expansion might not lead to good results. To tackle this challenge, we use the language model representations of the events detected from tweets and news articles in the expansion step. Therefore, in the second step of the framework, the events from tweets are expanded using their most similar events from news articles to include more relevant words to each event in their language model. The addition of more event-related words to the language model of each event modifies the words’ distribution such that more event-related tweets would be inclined to their corresponding clusters and also non-relevant tweets would become less similar to their wrongly assigned clusters. As shown in Figure 3, this step consists of three consecutive sub-steps:

Figure 3. Event expansion step.

Tweet–News similarity computation: First, all clusters whether from tweets or news articles, are organized in a weighted undirected bipartite graph. In this graph, the nodes are clusters and the weight of each edge is the similarity value between its connecting clusters. The edge weight between cluster i from tweets part and cluster j from news articles part is computed using Jensen-Shannon (JS) divergence (Fuglede and Topsoe Reference Fuglede and Topsoe2004) between the language models of cluster i, $\theta^T_i$ , and cluster j, $\theta_j^N$ , as follows:

(6) \begin{equation}D\left(\theta^T_i || \theta_j^N\right) = \sum_{w\in V}p\left(w|\theta^T_i\right)\log{\frac{p\left(w|\theta^T_i\right)}{p\left(w|\theta_j^N\right)}}\end{equation}

(7) \begin{equation}M = \frac{\theta^T_i+\theta_j^N}{2}\end{equation}

(8) \begin{equation}JS\_divergence\left(\theta^T_i , \theta_j^N\right) = \frac{D\left(\theta^T_i || M\right)+D\left(\theta_j^N || M\right)}{2}\end{equation}

(9) \begin{equation}JS\_SimilarityScore\left(\theta^T_i , \theta_j^N\right) = - JS\_divergence\left(\theta^T_i , \theta_j^N\right)\end{equation}

Next, the problem of finding the most similar news article cluster to each tweet cluster in the mentioned bipartite graph is solved as an assignment problem using the Hungarian method (Kuhn Reference Kuhn1955). The reason for choosing this method is that in this problem, we need to match each news cluster to at most one tweet cluster (i.e. a matching is needed) and also make the matching affected by the similarity values between clusters (i.e. the graph is weighted). Therefore, the maximum matching in a weighted bipartite graph is required which can be found using the Hungarian method.

In this method, the maximum weighted matching problem is mapped to finding a cover of minimum cost in the graph. Therefore, in this study, the weights of the bipartite graph that are JS similarity values are flipped to turn them into costs and also fit the problem setting. The input of the Hungarian algorithm is a square matrix of the assignment costs. The final assignments are found through a four-step procedure: (1) subtracting the minimum of each row from all row elements, (2) subtracting the minimum of each column from all column elements, (3) covering all zeros in the resulting matrix using a minimum number of horizontal and vertical lines, (4) if the minimum number of lines equals the number of rows in the matrix, an optimal assignment exists among the zeros of the matrix. Otherwise, the smallest element that is not covered by a line in the previous step should be subtracted from all uncovered elements and also added to all elements that are covered and the process is repeated from step 3.

Expansion documents construction: After applying the Hungarian algorithm on the graph, the language model of the matched news cluster for each tweet cluster is employed to compute the expansion data for enriching the corresponding tweet cluster’s language model. The idea of using the matched news cluster’s language model as useful data for expanding each tweet cluster lies in its similar distribution of words to the tweet cluster’s distribution, which facilitates finding relevant words suitable for constructing the expansion document. Let us assume the matched news cluster for tweet cluster i is M(i) and we want to find the best p expansion words. We assume that the best word candidates for expanding a language model are the representative words chosen from its most similar language model. The words having higher probabilities in the language model are considered as the representative ones, due to their substantial role in representing the words’ distribution. To this aim, top p words with the highest probabilities in the language model of the matched news cluster M(i) are employed as expansion words $Exp\_Words(i)$ to be added to tweet cluster i as follows.

(10) \begin{align} Exp\_Words\left(i\right) & = \left\{\left(w_t,P\left(w_t|\theta^N_{M\left(i\right)}\right)\right)|P\left(w_1|\theta^N_{M\left(i\right)}\right)\right.\nonumber\\[5pt]& \left.\geq P\left(w_2|\theta^N_{M\left(i\right)}\right) \geq..\geq P\left(w_{|V|}|\theta^N_{M\left(i\right)}\right), t=1,..,p\,,p < |V|\right\}\end{align}

In our method, the expansion content is employed by the clustering method through building an artificial document called expansion document and adding it to its corresponding tweet cluster. In other words, the set of expansion words obtained for each tweet cluster is employed to build an expansion document that is added to its tweet cluster. To this aim, two issues about the expansion document should be addressed:

1. 1. The frequency of expansion terms in the expansion document.

2. 2. The number of expansion documents to be added to the corresponding tweet cluster.

Our approach to resolving the first issue is considering the expansion words with higher probabilities in the news cluster language model to be more important than other words and consequently, should be more frequent than other words in the artificially built expansion document. Therefore, the goal is to estimate the frequency of expansion words in the expansion document using their probabilities in the news cluster language model. To this aim, we employed the normalized probabilities of p expansion words in the news cluster’s language model and the average documents’ size in the target cluster, $Avg\_TSize(i)$ , that is the average tweet size in the corresponding tweet cluster i. These two factors are used for computing the expansion words frequencies such that their relative frequencies would be proportional to their importance in the source news cluster language model while the expansion documents’ length would be comparable to other documents in the target cluster. Therefore, the frequency of each expansion word w in the expansion document $Exp\_Doc(i)$ which expands the tweet cluster i is computed as follows.

(11) \begin{align} Avg\_TSize(i) = avg_{t\in i}|t|\end{align}

(12) \begin{align}Term\_Freq \left(w,Exp\_Doc(i)\right) = \frac{P\left(w|\theta^N_{M(i)}\right)}{\sum_{w\in Exp\_Words(i)} P\left(w|\theta^N_{M(i)}\right)}\cdot Avg\_TSize(i), \!\!\quad \quad w\in Exp\_Words(i)\end{align}

The number of expansion words, p is one of the method’s parameters determined when tuning the parameters using the validation set. For the second issue, we heuristically make the number of expansion documents for each tweet cluster proportionate to the similarity between the corresponding tweet and news clusters such that more similar tweet clusters to the news cluster obtain more expansion documents. The idea is to have more expansion documents added to the tweet cluster when the tweet cluster and its matched news clusters are more similar compared to other tweet–news cluster pairs. In other words, higher tweet–news clusters’ similarity makes the obtained expansion content more reliable. To this aim, we used the JS similarity value between two corresponding tweet–news clusters, $\theta^T_i$ and $\theta^N_{M(i)}$ , normalized by the sum of similarity values between the same news cluster and all tweet clusters, calculated in the denominator. The JS similarity values are calculated using Equation (9). We also used the average number of tweets in tweet clusters, $Avg\_TNum(i)$ , to estimate the number of times that an expansion document $Exp\_Doc(i)$ should be added to the tweet cluster i according to the following formula.

(13) \begin{align} Avg\_TNum(i) = avg_{k\in i}|\{t|t\in k\}|\end{align}

(14) \begin{align}Doc\_Freq\left(Exp\_Doc(i)\right) = \gamma \cdot \frac{JS\_SimilarityScore\left(\theta^T_i , \theta^N_{M(i)}\right)}{\sum_{j\in I}JS\_SimilarityScore\left(\theta^T_j , \theta^N_{M(i)}\right)} \cdot Avg\_TNum(I)\end{align}

where I represents the set of tweet clusters, t represents tweet and $\gamma$ is the parameter (with a value between 0 and 1) defined to control the proportion of expansion documents to existing tweets of the cluster in the expansion step.

Cluster expansion: After constructing the expansion documents for each tweet cluster, we add them to their corresponding tweet clusters by updating the clusters’ statistics used by the GSDMM method (mentioned in Section 3.1) such as the number of documents in each cluster, the frequency of words in each cluster, the vocabulary size, and the total number of documents in the corpus. Using this expansion method, the assignments of all documents, including the original and expansion documents, to the clusters are recomputed using the new set of documents, which means the cluster-assignment probabilities based on formula (1) are recomputed considering the expanded set of documents. The impact of the original documents words on updating the cluster assignment probabilities is similar to the impact of expansion documents words, even if the words are common. In the next section, the enhanced event detection step is explained in more detail.

3.3 Step 3: Enhanced event detection

In the third step of the framework, the event detection method is applied to the expanded versions of tweet clusters computed in the previous step as shown in Figure 4. The rationale behind this step is to repeat the event detection method using an enhanced initialization of clusters rather than a random initialization to achieve better event detection results. In other words, the proposed framework through the mentioned steps tries to pause the event detection process in order to involve the content of another rich source of event-related information, that is news articles, to enhance the tweet clusters’ content and then resume the event detection process with the updated clusters.

Figure 4. Enhanced event detection step.

4.1 Datasets

To evaluate the performance of our framework, we needed a dataset of tweets with event labels and a dataset of news articles that are published during the same time interval with tweets. To this aim, we utilized a subset of the tweet dataset with event labels introduced in Kalyanam et al. (Reference Kalyanam, Quezada, Poblete and Lanckriet2016) and also a subset of the breaking news dataset (Ramisa et al. Reference Ramisa, Yan, Moreno-Noguer and Mikolajczyk2018) that was published in the same time period as the tweet dataset. The used tweet dataset contains 1,369,876 tweets that account for 268 real-world events and contains tweets published between the $1{\rm st}$ and the $13{\rm th}$ of January 2014. The description of the data collection methodology is provided in Kalyanam et al. (Reference Kalyanam, Quezada, Poblete and Lanckriet2016). As a brief description, the authors of Kalyanam et al. (Reference Kalyanam, Quezada, Poblete and Lanckriet2016) used the news headlines reported by a list of well-known news media accounts on Twitter (e.g., @CNN, @BBCNews, etc.) and collected tweets written about news headlines that are collected periodically every hour over approximately 1 year using Twitter API.Footnote ^b The collected events are also validated in Kalyanam et al. (Reference Kalyanam, Quezada, Poblete and Lanckriet2016) to ensure that each group of tweets corresponds to a meaningful and cohesive news event. Figure 5 shows the histogram of the frequency distribution of events’ sizes by bin. The size of an event is measured by the number of tweets having the same event label in the ground truth tweet dataset. The size of the bins is 1000. According to Figure 5, for 195 events (72.76% of the dataset) in the ground truth data, the event’s size is less than 4000 tweets and less than 9% of events have sizes higher than 10,000 tweets. The minimum, maximum, and average event size in the ground truth data is 1019, 165,785, and 5111.47, respectively. Some more statistics about the tweet dataset are as follows.

• The total number of sentences in the tweet dataset is 1,808,052 across all tweets and the average is 1.31 per tweet.

• The tweet dataset contains 19,501,927 words of which 525,402 are unique.

• The average number of words per tweet is 14.23.

Figure 5. The histogram of event sizes which is the number of tweets having the same event label in ground truth tweet dataset.

The news articles dataset contains 5473 news articles from several major newspapers and media agencies collected between the 1st and the 19th of January 2014. Therefore, the simultaneity requirement, which is having both tweets and news articles published in the same time frame, is met. We have done stop-word removal on both datasets. Some more statistics about the news article dataset are as follows.

• The total number of sentences in the news article dataset is 276,192 across all documents and the average is 50.46 per document.

• The news article dataset contains 5,395,437 words of which 218,832 are unique.

• The average number of words per document is 985.82.

4.2 Evaluation metrics

The tweet dataset contains the event labels of the tweets so the performance of the proposed method can be evaluated using supervised metrics including precision (P), recall (R), and F1. To compute these measures, we need to calculate true positive (TP), false positive (FP), true negative (TN), and false negative (FN) values for each pair of data. For example, TP shows how many pairs of data (i.e. tweets) that are in the same class (i.e. have the same event label) in gold data, are predicted to be in the same cluster by the clustering method. The measures are computed according to the following formulas.

\begin{equation*}P = \frac{TP}{TP+FP}\end{equation*}

\begin{equation*}R = \frac{TP}{TP+FN}\end{equation*}

\begin{equation*}F1 = \frac{2PR}{P+R}\end{equation*}

4.3 Baseline method

The baseline method used for comparison with the proposed method is an event detection method based on hashtags (Yang and Rayz Reference Yang and Rayz2017). In Yang and Rayz (Reference Yang and Rayz2017), the feature extraction method proposed in Feng et al. (Reference Feng, Zhang, Zhang, Han, Wang, Aggarwal and Huang2015) and K-means clustering algorithm (Lloyd Reference Lloyd1982) is employed for hashtag clustering. Each hashtag $h_i$ is represented as a weighted vector by combining two vectors: a word vector representing the words of the tweets that contain the hashtag $h_i$ and a hashtag vector representing other hashtags co-occurring with hashtag $h_i$ . The co-occurring hashtags are the ones that appear in the same tweets. The K-means algorithm is used to detect hashtags’ clusters. Finally, the hashtag clusters are used as their corresponding tweets’ clusters.

In the following subsections, the experiments evaluating the performance of the proposed framework from different viewpoints are presented. First, the impacts of the two main parameters of the framework, k and $\gamma$ , on the proposed framework’s performance are investigated. Afterward, the impact of the enhancement method on event detection performance is evaluated and the event detection result is compared with the baseline. Finally, the impact of using the Hungarian method to find the most similar news article cluster to each tweet cluster on the performance compared to a baseline method is studied.

4.4 Parameter tuning

We randomly selected one-fifth of the tweet dataset and used it as the validation set for tuning the parameters of the proposed method including $\alpha$ , $\beta$ , $\gamma$ , p, and k. Using grid search, the optimal values of the parameters that lead to the highest F1 value on the validation set are $\alpha= 0.5$ , $\beta= 0.1$ , $\gamma= 0.17$ , $p = 15$ , and $k = 40$ . These parameter values are used in all of the next experiments on the test set, that is the remaining four-fifth of the tweet dataset. Figure 6 shows the highest F1 obtained from the grid search for each value of k, that is the number of clusters, on the validation set. As shown in Figure 6, by increasing k from 10 to 40, F1 score rises until it reaches its maximum value at $k=40$ . The changes of precision by changing the values of k are similar to F1 while the recall scores at lower values of k are higher compared to larger k values. Higher recall values at low values of k is because of having many tweets in the same cluster due to the low number of clusters. In Yin and Wang (Reference Yin and Wang2014), the authors recommend setting k to larger values than the actual number of the clusters. To try this, we repeated the grid search using larger values of k from 200 to 300 increased by 10 and obtained the results. The results showed that the optimal value of k remains 40 and larger k values would not lead to higher performance of the clustering method.

Figure 6. The impact of k on the proposed framework performance on the validation set.

4.5 Impact of $\gamma$

In this experiment, we investigate the influence of $\gamma$ on F1 score of the proposed framework on the validation set. This parameter is defined to control the proportion of expansion documents to existing tweets in the cluster in the expansion step.

The experiment is done for six different values of $\gamma$ ( $\gamma\in\{0.01, 0.05, 0.09, 0.13, 0.17, 0.21\}$ ). Other parameters are set to the optimal values explained in Section 4.4. As shown in Figure 7, by increasing the value of $\gamma$ from $0.01$ to $0.17$ , the performance of the proposed method increases in terms of all three measures and reaches the maximum at $\gamma= 0.17$ and then all measures including precision, recall, and F1 decrease. This phenomenon can be explained considering how the event clusters change due to changes in $\gamma$ value. For $\gamma$ values smaller than the optimal value ( $0.17$ ), the impact of the expansion is restricted while the clusters still have improvement capacity and can include more event-related words from news articles. So, by increasing $\gamma$ from $0.01$ to $0.17$ , the precision, recall, and F1 increase. By increasing $\gamma$ more than $0.17$ , the event’s distribution of words deviates from the ideal distribution for that event by adding extra expansion words so the performance decreases.

Figure 7. The impact of $\gamma$ on the proposed framework performance.

4.6 Event detection enhancement (step 3) evaluation

In this section, the performance of the proposed framework is evaluated. In the proposed framework, after finding the tweet clusters and news clusters, the expansion documents are built for each tweet cluster as explained in Section 3.2. The proper functioning of the expansion document construction, that is the second step, directly impacts the framework’s performance. The improvement in the final results of the proposed framework shows that the second step performs well in finding suitable expansion words. We also investigated the content of the tweet clusters before the expansion as well as the expansion words found by the framework for those tweet clusters. As an example, the top 10 words with the highest weights in the language model of the tweet cluster number 4 without expansion are “Duggan”, “Mark”, “http”, “police”, “killed”,“verdict”, “jury”, “lawfully”, “inquest” and “duggan”. Using this set of words as a query and checking the top result retrieved by Google shows that no related document is retrieved that reports the particular event occurred in the time interval of our dataset (between the 1st and the 13th of January 2014). If we carry out the same Google search using the top 10 words followed by the corresponding expansion words, found by the framework, which are “Guardian”, “home”, “year”, “said”, “two”, “years”, “He”, “would”, “told”, “She”, “old”, “family” as a query, the top retrieved result is a news article published by the Guardian on Jan 8, 2014, about the event.Footnote ^c To demonstrate the impact of expansion words in retrieving the relevant news article, we repeated the Google search using the expanded query excluding the “Guardian” term which can help retrieve documents from this news website. Again, we obtained the relevant news article about the event as the first retrieval result.

As another example, the top 10 words with the highest weights in the language model of the tweet cluster number 28 such as “Best”, “Golden”, “TV”, “Series”, “Globe”, “wins”, and “Drama” does not clearly indicate the golden globe ceremony held in 2014, while its corresponding expansion words such as “years”, “Slave”, “12” that indicate the name of the movie that won the Golden Globe 2014 best motion picture drama award, enhances the clusters’ content regarding this event.

Expansion document construction is one of the principal parts of the proposed framework and its proper functioning directly impacts the framework’s performance. In Table 1, the content of a sample of four tweet clusters before the expansion as well as the expansion words found by the framework for those tweet clusters are reported to provide some more examples from the dataset in addition to the computed expansion terms for each of them. To represent the tweet cluster’s content, we show the top 10 words with the highest weights in each cluster’s language model. The expansion words for each tweet cluster, the links to the news webpages that report the corresponding event in addition to their publication dates are also shown in Table 1. The expansion words computed for each event provide more details reported about each event in news articles. For example, based on the top words in cluster number 0, such as “Best”, “Golden”, “Globes”, “Wins”, and “Comedy”, its corresponding expansion terms include the name of the movie whose actor won the best actor in Comedy award in Golden Globes 2014.

Table 1. Example tweet clusters and their computed expansion words

In the next experiment, the event detection result on the tweet dataset, that is the results of the first step, is compared with the enhanced event detection results obtained from the proposed framework, that is the results of the third step, to evaluate the performance of the event detection enhancement framework. Furthermore, the performance of the proposed method is compared with another event detection method used as a baseline (Yang and Rayz Reference Yang and Rayz2017). Table 2 shows the results of the original event detection method, which is GSDMM in this paper, in comparison with the proposed enhanced event detection and the baseline method results on the test set.

Table 2. The performance of the event detection enhancement framework in comparison with event detection without enhancement and the baseline method (Yang and Rayz Reference Yang and Rayz2017) on test set

According to Table 2, F1 value for the enhanced method on the test set achieves $47.13\%$ and improves the F1, $2.03\%$ compared to the original event detection method and $2.56\%$ compared to the baseline method. It also shows that the proposed method’s improvement is due to improving the recall measure from $51.62$ to $55.10$ . Comparing the results of the baseline method with the enhanced method shows that the precision has increased in the enhanced method. Hence, despite the higher recall value in the baseline method, the enhanced method outperforms the baseline in terms of F1.

4.7 Hungarian method performance in expansion evaluation

In this experiment, we try to study the performance of the Hungarian method in finding the most similar news article cluster to each tweet cluster. To this aim, we compare the results of the proposed framework using the Hungarian method with the results through which each tweet cluster is matched with its most similar news article cluster that can also be matched with other tweet clusters. We name the second run Closest Matching in Table 3. In both runs, the similarity between tweet clusters and news article clusters are computed using Jensen-Shannon (JS) divergence as explained in Section 3.2.

Table 3. The performance of the proposed framework using the Hungarian method compared to using Closest matching on test set

As shown in Table 3, the result using the Hungarian method outperforms the result obtained from matching each tweet cluster with its most similar news cluster. This can be because of several tweet clusters being matched with the same news cluster in the second run and consequently having several tweet clusters expanded similarly that do not occur in the Hungarian method. Moreover, news clusters with general content can cause more decrease in the performance of the second run rather than the first run (Hungarian method) since those clusters are similar to most tweet clusters and if they are the topmost similar clusters to tweet clusters, they can only be selected as the matched news cluster for one tweet cluster in the Hungarian method while they can be matched with several tweet clusters in Closest Matching run and cause more decrease in the performance.

The results of our experiments show that modifying the distribution of words to achieve better representations of events (i.e. better events’ language models) is an effective approach to improve the event detection performance (according to Table 2). One of the key points in this approach is to find event-related words and limit irrelevant or less-relevant words when updating the event’s language model. Our investigation into the results reveals that having some expansion words that are highly relevant to the event’s topic is more effective than adding more expansion words that are moderately relevant to the event (according to Figure 7).