3.1 Experimental settings

Dataset. We built a dataset consisting of pairs $(u,l)$ associating to each URL $u$ the language $l$ in which the linked document is written. We build on CommonCrawlFootnote ^b (CC) snapshots CC-MAIN-2023-06 and CC-MAIN-2023-14, which contain massive collections of web documents with language automatically identified using CLD2.Footnote ^c While language is not manually annotated, CLD2 shows a very high performance with both macro-precision and macro-recall above $98\%$ (CLD2Owners, 2014). Independent studies confirm a significant macro F1 even off-the-shelf (Lui and Baldwin Reference Lui and Baldwin2014).

The data in CC cover 161 languages that are represented following a Zipfian distribution.Footnote ^d In order to deal with over-represented languages, a maximum of 600,000 URLs per language are collected, resulting in a corpus of about 50.4 million URLs. We split the data and use 50 million URLs for training, and the rest for development and testing (about 220,000 URLs each). We divide our data in such a way that URLs from the same web domain only appear in one of the splits. Figure 2 reports the language distribution of each set.

Figure 2.

Language distribution for the training, development, and test sets used in the experiments for language identification.

Figure 2 Long description

The histogram displays the distribution of data samples for training, development, and testing sets in a language identification experiment. The x-axis represents the number of data samples on a logarithmic scale, ranging from 2 to 32. The y-axis lists various languages. The histogram consists of vertical bars divided into three colors: blue for training, red for development, and beige for testing. The blue section dominates the left side, indicating a higher number of training samples. The red section is narrower, representing development samples, while the beige section on the right shows the distribution of testing samples. The data shows a clear distinction between the sets, with training samples being the most numerous, followed by development and then testing samples. All values are approximated.

Evaluation metric. We present precision, recall, and F1 results, along with their aggregated counterparts: macro-precision, macro-recall, and macro-F1. These aggregated metrics represent the average values across all possible classes and offer a comprehensive overview of the systems’ performance.

Baseline systems. We compare our approach to a baseline that, inspired by El-Kishky et al. (Reference El-Kishky, Chaudhary, Guzmán and Koehn2020) and Baykan, Henzinger, and Weber (Reference Baykan, Henzinger and Weber2013), looks at different components of the URL that may indicate the language of the linked document: subdomain, public suffix, directory names, and value of the different parameters (if any). The value of each component is compared to the ISO 639-1 and ISO 639-2 language codes.

To address potential contradictions from various URL components, we examine them in a specific order established through evaluation on the development set. The optimal sequence, determined by assessing all possible combinations, is as follows: parameter values, directory names, public suffix, and subdomain.

Additionally, we include a baseline simpler than our transformer-based model (see Section 3) using FastText (Joulin et al. Reference Joulin, Grave, Bojanowski and Mikolov2017), a common approach for language identification (Facebook AI Research, 2017; NLLB Team et al. Reference Costa-Jussà, Cross, Çelebi, Elbayad, Heafield, Heffernan, Kalbassi, Lam, Licht, Maillard, Sun, Wang, Wenzek, Youngblood, Akula, Barrault, Gonzalez Mejia, Hansanti, Hoffman, Jarrett, Sadagopan, Rowe, Spruit, Tran, Andrews, Ayan, Bhosale, Edunov, Fan, Gao, Goswami, Guzmán, Koehn, Mourachko, Ropers, Saleem, Schwenk and Wang2022). This system is a linear classifier that represents text as a bag of character-level $n$ -grams. The embeddings of these $n$ -grams (extracted from each word independently) are averaged to form a hidden representation, which is then fed into a hierarchical softmax classifier to compute the probability distribution over the supported classes.

For this baseline, hereafter referred to as FastText, we use the same training data and data pre-processing pipeline as our approach. This ensures that both models support the same label set—enabling a direct comparison—and effectively adapts the model to classify URLs rather than text. Regarding hyperparameters, we adopt those used by the publicly available language identification model lid.176.bin (Facebook AI Research, 2017); namely, 16-dimensional embeddings, character $n$ -grams from $2$ to $4$ , learning rate of $0.1$ , and $10$ epochs.

3.2 Training of the model

We used HuggingFace Transformers (Wolf et al. Reference Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu, Le Scao, Gugger, Drame, Lhoest and Rush2020) and PyTorch (Paszke et al. Reference Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai and Chintala2019) to fine-tune XLM-RoBERTa $_{\mathrm{Base}}$ using the cross-entropy loss function. Based on preliminary experiments, we set the learning rate to $10^{-5}$ and used the AdamW optimizer (Loshchilov and Hutter, Reference Loshchilov and Hutter2019) with default parameters ( $\beta _1=0.9$ , $\beta _2=0.999$ , $\epsilon =10^{-8}$ , $\lambda =0.01$ ). We use a learning rate scheduler with an initial warm-up phase ( $10\%$ of the training steps in the first epoch) where the learning rate linearly increases to the configured value and then decays following an inverse square root pattern.

Due to the extensive data in CC, utilizing the entire training set in each training epoch is impractical. Therefore, we opted to incrementally process the training set in folds. Starting with one million URLs, we add an additional million URLs after each training epoch. Training stops when further data fails to enhance the macro-F1 metric on the development set for five epochs. The training data shown in Figure 2 correspond to the data actually used for training.

Finally, following Zhang et al. (Reference Zhang, Wu, Katiyar, Weinberger and Artzi2021)—who argue that layers near the output of a pretrained model retain task-specific knowledge—we experimented with re-initializing the top $n\in [0,5]$ layers of the transformer. Our findings reveal that reinitializing the top layer ( $n=1$ ) before fine-tuning for our downstream task slightly improves the results on the development set. Therefore, we adopted this configuration.

Table 1.

Results of the baseline, FastText, and our language identifier. Refer to Figure 3 for a breakdown per language

Table 1 Long description

The table presents a comparison of three language identification systems: Baseline, FastText, and a custom system. It includes three columns for Macro Precision, Macro Recall, and Macro F1 scores, and three rows for each system. The Baseline system has a Macro Precision of 65.29 percentage, Macro Recall of 34.42 percentage, and Macro F1 of 38.24 percentage. FastText shows a Macro Precision of 60.92 percentage, Macro Recall of 57.32 percentage, and Macro F1 of 57.41 percentage. The custom system, labeled as Ours, achieves a Macro Precision of 72.59 percentage, Macro Recall of 69.04 percentage, and Macro F1 of 69.22 percentage. Notably, the custom system outperforms both the Baseline and FastText systems across all metrics.

3.3 Results and discussion

Table 1 shows the results of the baseline system, the FastText system, and our model on the test set: our model surpasses the baseline and FastText systems across all metrics. While the baseline attains a notable macro-precision score, its macro-recall is low. This reduced recall is attributed to the absence of language information in some URLs, even in multilingual websites. This is corroborated by the significant percentage of test samples classified as unknown by the baseline (57.4%). The FastText system achieves better results than the baseline in terms of macro-F1 and exhibits more balanced performance across the reported metrics, similar to our transformer-based model; however, it still falls behind our approach by more than 10 points in macro-precision, macro-recall, and macro-F1.

Figure 3 offers a comprehensive breakdown of the F1 metric per language, revealing the consistent superiority of our model over the baseline (left subfigure) across all languages, except for a few instances with comparable F1—notably the baseline achieves better results in Amharic and Tatar. The FastText system (right subfigure) yields results closer to our approach across languages; however, the overall tendency indicates that our model consistently outperforms it, with the only exceptions being Sanskrit, Oriya, Turkish, Hindi, Thai, Urdu, and Azerbaijani.

It is noteworthy that all the systems—baseline, FastText, and our model—yield unexpectedly low results for English; the best performing model (ours) achieves a precision of $36.6\%$ and a recall of $58.5\%$ . Further analysis reveals a pattern of frequent confusion between non-English URLs and English (28 languages are confused with English in at least $10\%$ of their data). These findings align with those reported by the authors of CLD2, confirming a bias towards English, especially for Interlingua (CLD2 classifies $44.4\%$ of these documents as English). Noticeably, our model behaves similarly and $46\%$ of the URLs marked as Interlingua were misclassified as English.

Figure 3.

Language identification results on a per-language basis, comparing our model with the baseline (left subfigure) and with a FastText model trained on the same data (right subfigure). Only languages with a minimum of 100 URLs and 10 different web domains in the test set are included.

Figure 3 Long description

The bar graph compares language identification results on a per-language basis, featuring three data series: Baseline, FastText, and Ours. The x-axis lists language codes, while the y-axis represents F1 scores ranging from 0 to 100. Each language is represented by three horizontal bars, with Baseline in blue, FastText in red, and Ours in orange. The graph shows that the Ours model generally outperforms both the Baseline and FastText models across most languages. Notable trends include higher F1 scores for the Ours model, particularly in languages with sufficient data. The graph highlights the effectiveness of the Ours model in language identification tasks. All values are approximated.

Another source of noise in the data stems from URLs linking non-English documents, but that incorporate English text in the URL, such as the Spanish webpage https://www.dochaney.com/es/contact-us. This issue is exacerbated when the URL lacks any explicit language hint, as seen in the Spanish webpage https://hjmorelia.com/about-us. Upon manual inspection of the URLs in the test set linking to English documents, we found that $94.1\%$ of these URLs lack any language identifier mark (e.g., “/en/” or “/*-eng/”), compelling our model to rely on detecting the presence of English words or other intricate patterns, if any. This fact, along with the presence of webpages that link to non-English documents but contain English text in their URLs, such as the aforementioned examples of URLs linking to Spanish webpages, may be an additional reason why the model struggles with URLs linking to English documents.

The problems encountered are inherently related to the use of URLs to infer information about the content of the linked document before it is actually downloaded. URLs are often useful because they present similar semantics to the documents they link to. However, identifying the language of documents from their URLs is not always feasible due to insufficient or inaccurate semantics. During our analysis, we encountered several examples of URLs that appeared to link to a document in a particular language but instead led to a different one, as illustrated above.

4.1 Experimental settings

Dataset. Our model is trained on a labeled dataset of URL pairs, indicating parallel or non-parallel relationships. The most relevant publicly available dataset for this task is the one distributed for the WMT16 bilingual document alignment shared task (Buck and Koehn Reference Buck and Koehn2016a) (hereafter WMT16); it consists of a collection of documents in English and French obtained from 252 multilingual websites. It is divided into training (documents from 49 websites) and test (documents from 203 websites) sets. For every document, plain text and HTML content are available, along with its URL and automatically detected language. A gold standard consisting of 1,624 URL pairs for training and 2,402 for testing is also distributed. This dataset has three significant limitations: (i) it supports only one language pair; (ii) the gold standard is relatively small and includes only a subset of the actual parallel documents in the training and test sets; and (iii) the gold standard lacks negative samples, that is URL pairs labeled as non-parallel.Footnote ^f

We deal with this last limitation by extending the original WMT16 gold standard with negative samples obtained by emulating the behavior of a web crawler—which is the scenario in which we plan to use our model—as follows. For each URL in the gold standard, we extract all the URLs linked in the corresponding HTML content and discard those pointing to documents neither in English, nor in French. We then build a set of URL pairs by pairing the original URL with every linked URL. Finally, we check if one of these pairs appears in the gold standard, and if it does, we label the rest of pairs as non-parallel; otherwise, we discard all the pairs in the set.

We alleviate the other two limitations by extending our training data with the MaCoCu (Bañón et al. Reference Bañón, Esplà-Gomis, Forcada, García-Romero, Kuzman, Ljubešić, van Noord, Sempere, Ramírez-Sánchez, Rupnik, Suchomel, Toral, van der Werff and Zaragoza2022) v2 corpus, which covers 10 additional language pairs; namely, English aligned with Albanian, Bulgarian, Croatian, Icelandic, Macedonian, Maltese, Montenegrin, Slovenian, Serbian, and Turkish.Footnote ^g This corpus is distributed aligned both at the segment level and at the document level with URLs attached to each pair of aligned documents; although these alignments are not as accurate as those in WMT16 because documents were automatically aligned. Additionally, this corpus does not include the original HTML content, which prevents the application of the method described above for generating negative samples. To address this limitation, we devise strategies inspired by those proposed by Esplà-Gomis et al. (Reference Esplà-Gomis, Forcada, Ortiz-Rojas and Ferrández-Tordera2016) and Dara and Lin (Reference Dara and Lin2016) to generate negative samples without accessing the HTML content.

Creation of synthetic negative samples. In what follows we evaluate various methods for creating synthetic negative samples from the URL pairs $(u_A,v_B)$ in the MaCoCu v2 corpus, where $u_A$ and $v_B$ point to documents in languages $A$ and $B$ , respectively. These methods are:

• • Random match: A new URL pair $(u_A, v_B')$ is generated by replacing $v_B$ with $v_B'$ , a randomly selected URL from the pair collection. This approach is rather naive, as the resulting URLs are likely to be significantly different.

• • Remove random tokens: Both $u_A$ and $v_B$ are tokenized, and random tokens are removed; random tokens never include the scheme, authority, or the port.

• • Maximize Jaccard similarity: Similarly to random match, a new URL pair $(u_A,v_B')$ is created, but in this case $v_B'$ is chosen so that it maximizes the Jaccard similarity to $v_B$ . This method aims to create URL pairs very similar to the original one.

These methods assume that all URLs link documents in languages $A$ or $B$ , which is the case for the WMT16 dataset. However, in a realistic crawling scenario, the language cannot be identified until the document is downloaded, and it is common to encounter pairs of URLs linking documents in the same language. Hence, it is crucial to include negative samples corresponding to non-parallel documents in the same language for training. We achieve this by generating synthetic monolingual negative samples where $A=B$ , using pairs of identical URLs, $(u_A, u_A)$ and $(v_B, v_B)$ , as starting points.

We evaluated all combinations of the strategies outlined above on the WMT16 dataset, for which genuine negative samples can be obtained from HTML, as discussed earlier; the comprehensive results obtained in this evaluation are available in Appendix A. The results indicate that the most effective combination comprises random match (both monolingual and bilingual), maximize Jaccard similarity (both monolingual and bilingual), and remove random token (bilingual only). To create a cohesive dataset for our experiments, we apply these strategies to generate synthetic negative samples from both WMT16 and MaCoCu v2.

We use $60\%$ of the web domains in the WMT16 training split for training and the other $40\%$ for development.Footnote ^h The web domains in MaCoCu v2 are split as follows: $80\%$ for training, $10\%$ for development, and $10\%$ for testing; we made sure that URLs from the same web domain only appear in one of these splits. Table 2 reports the amount of positive and synthetic negative samples in each set per language pair.

Table 2.

Positive/synthetic negative samples per language pair in the training, development, and test sets are used to train the model for inferring the parallelness of two documents from their URLs

Table 2 Long description

The table presents the number of positive and synthetic negative samples for various language pairs used in training, development, and test sets. It includes data for pairs such as English to French, English to Chinese, and others. Each row lists the language pair, followed by the number of samples in the training, development, and test sets. For instance, the English to French pair has 0.9 million positive samples and 5 thousand synthetic negative samples for training, 0.7 million positive samples and 5 thousand synthetic negative samples for development, and 2 thousand positive samples and 15 thousand synthetic negative samples for testing. The table highlights the distribution of samples across different stages of model training, ensuring a balanced approach for inferring the parallelness of documents from their URLs.

Evaluation metrics. We assess our model on both the official WMT16 test set and the test set described in the previous paragraph. For the WMT16 test set, we employ recall and soft recall (Buck and Koehn Reference Buck and Koehn2016a), a variant of recall used in the shared task that considers near matches. Precision cannot be used due to the non-exhaustive nature of the WMT16 gold standard, rendering the actual correct URL pairs unavailable for the test set.

As regards the evaluation on the test set described above, the same metrics described in Section 3.1 are used: precision, recall, F1, macro-precision, macro-recall, and macro-F1.

Baseline system. We use the official baseline of the WMT16 shared taskFootnote ⁱ (Buck and Koehn Reference Buck and Koehn2016a).Footnote ^j This baseline defines a set of tokens considered to be potential language identifiers in a URL; a given pair of URLs is aligned if they can be transformed into the same string by removing one or more of these tokens. We have extended this baseline to support all the languages covered in our experiments. Similar to the already supported languages, the set of tokens for additional languages includes the name of the language in English, its endonym, the ISO 639-1 and ISO 639-2 language codes, and combinations of the ISO 639-1 language codes and the ISO 3166-1 country codes where the language is official, separated by a hyphen; for example for Albanian: albanian, shqip, sqp, alb, sq, sq-sq, sq-ks.

4.2 Training of the model

We employed the same training configuration as regards learning rate, optimizer, hyperparameters, and loss function, as described in Section 3.2. Training stops if the macro-F1 computed on the development set does not improve after five epochs. In addition, the top layer of the XLM-RoBERTa $_{\mathrm{Base}}$ model used is re-initialized before training.

4.3 Results and discussion

Table 3 reports the results in terms of recall and soft recall on the WMT16 test set. The figures in this table can be directly compared to those achieved by the systems submitted to the shared task (Buck and Koehn (Reference Buck and Koehn2016a, Tables 3 & 4); 19 submissions from 11 distinct research groups. To compute them, we obtain the Cartesian product of all English and French URLs in the WMT16 test set and apply our model and the baseline to each URL pair. As WMT16 only allows 1-to-1 URL alignments, our model outputs the most probable alignment among those identified. The results obtained are promising: our approach not only outperforms the baseline in almost 7 points of soft recall but also four of the nineteen systems submitted to the shared task. This is especially relevant given that all the systems submitted compare the content of documents to determine whether they are parallel or not, while our model solely relies on URLs.

Table 3.

Recall and soft recall obtained by our parallel URL identifier and the baseline on the WMT16 test set

Table 3 Long description

The table presents a comparison of recall and soft recall metrics for two systems, labeled as Baseline and Ours, on the WMT16 test set. It consists of three columns: System, Recall, and Soft Recall, with two rows of data. The Baseline system has a recall of fifty-nine point nine one percentage and a soft recall of fifty-nine point nine one percentage. The Ours system has a recall of sixty-two point nine nine percentage and a soft recall of sixty-six point six nine percentage. The data indicates that the Ours system outperforms the Baseline system in both recall and soft recall metrics.

Table 4.

Results for the parallelness identifier from URLs on the dataset described in Section 4.1, considering all language pairs collectively

Table 4 Long description

The table presents a comparison of precision, recall, and F1 scores for both positive and negative classes between a baseline model and a new model. It includes macro averages for these metrics. The table has four rows and three columns, with the columns labeled as Class, Metric, Baseline, and Ours. The positive class metrics show that the new model significantly outperforms the baseline in precision, recall, and F1 score. For the negative class, the new model also shows improved precision and F1 score, though the recall is slightly lower. The macro averages indicate an overall improvement in performance with the new model, particularly in recall and F1 score. The new model achieves a soft recall that is nearly 7 points higher than the baseline, demonstrating its effectiveness in identifying parallel documents using URLs.

Figure 5.

Macro-F1 scores per language (paired with English) for the parallelness identifier from URLs on the dataset described in Section 4.1.

Figure 5 Long description

The bar graph compares Macro F1 scores for different languages using two methods, Baseline and Ours. The x-axis lists languages with abbreviations such as fra, hrv, bul, cnr, mlt, isl, tur, slv, mkd, srp, and sqi. The y-axis ranges from 0 to 100, representing Macro F1 scores. Each language has two bars: a blue bar for Baseline and a red bar for Ours. The red bars consistently show higher scores than the blue bars across all languages. Notable trends include higher scores for languages like fra, hrv, and srp using the Ours method. All values are approximated.

Table 4 presents the results on the test set described in Section 4.1 for all language pairs combined; Figure 5 displays the macro F1 score for each language pair individually. Table 4 reveals the baseline’s poor performance for the positive class, with a low recall below $15\%$ , in contrast to the high recall (above $99\%$ ) for the negative class. This suggests that the baseline’s simplicity struggles with the complexities of parallel URLs. However, it achieves a competitive macro-precision, although not surpassing our system. This is because it looks for explicit language hints in the URLs and if a URL pair is deemed parallel, it is likely to be a true positive, except for potential confusion with common strings in ISO-639 codes (e.g., “isl” confused with “island” instead of Icelandic).

Figure 5 shows that our model systematically outperforms, in terms of macro-F1, the baseline in all languages, but French. Manual inspection confirmed a significant difference between the English–French test set, derived from the WMT16 dataset, and the rest of the test sets, derived from the MaCoCu v2 corpus. About 60% of the positive samples in the English–French dataset exclusively use language identification tokens in URLs as the method to identify translated documents.Footnote ^k Conversely, for the datasets used for the rest of languages this percentage ranges from 10% to 15%. The fact that the baseline designed for the WMT16 task exclusively builds on language identification tokens to detect parallel documents explains the high performance on this specific test set. While the details about the methodology followed to build the WMT16 gold standard are not public, it seems likely that the procedure followed somehow biased this dataset towards over-representing websites using language identification tokens.

5.1 Experimental settings

Dataset. We evaluate the crawling strategy described above on four language pairs: English–Icelandic (eng-isl), English–Maltese (eng-mlt), English–Finnish (eng-fin), and Spanish–Basque (spa-eus). It is worth noting that neither Finnish, Spanish, nor Basque were used to fine-tune the model for inferring parallelness from URLs, and Maltese was not even used to train the base model, XLM-RoBERTa, on which our models build.

We built a collection of seed URLs for each of the four bilingual crawling tasks. To ensure that the websites to be crawled actually contained parallel data, we used the Paracrawl v9 parallel corpus to identify productive websites for the languages covered in our experiment. Paracrawl consists of parallel sentences for 43 language pairs, along with their corresponding source URLs. For each language pair, we built the list of unique URLsFootnote ⁿ and grouped them by websites; we then ranked websites by the amount of URLs in this list. Websites present in the training and development sets used to fine-tune the models described in Sections 3 and 4 were excluded. Ultimately, a list of seed URLs was created for each language pair by collecting the home page of the 200-top websites.Footnote ^o

Evaluation. We measure the performance of our approach in terms of the amount of parallel data downloaded at different moments of a crawling process. We include three models in our comparison: (a) the original implementation of Heritrix; (b) the proposed approach that integrates the two models proposed in Sections 3 and 4, along with CLD2 (Heritrix+CLD2+smart); and (c) a crawler integrating only CLD2 to exclude documents not in the targeted languages (Heritrix+CLD2). The inclusion of the last crawler helps determine the individual contributions of the proposed models and CLD2. We run each version of Heritrix on each website included in the experiment for a maximum of 48 hours. We then evaluate the amount of parallel data obtained by deciles of the total data downloaded per website.

To measure the amount of parallel data acquired, we used BitextorFootnote ^p because manual assessment would have been prohibitively expensive. Although Bitextor may introduce some noise, this equally affects all evaluated approaches. In any case, it is important to note that Bitextor is a state-of-the-art tool for harvesting parallel content (Bañón et al. Reference Bañón, Chen, Haddow, Heafield, Hoang, Esplà-Gomis, Forcada, Kamran, Kirefu, Koehn, Ortiz Rojas, Pla Sempere, Ramírez-Sánchez, Sarrías, Strelec, Thompson, Waites, Wiggins and Zaragoza2020; Esplà-Gomis et al. Reference Esplà-Gomis, Forcada, Ortiz-Rojas and Ferrández-Tordera2016) widely used and that has been utilized to create popular datasets such as ParaCrawl (Bañón et al. Reference Bañón, Chen, Haddow, Heafield, Hoang, Esplà-Gomis, Forcada, Kamran, Kirefu, Koehn, Ortiz Rojas, Pla Sempere, Ramírez-Sánchez, Sarrías, Strelec, Thompson, Waites, Wiggins and Zaragoza2020) and MaCoCu (Bañón et al. Reference Bañón, Esplà-Gomis, Forcada, García-Romero, Kuzman, Ljubešić, van Noord, Sempere, Ramírez-Sánchez, Rupnik, Suchomel, Toral, van der Werff and Zaragoza2022).

5.2 Results and discussion

Figure 7 illustrates the results obtained by the three models; the x-axis corresponds to the percentageFootnote ^q of the total number of documents downloaded for each website after 48 hours of crawling, while the y-axis represents the amount of parallel data identified at each moment. The number of parallel documents for a given percentage of downloaded data was computed separately for each website, and the figure shows the aggregated results. It is worth noting that the number of parallel documents identified can serve as a proxy for crawling speed, avoiding the drawbacks of measuring crawling times, such as server workload or computing resource constraints. If more parallel documents are identified with the same amount of downloaded documents, then we are potentially improving crawling speed and bandwidth usage with respect to the downloaded documents.

Figure 7.

For the three crawlers (Heritrix, Heritrix+CLD2, and Heritrix+CLD2+smart) and the language pairs eng-isl, eng-mlt, eng-fin, and spa-eus, thousands of parallel documents retrieved (y-axis) as a function of the percentage of documents downloaded (x-axis) from each website. Each data point accumulates the number of parallel documents downloaded from each website up to the percentage indicated on the x-axis.

Figure 7 Long description

Four line graphs compare the performance of three crawlers (Heritrix, Heritrix+CLD2, and Heritrix+CLD2+smart) in retrieving thousands of parallel documents for four language pairs (eng-isl, eng-mlt, eng-fin, and spa-eus). The y-axis represents the number of parallel documents retrieved, while the x-axis shows the percentage of documents downloaded from each website. Each data point accumulates the number of parallel documents downloaded from each website up to the percentage indicated on the x-axis. The Heritrix+CLD2+smart crawler consistently retrieves the most documents across all language pairs, followed by Heritrix+CLD2 and Heritrix. The graphs illustrate the efficiency of each crawler in gathering parallel documents, with notable differences in performance for each language pair.

The results obtained for all language pairs confirm that the Heritrix+CLD2+smart approach is capable of identifying most of the available parallel data earlier than the other two approaches. As expected, the difference between the three approaches diminishes as a higher fraction of the available data is downloaded, and they would eventually converge if the crawling time was enough to allow crawling all the data in these websites. Another notable observation is that the trend is consistent across all language pairs, regardless of whether they were seen during training, partially confirming the zero-shot capabilities of our approach (Heritrix+CLD2+smart). Notice that while our approach for inferring parallelness is language-independent, the model used to identify the language of documents from their URLs (see Section 3) was fine-tuned on all languages included in the experiments. Hence, a more extensive evaluation on a larger number of language pairs is required to robustly confirm the zero-shot capabilities of our method.