A full-text index for a string T = t1t2 … tn is a data structure that is built once, and that is kept in memory for answering an arbitrarily large number of queries on the position and frequency of substrings of T, with a time complexity that depends sublinearly on n. Consider for example a set R = {R1, R2, …, Rd} of reads generated by a highthroughput sequencing machine from a genome G: assuming that each Ri is an exact copy of a substring of G, a routine question in read analysis is finding the position of each Ri in G. The lens of full-text indexes can also be used to detect common features of a set of strings, like their longest common substring: such analyses arise frequently in evolutionary studies, where substrings that occur exactly in a set of biological sequences might point to a common ancestral relationship.

The problem of matching and counting exact substrings might look artificial at a first glance: for example, reads in R are not exact substrings of G in practice. In the forthcoming chapters we will present more realistic queries and analyses, which will nonetheless build upon the combinatorial properties and construction algorithms detailed here. Although this chapter focuses on the fundamental data structures and design techniques of classical indexes, the presentation will be complemented with a number of applications to immediately convey the flexibility and power of these data structures.

k-mer index

The oldest and most popular index in information retrieval for natural-language texts is the so called inverted file. The idea is to sort in lexicographic order all the distinct words in a text T, and to precompute the set of occurrences of each word in T. A query on any word can then be answered by a binary search over the sorted list. Since biological sequences have no clear delimitation in words, we might use all the distinct substrings of T of a fixed length k. Such substrings are called k-mers.