5.1.4. Token indexing

After the tokens have gone through zero or more token filters, they’re sent to Lucene to be indexed for the document. These tokens make up the inverted index we discussed back in chapter 1.

Together, these different parts make up an analyzer, which can also be defined as zero or more character filters, a tokenizer, and zero or more token filters. There are some prebuilt analyzers we’ll talk about later on in this chapter that you can use without having to construct your own, but first we’ll talk about the individual components of an analyzer.