9.3. Removing nodes from a cluster

Adding nodes is a great way to scale, but what happens when a node drops out of the Elasticsearch cluster or you stop the node? Use the three-node example cluster you created in figure 9.2, containing the test index with five primary shards and one replica spread across the three nodes.

Let’s say Joe, the sys admin, accidentally trips over the power cord for Node1; what happens to the three shards currently on Node1? The first thing that Elasticsearch does is automatically turn the test0 and test3 replica shards that are on Node2 into primary shards, as shown in figure 9.6. This is because indexing first goes to the primary shards, so Elasticsearch tries hard to make sure there are always primaries assigned for an index.

Figure 9.6. Turning replica shards into primaries after node loss

Elasticsearch can choose any of the replicas to turn into a primary shard. It just so happens in this example that there’s only one replica for each primary shard to choose from: the replicas on Node2.

After Elasticsearch turns the replicas for the missing primary shards into primaries, the cluster looks like figure 9.6.

After turning the replica shards into primaries, the cluster is now in a yellow state, meaning that some replica shards aren’t allocated to a node. Elasticsearch next needs to create more replica shards to maintain the high-availability setup for the test index. Because all the primaries are available, the data from the test0 and test3 primary shards on Node2 will be replicated into replicas on Node3, and the data from the test1 primary shard on Node3 will be replicated onto Node2, as shown in figure 9.7.

Figure 9.7. Re-creating replica shards after losing a node

Once the replica shards have been re-created to account for the node loss, the cluster will be back in the green state with all primary and replica shards assigned to a node. Keep in mind that during this time the entire cluster will be available for searching and indexing because no data was actually lost. If more than a single node is lost or a shard with no replicas is lost, the cluster will be in a red state, meaning that some amount of data has been lost permanently, and you’ll need to either reconnect the node that has the data to the cluster or re-index the data that’s missing.

It’s important to understand how much risk you’re willing to take with regard to the number of replica shards. Having a single replica means that one node can disappear from the cluster without data loss; if you use two replicas, two nodes can be lost without data loss, and so on, so make sure you choose the appropriate number of replicas for your usage. It’s also always a good idea to back up your indices, which is a subject we’ll cover in chapter 11 when we talk about administering your cluster.

You’ve seen what adding and removing a node looks like, but what about shutting down a node without having the cluster go into a yellow state? In the next section we’ll talk about decommissioning nodes so that they can be removed from the cluster with no interruption to the cluster users.

9.3.1. Decommissioning nodes

Having Elasticsearch automatically create new replicas when a node goes down is great, but when maintaining a cluster, you’re eventually going to want to shut down a node that has data on it without the cluster going into a yellow state. Perhaps the hardware is degraded or you aren’t receiving the same number of requests you previously were and don’t need to keep the node around. You could always stop the node by killing the Java process, and Elasticsearch would recover the data to the other nodes, but what about when you have zero replicas for an index? That means you could lose data if you were to shut down a node without moving the data off first!

Thankfully, Elasticsearch has a way to decommission a node by telling the cluster not to allocate any shards to a node or set of nodes. In our three-node example, let’s assume that Node1, Node2, and Node3 have the IP addresses of 192.168.1.10, 192.168.1.11, and 192.168.1.12, respectively. If you wanted to shut down Node1 while keeping the cluster in a green state, you could decommission the node first, which would move all shards on the node to other nodes in the cluster. You decommission a node by making a temporary change to the cluster settings, as shown in the following listing.

Listing 9.3. Decommissioning a node in the cluster

Once you run this command, Elasticsearch will start moving all the shards from the decommissioned node to other nodes in the cluster. You can check where shards are located in the cluster by first determining the ID of the nodes in the cluster with the _nodes endpoint and then looking at the cluster state to see where each shard in the cluster is currently allocated. See the next listing for example output of these commands.

Listing 9.4. Determining shard location from the cluster state

This is a long and ugly listing! Don’t worry, though; later on this chapter we’ll talk about a more humanreadable version of this API called the _cat API.

Here you can see that there are no shards on the lFd3ANXiQlug-0eJztvaeA node, which is the 192.168.1.10 node that was decommissioned, so it’s now safe to stop ES on that node without causing the cluster to leave a green state. This process can be repeated one node at a time to decommission each node you want to stop, or you can use a list of comma-separated IP addresses instead of 192.168.1.10 to decommission multiple nodes at once. Keep in mind, however, that the other nodes in the cluster must be able to handle allocating the shard in terms of disk and memory use, so plan accordingly to make sure you have enough headroom before decommissioning nodes!

How much data can an Elasticsearch index handle?

Good question! Unfortunately, the limitations of a single index depend on the type of machine used to store the index, what you’re planning to do with the data, and how many shards the index is backed by. Generally, a Lucene index (also known as an Elasticsearch shard) can’t have more than 2.1 billion documents or more than 274 billion distinct terms (see

https://lucene.apache.org/core/4_9_0/core/org/apache/lucene/codecs/lucene49/packagesummary.html#Limitations), but you may be limited in disk space before this point. The best way to tell whether you’ll be able to store your data in a single index is to try it out in a nonproduction system, adjusting settings as needed to get the performance characteristics desired. You can’t change the number of primary shards once an index has been created; you can only change the number of replica shards, so plan accordingly!

Now that you’ve seen how nodes are added and removed from the cluster, let’s talk about how to upgrade Elasticsearch nodes.