9.2. Discovering other Elasticsearch nodes

You might be wondering exactly how the second node you added to your cluster discovered the first node and automatically joined the cluster. Out of the box, Elasticsearch nodes can use two different ways to discover one another: multicast or unicast. Elasticsearch can use both at once but by default is configured to use only multicast because unicast requires a list of known nodes to connect to.

9.2.1. Multicast discovery

When Elasticsearch starts up, it sends a multicast ping to the address 224.2.2.4 on port 54328, which in turn is responded to by other Elasticsearch nodes with the same cluster name, so if you notice a coworker’s local copy of Elasticsearch running and joining your cluster, make sure to change the cluster.name setting inside your elasticsearch.yml configuration file from the default

elasticsearch to a more specific name. Multicast discovery has a few options that you can change or disable entirely by setting the following options in elasticsearch.yml, shown with their default values:

Generally, multicast discovery is a decent option when dealing with very flexible clusters on the same network, where the IP address of nodes being added changes frequently. Think of multicast discovery as shouting “Hey, are there any other nodes out there running an Elasticsearch cluster named ‘xyz’?” and then waiting for a response. Figure 9.3 shows what multicast discovery looks like graphically.

Figure 9.3. Elasticsearch using multicast discovery to discover other nodes in the cluster

Although multicast discovery is great for local development and a quick proof-of-concept test, when developing a production cluster, a more stable way of having Elasticsearch discover other nodes is to use some or all of the nodes as “gossip routers” to discover more information about the cluster. This can prevent the situation where nodes accidentally connect to a cluster they shouldn’t have when someone connects a laptop to the same network. Unicast helps combat this by not sending a message to everyone on a network but connecting to a specific list of nodes.

9.2.2. Unicast discovery

Unicast discovery uses a list of hosts for Elasticsearch to connect to and attempt to find more information about the cluster. This is ideal for cases where the IP address of the node won’t change frequently or for production Elasticsearch systems where only certain nodes should be communicated with instead of the entire network. Unicast is used by telling Elasticsearch the IP address and, optionally, the port or range of ports for other nodes in the cluster. An example of a unicast configuration would be setting

discovery.zen.ping.unicast.hosts: ["10.0.0.3", "10.0.0.4:9300",

"10.0.0.5[9300-9400]"] inside elasticsearch.yml for the Elasticsearch nodes on your network. Not all of the Elasticsearch nodes in the cluster need to be present in the unicast list to discover all the nodes, but enough addresses must be configured for each node to know about a gossip node that’s available. For example, if the first node in the unicast list knows about three out of seven nodes in a cluster, and the second node in the unicast list knows about the other four out of the seven nodes, the node performing the discovery will still be able to find all seven nodes in the cluster. Figure 9.4 shows a graphical representation of unicast discovery.

Figure 9.4. Elasticsearch using unicast discovery to discover other nodes in the cluster

There’s no need to disable unicast discovery. If you’d like to use only multicast discovery to find other Elasticsearch nodes, leave the list unset (or empty) in the configuration file. After discovering other nodes that are part of the cluster, the Elasticsearch nodes will hold a master election.

9.2.3. Electing a master node and detecting faults

Once the nodes in your cluster have discovered each other, they’ll negotiate who becomes the master. The master node is in charge of managing the state of the cluster—that is, the current settings and state of the shards, indices, and nodes in the cluster. After the master node has been elected, it sets up a system of internal pings to make sure each node stays alive and healthy while in the cluster; this is called fault detection, which we’ll talk more about at the end of this section. Elasticsearch considers all nodes eligible to become the master node unless the node.master setting is set to false. We’ll talk more in this chapter about why you may want to set the node.master setting, and the different types of Elasticsearch nodes, when we talk about how to search faster. In the event that your cluster has only a single node, that node will elect itself as the master after a timeout period if it doesn’t detect any other nodes in the cluster.

For production clusters with more than a couple of nodes, it’s a good idea to set the minimum number of master nodes. Although this setting may make it seem like Elasticsearch can have multiple master nodes, it actually tells Elasticsearch how many nodes in a cluster must be eligible to become a master before the cluster is in a healthy state. Setting the minimum number of eligible master nodes can be helpful in making sure your cluster doesn’t try to perform potentially dangerous operations without first having a complete view of the state of your cluster. You can either set the minimum number to the total number of nodes in your cluster if the number of nodes doesn’t change over time or set it according to a common rule, which is the number of nodes in your cluster divided by 2, plus 1. Setting the minimummaster_nodes setting to a number higher than 1 can help prevent what’s called a _split brain in the cluster. Following the common rule for a three-node cluster, you’d set minimum_master_nodes to 2, or for a 14-node cluster, you’d set the value to 8. To change this setting, change

discovery.zen.minimum_master_nodes in elasticsearch.yml to the number that fits your cluster.

What’s a split brain?

The term split brain describes a scenario where (usually under heavy load or network issues) one or more of the nodes in your Elasticsearch cluster loses communication to the master node, elects a new master, and continues to process requests. At this point, you may have two different Elasticsearch clusters running independently of each other—hence the term split brain, because a single cluster has split into two distinct parts, similar to the hemispheres in a brain. To prevent this from happening, you should set

discovery.zen.minimum_master_nodes depending on the number of nodes in your cluster. If

the number of nodes won’t change, set it to the total number of nodes in the cluster; otherwise the number of nodes divided by 2 plus 1 is a good setting, because that means that if one or two nodes lose communication to the other nodes, they won’t be able to elect a new master and form a new cluster because they don’t meet the required number of master-eligible nodes.

Once your nodes are up and have discovered each other, you can see what node your cluster has elected as master by using the curl command shown in the following listing.

Listing 9.2. Getting information about nodes in the cluster with curl

9.2.4. Fault detection

Now that your cluster has two nodes in it, as well as an elected master node, it needs to communicate with all nodes in the cluster to make sure everything is okay within the cluster; this is called the fault detection process. The master node pings all other nodes in the cluster and each node pings the master to make sure an election doesn’t need to be held, as shown in figure 9.5.

Figure 9.5. Cluster fault detection by the master node

As the figure shows, each node sends a ping every discovery.zen.fd.ping_interval (defaulting to 1s), waits for discovery.zen.fd.ping_timeout (defaulting to 30s), and tries a maximum number of discovery.zen.fd.ping_retries times (defaulting to 3) before declaring a node disconnected and routing shards or holding a master election as necessary. Be sure to change these values if your environment has higher latency—say, when running on ec2 nodes that may not be in the same Amazon AWS zone.

Inevitably, one of the nodes in your cluster will go down, so in the next section, we’ll talk about what happens when nodes are removed from the cluster and how to remove nodes without causing data loss in a distributed system.