11.2. Allocation awareness

This section covers the concept of laying out cluster topology to reduce central points of failure and improve performance by using the concept of allocation awareness. Allocation awareness is defined as knowledge of where to place copies (replicas) of data. You can arm Elasticsearch with this knowledge so it intelligently distributes replica data across a cluster.

11.2.1. Shard-based allocation

Allocation awareness allows you to configure shard allocation using a self-defined parameter. This is a common best practice in Elasticsearch deployments because it reduces the chances of having a single point of failure by making sure data is evened out among the network topology. You can also experience faster read operations, as nodes deployed on the same physical rack will potentially have a proximity advantage of not having to go over the network.

Enabling allocation awareness is achieved by defining a grouping key and setting it in the appropriate nodes. For instance, you can edit elasticsearch.yml as follows: cluster.routing.allocation.awareness.attributes: rack

Note

The awareness attribute can be assigned more than one value.

cluster.routing.allocation.awareness.attributes: rack, group, zone

Using the previous definition, you’ll segment your shards across the cluster using the awareness parameter rack. You alter the elasticsearch.yml for each of your nodes, setting the value the way you want your network configuration to be. Note that Elasticsearch allows you to set metadata on nodes. In this case, the metadata key will be your allocation awareness parameter: node.rack: 1

A simple before-and-after illustration may help in this case. Figure 11.1 shows a cluster with the default allocation settings.

Figure 11.1. Cluster with default allocation settings

This cluster suffers from primary and replica shard data being on the same rack. With the allocation awareness setting, you can remove the risk, as shown in figure 11.2.

Figure 11.2. Cluster with allocation awareness

Wwith allocation awareness, the primary shards were not moved, but the replicas were moved to nodes with a different node.rack parameter value. Shard allocation is a convenient feature that insured against a central point of failure. A common use is separating cluster topology by location, racks, or even virtual machines.

Next, we’ll take a look at forced allocation with a real-world AWS zone example.

11.2.2. Forced allocation awareness

Forced allocation awareness is useful when you know in advance the group of values and want to limit the number of replicas for any given group. A real-world example of where this is commonly used is in cross-zone allocation on Amazon Web Services or other cloud providers with multizone capabilities. The use case is simple: limit the number of replicas in one zone if another zone goes down or is unreachable. By doing this, you reduce the danger of overallocating replicas from another group.

For example, in this use case you want to enforce allocation at a zone level. First you specify your attribute, zone, as you did before. Next, you add dimensions to that group: us-east and us-west. In your elasticsearch.yml, you add the following:

cluster.routing.allocation.awareness.attributes: zone cluster.routing.allocation.force.zone.values: us-east, us-west

Given these settings, let’s play out this real-world scenario. Let’s say you start a set of nodes in the East region with node.zone: us-east. You’ll use the defaults here, leaving an index with five primary shards and one replica. Because there are no other zone values, only the primary shards for your indices will be allocated.

What you’re doing here is limiting the replicas to balance only on nodes without your value. If you were to start up your West region cluster, with node.zone: us-west, replicas from us-east would be allocated to it. No replicas will ever exist for nodes defined as node.zone: us-east. Ideally, you’d do the same on node.zone: us-west, thereby ensuring that replicas never exist in the same location. Keep in mind that if you lose connectivity with us-west, no replicas will ever be created on us-east, or vice versa.

Allocation awareness does require some planning up front, but in the event that allocation isn’t working as planned, these settings can all be modified at runtime using the Cluster Settings API. They can be persistent, where Elasticsearch applies the settings even after a restart, or temporary (transient):

curl -XPUT localhost:9200/_cluster/settings -d '{

"persistent" : {

"cluster.routing.allocation.awareness.attributes": zone

"cluster.routing.allocation.force.zone.values": us-east, us-west

}

}

Cluster allocation can make the difference between a cluster that scales and is resilient to failure and one that isn’t.

Now that we’ve explored some of the finer adjustments that you can make to Elasticsearch default settings with shard allocation, let’s look at how to monitor the general health of your cluster for performance issues.