The Rules of the Search Head Cluster (SHC)

When you’re looking for a way to scale your search capabilities in Splunk, look no further than the Search Head Cluster.

A Splunk search head cluster is a group of Splunk Enterprise search heads that serves as a central resource for searching. As the central source for searching, the cluster is a group of networked searched heads that share configurations, apps, search artifacts, and job scheduling. For example, you can view the same dashboards, access the same search results, and run the same searches from any member of the search head cluster.

Why use a search head cluster in Splunk? SHCs allow for horizontal scaling to increase capacity within your Splunk searches. With high availability, you have increased accessibility in your environment. For example, you can create a dashboard on one cluster member and then access the same dashboard on another cluster member. With a SHC, there is no single point of failure. The cluster uses a dynamic captain to manage the cluster. If the captain goes down, another member automatically takes over the management of the cluster.

 

The Search Head Basics

When creating a search head cluster, the cluster must contain at least 3 search heads. These search heads must share configurations, jobs, and search artifacts. Don’t forget about these two key components to the cluster…

 

The deployer –A Splunk instance that distributes apps and other configuration to the cluster members. The deployer is not part of a cluster member, it cannot run on the same instance as a cluster member.

Search peers – the indexers that cluster member run their searches across.

 

Oh Captain, My (Search) Captain

In a search head cluster, you’ll need a captain. Similar to the role a captain plays manning their ship, a SHC captain is a cluster member that regulates the functioning of the cluster.

The captain coordinates job scheduling and replication activities among all the members.  It also serves as a search head like any other member, running a search job, and so on. Over time, the role of the captain can shift among the cluster members.

 

Set Up Your Search Head

 Before we jump into the install process, let’s look at these few items to consider…

1. A search head cluster have at least three members
2. You must use identical hardware and specification for all SHC members
3. It’s recommended to always use a new Splunk instance for your cluster

Now that we’ve covered our bases, let’s jump into the SHC set up.

In order to set up the SHC, it’s recommended that we set up the Deployer first. You’ll use the deployer to distribute apps and updated configuration to the cluster members.

 

 

1. Set up the Deployer

To set up the deployer, follow these steps…

1. Choose a Splunk Enterprise instance for the deployer
2. Create a security key for SHClustering in server.conf. This same security key would also be used for the SHC members.
3. Restart the deployer (Splunk restart)

2. Initialize the Cluster

Now that we’ve set up our deployer, let’s initialize the cluster members. Run the following command on all SHC members (in $Splunk_Home/bin).

Splunk init SHCluster-config -mgmt_uri <sh uri>:8089 replication_port <port_number> -replication_factor 2 -conf_deploy_fetch_url https://deployerurl:8089 -secret <security password here>

 

For example, for shearhead1, you could write this…

Splunk init SHCluster-config -mgmt_uri https://searchhead1url:8089 -replication_port 34321 -replication_factor 2 -conf_deploy_fetch_url https://deployerurl:8089 -secret securitykeyhere

Then, run a Splunk restart after initializing the cluster.

 

3. Bring up the Cluster Captain

Once the cluster is up and running, you’ll select your captain. Select any one of the initialized instances to be the first cluster captain. Run the Splunk bootstrap SHCluster-captain command on one of the instances.

For example, you could run the following command:

Splunk bootstrap SHCluster-captain -servers_list “https://searchhead1url:8089, https://searchhead2url:8089, https://searchhead3url:8089: -auth <admin>:<adminpassword>

Once you select a captain, you’ve successfully set up a SHC!

Splunk Pro Tip: This type of work can be a considerable resource expense when executing it in-house. The experts at Kinney Group have several years of experience architecting, creating, and solving in Splunk. With Expertise on Demand, you’ll have access to some of the best and brightest minds to walk you through simple and tough problems as they come up.

See for Yourself

Search No Further

Creating search head clusters should be an easy task, that optimizes your Splunk environment. Let’s say you already have too much Splunk work on your plate. Maybe search head clustering isn’t your only issue with Splunk. Whatever the Splunk task, small or large, Kinney Group has the team who can help. Let us know in the form below!