What is Splunk Deployment?

Splunk Deployment is the process of managing configuration files among different Splunk instances. Depending on the role of each instance, certain configuration files must be deployed.

Most Splunk environments start with a very small number of servers (50 or less), then gradually increase to tens of thousands of servers.

Managing a small number of instances is much easier than having to manage a very large number of instances. However, following Splunk deployment best practices makes it easier to manage regardless of the size and number of servers involved.

In this post, we walk you through the components of Splunk deployment, best practices for Splunk deployment, and a few examples to get you started on the right foot.

Splunk Deployment Components

Deployment Server: The Splunk instance that acts as a centralized configuration manager and manages the distribution of configuration files to other Splunk instances.   

Deployment Client: The Splunk instance that is remotely configured by a deployment server. 

Server Class: Represents a group of deployment clients as a single unit. It can be grouped by application used, OS, data type to be indexed, or any other feature matching the group of clients.  

Deployment App: A set of configuration files (bundle) deployed as a single unit to clients grouped by a server class. It is in $SPLUNK_HOME/etc/deployment-apps on the deployment server and is pushed to deployment client’s $SPLUNK_HOME/etc/apps folder. 

Splunk Deployment Best Practices

Below are Splunk deployment best practices related to the components listed above. 

• For 50 or fewer clients, a deployment server can co-exist with any Splunk Instance, like a search head or an indexer. A dedicated instance of Splunk is required if you have more than 50 clients.

• A Deployment Server requires many active TCP sessions—one for each connected client. In this case, consider an instance that has enough TCP sessions to manage all of the required clients.

• Use a DNS hostname for the deployment server instead of an IP Address used by deployment clients, this will make it easier to migrate or change deployment server, if needed, without having to make changes at the deployment clients. 

• Consider using Splunk’s clientName feature in deploymentclient.conf to allow subdividing a class of servers into further sub-classes. For example, you might have a Linux server class that groups all Linux servers together, adding clientName=Apache for Apache servers, or clientName=DB for DB servers can subdivide the Linux Servers class into multiple sub-classes, one for Apache servers and another for DB servers and so on. This way you can deploy apps to the whole group of Linux servers, as well as individual apps for each application.

• For large environments, consider adjusting the polling period for clients that are less important. This allows the deployment server to handle more clients if needed. 

• Splunk deployment server pushes App bundles into $SPLUNK_HOME/etc/apps folder on clients. However, configuration files in $SPLUNK_HOME/etc/system/local have the highest precedence and cannot be overridden by deployment server. Because of this, take our advice and avoid storing any configurations in $SPLUNK_HOME/etc/system/local. 

• If you are using Puppet, Chef of any other deployment tool other than deployment server, utilize the concept of deployment apps, as this is easier to manage and troubleshoot. 

• For deployment apps, use a single function for any deployment apps you are trying to deploy. For example, if you need to set up inputs and props entries for a class of servers, create two deployment apps, one for inputs and the other for props, and don’t mix up the two sets into one app. 

• Use deployment app naming convention to simplify the process of deployment and assist in troubleshooting issues if needed.

Splunk Deployment Examples

1. kgi_marketing_apache_inputs  

These are the inputs files for marketing group of Apache servers at Kinney Group. 

2. kgi_finance_iis_props  

These are the props files for finance group of IIS servers at Kinney Group. 

3. kgi_all_indexer_base 

These are the base configuration files for all indexers at Kinney Group. 

Conclusion  

The Splunk Deployment process is done differently from one company to another and can become very difficult to manage in large environments, especially when having to change administrators at different times. Following these Splunk deployment best practices will give you a uniform way to manage the deployment process and make it easier to troubleshoot issues when they arise. 

If you found this helpful…

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment: Instantly see where your Splunk environment is excelling and opportunities for improvement. From download to results, the whole process takes less than 30 minutes using the button below: