Splunk Enterprise utilizes multiple configuration files to control which functional role(s) a given instance performs and to set a myriad of operating parameters that determine nearly every aspect of its behavior. Examples of the configuration files that are commonly adjusted by Splunk admins include inputs.conf, outputs.conf, server.conf, indexes.conf, and many more. Note that all Splunk configuration files utilize the .conf extension.
Configuring the Splunk server.conf file
The server.conf file is a key configuration file in a distributed Splunk Enterprise deployment in that it controls what functional role each node will play in the solution, and how these nodes communicate with each other. This article will examine the Splunk server.conf file in-depth and the most typical entries required to configure a Splunk server for each role.
Splunk Configuration File Folder Precedence
Any discussion of Splunk configuration files must be considered in the context of directory location ‘precedence’, which is the hierarchy in which Splunk assigns the various .conf file entry values to memory during startup and utilizes them during its operation.
When the configuration file context is global, directory priority descends in this order:
- System local directory — highest priority
- App local directories
- App default directories
- System default directory — lowest priority
By way of further explanation, there are a number of .conf files, including a server.conf file, located in the $SPLUNK_HOME/etc/system/default folder. Each of these .conf files contain a large number of default configuration settings. When you make configuration changes that affect the operation of Splunk either via Splunk Web or from the command line, deviations from the default settings in server.conf in the default folder will be stored in another server.conf file located in the $SPLUNK_HOME/etc/system/local folder. You’ll note from the directory precedence outlined above those settings that reside in a file in the local folder will take precedence over those in the default folder.
For example, one of the stanzas and entries in the /etc/system/default/server.conf file on a Splunk server that is part of a clustered solution with a License Manager is:
[license] manager_uri = self In the /etc/system/local/server.conf file on this server we find this setting that points to the License Manager to obtain licensing – this setting takes precedence over the default setting: [license] manager_uri = https://10.2.6.36:8089
This folder precedence convention is a convenient way of managing a complex environment but is also why Splunk docs always warn against making any changes to .conf files in default folders – the local folder settings take precedence and will be preserved in the case of a Splunk version upgrade. Said another way, any changes made to a .conf file in the $SPLUNK_HOME/etc/system/default folder will be overwritten during an upgrade. Any changes you manually enter that will alter or supplement default settings should therefore be placed in a same-named .conf file residing in a /local folder, for both system and app level configurations.
A final consideration is that while you could place system-level configuration settings into .conf files located in the /default or /local folder of an app folder in $SPLUNK_HOME/etc/apps/, this will result in these significant server settings being scattered all over the server such that it is difficult to determine what the total/final configuration really is without using btool or a lot of hunt-and-peck searching. Best practice suggests that non-default system level .conf files should be kept in $SPLUNK_HOME/etc/system/local, with very few exceptions.
server.conf Settings for Various Splunk Functions
In the following sections the significant server.conf settings that determine which role a Splunk server plays and how it interacts with other Splunk nodes in a distributed deployment are explained and compared.
Note: The use of the term ‘master’ or ‘slave’ for some configuration settings has been replaced with ‘manager’ or ‘peer’ in more recent versions of Splunk. You may still see ‘master_uri’ or similar entries in older versions.
Standalone Splunk Enterprise or License Manager
The server.conf in the $SPLUNK_HOME/etc/system/local folder of a standalone instance of Splunk, or a Splunk server that is serving as a License Manager and has a non-free Splunk license installed will have the following entries:
[general] serverName = ip-10-2-6-36.ec2.internal pass4SymmKey = $7$ZEqxybfpYs1H2DNH4eny19ih84dz4suy+RLmhMJeoS+YzGdt8YbNEsgBbjs1 [sslConfig] sslPassword = $7$p35gM9btomGgv2OUzJBUQ7wlcZflYy9SHDW3o0n6QYD2jhuw6MOwUQ== [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial peers = * quota = MAX stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder peers = * quota = MAX stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free peers = * quota = MAX stack_id = free [lmpool:auto_generated_pool_enterprise] description = auto_generated_pool_enterprise peers = * quota = MAX stack_id = enterprise [license] active_group = Enterprise The significant stanzas and entries include: [general]
serverName: the name that identifies this Splunk software instance for features such as distributed search.
pass4SymmKey: this hashed password key is used to authenticate traffic between a license manager and its license peers, members of a cluster, or a deployment server and its deployment clients. The keys must match between these members.
[sslConfig] sslPassword: the server certificate password, if any, used for SSL/TLS communications.
[lmpool_auto_generated_pool_<function>]: auto-generated license pool configurations to support the use of download-trial, forwarder, free, or enterprise license types.
[license] active_group: this sets the license type.
With the exception of the license related entries, there are no function related entries in the local folder server.conf for a standalone or license manager node.
Cluster Manager
The Cluster Manager is responsible for pushing indexer configurations and managing bucket replication across the indexing cluster. In the following server.conf entries, the [general] and [sslConfig] stanzas and entries have been omitted (as they will be in further examples) as they are the same as for the previous example.
[license] manager_uri = https://10.2.6.36:8089 [indexer_discovery] pass4SymmKey = $7$2N4Q7EspaU2oX8AaeleICi05sDJJg7GAT9gMwWxYc9rZf86bE0CHEF3ht0mb polling_rate = 1 [clustering] mode = manager replication_factor = 2 search_factor = 2 pass4SymmKey = $7$/gdzL5bjZEAv0v6Dy3yMKtdy/JqTGQblPKUhkHRbcTa+VmBmrKmzZhNtxw== cluster_label = mdi_svac3_idxc access_logging_for_heartbeats = 0 cm_heartbeat_period = 3 precompress_cluster_bundle = 1 rebalance_threshold = 0.9
The significant stanzas and entries for a Cluster Manager include:
[license] manager_uri: this is the URL of the license manager for licensing purposes.
[indexer_discovery]:
pass4SymmKey: hashed password for authenticating with Universal Forwarder nodes that use this feature.
polling_rate: set the indexer discovery polling rate.
[clustering]
mode: sets operational mode for this cluster node – manager = cluster manager.
replication_factor, search_factor: set the RF/SF for the indexing cluster.
pass4SymmKey: hashed password for authenticating with indexing cluster nodes.
cluster_label: specifies the label of the indexer cluster – only valid for mode = manager.
access_logging_for_heartbeats: enables/disables logging to the splunkd_access.log file
for peer heartbeats – only valid for mode = manager.
cm_heartbeat_period: frequency, in seconds, of cluster manager to cluster.
manager heartbeat for mode = manager and manager_switchover_mode = auto|manual
precompress_cluster_bundle: whether or not the manager compresses the configuration bundle files before it pushes them to peers.
rebalance_threshold: used as a percentage to determine when the cluster is balanced. 1 = 100% balanced.
The last four entries above are only valid for cluster manager nodes; cm_heartbeat_period is related to the redundant cluster manager capability introduced in Splunk v9.0.
Indexer
The server.conf entries for indexers are fairly simple – the [general] and [sslConfig] stanzas and entries are omitted below as they reflect examples already given.
[license] manager_uri = https://10.2.6.36:8089 [clustering] manager_uri = https://10.2.6.33:8089 mode = peer pass4SymmKey = $7$/gdzL5bjZEAv0v6Dy3yMKtdy/JqTGQblPKUhkHRbcTa+VmBmrKmzZhNtxw== [replication_port://9887]
The significant stanzas and entries for an indexer include:
[license] manager_uri: this is the URL of the license manager for licensing purposes. [clustering] manager_uri: URL of the cluster manager(s).
mode: sets operational mode for this cluster node – peer = an indexer
pass4SymmKey: hashed password for authenticating with indexing cluster nodes.
replication_port: TCP port for replicating bucket data to/from other cluster members.
Indexers communicate with the cluster manager via the master_uri to discover other indexers in the cluster, and replicate data buckets to those other indexers per the replication factors specified in the cluster manager server.conf.
Search Head in a Search Head Cluster
The server. conf entries for clustered search heads are a bit more involved than for other nodes because they communicate with the Deployer to fetch configuration bundles, communicate with a Captain and other search heads for knowledge object replication and scheduled search management, communiate with the Cluster Manager to discover indexer nodes, and directly with indexers for search operations. The [general], [sslConfig], and [license] stanzas and related entries are omitted below as they reflect examples already given.
[replication_port://9777] [shclustering] conf_deploy_fetch_url = https://10.2.6.39:8089 mgmt_uri = https://ip-10-2-4-40.ec2.internal:8089 replication_factor = 2 pass4SymmKey = $7$cCuCd3Yv0HloWIvJwVZsw7vrI3LHuxNf21dJnhe6U5bX3TcIHoNoC+ugKg== shcluster_label = mdi_svac3_shc id = CE32AC4C-7715-4907-B1D2-C955D140E8C0 [clustering] manager_uri = https://10.2.6.33:8089 mode = searchhead pass4SymmKey = $7$/gdzL5bjZEAv0v6Dy3yMKtdy/JqTGQblPKUhkHRbcTa+VmBmrKmzZhNtxw==
The significant stanzas and entries for a clustered search head include:
replication_port: TCP port for replicating knowledge objects to/from other search head cluster members.
[shclustering] (communicating with the deployer and other search head cluster members)
conf_deploy_fetch_url: URL of a deployer from which members fetch configuration bundles.
mgmt_uri: used to identify the cluster member’s own address to itself.
replication_factor: how many copies of search artifacts are created in the cluster.
pass4SymmKey: hashed password for authenticating with the Deployer and other search head cluster nodes.
shcluster_label: the label of the search head cluster.
[clustering] (settings for communicating with the cluster manager and indexers)
manager_uri: URL to the cluster manager(s).
mode: sets operational mode for this cluster node
pass4SymmKey: hashed password for authenticating with the cluster manager and indexing cluster nodes.
Deployer
The Deployer distributes configuration bundles (apps and private user folders for search heads) to the search head cluster members. The [general], [sslConfig], and [license] stanzas and related entries are omitted below as they reflect examples already given.
[shclustering] pass4SymmKey = $7$cCuCd3Yv0HloWIvJwVZsw7vrI3LHuxNf21dJnhe6U5bX3TcIHoNoC+ugKg== shcluster_label = mdi_svac3_shc
The significant stanzas and entries for a Deployer include:
[shclustering] (communicating with the search head cluster members)
pass4SymmKey: hashed password for authenticating with search head cluster nodes.
shcluster_label: the label of the search head cluster.
Deployment Server and Heavy Forward
The deployment server’s only role is to distribute applications to Splunk Universal Forwarders and any other Splunk nodes as specified by entries in the serverclass.conf that resides in $SPLUNK_HOME/etc/system/local; the presence of this file invokes this functionality.
Heavy forwarders are typically populated with apps containing other .conf files that configure inputs and any event data parsing before forwarding data to the indexing tier via outputs.conf settings.
Since these are basically standalone Splunk Enterprise servers the non-default entries in the /local server.conf file are pretty generic.
[general] serverName = ip-10-2-6-5.ec2.internal pass4SymmKey = $7$ZEqxybfpYs1H2DNH4eny19ih84dz4suy+RLmhMJeoS+YzGdt8YbNEsgBbjs1 [sslConfig] sslPassword = $7$CKCCf2zYemSzb+IBpqGg6Nfxt78GMIbz1ZSREemWiP1tfxGmhPon/Q== [license] manager_uri = https://10.2.6.36:8089
This concludes an overview of typical server.conf entries for the various Splunk Enterprise nodes in a distributed, clustered environment.
Note that the hashed pass4SymmKey password in the [shcluster] section of server.conf should be the same across the Deployer and all clustered search heads. The hashed pass4SymmKey password for the [clustering] section should be the same across the Cluster Manager and all indexer nodes, as well as the [clustering] section of server.conf on search heads. Finally, the hashed pass4SymmKey password for Indexer Discovery must be the same between the Cluster Manager and the UF nodes. If Splunk configurations look correct but nodes aren’t connecting and functioning correctly, checking that these passwords are properly matched is essential.
Conclusion
This tour of the typical non-default server.conf stanzas and entries should be helpful for understanding how these settings work together across a distributed, clustered Splunk deployment, and for verifying that the appropriate entries and reasonable settings are in place while troubleshooting a functional issue within a new or updated Splunk environment.
You can view all of the server.conf settings and their default and allowable settings in the Splunk docs.
If you found this helpful…
You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.
Cue Atlas Assessment: Instantly see where your Splunk environment is excelling and opportunities for improvement. From download to results, the whole process takes less than 30 minutes using the button below:
