Skip to content
Article

The Ultimate Guide to Splunk Universal Forwarders

 

Written by: The Kinney Group Team | Last Updated:

 
November 14, 2022
 
 
 

Originally Published:

 
November 11, 2022

For many getting started with Splunk, the question of “How do I get my data into Splunk” comes up quite regularly. The answer to that question is most often: “use the Universal Forwarder.”

What is a Splunk Universal Forwarder?

The Universal Forwarder is a Splunk instance that can be installed on just about any operating system (OS). Once installed, the Universal Forwarder can be configured to collect systems data and forward it to Splunk Indexers. The Universal Forwarder can also be configured to send data to other forwarders or third-party systems as well if you so desire.

New call-to-action

Universal Forwarders use significantly fewer resources than other Splunk products. You can install literally thousands of them without impacting network performance and cost. The Universal Forwarder does not have a graphical user interface, but you can interact with it through the command line or REST endpoints. The Universal Forwarder also comes with its own license pre-installed, so there is no need to purchase a license for it.

There are many benefits to using a Universal Forwarder to forward your logs as opposed to other solutions.

  1. Forwarding data from a Universal Forwarder is reliable right out of the box, but it can be configured to further protect in-flight data using Indexer Acknowledgement.
  2. The Universal Forwarder uses an internal index called the fishbucket which is used to track previously read files and directories so that Splunk does not send the same data twice.
  3. Universal Forwarders can be configured for load balancing which enables scaling and improved performance.
  4. Using a Universal Forwarder gives you the ability to remotely manage its configurations using apps and add-ons that are deployed by a Deployment Server.

Splunk Pro Tip: There’s a super simple way to get visibility into the health of your universal forwarders using Forwarder Awareness in the Atlas app on Splunkbase. Here’s a snapshot of the data you’ll see with the click of a button. With this information, you’ll never have to wonder if all of your data is being ingested or whether it’s vulnerable. You can give it a go and decide for yourself right now, completely free.

Atlas Forwarder Awareness - A Splunkbase app for forwarder visibility

Try the Atlas Forwarder Awareness Tool Free in Splunkbase

Types of Splunk Forwarders

There are two types of forwarders: the Universal Forwarder, and the Heavy Forwarder.

1. Splunk Universal Forwarder

Universal Forwarders are more commonly utilized in most environments; Heavy Forwarders are used for specific use cases. In most situations, users simply want to collect data from a file or directory on a host and forward it to Splunk as is.

2. Splunk Heavy Forwarder

There are some instances where the format of the data might not be very pretty or even readable, or the data contains Personally Identifiable Information (PII), credit card information, etc. which needs to be masked or omitted—this is where the Heavy Forwarder comes in. A Heavy Forwarder can be configured to parse and perform transformative changes on the data BEFORE it is forwarded to Splunk indexers or another destination. Parsing data is done using props.conf and transforms.conf files.

How to Download and Install the Universal Forwarder

Step 1: Login to Splunk.com

How to Download and Install the Universal Forwarder: Step 1 - Login to Splunk.com

The Universal Forwarder can be downloaded two ways, and both involve logging into Splunk.com. Don’t panic, creating a Splunk account is quick, easy, and most importantly, free.

Step 2: Find the Universal Forwarder Install Package

Once logged into Splunk.com, hover over the Products tab at the top of the page and click on “Free Trials & Downloads”.

How to Download and Install the Universal Forwarder: Step 2 - Choose the Download type of the Universal Forwarder from Splunk.com

From the downloads page, scroll down toward the bottom until you see the “Download Now” link for the Splunk Universal Forwarder and click it.

How to Download and Install the Universal Forwarder: Step 2 - Download the Universal Forwarder from Splunk.com

It is on this page that you will be presented with a variety of choices for which OS you wish to install your Universal Forwarder package on. The Universal Forwarder can be installed on a wide variety of platforms such as Windows, Linux, Mac OS, Free BSD, Solaris, and AIX. This is where you get to choose how you want to download your Universal Forwarder package. Once you click the download now button, the package should automatically download to your system.

How to Download and Install the Universal Forwarder: Step 2 Find the Universal Forwarder Install Package

 

Step 3: Download the Universal Forwarder

Clicking the download button also loads a new page, it is here where you will have the option to copy a wget command (my preferred method) and download the install package directly to your system or any other system that has wget installed. I will demonstrate this process below.

How to Download and Install the Universal Forwarder: Step 3 - Download the Universal Forwarder onto your machine

Copy and paste the wget command into your terminal to download the Universal Forwarder install package (Yes, I am using root for the sake of ease during this tutorial)

How to Download and Install the Universal Forwarder: Step 3 - copy and paste the wget command

wget -O splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz “https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz”

Since I chose the tarball download, there is an additional step that needs to be done before installing and that is creating the Splunk user. This can be done with the following command:

“sudo adduser splunk”

How to Download and Install the Universal Forwarder: Step 3 - copy and paste the wget command use the sudo adduser splunk command

Step 4: Install the Universal Forwarder

Now that we have downloaded the Universal Forwarder, we need to extract the archive file to the “/opt” directory. To perform this action, I will use the following command:

tar –zxf  splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz -C /opt/

How to Download and Install the Universal Forwarder: Step 5 - install the universal forwarder

Next, we will start Splunk for the very first time and accept the license agreement so that we don’t get bombarded with a wall of text. I will also use the default admin username and password of admin/changeme

/opt/splunkforwarder/bin/splunk start –accept-license

How to Download and Install the Universal Forwarder: Step 5 - install the universal forwarder and start splunk for the first time

*NOTE* there is a message that you will get about an invalid stanza, this is currently a known issue with the 9.0.1 Universal Forwarder. It won’t cause you any problems though.

We also get some warnings about the permissions on $SPLUNK_HOME, this is to be expected since we are starting the service as root. Starting in 9.0, Splunk does not like to be issued commands by root and will complain every time you do so.

How to Configure the Splunk Universal Forwarder

Now that we have the Universal Forwarder installed, it’s time to configure it. The Universal Forwarder has two main files that need to be configured for it to collect and forward data, inputs.conf and outputs.conf. If you will be utilizing a Deployment Server to manage your Universal Forwarders, you will also need to configure a deploymentclient.conf file that tells the Universal Forwarder where to ‘phone home’ to retreive the appropriate Splunk apps and any other configuration information. We will assume this is a simple, no Deployment Server installation for this article.

Inputs.conf is where you configure the Universal Forwarder to collect data. For this tutorial, we will add a monitor stanza for /var/log/messages. But first, we will need to create an inputs.conf within /opt/splunkforwarder/etc/system/local. Now you may be wondering why we are creating the file in this location and why one already exists in /opt/splunkforwarder/etc/system/default. As a rule of thumb and best practice, you should never modify files within the default directory. These files exist to provide default settings and can help you identify certain settings that you may need to adjust, and this directory will be overwritten during upgrades so any changes you might make in the .conf files in the default directory would be lost. So, take only what you need from default and place it into same name .conf files in the local folder.

Step 1: Create an inputs.conf

So now we will create an inputs.conf in the local directory and add the monitor stanza. You can use your text editor of choice for this task. Underneath the stanza we will apply two additional settings, one for the index our events will be sent to and one for enabling the input. There are many other settings that can also be applied here – typically, you will also specify a sourcetype – but we will stick to what is needed for data collection to function for this tutorial.

How to configure the splunk universal forwarder: step 1 - create an inputs.conf

*NOTE* the three slashes represent that this file is contained on the localhost.

Step 2: Create an outputs.conf

Next up we will create an outputs.conf in the same directory and configure our forwarder to forward data to two indexers. Here you will specify the IP address of the Indexers that you want to forward data to and the port that you want that data to be forwarded over. Even though Splunk does not list it as a default port, 9997 is typically used as the standard port for data forwarding.

How to configure the splunk universal forwarder: step 2 - create an outputs.conf

Step 3: Restart Splunk

With our settings applied, we now must restart the forwarder for our changes to be committed to disk. This is a key aspect of Splunk to remember as well, that ANY changes you make to Splunk will require a restart.

/opt/splunkforwarder/bin/splunk restart

How to configure the splunk universal forwarder: step 3 - restart splunk

Congratulations! You now have a working Splunk Universal Forwarder and should see data in your “os” index.

Your Crash Course to Splunk Universal Forwarders

As you can see, installing the Universal Forwarder Is straightforward and takes minimal configuration to get up and running. Whether you are an aspiring Splunk admin or someone that has used Splunk for a while but has never gone through the process of installing a forwarder, this is a good process to get familiar with as the process for installing other Splunk Enterprise components is not any different.

If you found this helpful…

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. Once you download the app below, you’ll get your report in just 30 minutes.

New call-to-action