The Ultimate Guide to Splunk Universal Forwarders

For many getting started with Splunk, the question of “How do I get my data into Splunk” comes up quite regularly. The answer to that question is most often: “use the Universal Forwarder.”

What is a Splunk Universal Forwarder?

The Universal Forwarder is a Splunk instance that can be installed on just about any operating system (OS). Once installed, the Universal Forwarder can be configured to collect systems data and forward it to Splunk Indexers. The Universal Forwarder can also be configured to send data to other forwarders or third-party systems as well if you so desire.

New call-to-action

Universal Forwarders use significantly fewer resources than other Splunk products. You can install literally thousands of them without impacting network performance and cost. The Universal Forwarder does not have a graphical user interface, but you can interact with it through the command line or REST endpoints. The Universal Forwarder also comes with its own license pre-installed, so there is no need to purchase a license for it.

There are many benefits to using a Universal Forwarder to forward your logs as opposed to other solutions.

  1. Forwarding data from a Universal Forwarder is reliable right out of the box, but it can be configured to further protect in-flight data using Indexer Acknowledgement.
  2. The Universal Forwarder uses an internal index called the fishbucket which is used to track previously read files and directories so that Splunk does not send the same data twice.
  3. Universal Forwarders can be configured for load balancing which enables scaling and improved performance.
  4. Using a Universal Forwarder gives you the ability to remotely manage its configurations using apps and add-ons that are deployed by a Deployment Server.

Splunk Pro Tip: There’s a super simple way to get visibility into the health of your universal forwarders using Forwarder Awareness in the Atlas app on Splunkbase. Here’s a snapshot of the data you’ll see with the click of a button. With this information, you’ll never have to wonder if all of your data is being ingested or whether it’s vulnerable. You can give it a go and decide for yourself right now, completely free.

Atlas Forwarder Awareness - A Splunkbase app for forwarder visibility

Try the Atlas Forwarder Awareness Tool Free in Splunkbase

Types of Splunk Forwarders

There are two types of forwarders: the Universal Forwarder, and the Heavy Forwarder.

1. Splunk Universal Forwarder

Universal Forwarders are more commonly utilized in most environments; Heavy Forwarders are used for specific use cases. In most situations, users simply want to collect data from a file or directory on a host and forward it to Splunk as is.

2. Splunk Heavy Forwarder

There are some instances where the format of the data might not be very pretty or even readable, or the data contains Personally Identifiable Information (PII), credit card information, etc. which needs to be masked or omitted—this is where the Heavy Forwarder comes in. A Heavy Forwarder can be configured to parse and perform transformative changes on the data BEFORE it is forwarded to Splunk indexers or another destination. Parsing data is done using props.conf and transforms.conf files.

How to Download and Install the Universal Forwarder

Step 1: Login to Splunk.com

How to Download and Install the Universal Forwarder: Step 1 - Login to Splunk.com

The Universal Forwarder can be downloaded two ways, and both involve logging into Splunk.com. Don’t panic, creating a Splunk account is quick, easy, and most importantly, free.

Step 2: Find the Universal Forwarder Install Package

Once logged into Splunk.com, hover over the Products tab at the top of the page and click on “Free Trials & Downloads”.

How to Download and Install the Universal Forwarder: Step 2 - Choose the Download type of the Universal Forwarder from Splunk.com

From the downloads page, scroll down toward the bottom until you see the “Download Now” link for the Splunk Universal Forwarder and click it.

How to Download and Install the Universal Forwarder: Step 2 - Download the Universal Forwarder from Splunk.com

It is on this page that you will be presented with a variety of choices for which OS you wish to install your Universal Forwarder package on. The Universal Forwarder can be installed on a wide variety of platforms such as Windows, Linux, Mac OS, Free BSD, Solaris, and AIX. This is where you get to choose how you want to download your Universal Forwarder package. Once you click the download now button, the package should automatically download to your system.

How to Download and Install the Universal Forwarder: Step 2 Find the Universal Forwarder Install Package

 

Step 3: Download the Universal Forwarder

Clicking the download button also loads a new page, it is here where you will have the option to copy a wget command (my preferred method) and download the install package directly to your system or any other system that has wget installed. I will demonstrate this process below.

How to Download and Install the Universal Forwarder: Step 3 - Download the Universal Forwarder onto your machine

Copy and paste the wget command into your terminal to download the Universal Forwarder install package (Yes, I am using root for the sake of ease during this tutorial)

How to Download and Install the Universal Forwarder: Step 3 - copy and paste the wget command

wget -O splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz “https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz”

Since I chose the tarball download, there is an additional step that needs to be done before installing and that is creating the Splunk user. This can be done with the following command:

“sudo adduser splunk”

How to Download and Install the Universal Forwarder: Step 3 - copy and paste the wget command use the sudo adduser splunk command

Step 4: Install the Universal Forwarder

Now that we have downloaded the Universal Forwarder, we need to extract the archive file to the “/opt” directory. To perform this action, I will use the following command:

tar –zxf  splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz -C /opt/

How to Download and Install the Universal Forwarder: Step 5 - install the universal forwarder

Next, we will start Splunk for the very first time and accept the license agreement so that we don’t get bombarded with a wall of text. I will also use the default admin username and password of admin/changeme

/opt/splunkforwarder/bin/splunk start –accept-license

How to Download and Install the Universal Forwarder: Step 5 - install the universal forwarder and start splunk for the first time

*NOTE* there is a message that you will get about an invalid stanza, this is currently a known issue with the 9.0.1 Universal Forwarder. It won’t cause you any problems though.

We also get some warnings about the permissions on $SPLUNK_HOME, this is to be expected since we are starting the service as root. Starting in 9.0, Splunk does not like to be issued commands by root and will complain every time you do so.

How to Configure the Splunk Universal Forwarder

Now that we have the Universal Forwarder installed, it’s time to configure it. The Universal Forwarder has two main files that need to be configured for it to collect and forward data, inputs.conf and outputs.conf. If you will be utilizing a Deployment Server to manage your Universal Forwarders, you will also need to configure a deploymentclient.conf file that tells the Universal Forwarder where to ‘phone home’ to retreive the appropriate Splunk apps and any other configuration information. We will assume this is a simple, no Deployment Server installation for this article.

Inputs.conf is where you configure the Universal Forwarder to collect data. For this tutorial, we will add a monitor stanza for /var/log/messages. But first, we will need to create an inputs.conf within /opt/splunkforwarder/etc/system/local. Now you may be wondering why we are creating the file in this location and why one already exists in /opt/splunkforwarder/etc/system/default. As a rule of thumb and best practice, you should never modify files within the default directory. These files exist to provide default settings and can help you identify certain settings that you may need to adjust, and this directory will be overwritten during upgrades so any changes you might make in the .conf files in the default directory would be lost. So, take only what you need from default and place it into same name .conf files in the local folder.

Step 1: Create an inputs.conf

So now we will create an inputs.conf in the local directory and add the monitor stanza. You can use your text editor of choice for this task. Underneath the stanza we will apply two additional settings, one for the index our events will be sent to and one for enabling the input. There are many other settings that can also be applied here – typically, you will also specify a sourcetype – but we will stick to what is needed for data collection to function for this tutorial.

How to configure the splunk universal forwarder: step 1 - create an inputs.conf

*NOTE* the three slashes represent that this file is contained on the localhost.

Step 2: Create an outputs.conf

Next up we will create an outputs.conf in the same directory and configure our forwarder to forward data to two indexers. Here you will specify the IP address of the Indexers that you want to forward data to and the port that you want that data to be forwarded over. Even though Splunk does not list it as a default port, 9997 is typically used as the standard port for data forwarding.

How to configure the splunk universal forwarder: step 2 - create an outputs.conf

Step 3: Restart Splunk

With our settings applied, we now must restart the forwarder for our changes to be committed to disk. This is a key aspect of Splunk to remember as well, that ANY changes you make to Splunk will require a restart.

/opt/splunkforwarder/bin/splunk restart

How to configure the splunk universal forwarder: step 3 - restart splunk

Congratulations! You now have a working Splunk Universal Forwarder and should see data in your “os” index.

Your Crash Course to Splunk Universal Forwarders

As you can see, installing the Universal Forwarder Is straightforward and takes minimal configuration to get up and running. Whether you are an aspiring Splunk admin or someone that has used Splunk for a while but has never gone through the process of installing a forwarder, this is a good process to get familiar with as the process for installing other Splunk Enterprise components is not any different.

If you found this helpful…

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. Once you download the app below, you’ll get your report in just 30 minutes.

New call-to-action

Meet Atlas Migration Helper

Whether you’re moving to Splunk Cloud or migrating your Splunk instance from one server, system, architecture, or filesystem to another, there are a lot of factors to consider before making the move. What apps need to move to the new location (and are they even compatible)? Do I have all of my datasets and forwarders? As if that weren’t enough, what about local knowledge objects?

Migrating Splunk isn’t for the faint of heart. But you’re in luck! The Atlas Platform for Splunk takes the pain and guesswork out of migration, and provides you with a step-by-step plan for moving your instance.

When Should I Migrate?

The reasons for migration are as varied as the organizations that use Splunk themselves. But some reasons for moving from Point A to Point B might include:

  • You want to move Splunk to a new or different file system
  • You’re upgrading from a 32-bit to 64-bit architecture for performance gains
  • You need to switch operating systems (from Windows to Unix, for example)
  • You’re upgrading infrastructure components or retiring hardware
  • You’ve decided to move to Splunk Cloud (or AWS or Azure)

Considerations When Migrating Splunk

Atlas’s Splunk Migration Helper is a powerful element geared towards helping Splunk owners move their Splunk environment with precision, reporting, and speed. Atlas Migration Helper contains everything you need for identifying what is useful and necessary to move to your new Splunk environment, empowering you to enter your new Splunk environment worry-free.

Some of the ways Atlas Migration Helper guides and supports the migration process:

  • Easily identify Splunk Applications that should be moved and select them for migration
  • Identify what data ingests are being utilized by users and apps and select them for migration
  • Identify Forwarders required to support selected data ingests and applications and select them for migration
  • Analyze your Splunk Environment for issues to resolve before migration to ensure stability
  • Track the Migration using automated dashboarding that reports current status.

All of these features and more are summarized with top level KPIs for simplified tracking.

Now that you know how the Atlas Migration Helper application can take the guess work out of your Splunk migration, why not check out our Atlas documentation for a closer look, or schedule a 1:1 discovery session to answer any questions you may have?

Meet Atlas Monitor

Creating incredible results and outcomes in Splunk requires what we call “data certainty.” Meaning, you know you have all the data you need, and you have a way to be alerted when you don’t (due to a data source going “offline,” for example).

Atlas Monitor provides unparalleled visibility into your Splunk data, and powerful alerting capabilities. Dashboards, alerts, visualization, and Enterprise Security all rely on a constant and reliable feed of data flowing in to Splunk. Without pro-active measures to monitor and alert, these data streams can fail, causing inaccurate reporting and cascading failures.

And bad data is worse than having no data at all.

Track and alert on data failures

Splunk admins can utilize Atlas’s simple interface to create “Monitors,” which track and alert on data ingest failures, preventing errors and increasing reliability. Monitors efficiently utilize Splunk resources to do more with less, while providing highly-detailed reporting (without adding complexity).

Further, admins are able to create Monitor Groups within the application to consolidate reporting, and can leverage lookup tables and custom searches to make effective use of Change Management Knowledge Objects. With Atlas Monitor, admins have a powerful tool that will increase data flow stability and awareness.

Monitor Capabilities

  • At-a-glance summaries to empower creators to quickly assess data flow health with custom thresholding
  • Group related data flows together for visualizations and simplified reporting
  • Leverage metric indexes and enhanced searching to reduce resource utilization
  • Report on outages to enable historical tracking of downtime
  • Automatically send alerts by email when Monitors breach thresholds
  • Integrate lookup tables for integrating CMDB and assets & identities files
  • Create custom searches to monitor unique data sets utilizing advanced base searches

Now that you know how the Monitor application can take the guess work out of your Splunk data, why not check out our Atlas documentation for a closer look, or schedule a 1:1 discovery session to answer any questions you may have?

See More: Data Awareness in Splunk

Every beautiful dashboard and impressive visual Splunk is capable of producing is, ultimately, driven by two things: data and search.

And while search is the primary driver behind the analytics and visualizations in Splunk, all the perfectly written and executed searches in the world can’t help you if you’re missing the most important resource of all — quality data.

If you’ve ever put together a Lego® set, you know you’ve got to have all the pieces if you’re going to be able to build the Lego Death Star. Even one missing piece could leave you frustrated and incapable of building what you set out to create.

In short: you have to know what data you’re working with, and that you have all of it.

Data Awareness Defined

“Data awareness” refers to your organization’s ability to look at the infrastructure bringing data into your Splunk environment, the visibility you have (or don’t have) when there’s a failure in your data pipeline, and the health of your forwarder infrastructure.

To get the most from Splunk, and to empower you to do more with the platform, there are two critical questions you must address:

Question 1: Do you have an alerting system in place when critical data streams fail in your Splunk environment?

Setting alerts for critical data streams is important for ensuring your dashboards and processes are up to best practices. You want to be the first person to know an issue has occurred so it can be solved before it becomes a larger problem.

Some may read that and think, “We check our data streams monthly or weekly, so we have a pretty good idea of how healthy our data pipeline is.” But what about those moments when a data stream or forwarder goes offline in between those manual checks?

Maybe it’s not important data… but maybe it is.

If you’re using Splunk for compliance, even a few moments of downtime can cause huge problems down the road. If you’re using Splunk for security, you know all too well how much meaningful (and dangerous) activity can occur within even a few minutes.

Alerts are best practice for a reason. Your team and those throughout the organization who rely on Splunk dashboards and visualizations for day-to-day operations, security, insights, and decision-making have to be able to trust the data. If your alerting isn’t strong, that means you could be missing data. And bad data is worse than no data at all.

Question 2: If you’re using Splunk Enterprise Security (ES), are you confident you’re ingesting all the appropriate data to get the most from your investment?

Splunk ES is an incredible tool, but depends on being fed the proper data for it to really shine. Without the ability to ensure you have full coverage for your priority data and clear eyes on that data’s acceleration, you could be leaving yourself vulnerable. The continuous security monitoring, advanced threat detection, and your ability to rapidly investigate and respond to threats is all contingent on priority data being fed into the system.

At a minimum, without that information, you’re certainly not maximizing your use of ES (or the dollars you’ve invested in the platform).

Solving Alerts

It’s possible that you can manually create alerts for any number of situations and needs within Splunk. Once again, that’s the power of having such a versatile platform at your fingertips. The downside to that approach, however, is that it’s time consuming and requires, many time, a degree of technical proficiency with Splunk that many internal teams lack.

What would be ideal is a single pane of glass that shows a complete inventory of every sourcetype in your Splunk deployment. Even better would be if that inventory could also show how much data is being received by that sourcetype, its status, a use case or description, admin notes, who owns it… you get the idea. And the cherry on top of this magical solution would be a push-button simple way to create an alert for that sourcetype.

This is exactly what the Data Management application within the Atlas Platform provides.

The Data Inventory component of Data Management allows you to easily see every sourcetype, the last time it was ingested, how much of your license that data is utilizing, and a host of other important information.

Utilizing the Data Watch feature by clicking on the alert icon inline with this information, you can also utilize Splunk to keep a watchful eye on the sourcetype and alert you when there’s a certain percentage drop in hosts or events:

Of course, watching sourcetypes is only a piece of the puzzle. You also need a way to provide that same level of protection and visibility to your forwarders. This is where the Atlas Forwarder Awareness application swoops in to save the day with a system-wide overview of forwarder status that you can group however you wish, with the ability to dive into each group for details.

Within each group, you’ll have visibility into missing forwarders, the SSL status of each forwarder, the version of Splunk each forwarder is running, and a variety of other information that allows you peace of mind that data is reliable and being brought in to your environment properly.

Solving Enterprise Security

As stated, Enterprise Security depends on the right data — and especially priority data with clear eyes on acceleration of that data.

That’s why we’ve developed the Atlas ES Helper application to guide the process and ensure you have the coverage you need and you’re utilizing the platform effectively.

In addition to a comprehensive inventory of ES-related data models, the power of ES Helper is its ability to give you an understanding of your environment’s overall utilization, data coverage, and acceleration at a glance.

The proprietary ES Utilization score is based on scoring your system’s Priority Data Coverage, Priority Data Acceleration, Lower Data Coverage, and Lower Data Acceleration. The takeaway is an easy to understand and actionable report that tells you, with certainty, if you’re getting the most and doing the most with your investment in Splunk Enterprise Security.

Wrapping Up

Whether it’s a comprehensive understanding of your sourcetypes, data models, and forwarders or getting more from Splunk ES, the value of Data Awareness can’t be overstated. Downloading the free Atlas Assessment application from Splunkbase is the perfect way to see if Atlas is the right fit to solve these challenges in your environment. Still not convinced? A free 30-day trial of Atlas will provide you with the opportunity to see for yourself. If you’d like to read more, grab our free “Do More with Splunk” ebook (just tap the button below — no email required). You’ll learn what a Splunk “Creator” is (and does), and get actionable next steps for accelerating your Splunk journey.

New call-to-action

What is a Splunk Creator?

Before we dive into our three primary topics, we want to set the stage by explaining a term you’ll see used throughout this piece:

“Creator.”

Typically when we think of “creators,” we think of, well, creative occupations. Painters, designers, architects, and the like. Or maybe you think in more modern terms, such as YouTubers and TikTok content creators. But most individuals working with Splunk wouldn’t consider the work they do with and in the platform to be creative.

Except… it really is.

Think about it for a moment. An architect or product designer gathers a list of requirements and then creates the best possible solution with existing materials, working within the budget and time constraints the client has laid out for them.

Isn’t that exactly what you do each and every day in Splunk?

Someone needs an outcome:

“We want a dashboard to view security incidents.”

“I need a visualization for revenue vs targets.”

“We need a way to collect Remote Work Insights.”

And what do you do? What every creator does. You look at the available materials — the data you’re pulling in to your environment — and then you begin to use the platform to create a solution. An outcome.

A masterpiece.

A dashboard in Splunk is more than graphs and charts and tables. It’s the output of one of the most complex functions asked of any technical professional — telling a story with data.

“Is our organization safe from threats today?”

“Are we delivering on our promise to our customers and stakeholders?”

You tell those stories — and many others like them — every day with the solutions and outcomes you create in Splunk.

Look at you go.

Want to learn how?

If you want to create incredible outcomes in Splunk that empower you to see more, create more, and save more with the platform, grab our free “Do More with Splunk” ebook (just tap the button below — no email necessary). You’ll learn what a Splunk “Creator” is (and does), and get actionable next steps for accelerating your Splunk journey.

New call-to-action

 

Do More With Splunk

In every recession, organizations find themselves in uncharted waters — after all, no two recessions or downturns are the same. What worked for Company A during the early 2000’s
recession may or may not work for Company B facing the economic downturn ahead of us today.

What organizations can do, however, is identify the patterns and behaviors of the companies that managed to thrive in challenging economic times. Harvard Business Review conducted a year-long study of nearly 5,000 companies and their behaviors in the periods immediately preceding, during, and after an economic downturn.

While 17% of the companies they studied didn’t survive (for a wide variety of reasons), and the overwhelming majority were unable to regain their pre-recession rate of growth, 9% were able to gain ground and outperform competition.

What Do the Winners Do Differently?

Harvard Business Review’s study effectively found that the key to coming out ahead, during and after a recession, is an adept combination of “defensive” and “offensive” moves. Defensive moves are those that are, perhaps, the most common response to a downturn — spend less and cut costs.

But “offensive” moves?

The companies that fared best in a downturn were those that focused on maximizing what they already had. This approach to improving operational efficiency had the same net effect as reducing headcount or slashing expenses without actually doing those things at the same levels as their competition.

Doing More

Regardless of economic conditions around us, the reality is we could all use some help in getting more from what we’ve got. The average utilization of a software platform across all industries looks something like this:

Take a moment to let that sink in. What is your company’s annual investment in the Splunk platform? What could it mean to your organization if you could unlock that 50-80% of the platform that you may not be utilizing to the fullest?

The key to your future success will be found in unlocking the underutilized potential of the most amazing security and observability platform available… and creating incredible outcomes in the process.

Want to learn how?

If you want to create incredible outcomes in Splunk that empower you to see more, create more, and save more with the platform, grab our free “Do More with Splunk” ebook (just tap the button below — no email necessary). You’ll learn what a Splunk “Creator” is (and does), and get actionable next steps for accelerating your Splunk journey.

New call-to-action