Splunk Forwarders are critical components for any Splunk environment, both for Splunk Enterprise running on-prem or Splunk Cloud. Forwarders are the critical element in any Splunk architecture for successfully getting data ingested into Splunk.
For customers operating Splunk at scale, the population of forwarders can be in the hundreds, thousands, or tens of thousands. Like any critical piece of enterprise software, Splunk forwarders must be deployed and configured correctly, and must be maintained to ensure operational integrity.
In this article, we explain the most detrimental risks of not maintaining Splunk forwarders and how to avoid them.
The Risks of Poor Splunk Forwarder Configuration and Management
There are three primary risks if Splunk forwarders are not actively maintained:
- Forwarder versions are interdependent with versions of Splunk Enterprise and Splunk Cloud – if forwarder versions are not kept up to date, they will not function properly with the Splunk instance(s) that are receiving data.
- Forwarders should be secured – just like any other enterprise software application, the forwarder software and underlying OS should be secured and patched to address vulnerabilities. Active configuration management of Splunk forwarders is basic hygiene for security.
- Forwarders are the first line of defense for preventing unauthorized data being ingested into Splunk – many Splunk deployments prohibit the ingest of PII, PHI, or other sensitive data. Forwarder configurations enable “black-listing,” the ingested files that contain prohibited data. If the configurations are not managed, prohibited data could be ingested accidentally.
The risks outlined above are driving the need for a comprehensive Splunk forwarder management solution – using a traditional Splunk deployment server isn’t enough.
A Comprehensive Splunk Forwarder Management Solution
Solution Architecture
Using a robust architecture to automate the deployment, management and maintenance of Splunk Forwarders at scale is the crux of a comprehensive Splunk forwarder management solution. This highly secure solution is designed so that it can be deployed in the cloud or on a server on the local network. The solution uses Puppet Enterprise, an unparalleled infrastructure and automation delivery platform with security built right into it. Here are some notable features of it:
- Extensively hardened enterprise platform that operates on classified networks
- Walmart and New York Stock Exchange depend on Puppet to manage highly secure environments at massive scale
- All communication happens over SSL with two-way peer verification at every step
- Trusted facts, extension to x509 certs, ensure un-spoofable inventory data
- Role-based access control to limit who sees what and change authorization
Agentless Upgrade and Management
With this Splunk forwarder management platform, you can update forwarders faster — no agent required. In this process, Puppet Bolt is used to connect via SSH (*nix) or WinRM (Windows) to all forwarder hosts. Once it’s connected, the following steps are performed:
- Download latest version of the forwarder files from a centralized repository
- Backup config files and settings
- Upgrade forwarder to downloaded version
- Start forwarder
- Monitor puppet logs for errors, and respond accordingly
- Execute test scripts to confirm running state of forwarder
Agent-based Upgrade and Management
If you’d prefer to opt for an agent-based solution, here are the steps involved:
- Puppet agent is installed on all forwarder hosts
- Puppet agent checks in with Puppet Master to receive configuration for forwarder hosts
- Puppet Master sends configuration payload over SSL. Configuration includes
- URL to centralized repository for latest forwarder
- Backup and install instructions for the agent to execute the upgrade process and validation
- Puppet Agent send log information back to Puppet Master
- Puppet Agent applies configuration changes to forwarder host.
- Manage infrastructure configuration and compliance at scale. The Agent-based solution, can be configured to enforce state on key .conf files minimizing impact of accidental changes
- Puppet Agent will monitor specific files – outputs.conf, inputs.conf, system.conf, web.conf and others
- Any unplanned changes are blocked by the agent
Advantages of an automated secure forwarder management process vs manual upgrade and management methodology:
An automated solution addresses each of the three primary risks associated with not actively managing Splunk Forwarder.
The ROI of Splunk Forwarder Management
Forwarder management solution provides tangible returns on investment in three areas – mission, financial, and human. We call these the “3 ROIs” that all should be present in any enterprise software investment.
For the Splunk forwarder management solution, Kinney Group sees the following application of the 3 ROIs:
Mission ROI: all organizations are investing in Splunk to help address identified business and mission objectives. If the application for Splunk is security, the capabilities provided by Splunk are critical to an organization’s security posture.
Given that forwarders are a critical component for a successful Splunk implementation, there is direct correlation between effective active management of Splunk forwarders and success with Splunk – hence the connection to mission success and a tangible mission ROI.
Financial ROI: all effective management solutions should help drive out costs. This includes both direct costs and opportunity costs.
On average, there are 4-6 hours of engineering time required annually to properly deploy, configure, patch, update, and manage an ecosystem of Splunk forwarders. For organizations that have deployed thousands or tens of thousands of forwarders, the engineering costs associated with forward management can add up fast.
Human ROI: managing a distributed ecosystem of Splunk forwarders using manual techniques is painful for engineers. Having to manage thousands of forwarders with legacy approaches is unpleasant work that most engineers would like to avoid.
Happy engineers produce extraordinary results for the enterprise. Our forwarder management solution helps reduce the time to execute rote tasks that are tied to forwarder management. This Human ROI is perhaps the most tangible of all.
If you found this helpful…
The Kinney Group Atlas Forwarder Management solution drives out 80-90% of the time needed to manage forwarders using legacy manual- or script-oriented approaches. This time reduction will drive measurable direct cost savings.
When engineers are freed up from rote tasks associated with software management, this is time that is now available to deliver more mission-oriented results from Splunk – said another way, this is the area of opportunity costs. While opportunity costs can be a challenge to quantify, they remain very real. Our Forwarder Management solution helps reduce opportunity costs.
With Kinney Group’s Atlas Forwarder Management solution, organizations can now be better equipped to address the challenges of Splunk forwarder management and do so in a way that provides real returns on investment.
Let’s get started! Contact us directly at (317) 721-0522.
