Skip to content

Part 2 – A Comprehensive Splunk Forwarder Management Solution

Part 2 – A Comprehensive Splunk Forwarder Management Solution

Kinney Group’s team of 40+ Splunk and Automation engineers has experience working with deployments of all sizes, various stages of execution, and across a variety of use cases. We’ve helped companies identify end goals, design, develop and implement technology. Our team of certified consultants have helped organization architect, scale and optimize environments for over 5 years. We architect complete application and infrastructure automation capabilities using platforms such as Puppet, Jenkins, and VMware vRealize – the same platforms found in the most advanced data centers in the world. Current patching and robust configuration management is basic hygiene for secure systems. Every day, the speed and volume of security threats outpaces legacy manual methods of patching systems and configuration management. We automate solutions for systems patching, configuration management, monitoring, and compliance that enable organizations to keep their critical systems current, and thus, secure.

Solution Architecture

At Kinney Group, we have developed a robust architecture to automate the deployment, management and maintenance of Splunk Forwarders at scale. This highly secure solution is designed such that it can be deployed in the cloud or on a server on the local network. The solution uses Puppet Enterprise, an unparalleled infrastructure and automation delivery platform with security built right into the platform

  • Extensively hardened enterprise platform that operates on classified networks
    • Walmart and New York Stock Exchange depend on Puppet to manage highly secure environments at massive scale
  • All communication happens over SSL with two-way peer verification at every step
  • Trusted facts, extension to x509 certs, ensure un-spoofable inventory data
  • Role-based access control to limit who sees what and change authorization



Agentless Upgrade and Management

With KGI Splunk forwarder management platform, you can update forwarders faster via agentless process. In this process, Puppet Bolt is used to connect via SSH (*nix) or WinRM (Windows) to all forwarder hosts. Once connected, the following steps are performed:

  • Download latest version of the forwarder files from a centralized repository
  • Backup config files and settings
  • Upgrade forwarder to downloaded version
  • Start forwarder
  • Monitor puppet logs for errors, and respond accordingly
  • Execute test scripts to confirm running state of forwarder


Agent-based Upgrade and Management

  • Puppet agent is installed on all forwarder hosts
  • Puppet agent checks in with Puppet Master to receive configuration for forwarder hosts
  • Puppet Master sends configuration payload over SSL. Configuration includes
    • URL to centralized repository for latest forwarder
    • Backup and install instructions for the agent to execute the upgrade process and validation
    • Puppet Agent send log information back to Puppet Master
  • Puppet Agent applies configuration changes to forwarder host.
  • Manage infrastructure configuration and compliance at scale. The Agent-based solution, can be configured to enforce state on key .conf files minimizing impact of accidental changes
    • Puppet Agent will monitor specific files – outputs.conf, inputs.conf, system.conf, web.conf and others
    • Any unplanned changes are blocked by the agent


Advantages of an automated secure forwarder management process vs manual upgrade and management methodology:

An automated solution addresses each of the three primary risks associated with not actively managing Splunk Forwarder referenced in Part 1 of this blog series by Jim Kinney.

Advantages of Automated Forwarder Management Over Manual Processes

In Part 3 of this blog post series, Jim Kinney will explain how the forwarder management solution provides tangible returns on investment in three areas – mission, financial, and human.