Whether you’re just getting started with Splunk or you’re a seasoned expert, odds are you’ve already had data to work with in the platform. But what if you need to add new data, or if you’re starting a brand new Splunk environment at your organization?
This article will walk you through exactly how to get your data into Splunk. If you want to customize this guide even further to your Splunk environment, download your free, personalized Splunk report from Splunkbase using the Atlas Assessment.
4 Ways to Get Data Into Splunk
1. Use a Universal Splunk Forwarder.
If most of the data you will ingest into Splunk comes from the logs of applications and web servers, a universal forwarder on a server will be your best bet to successfully get data into Splunk. This option is completely free to download. You should configure it to monitor specific logs and send that data to Splunk for indexing, along with some configuration settings that tell Splunk how to parse the logs and which index to store the data in.
2. Use an HTTP Event Collector.
Another very versatile and highly scalable way of getting data into Splunk is via the HTTP Event Collector (HEC). This is a solution that listens for HTTP requests containing JSON objects. The HTTP Event Collector can collect data at extremely high volumes from many devices and data sources, all on a single port. Another interesting feature of using HEC is that the host, index, source, and sourcetype associated with a given data source can be specified within the JSON object of each received event.
To set up the HTTP Event Collector, you first configure the global settings, which includes the HTTP endpoint it will listen on – by default, this is port 8088. You should also configure a default index, and optionally, a default sourcetype. This is one activity that is best done in Splunk Web the first time, so you can see all the options available to you.
3. Use a Splunkbase App.
Apps can help parse incoming data so that it’s more useful and valuable. An app created by a Splunk user, or any of the apps that come with Splunk, contain a collection of all the objects that make up the app and how it functions within the Splunk environment. objects can include configuration (.conf) files to control inputs, indexes, saved searches, reports, dashboards, alerts, and more, as well as static files for creating web browser views and menu navigation items. Each of these objects have permissions associated with them to determine who can view or alter them.
The launcher, which is a fairly simple app that controls the view and menu when you first log in to Splunk (unless you’ve selected another Default Dashboard), or the search folder, which contains all the files used to manifest the Search & Reporting interface. The more complex search app, by comparison, contains these and several other folders for lookups, scripts, and so on. But an app can be as simple as just an app folder, with a /local folder inside, that contains a single inputs.conf or indexes.conf file–there are no minimal requirements for contents,
4. Use a Splunk Heavy Forwarder.
If you aren’t able to use a universal forwarder as Splunk recommends, you could consider a heavy forwarder instead. You’ll need a forwarder license to use it, but it can be beneficial if your use case calls for this extra power, such as with intermediate forwarders between UFs and indexers.
A heavy forwarder is a type of universal forwarder that sends data to another Splunk Enterprise instance or third-party system. It’s a specially configured instance of Splunk Enterprise that operates similarly to an indexer. Heavy forwarders leave a smaller footprint than a traditional Splunk Enterprise indexer because it disables some larger services like Splunk Web, but retains most of the capabilities of an indexer. The specifications for your heavy forwarder (CPU, memory, disk) will depend upon the app(s) you install, and how you plan to use it, but you should build it to reference server specs as a minimum (16 cores, 12 GB RAM, 300 GB disk).
There are a few benefits to using heavy forwarders to get data into Splunk:
- Parses data before forwarding it.
- Routes data based on source or event criteria.
- Can index data locally while forwarding data to another indexer.
- Can host specialized Splunk applications from Splunkbase.
But this single detractor can give some Splunk users pause before ingesting data in this way: Heavy forwarders can be expensive. Purchasing a forwarder license is an added cost to consider, and if you’re planing to use it as an indexer, that can nearly double your normal ingestion charges. Luckily, you don’t have to use a heavy forwarder as an indexer, but if you do, make sure there’s a strong business case for it.
Since you won’t find a download file for a heavy forwarder on the Splunk site, nor will you find a specific document or direct references to setting up a heavy forwarder, you’ll need to configure it manually.
Getting Your Data Into Splunk the Easy Way
Each of these options has its pros and cons, but all of them will work seamlessly to get your data into Splunk. The even better part is that you don’t have to do this alone. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate
Cue Atlas Assessment 30-day free trial: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. You’ll get your report in just 30 minutes.
