The Ten Command(ment)s of Splunk

There are many do’s and don’t’s when it comes to Splunk. In our time supporting Splunk customers through Expertise on Demand, Team Tech Ops has seen the good, the bad, and the ugly situations customers can fall into with Splunk.

We’re happy to present the Tech Ops Ten (Command)ments of Splunk best practices.

1) Thou shalt NEVER search index=*

This one is pretty self-explanatory. Splunk has A LOT of data.

Figure 1: "Splunk Slaps" meme
Figure 1: “Splunk Slaps” meme

In most cases, hundreds of gigabytes, maybe even terabytes of data. I’m sure you tried running a search that looks in one index across millions of events and found it took a very long time to complete.

Now, imagine that across all of your indexes. Not many Splunkers can see this full picture because there’s always a search that will not complete (unless you have a tiny environment or use something like tstats).

Searching “index=*” goes into what I like to call the worst practices box.

2) Thou shalt remove real-time & All Time as an option for basic users

Right in line with never looking at every index humanly possible, we also want to avoid looking at every event (that does exist or will exist).

Running a search in real-time or across all-time causes a resource strain on the environment and may even cause disruption for your fellow Splunk users.

3) Thou shalt not ingest data into the main index

Main index is a default index for Splunk Enterprise. Without specifying an index for your inputs, all your data will default to the main index.

Typically, it is the best practice if you never send information to the main index. Ever. If you thought it was confusing to find data when it nicely organized in your indexes and source types, try finding anything when it’s completely jumbled up in one place.

4) Thou shalt leave on ALL search formatting settings

This one is a “You v.s. The Guy she tells you not to worry about” situation.


Figure 2 - Splunk without search formatting settings
Figure 2 – Splunk without search formatting settings

The Guy she tells you not to worry about:

Figure 2 - Splunk with ALL search formatting settings
Figure 3 – Splunk with ALL search formatting settings

5) Thou shalt view the monitoring console before requesting a performance dashboard be built

Most of the information is already there, you don’t need to reinvent the wheel or in this case… the monitoring console.

Figure 3 - Splunk monitoring console meme
Figure 4 – Splunk monitoring console meme

6) Thou shalt look for an add-on first before onboarding new data

If you’re onboarding data, there is probably an app or add-on that can help. You’re going to save a lot of time (and aspirin) utilizing one of Splunk’s app or add-on tools. Note: this does not apply if you’re ingesting something completely unique to you or your company.

Figure 5 - Save time with a Splunk app or add-on
Figure 5 – Save time with a Splunk app or add-on

7) Thou shalt follow correct directory precedence

NEVER save a .conf file in /default. It’s that simple. Just don’t do it.

Figure 6 - Don't use .conf file in /default in Splunk
Figure 6 – Don’t use .conf file in /default in Splunk

8) Thou shalt have all instances of Splunk on a supported version

If you wouldn’t allow an unsupported version of Windows Server in your environment, then why would you allow an unsupported version of Splunk in?

Figure 7 - update your version on Splunk
Figure 7 – Update your Splunk version

9) Thou shalt use forwarder management

Think smarter not harder. Forwarder management makes it easier to keep all your forwarders buttoned up and working properly. The alternative is to make changes and updates manually and individually, and depending on how many clients you have…that might take a while. Splunk’s native forwarder management tool is cool, but Kinney Group’s Forwarder Awareness application (through Atlas) is cooler. Check out this incredible tool that will save you a TON of time in Splunk.

Figure 8- Utilize forwarder awareness so you don't look like this guy
Figure 8- Utilize forwarder awareness so you don’t look like this guy

10) Thou shalt not use join/subsearches unless absolutely necessary

I want to start off by saying that sub-searches aren’t bad, they’re just not as efficient as other solutions. There’s more to come on this rule, but trust our advice, for now, avoid this at all causes.


Your data is important, so how you work with it in Splunk makes all the difference in the value you’ll get out of the platform. The Tech Ops team has worked with hundreds of Splunk customers, from our experience, these tips are a great place to start in adopting Splunk best practices. If you’d like to work directly with us, the experts, please fill out the form below!

Clearing the Air: Apps vs Add-ons in Splunk

When talking about apps that we need to bring into Splunk, the conversation can get very confusing, very quickly. This is because apps serve different purposes and come from different sources.

Let’s look at AWS data for example. If I do a cursory search on Splunkbase, the center for Splunk’s Apps and Add-ons, for an app to bring in my data, I might find the following results:

  • Splunk App for AWS
  • Splunk Add-on for Amazon Web Service
  • Splunk Add-on for Amazon Kinesis Firehose


Figure 1: Search results in Splunkbase
Figure 1: Search results in Splunkbase

This is just to name a few on the list out of the 38 results that pop up. Of those 38, which do you choose?

There are a number of similarly named apps built around the same data. Without doing extensive research before your search, you probably couldn’t clearly identify when each app needs to be used. Which app is the best fit for the AWS data I’m consuming? How many users have installed this app? What are the users saying about this app?


The Tricky Part

There is a lot to decipher when choosing which tool to utilize. And it gets even trickier than that– Splunk provides both apps and add-ons built for users to enhance and extend the value of the Splunk platform. Although the two have very different functions, both apps and add-ons are listed the same on your Splunkbase results: all results come up listed as a “app.”

This can make the process of identifying the correct app or add-on extremely difficult for users within Splunk. That’s why the Tech Ops team has some tips that should make the choice clear.


Apps vs Add-ons: The Difference

Let’s see if we can make it easier to decipher this in the future. First, we’ll breakdown the different types of “apps”:

Add-on (TA)

These are the bread and butter of bringing in data from your machines. Add-ons are built to have props, transforms, inputs, are various other configuration files to ensure that the data sources being ingested are parsed, extracted, and indexed correctly.


In most cases, an app usually brings in Knowledge objects for the user to utilize. This could be dashboards, alerts, reports, and macros. It uses the data brought in via the add-on to populate the Knowledge objects

To take full advantage of the data we’re bringing in, we generally want to use both Add-ons and Apps in tandem. While neither of these products are required to bring in your data, they certainly make it much easier. Start with your add-ons to help you bring your data in from machines. Then, utilize your apps to do the heavy lifting to help you visualize and analyze your data.

Tips from Team Tech Ops

Here at Kinney Group, the Tech Ops team is dedicated to helping customers fix any issue they face with Splunk (really, we mean anything) through our Expertise on Demand offering through the Atlas Platform. We work with different Apps and Add-ons all day, every day and are constantly recommended the best of these products to our customers. If you want to see the full picture of Splunk, all while snagging out best practice help and guidance, fill out the form below to talk with one of our Splunkers.