Skip to content
Article

The Ten Command(ment)s of Splunk

There are many do’s and don’t’s when it comes to Splunk. In our time supporting Splunk customers through Expertise on Demand, Team Tech Ops has seen the good, the bad, and the ugly situations customers can fall into with Splunk.

We’re happy to present the Tech Ops Ten (Command)ments of Splunk best practices.

1) Thou shalt NEVER search index=*

This one is pretty self-explanatory. Splunk has A LOT of data.

Figure 1: "Splunk Slaps" meme
Figure 1: “Splunk Slaps” meme

In most cases, hundreds of gigabytes, maybe even terabytes of data. I’m sure you tried running a search that looks in one index across millions of events and found it took a very long time to complete.

Now, imagine that across all of your indexes. Not many Splunkers can see this full picture because there’s always a search that will not complete (unless you have a tiny environment or use something like tstats).

Searching “index=*” goes into what I like to call the worst practices box.

2) Thou shalt remove real-time & All Time as an option for basic users

Right in line with never looking at every index humanly possible, we also want to avoid looking at every event (that does exist or will exist).

Running a search in real-time or across all-time causes a resource strain on the environment and may even cause disruption for your fellow Splunk users.

3) Thou shalt not ingest data into the main index

Main index is a default index for Splunk Enterprise. Without specifying an index for your inputs, all your data will default to the main index.

Typically, it is the best practice if you never send information to the main index. Ever. If you thought it was confusing to find data when it nicely organized in your indexes and source types, try finding anything when it’s completely jumbled up in one place.

4) Thou shalt leave on ALL search formatting settings

This one is a “You v.s. The Guy she tells you not to worry about” situation.

You:

Figure 2 - Splunk without search formatting settings
Figure 2 – Splunk without search formatting settings

The Guy she tells you not to worry about:

Figure 2 - Splunk with ALL search formatting settings
Figure 3 – Splunk with ALL search formatting settings

5) Thou shalt view the monitoring console before requesting a performance dashboard be built

Most of the information is already there, you don’t need to reinvent the wheel or in this case… the monitoring console.

Figure 3 - Splunk monitoring console meme
Figure 4 – Splunk monitoring console meme

6) Thou shalt look for an add-on first before onboarding new data

If you’re onboarding data, there is probably an app or add-on that can help. You’re going to save a lot of time (and aspirin) utilizing one of Splunk’s app or add-on tools. Note: this does not apply if you’re ingesting something completely unique to you or your company.

Figure 5 - Save time with a Splunk app or add-on
Figure 5 – Save time with a Splunk app or add-on

7) Thou shalt follow correct directory precedence

NEVER save a .conf file in /default. It’s that simple. Just don’t do it.

Figure 6 - Don't use .conf file in /default in Splunk
Figure 6 – Don’t use .conf file in /default in Splunk

8) Thou shalt have all instances of Splunk on a supported version

If you wouldn’t allow an unsupported version of Windows Server in your environment, then why would you allow an unsupported version of Splunk in?

Figure 7 - update your version on Splunk
Figure 7 – Update your Splunk version

9) Thou shalt use forwarder management

Think smarter not harder. Forwarder management makes it easier to keep all your forwarders buttoned up and working properly. The alternative is to make changes and updates manually and individually, and depending on how many clients you have…that might take a while. Splunk’s native forwarder management tool is cool, but Kinney Group’s Forwarder Awareness application (through Atlas) is cooler. Check out this incredible tool that will save you a TON of time in Splunk.

Figure 8- Utilize forwarder awareness so you don't look like this guy
Figure 8- Utilize forwarder awareness so you don’t look like this guy

10) Thou shalt not use join/subsearches unless absolutely necessary

I want to start off by saying that sub-searches aren’t bad, they’re just not as efficient as other solutions. There’s more to come on this rule, but trust our advice, for now, avoid this at all causes.

Conclusion

Your data is important, so how you work with it in Splunk makes all the difference in the value you’ll get out of the platform. The Tech Ops team has worked with hundreds of Splunk customers, from our experience, these tips are a great place to start in adopting Splunk best practices. If you’d like to work directly with us, the experts, please fill out the form below!

Author