Splunk Best Practices: The 10 Commandments You Need to Know

There are many do’s and don’t’s when it comes to Splunk. In our time supporting Splunk customers through Expertise on Demand, Team Tech Ops has seen the good, bad, and ugly situations customers can fall into with Splunk.

You don’t want to fall victim to similar situations that deprive you of your time, Splunk bandwidth, and quite frankly, your sanity. So, here are the 10 Commandments of Splunk best practices you absolutely need to know.

1) Thou shalt NEVER search index=*

Why? Splunk has A LOT of data. In most cases, it has hundreds of gigabytes, maybe even terabytes of data. Searching all of that data puts your Splunk instance under enormous strain.

What happens if you search index=*? By searching index=* you’re searching every single index that you have access to. 

For example, I searched index=* and the search took 13.5 seconds. 

Now, imagine that across all of your indexes. Yikes.

Do this instead: Specify a specific index like index=windows. That search took 2.8 seconds — much better! 

Moral of the story: when you know your index, always search for it.

Splunk Best Practices Meme: "Slaps Splunk" meme
Figure 1: “Splunk Slaps” meme

2) Thou shalt remove real-time & All Time as an option for basic users

Why? Just as we don’t want to search every index because of the speed required for the search, we also want to avoid looking at every event that already exists or will exist in the future.

What happens if you search using real-time and All Time? Running a search in real-time or across All Time causes a resource strain on the environment and may even cause disruption for your fellow Splunk users.

Do this instead: Narrow your search using a more specific date range such as “7 days”  rather than All Time. The difference is seven seconds for the former and 139 seconds for the latter.

3) Thou shalt not ingest data into the main index

Why? Main index is a default index for Splunk Enterprise. 

What happens if you ingest data into the main index? Without specifying an index for your inputs, all your data will default to the main index, and you’ll have to search a huge, unorganized index every time you search.

Do this instead: Typically, it is best practice if you never send information to the main index. Ever. If you thought it was confusing to find data when it is nicely organized in your indexes and source types, try finding anything when it’s completely jumbled up in one place.

4) Thou shalt leave on ALL search formatting settings

Why? Without data formatting, your search will look like one big wall of text, making it more difficult to read and edit.

What happens if you don’t leave on all search formatting settings? Your search will look something like this.

Figure 2 - Splunk without search formatting settings
Figure 2 – Splunk without search formatting settings

Do this instead: Go to your SPL editor and choose “Line numbers” and Search auto-format” to keep your search looking clean and neat.

Figure 2 - Splunk with ALL search formatting settings
Figure 3 – Splunk with ALL search formatting settings

5) Thou shalt view the monitoring console before requesting a performance dashboard be built

Why? Most of the dashboards you’re looking for are already there, so you don’t need to reinvent the wheel or in this case… the monitoring console.

What happens if you don’t view the monitoring console before requesting a performance dashboard? You could be doing duplicate work which takes you away from other tasks you could be completing.

Do this instead: Explore the dozens of dashboards in the monitoring console to see what dashboards are available and whether they meet your needs.

Splunk Best Practices Meme: dragon ball z Splunk monitoring console meme
Figure 4 – Splunk monitoring console meme
New call-to-action

6) Thou shalt look for an add-on first before onboarding new data

Why? If you’re onboarding data, there is probably an app or add-on that can help.

What happens if you don’t use an add-on to onboarding new data? You’ll spend far more time and have far more difficulty adding new data to Splunk without a helpful add-on.

Do this instead: You’re going to save a lot of time (and aspirin) using one of Splunk’s apps or add-on tools. Note: this does not apply if you’re ingesting something completely unique to you or your company.

Splunk Best Practices Meme: Splunk app or add-on meme
Figure 5 – Save time with a Splunk app or add-on

7) Thou shalt follow correct directory precedence

Why? Correct directory precedence in Splunk is to save your files to a local folder. Saving it anywhere else puts your data at risk of being lost.

What happens if you don’t follow the correct directory precedence? WHen you’re ready to update Splunk, you’ll lose all of the data that was stored in the wrong directory. 

Do this instead: NEVER save a .conf file in /default. Instead, save them to a local folder.

Splunk Best Practices Meme: South Park /conf meme
Figure 6 – Don’t use .conf file in /default in Splunk

8) Thou shalt have all instances of Splunk on a supported version

Why? If your Splunk environment is not on a supported version, all of your data is at risk of being corrupted or lost when it’s time for the next update.

What happens if you don’t have all instances of Splunk on a supported version? If you have multiple instances of Splunk on one version and multiple instances on another, your data could be ingesting differently, causing your database to run more slowly than it normally would. 

Do this instead: Go to Help> About to review the version of Splunk you’re running. If it’s out of date, work with your team on a plan to update it to the newest version.

Splunk Best Practices Meme: update your version on Splunk mean girls meme
Figure 7 – Update your Splunk version

9) Thou shalt use forwarder management

Why? Forwarder management makes it easier to keep all your forwarders buttoned up and working properly. 

What happens if you don’t use forwarder management? The alternative to forwarder management is to make changes and updates manually and individually. Depending on how many clients you have, that could take a while. 

Do this instead: You could use Splunk’s native forwarder management tool, but Kinney Group’s Forwarder Awareness application (through Atlas) is a better option because it lets you see all of your forwarders in one place. Rather than wondering how your forwarders are performing, you can see them in real-time and act on any issues the second they happen. Check out this incredible tool that will save you a TON of time in Splunk.

Splunk Best Practices Meme: Utilize forwarder awareness so you don't look like this guy
Figure 8- Utilize forwarder awareness so you don’t look like this guy

10) Thou shalt make searches as performant as possible

Why? While there are many ways to run a search, just because you can run it a certain way doesn’t mean you should. 

What happens if your search isn’t as performant as possible? Searches that are bulky and slow not only slow you down, but they can make your Splunk environment sluggish and even cause bigger issues, like failed searches, down the line.

Do this instead: Familiarize yourself with several search commands and practice using them in combination with one another. We have additional blog posts that walk you through the most popular Splunk commands, how to use them, and examples of using them with other commands.

Conclusion

Your data is important, so how you work with it in Splunk makes all the difference in the value you’ll get out of the platform. The Tech Ops team has worked with hundreds of Splunk customers, from our experience, these tips are a great place to start in adopting Splunk best practices. If you’d like to work directly with us, the experts, please fill out the form below!

New call-to-action