There are many do’s and don’t’s when it comes to Splunk. In our time supporting Splunk customers through Expertise on Demand, Team Tech Ops has seen the good, the bad, and the ugly situations customers can fall into with Splunk.
We’re happy to present the Tech Ops Ten (Command)ments of Splunk best practices.
1) Thou shalt NEVER search index=*
This one is pretty self-explanatory. Splunk has A LOT of data.
In most cases, hundreds of gigabytes, maybe even terabytes of data. I’m sure you tried running a search that looks in one index across millions of events and found it took a very long time to complete.
Now, imagine that across all of your indexes. Not many Splunkers can see this full picture because there’s always a search that will not complete (unless you have a tiny environment or use something like tstats).
Searching “index=*” goes into what I like to call the worst practices box.
2) Thou shalt remove real-time & All Time as an option for basic users
Right in line with never looking at every index humanly possible, we also want to avoid looking at every event (that does exist or will exist).
Running a search in real-time or across all-time causes a resource strain on the environment and may even cause disruption for your fellow Splunk users.
3) Thou shalt not ingest data into the main index
Main index is a default index for Splunk Enterprise. Without specifying an index for your inputs, all your data will default to the main index.
Typically, it is the best practice if you never send information to the main index. Ever. If you thought it was confusing to find data when it nicely organized in your indexes and source types, try finding anything when it’s completely jumbled up in one place.
4) Thou shalt leave on ALL search formatting settings
This one is a “You v.s. The Guy she tells you not to worry about” situation.
The Guy she tells you not to worry about:
5) Thou shalt view the monitoring console before requesting a performance dashboard be built
Most of the information is already there, you don’t need to reinvent the wheel or in this case… the monitoring console.
6) Thou shalt look for an add-on first before onboarding new data
If you’re onboarding data, there is probably an app or add-on that can help. You’re going to save a lot of time (and aspirin) utilizing one of Splunk’s app or add-on tools. Note: this does not apply if you’re ingesting something completely unique to you or your company.
7) Thou shalt follow correct directory precedence
NEVER save a .conf file in /default. It’s that simple. Just don’t do it.
8) Thou shalt have all instances of Splunk on a supported version
If you wouldn’t allow an unsupported version of Windows Server in your environment, then why would you allow an unsupported version of Splunk in?
9) Thou shalt use forwarder management
Think smarter not harder. Forwarder management makes it easier to keep all your forwarders buttoned up and working properly. The alternative is to make changes and updates manually and individually, and depending on how many clients you have…that might take a while. Splunk’s native forwarder management tool is cool, but Kinney Group’s Forwarder Awareness application (through Atlas) is cooler. Check out this incredible tool that will save you a TON of time in Splunk.
10) Thou shalt not use join/subsearches unless absolutely necessary
I want to start off by saying that sub-searches aren’t bad, they’re just not as efficient as other solutions. There’s more to come on this rule, but trust our advice, for now, avoid this at all causes.
Your data is important, so how you work with it in Splunk makes all the difference in the value you’ll get out of the platform. The Tech Ops team has worked with hundreds of Splunk customers, from our experience, these tips are a great place to start in adopting Splunk best practices. If you’d like to work directly with us, the experts, please fill out the form below!