The Secret to a Great Splunk Search

The secret to a great Splunk basic search is… well, it’s simple. Eliminate as many options as possible, as early as possible, and make your search simple. By keeping your Splunk searches constrained, you narrow down the work that’s done on the back end. Focusing your search not only improves your speed to results but also leaves more resources available so your environment can run efficiently. Let’s take a look at a few ways to creating a great search in Splunk.

Narrow Your Focus

When first constructing your basic search query, define exactly what you are looking for. Once you have that in mind, you can determine where that data resides within Splunk. By doing this, you also avoid forcing Splunk to reference a lot of unrelated events.

If you are looking for inbound network traffic that has been bounced by your organization’s firewalls, there is no reason for Splunk to search the indexes and sourcetypes used for apache weblogs.

Limiting the scope of your Splunk search by specifying the index(es), and if possible the sourcetypes, that contain the information you are looking for is the most basic and useful limit you can place to control the pool of data used. If you know the source or host associated with the log you are interested in, you can hone in even more tightly on your goal.

Set Time Frames

If you know that the information you are searching for would be found during a specific time window, then use that to your advantage. This can dramatically reduce the amount of information Splunk has to retrieve and process, both speeding up your query and reducing the performance impact on the Indexers.

Specifying time ranges can either be achieved via the time picker or through hardcoding the information into the search query you use. Detailed information on usage of the time picker is available from Splunk here, while information on applying time ranges directly to the queries can be found here.

Figure 1 - Utilize Splunk time range picker to narrow basic search results

Figure 1 – Utilize Splunk time range picker to narrow basic search results

Filter Your Results

Filter your results early and often. Whenever you reach a step in the query where results can be eliminated, do so. In our example, we are only interested in firewall results where the traffic was blocked. Therefore in the initial query, we can eliminate any results that were allowed.

Let’s say our end goal is to find information related to the top 5 source IPs that have been blocked. Once we have calculated how many times each source has been rejected, we can drop all of the non-relevant hits.

This reduction isn’t just related to dropping entire events from your results, it can also take the form of killing off fields in the results that are of no interest to you. This reduces the amount of overhead Splunk has to deal with for each event.

This reduction of results can take many forms. From specifying values in the query directly to the use of commands such as: fields, head, rare, tail, and top.

Avoid Subsearches

Whenever possible, avoid using subsearches in your queries. While they can be a powerful tool in certain use cases, they can also carry a significant overhead.

In general, a subsearch should be used to narrow the results of a previous portion of the search, not bring in new results. Remember that the initial query can span multiple indexes and sourcetypes, so it is rarely necessary to use a subsearch for this purpose.

Sharpen Your Splunk

We know that tuning up your Splunk searches may fall short in priority with your larger Splunk tasks at hand. That’s why the Kinney Group service offering, Expertise on Demand, gives your team time back in their day to tackle the larger Splunk tasks. We have the best practice knowledge and a team of experts ready to sharpen up your Splunk efficiency. If you’re looking for some extra support to ensure your Splunk is performing in top shape, let us know in the form below.


Start typing and press Enter to search