Does Slow Splunk Search Performance Have You Stuck?

Does slow Splunk search performance have you stuck?

Get Unstuck with Jumpstart Services for Splunk by Kinney Group Inc.

Slow Splunk search performance can be attributed to several different factors. Perhaps the hardware Splunk is running on is old or outdated. Maybe you are running an older version of Splunk. But what if you are running a newer version of Splunk on minimum or above recommended hardware? One possible cause could be attributed to inefficient searches.

Why stats?

You may have a search/report that you dread to run due to the excessive duration it takes to complete. Splunk’s search processing language (SPL) is very flexible. This means one search can be written many different ways to accomplish the same end goal. With this in mind, we can craft searches to avoid certain functions that may be negatively contributing to our overall search performance.

Certain search functions can be used in the place of others to greatly improve performance. Did you know most transaction and join commands can be replaced with stats? Transactions and joins can be very costly in regard to search time. Below we will walk through a simple example of how to switch a join command to stats while preserving the original goal of the search.

For this example, we will want to produce a count of orders by purchaser country:

Screenshot of Slow Splunk Search Performance Example - Orders by Country - Kinney Group - Jumpstart for Splunk

Figure 1 – Order count by Country

The following search is our original search to output the desired results:

index=orders source=orders.csv

| table order_id,customer_id,order_date

| join type=left customer_id

     [search index=customers source=customers.csv

     | fields customer_id,customer_name,contact_name,country

     | table customer_id,customer_name,contact_name,country]

| table customer_id,customer_name,contact_name,country

| stats count by country

| sort 5 -count

| rename country as Country, count as "Total Orders"

This search achieves our goal, however, takes a considerable amount of time to run. This is due to the join command. Join utilizes a sub search which greatly increases search time. To improve performance while preserving the final result, the stats command will be used.

First, the data from the sub search will need to be brought up to the base search:

(index=orders source=orders.csv) OR (index=customers source=customers.csv)

Now we will create a filter field using the eval command. This will allow us to filter out only the rows that are applicable for the end result:

(index=orders source=orders.csv) OR (index=customers source=customers.csv)

| eval filter=if(index==”a”, 1, 0)

This filter field we created enables us to perform simple math to determine which fields need to be kept. This will assign every event from ‘index=orders’ to have a 1 and everything else would be 0. The original base search is performing a left join on the customer_id field.

To simulate this, we will be running a preliminary stats command to aggregate on the customer_id field:

(index=orders source=orders.csv) OR (index=customers source=customers.csv)

| eval filter=if(index==”orders”, 1, 0)

| stats values(country) as country, sum(filter) as filter by customer_id

The result will output the following table:

Screenshot of Splunk Search Output Table - Kinney Group - Jumpstart for Splunk

Figure 2 – Portion of the statistics table produced by the search.

Notice if the customer_id field is not present in index=orders, it will have a filter value of 0. This will allow us to filter where this value is greater than 0 effectively creating a left join.

Putting it all together:

(index=orders source=orders.csv) OR (index=customers source=customers.csv)

| eval filter=if(index==”orders”, 1, 0)

| stats values(country) as country, sum(filter) as filter by customer_id

| where filter>0

| stats count as “Total Orders” by country

| sort 5 –“Total Orders”

This search will provide us the same results as the original join search while giving a significant difference in search speed:

Screenshot - Splunk Optimized Search Stats - Kinney Group

Figure 3 – Result from optimized stats search showing same results from original search.


Screenshot Original Search

Figure 4 – Original search

Getting Unstuck

This was a simple example of how to improve slow Splunk search performance by switching to a stats from join. Kinney Group Splunk consultants are highly experienced and know how to get you unstuck.

The Kinney Group team began working with the Splunk technology all the way back in 2013. Since that time, we have consistently seen organizations run into challenges as they work to fully activate and adopt the platform.

Get Your Team “Unstuck” with Splunk Implementations

We have tested this service concept with “stuck” Splunk implementations and have experienced tremendous success. Organizations will be amazed at how much they can accomplish in just one day of focused immersion with one of our top-notch Splunk consultants.

Jumpstart Service for Splunk is designed to help companies that are:

  • Evaluating new use cases for Splunk
  • Limited on technical resources
  • Struggling to capture ROI with Splunk
  • Seeking documentation and adoption support
  • Short-staffed or onboarding new Splunk resources

If you are “stuck” or “stalled” with your Splunk implementation, get rolling again with our Jumpstart Service for Splunk—it just might be the best thousand bucks your organization has ever spent!

Jumpstart Services for Splunk - Kinney Group

Start typing and press Enter to search