The Beginner’s Guide to Splunk Search

 

What are the basics of Splunk search?

The basics of Splunk search include the Splunk Web Interface, search controls, timeline and events, search commands, and data visualizations. Together, these elements make up the building blocks of all SPL search strings to view, analyze, and manipulate the data in your Splunk environment.

New call-to-action

Splunk Web Interface

Simply put, the Splunk Web Interface is Splunk’s UI (user interface). It’s where you’ll perform searches, use search commands, and visualize data. To begin the basics of search in Splunk, you’ll need to get your data into the Splunk Web Interface.

Search Controls

Search controls in Splunk are responsible for helping you navigate the Splunk Web Interface as you conduct your searches. There are several functions available within the search controls, including:

  • The Splunk Logo: Go back to your Splunk home screen.
  • App: Search & Reporting: Manage and download other Splunk apps from Splunkbase. 
  • Administrator (username): Personalize your settings in Splunk.
  • Messages: View health notifications and other pertinent messages from Splunk.
  • Settings: Customize your Splunk features.
  • Activity: View triggered alerts.
  • Help: Discover Splunk resources and learning material.
  • Find (Search): Search for items or keywords in your Splunk environment.

Timeline and Events

There are two additional elements alongside the search controls in Splunk: the timeline and events tab. The timeline provides a visual representation of the events that are taking place in your Splunk environment over time. Timeline and events are especially useful to identify patterns within your Splunk data so that your team can act quickly when activity that is outside the norm occurs.

Now that we’ve covered the basics of Splunk search, let’s go over how to create a Splunk search from scratch.

How to Create a Splunk Search

Splunk searches use SPL commands and arguments to retrieve, organize, and display data. A pipe character is used to start each new search string, followed by the command. Here’s the format for creating a Splunk search:

  1. Choose an index and a time range.
  2. Include filters to narrow down your search to only the data you want to see.
  3. Add commands and arguments one by one using the pipe character.
  4. Include the table, chart, or other visualization after all of your search strings are included.
  5. Save your search as a report, dashboard, or alert.

Basic Search Commands in Splunk

There are several search commands in Splunk that you can use to add further context to your data. Three of the most common elements you’ll use in nearly every search you run are index, time-range selection, and search filters. While these aren’t technically commands, they do help your commands work better, so we included them in this section.

  • Index: Specifying an index is step one of any basic or complex search because it tells Splunk where to look up the data. There are virtually no use cases where you won’t need to specify an index for your search, so you should always specify an index, here’s why.
  • Time-range selection: Choosing a time-range makes your search faster and more efficient. Selecting too small of a time-range can limit the data that returns from your search, but too large of a time-range can be wasteful of your Splunk resources and take much longer than necessary. If you’re not sure exactly what time-range to start with, do not choose real-time or all time.
  • Search Filters: Similar to index and time-range selection, search filters help you further sort your data to retrieve only the most relevant information to your search. Using search filters also results in more efficient searches. 

Eval Command: The eval command is incredibly robust and one of the most commonly used commands that calculates an expression and applies that value to a brand new destination field.

Stats Command:  The stats command calculates statistics. By using the stats command, you can find a high-level calculation of what’s happening to your machines.

Rex Command: When extracting fields, you can use rex commands to manipulate, calculate, or display specific data by itself.

Dedup Command: When you have events that include the same values in certain fields, you can use the dedup command to remove them.

Where Command: Another way to filter search results is the where command which uses true or false expressions to return search results.

Rename: This formatting command is used to rename a field. The syntax is <fieldname><newfieldname>

Head/Tail: By default, the head command returns the last 10 results of a search in descending order. The tail command does the same thing, only it starts at the end of the result set.

Top/Rare: The top command allows you to easily find the most common values in fields while the rare command returns the least common values in fields. These commands will also help you find information behind your event values like count and percentage of the frequency.

Sort/Reverse: The sort command is responsible for sorting search output data by field. The reverse command reverses a set of events.

Subsearches: A subsearch returns data that a primary search requires. The subsearch always runs before the primary search. It uses square brackets [ ] and an event-generating command. 

Join Command: To combine a primary search and a subsearch, you can use the join command. All you need to use this command is one or more of the exact same fields between both searches.

Transaction Command: To find, group, break up, or display raw data for similar events, use the transaction command. 

Visualizing Data and Splunk Search Results

We’ve created our search and formatted it. Now it’s time to visualize the data in a way that makes it easy to digest for yourself and any stakeholders who’ll need to view it. To do that, we use a variety of data visualizations in Splunk.

  • Table/Fields: The table and fields commands in Splunk allow you to retrieve specific fields within your data without conducting a search for all the fields in the data. The table command, specifically, pulls the data into a tabular format.
  • Chart/Timechart: The chart command is a transforming search command that allows you to put your data into a graphical visualization. The timechart command formats the results into an x and y chart where time is the x -axis and the y -axis is a specified field.

Splunk Search Basics

Searching in Splunk is quite straightforward when you have all the options at your fingertips. That’s why we created this guide. Bookmark this article for future reference as you become familiar with each search command. Practice using them and soon enough, you’ll be a Splunk search pro!

If you found this helpful… 

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. Once you download the app, you’ll get your report in just 30 minutes.

New call-to-action

Splunk Search Command Series: mvzip

 

 

Need some help zipping up your data in Splunk? This week’s Search Command should do the trick. The Splunk Search Command, mvzip, takes multivalue fields, X and Y, and combines them by stitching together.

Today, we are going to discuss one of the many functions of the eval command called mvzip. This function can also be used with the where command and the fieldformat command, however, I will only be showing some examples of this function using the eval command.

If you have been following our eval series, I am sure by now you know that the eval command is very versatile. Now let’s dive into another tool in the eval command’s tool belt! Let’s also use another command that we just learned called makemv to help facilitate this lesson. First, let’s make some data that has multiple field values.

Figure 1 - Data with multiple fields in Splunk
Figure 1 – Data with multiple fields in Splunk

 

New call-to-action

 

I’ve created three new fields called name, grade, and subject. Within each of these fields, we have multiple values. Let’s say we want to create a new field with these values “zipped” together. For example, I want to know what subjects Mike is taking all in one field. This is where mvzip comes in.

Figure 2 - mvzip example in Splunk
Figure 2 – mvzip example in Splunk

 

Here, I have created a new field called “zipped” with the values from the name and subject fields. Now we can see that Mike is taking Math, Science, History, and English. Next, I want to know what grades Mike has in those subjects (a.k.a. report card time!).

Figure 3 - Using mvzip in Splunk
Figure 3 – Using mvzip in Splunk

 

Using mvzip, we can see what grades Mike has in each subject. As you can see from the SPL above, I have mvzip the third field “grade” to the other two by adding another mvzip function. Splunk only allows you to zip three fields together, so this is our limit here! Also, if you noticed I added a different delimiter to our final results. I have a pipe separating my values instead of a comma in my first example. You can use whatever delimiter you want when using the mvzip function by putting quotes around the delimiter.

That is it for now, I hope you enjoyed this lesson and I hope you try this out in your own environment, happy Splunking! P.S. I think Mike could use some tutoring in History and English??

 

Ask the Experts

Our Splunk Search Command Series is created by our Expertise on Demand (EOD) experts. Every day, our team of Splunk certified professionals works with customers through Splunk troubleshooting support, including Splunk search command best practice. If you’re interested in learning more about our EOD service or chat with our team of experts, fill out the form below!

New call-to-action

How To Use Splunk Table and Fields Commands

 

Are you working with the same set of data on a regular basis? Are your searches taking more than a couple of seconds to load? Splunk’s table command and fields command can make this process faster for you. 

These two commands are similar, but they have different functions. In this guide, I’ll walk you through what table and field commands are and how to use them.

 

What is the fields command in Splunk?

The fields command is a Splunk search command that allows you to retrieve specific fields within your data. You can retrieve these fields without conducting a search for all the fields in the data. The benefit of using this command is that it reduces the time it takes for Splunk to retrieve the events associated with those fields.

 

How to Use the Fields Command

Step 1: Start a base search.

In this example, we’re using this search:

index=”splunk_test” sourcetype=”access_combined_wcookie”

Using job inspector, we can see it took about 7.3 seconds to run this search. This search includes all the events associated with each field in this set of data. You can see this on the right-hand side. 

Before Using Splunk Fields Command Search Speed Using the Job Inspector Tool

Step 2: Add the fields command.

index=”splunk_test” sourcetype=”access_combined_wcookie”
|fields JSESSIONID req_time referrer_domain

This fields command is retrieving the raw data we found in step one, but only the data within the fields JSESSIONID, req_time, and referrer_domain.

It took only three seconds to run this search — a four-second difference!After Using the Splunk Fields Command Search Speed Using the Job Inspector Tool

Running the Fields Command and Stats Command Together

You can use the fields and stats commands together for even faster searches. In this example, we’re running a stats command, but excluding a field from the search after we’ve run it.

Let’s start with the base search and the stats command:

index=”splunk_test” sourcetype=”access_combined_wcookie”
|stats count by action, status, JSESSIONID

How to Run the Fields Command and Stats Commands TogetherNext, we’ll include the fields command. We’ll be excluding the count field.

index=”splunk_test” sourcetype=”access_combined_wcookie”
|stats count by action, status, JSESSIONID
|fields - count

Now our search displays all of the same data it displayed before, but without the column dedicated to the count field.How to Use the Fields Command to Exclude a Count Field in Splunk Data

Splunk Tip: The fields command automatically includes the plus (+) so you don’t have to manually type it in when using this search command. This is also why using a minus (-) returns all the fields except those you’ve specified in the search.

 

New call-to-action

 

What is the table command in Splunk?

The table command does the exact same thing as the fields command where it pulls the raw data from a search quickly, using the fields you specify. The difference is that it pulls this data into a tabular format.

 

How to Use the Table Command

Step 1: Start a base search.

In this example, we’re using this search:

index=”splunk_test” sourcetype=”access_combined_wcookie”

Using job inspector, we can see it took about 7.3 seconds to run this search. This search includes all the events associated with each field in this set of data. You can see this on the right-hand side.

Step 2: Add the table command.

index=”splunk_test” sourcetype=”access_combined_wcookie”

|table JSESSIONID req_time referrer_domain

This table command is retrieving the raw data we found in step one, but only the data within the fields JSESSIONID, req_time, and referrer_domain. It’s placing that data in a tabular input.

How to Use the Table Command: Add the Table Command to the Search Bar

Splunk Tip:  The table command can also pull in fields that were not originally in your data — even fields that have been created after your data has been ingested.

Running the Table and Eval Commands Together

You can use the table and eval commands together for even faster searches. In this example, we’re looking for the error check field — this field doesn’t appear in our data until we run the eval command.

Let’s start with the base search and the eval command:

index=”splunk_test” sourcetype=”access_combined_wcookie”

|eval errorcheck=if(status>=400, “error”. “Non-error”}

Remember, the error check field won’t appear unless we search for this data using the eval command.How to Run the Table and Eval Commands Together Using Error Check as an Example Field

Now let’s add the table command so we can see the data in tabular format.

index=”splunk_test” sourcetype=”access_combined_wcookie”

|eval errorcheck=if(status>=400, “error”. “Non-error”} 
|table action errorcheck itemId
How to Use the Table Command in Tabular Format

Running the Stats and Table Commands Together

It’s important to note that the stats and table commands can be used together, but your table command results will be limited because the stats command is a transforming command. Put simply, that means any fields you’ve specified for the stats command will be the only fields that appear in your table, even if there is additional data in the base search.

Here’s what that looks like in practice:

index=”splunk_test” sourcetype=”access_combined_wcookie”

|stats count by action, status, JSESSIONID

|table action status req_timeHow to Run the Stats and Table Commands Together

Here, we can see that req_time has no values because running a transforming command like the stats command, our data is limited to the three fields we’ve specified (action, status, and JSESSIONID). Therefore, our table command can’t pull additional fields outside of the stats command.

 

Table and Fields Commands Made Easy In Splunk

There you have it! The fields command and the table command: two very useful and powerful commands that you should definitely add to your arsenal of search commands. Enjoy! 

Ask the Experts

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment 30-day free trial: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. You’ll get your report in just 30 minutes.

New call-to-action

Splunk 101: Basic Reporting and Dashboarding

 

It’s Mike again, one of Kinney Group’s resident Splunk experts. This week, I’ll review basic reporting and dashboarding functions following best practice methods in this video tutorial.

Basic Reporting and Dashboarding is one of many Splunk troubleshooting issues that is covered by our Expertise on Demand service offering. Within this video, I’ll break down the basics behind these essential functions of Splunk…

New call-to-action

 

Splunk Help At Your Fingertips

If you’re a Splunker, or work with Splunkers, you probably have a full plate. Finding the value in Splunk comes from the big projects and the small day-to-day optimizations of your environment. Cue Expertise on Demand, a service that can help with those Splunk issues and improvements to scale. EOD is designed to answer your team’s daily questions and breakthrough stubborn roadblocks. We have the team here to support you. Let us know below how we can help.

New call-to-action

Splunk Search Command of the Week: chart

 

This week, let’s chat about chart command.

The chart command is a transforming search command that allows you to put your data into a graphical visualization and like the stats command, the chart command can perform statistical functions such count, avg, min, max, etc. Chart command is going to be most utilized when you have fields that you want to build your chart with that do not involve time. Timechart and chart are similar. However, when you use the timechart command, your charts x-axis value is always going to represent time. With chart command, you can represent the x-axis using the over clause with any field you specify.  

 

Chart in Action

Let’s check out this dataset reviewing the ratings from IMBd on Netflix TV shows and movies.

 

Over and By Clause

Here’s an example of chart command and the over clause in action.

 

Figure 1 - Chart command and the over clause
Figure 1 – Chart command and the over clause

Notice that the x-axis is represented by the Age field. This is a product of using the over clause and letting Splunk know that you want Age to be on the x-axis.  The chart command also allows you to manipulate the y-axis by using the by clause.

New call-to-action

Here is an example of using the over clause and the by clause together. You can see the chart broken down over Age by IMDb which is the ratings of those movies in that specific age group.

 

Figure 2 - Chart command and the over clause and by clause
Figure 2 – Chart command and the over clause and by clause

 

Remove NULL and OTHER

The legend on the right-hand side has all the ratings in different colors. You’ll also see two values you may not necessarily be interested in… NULL and OTHER. Chart and timechart commands automatically filter results to include the ten highest values while the surplus values are grouped into the OTHER category. In this particular search, our results are skewed by the NULL and OTHER values.

To remove the NULL and OTHER values, you will use these two arguments “useother=f & usenull=f”. After applying the useother=f and usenull=f, you get the results you see below. You can see how the data looks better and cleaner without the OTHER and NULL values.

 

Figure 3 - Remove NULL and OTHER from your chart legend
Figure 3 – Remove NULL and OTHER from your chart legend

 

The Limit Argument

If you want to adjust the number of series that Splunk returns back, use the limit argument. With limit, specify how many values you’d like Splunk to return with.  If you want Splunk to return an unlimited amount of values, use limit=0. Let’s take a look at this in action. After applying the limit argument of 20, this is what Splunk brings back.

 

Figure 4 - Chart command series limit of 20
Figure 4 – Chart command series limit of 20

Next, let’s take a see what an unlimited amount of values looks like.

 

Figure 5 - Chart command series unlimited
Figure 5 – Chart command series unlimited

There you have it. Splunk has brought back all of the IMDb ratings associated with the movies in each age group. Now, you’ve seen chart command in action and its visualization options.

Ask the Experts

Our Splunk Search Command of the Week series is created by our Expertise on Demand (EOD) experts. Every day, our team of Splunk certified professionals works with customers through Splunk troubleshooting support, including Splunk search command best practice. If you’re interested in learning more about our EOD service or chat with our team of experts, fill out the form below!

New call-to-action