Pivots are the perfect way to build personal a dashboard in Splunk without creating search queries manually. Whether you’re a beginner or an expert, learning how to build a Pivot dashboard can save you a ton of time (and headaches) when pulling data from your Splunk environment.  

Here’s a crash course on everything you’ll need to know about Pivots in Splunk. 

What is Pivot in Splunk? 

A Pivot is a dashboard panel in Splunk. The goal of Pivots is to make searching easier in Splunk by using existing data sets instead of SPL queries to populate the Pivot. 

Do I need to know SPL to build a Pivot or dashboard in Splunk? 

No, you don’t have to know SPL to build a Pivot dashboard in Splunk. By using data models and data sets, you can build a robust Pivot dashboard without using SPL or running queries manually. 

Who can build Pivots in Splunk? 

Anyone who uses Splunk to understand that data in their organization can build a Pivot dashboard in Splunk. Because it doesn’t require any SPL knowledge, anyone from a summer intern to the VP of Technology can build their own Pivot dashboards in Splunk. 

How to Create a Pivot in Splunk 

The drag and drop UI of Pivots makes it easy to build a Pivot dashboard in Splunk.  

Step 1: Create a New Data Model or Use an Existing Data Model 

To begin building a Pivot dashboard, you’ll need to start with an existing data model. If you don’t have an existing data model, you’ll want to create one before moving through the rest of this tutorial. 

Go to data models by navigating to Settings > Data Models. 

For this example, we’re using the standard data model Internal Audit Logs, but you can choose any data model in your environment. 

Splunk Tip: When your Splunk environment was created, it automatically came with the Internal Audit Logs data model. This data model includes all of your internal audit log data, so you can be sure that the Pivot table you’re creating will reflect real and accurate data. 

Step 2: Select Pivot 

In the top right corner of the screen select Pivot. 

Once you open your data model and select Pivot, you’ll see at least one (but likely more) data sets in the model. 

Step 3: Review the Data for Your Pivot 

It’s important to click on each data set and review the fields within it in order to find the data you want to include in your Pivot table. Once you find your desired fields, click on the name of the data set again to open your new Pivot.  

Step 4: Build Your Pivot 

Building your Pivot is both an art and a science. Here are the overarching elements you can manipulate to build your Pivot table. Play around here and see what data populates. Keep what you like, and remove what you don’t. 

• Filter Your Data: You should filter your data so that it pulls information from the right time period. To do this click Filters and choose from Real Time, Relative, or All Time. In this example, we’re choosing a Relative time filter of Last 7 Days. 
• Check the Count of Audit: After filtering your data, you’ll see how many audited events happened in the time frame you selected. In our case, we see 1,247 audits in the last 7 days. If you’re seeing zero audits, double-check that the data set you’re using actually has data, or try refiltering your data using a larger time frame. 
• Add Fields to Your Pivot: Select Split Rows to reveal a dropdown of all the fields available in the rows of your Pivot. To start, we’re choosing the Action field. This will show us all the actions that happened in our audit and how many of each action occurred. You can continue to add rows to your Pivot for more details about the data. 
• Add Fields to Your Pivot: Select Split Columns to reveal a dropdown of all the fields available in the columns of your Pivot. In this example, we’re choosing the Host field. This will show us all the hosts for each action in our audit. You can continue to add columns to your Pivot for more details about the data. 

Splunk Tip: The smaller and less complex your data set, the fewer fields you’ll have to choose from when splitting rows and columns. Don’t get discouraged if you have only a couple of fields to include in your Pivot. If there are additional fields you’d like to pull into your Pivot in the future, you can work with your Splunk team or ask the experts at Kinney Group to help you set them up. 

• Add Visualizations: Although the default for building a Pivot in Splunk is to use a table, you can change the visualizations to display your data in different ways. On the right-hand side of your screen, you’ll see a vertical list of icons, each of which will display your data in a different graph or chart. In this example, we’re using the line graph visualization represented by the line graph icon. With any visualization, you can adjust the X-axis, y-axis, color, and other properties of your graph or chart. 

Splunk Tip: Visualizations represent what your data will look like in the finished Pivot dashboard. If you don’t choose a visualization, you’ll simply see the table and raw data in your Pivot. This makes it hard to see your data at a glance which is the point of building the Pivot dashboard. We highly recommend you choose a visualization for your data so that it reflects the information you want to see in your finished Pivot in an accurate and appealing way.  

Step 5: Save Your Pivot 

In the top right of the screen, select Save As > Dashboard Panel. 

Give your new Pivot a title and description. Then choose whether your Pivot will be private or public.  

Choose Save > View Dashboard to see your new Pivot. 

Step 6: Title your line chart.

Splunk Tip: We already named the Pivot dashboard, but you’ll still want to title your line chart so that you know what data is represented in it. As you add more visualizations of different data sets, you’ll find that naming each one makes your Pivot dashboard easier to use. 

Can Pivots be saved as reports panels in Splunk? 

Unfortunately, Pivots cannot be saved as reports panels in Splunk. If your team wants access to your Pivot dashboard for their own reporting needs, you can make the Pivot public and share it with them so they have access to it on demand. 

Key Takeaways for Creating Pivots in Splunk 

Pivots are an amazing tool for Splunk users who aren’t well-versed in SPL or building search queries. You don’t have to make decisions in the dark because you don’t understand Splunk as well as your engineers do. With Pivot dashboards, you’ll have the most important data at your fingertips when you need it, all without creating a single SPL search query. 

If you found this helpful… 

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate. 

Cue Expertise on Demand, a service that can help with those Splunk issues and improvements to scale. EOD is designed to answer your team’s daily questions and breakthrough stubborn roadblocks. Book a free consultation today, our team of experts is ready to help. 

Join Kinney Group Splunk Enterprise Security Certified Admin Hailie Shaw as she walks through the process of data model mapping in Splunk for CIM compliance. Catch the video tutorial on our YouTube channel here.

Note: the data visible in this video and blog post are part of a test environment and represent no real insights. 

Starting Out with Data Model Mapping

As Splunk engineers, we constantly deal with the question: How do I make my data, events, and sourcetypes Common Information Model (CIM) compliant? This is especially crucial with Enterprise Security use cases, when data will need to map to a CIM compliant data model.  When we search the environment pre-CIM compliance, our query will return no results. This is what we aim to change with data model mapping. 

Begin by downloading the Splunk Add-on Builder app, which you’ll use later. Let’s transition to Splunk docs. The left panel of the webpage lists all data models. For any option, Splunk docs will list out all fields to which users can map. This should be the first step when choosing the data model that matches your environment’s existing fields as closely as possible.  

Creating the Add-on

Turn to Splunk’s Search & Reporting app and navigate to the data that you’re looking to map; the fields will populate on the left side of the screen. Pay special attention to the sourcetype that you wish to match—you’ll need to supply this to the Splunk Add-on Builder app that will map the fields for CIM compliance. Within Splunk Add-on Builder, select “New Add-on.” The default settings will be sufficient with the exception of the required Add-on name. Click “Create.”  

Click Manage Source Types on the top banner menu. Click “Add,” and then “Import from Splunk” from the dropdown menu. Select your sourcetype, which should populate within the menu after you import data from Splunk. Click Save, and the events will be uploaded. Next, click Map to Data Models on the top banner menu.  

Mapping the Data Model

From here, select “New Data Model Mapping.” You’ll be prompted to enter a name for the event type and select the sourcetype you’re using. The search will populate below, automatically formatted with the correct name. Click “Save.” The resultant data model will include, on the left-hand side of the screen, a list of the event type fields. Hovering over each entry within this list will reveal a preview of the data that exists within the field. 

Essentially, the app takes the fields from your initial data and transfers them onto the data model. On the right of the screen is where you’ll select a data model for the fields to map to. Each data model is filtered through Splunk’s CIM, and you can select which is most appropriate based on the Splunk documentation with which we began.  

When you select a data model, the Add-on Builder will provide supported fields, which you can cross-reference with Splunk Docs; the app is a field-for-field match. This step will give you a list of CIM-approved fields on the right to complement the original fields on the left. To map them together, click “New Knowledge Object” and select “FIELDALIAS.” This is where you’ll need to select which data model field most closely matches the initial event type or expression.

Once you’ve made a match, select OK, and the app will provide a field alias. Repeat this process for each field you wish to include. Once you’re satisfied, click “Done.” As you can see, the data has now populated with the sourcetype listed. 

Validate and Package 

It’s important to validate your package to ensure that it follows best practices. To do so, click “Validate & Package” from the top banner menu. Click the green “Validate” button. When the validation reaches 100%, you can download the package. This page will also generate an Overall Health Report detailing which elements, if any, should be addressed. Once downloaded, change the name of the file to a .zip. By double-clicking, the file will extract, and you can open it to view details of the sourcetype and events within.  

Return to the Splunk instance through Search and Reporting and run the data model command again as a search. Now, your events will populate in the data model! You can also view the CIM compliant aliases in Settings > All Configurations.  

Compliance, Achieved

The resultant data model is good to go—CIM compliant and ready to be exported. Check out more Splunk tips on our blog and our YouTube channel. To get in touch with engineers like Hailie, fill out the form below: 

When users import a data file into Splunk, they’re faced with a dense, confusing block of characters in the data preview. What you really need is to make your data more understandable and more accessible. That’s where data parsing and event breaking come in.

In this brief video tutorial, TechOps Analyst, Hailie Shaw, walks you through an easy way to optimize and configure event breaking— spoiler alert: it boils down to two settings in props.conf.

This function within Splunk can greatly simplify the way you interact with your data. After learning it, you’ll get the insights you need, faster.

Event Breaking

When importing data into your Splunk instance, you’ll want to be able to separate it based on events, which enables legibility and ease of interpretation. Because the imported data file isn’t pre-separated, event breaking is an essential skill. The most important part in separating data is the line breaker within the event boundary, which is how Splunk decides how to group and separate certain events.

LINE_BREAKER is how Splunk separates events into different lines. The default = a sequence of new lines, followed by all carriage returns within parentheses.

SHOULD_LINEMERGE is how Splunk merges separate lines into individual events. The default value = true, but should always be set to false as a best practice. If set to true, the value will automatically set to false once Splunk is supplied with your new regex, which provides a specific guide for breaking up new lines and preventing them from merging.

When you import the data file, the term “<event>” appears several times intermittently within the block of data, which is listed as one large event. By pasting the original block of data into regex101.com, users can identify the <event>s that should ideally begin new lines. Then enter the regex resulting from this step into Splunk in props.conf, which will populate the desired fields.

By selecting regex as the event-breaking policy and entering the pattern from regex101.com, the data preview will display your data separated into events (each beginning with “<event>”). The regex in props.conf defines both terms listed above. This data preview can now be saved as a new sourcetype.

Learn More!

Learning event breaking can help make your data more organized and legible for members of your team. Kinney Group’s Expertise on Demand service offers a wealth of Splunk tips and knowledge, tailored by analysts like Hailie to your team’s specific needs. Check out the blog for additional tutorials from our engineers. To learn more about Expertise on Demand, fill out the form below!

With Splunk, getting data in is hard enough. After uploading a CSV, monitoring a log file, or forwarding data for indexing, more often than not the data does not look as expected. These large blocks of unseparated data are hard to read and unable to be searched. If the data is not separated into events, you may be wondering how to correctly parse, and perform advanced search commands using fields.

This is where field extraction comes in handy.

Field Extraction via the GUI

Field extractions in Splunk are the function and result of extracting fields from your event data for both default and custom fields. Basically, organize your data with field extractions in order to see the results you’re looking for.

Pictured above is one of Splunk’s solutions to extracting searchable fields out of your data via Splunk Web. Within the Search and Reporting App, users will see this button available upon search. After clicking, a sample of the file is presented for you to define from events the data. The image below demonstrates this feature of Splunk’s Field Extractor in the GUI, after selecting an event from the sample data.

From here, you have two options: use a regular expression to separate patterns in your event data into fields, and the ability to separate fields by delimiter. Delimiters are characters used to separate values such as commas, pipes, tabs and colons.

If you have selected a delimiter to separate your fields, Splunk will automatically create a tabular view in order to allow you to see what all events properly parsed would look like compared to its _raw data pictured above.

Functionality is provided to rename all fields parsed by the selected delimiter. After saving, you will be able to search upon these fields, perform mathematical operations, and advanced SPL commands.

What’s Next? Rex and Erex Commands 

After extracting fields, you may find that some fields contain specific data you would like to manipulate, use for calculations, or display by itself. You can use the Rex and Erex commands to help you out.

Rex

The Rex command is perfect for these situations. With a working knowledge of regex, you can utilize the Rex command to create a new field out of any existing field which you have previously defined. This new field will appear in the field sidebar on the Search and Reporting app to be utilized like any other extracted field.

Syntax

| rex [field=<field>] (<regex-expression>)

For those who would like to use the Rex command, and would like resources to learn, please utilize websites such as https://regex101.com/ to further your development.

In order to define what your new field name will be called in Splunk, use the following syntax:

| rex [field=<field>] (?<field_name>”regex”)

Erex

Many Splunk users have found the benefit of implementing Regex for field extraction, masking values, and the ability to narrow results. Rather than learning the “ins and outs” of Regex, Splunk provides the erex command, which allows users to generate regular expressions. Unlike Splunk’s rex and regex commands, erex does not require knowledge of Regex, and instead allows a user to define examples and counterexamples of the data to be matched.

Syntax

 | erex <field_name> examples="<example, <example>" counterexamples="<example,

<example>"

Here’s an example of the syntax in action:

 | erex Port_Used examples=”Port 8000, Port 3182”

That’s a Wrap

There is a ton of incredible work that can be done with your data in Splunk. When it comes to extracting and manipulating fields in your data, I hope you found this information useful. We have plenty of Splunk tips to share with you. Fill out the form below if you’d like to talk with us about how to make your Splunk environment the best it can be.

There is immense power built within the Splunk data center model. However, with great power comes even greater complexity. The traditional model is difficult to scale for performance, and Splunk teams already working against license costs are constantly faced with performance issues.

We know your teams have more to do with their time, and your organization has better ways to spend their budget.

With Atlas’s Reference Designs, we’ve found a better path forward.

Background

The Atlas Reference Designs provide a blueprint for modern data center architecture. Each provides blazing-fast performance increases and reduces indexer requirements, leading to a lower cost of ownership for your environment. Our Reference Designs are built to remove any friction caused by building and deploying your Splunk environment, serving all Splunk customers. Whether you’re running Splunk on-prem, in a hybrid fashion, or in the cloud, Atlas’s Reference Designs provide you with a simplified and proven solution to scaling your environment. 

There is a common problem that every organization faces with Splunk: ingesting data effectively.

The Atlas Reference Designs present you with a solution that pairs the Splunk Validated Architectures with the needed hardware or environment in order to deliver proven performance results.

Our first set of Reference Designs were built for on-prem Splunk instances in partnership with Pure Storage, a leader in the enterprise platform and primary storage world. Check out Pure Storage’s many nods in the Gartner Magic Quadrant for their innovative solutions.

With the matched power of Pure Storage with Kinney Group’s Splunk solutions, we’ve been able to decrease the indexers needed through Pure’s speed and increased IO on the back end—and we have the tested results to prove it.

Proven Results

Splunk recommends that for every 100 GB you ingest, you create at least one indexer. Let’s look at a real-world example delivered by the Atlas team: a company is ingesting 2 TB/day in Splunk—based on Splunk’s recommendations, this company relied on 20 indexers.

By applying the Atlas Reference Design, we were able to reduce that physical indexer count to just five indexers. This significantly reduces the cost of owning Splunk while increasing performance by ten times.

For those who invest in Splunk infrastructure on-prem, this means huge savings. Think of the costs that 20 indexers entail versus five—the cost savings are huge. Then factor in the impact of reducing the exhausting manual hours spent on Splunk to just a few minutes of human interactions.

Conclusion

To sum it up, we took an incredibly complex design and made it shockingly easy. Through automation and the great minds behind Atlas, your Splunk deployments can now launch with the push of a button, less time and fewer resources, and guaranteed results. The Atlas Reference Designs are built for all customers, with many releases to come; get in touch using the form below to inquire about our other Reference Designs.

Last week, we shared some big news! Kinney Group launched Atlas — an exciting new platform for navigating and achieving success with Splunk. We’re thrilled to reveal this revolutionary platform to the world and share more about its vision and technical components.

The mission of Atlas is to help customers derive meaningful value from Splunk for their organization and their colleagues.

Background

What’s stopped organizations so far from reaching their goals in Splunk? The answer is all too real for our friends in the industry: IT teams are under siege.

Splunk is and should always be a key enabler for organizations to reach business, security, and operational objectives. Splunk gives IT teams meaningful insights into security posture, IT Ops, and other analytics. Here’s the issue: these teams are buried in work. IT teams are already charged with an ever-growing list of IT demands. And to top it off, they’re now tasked with managing, implementing, and learning Splunk.

Atlas enables these teams to derive value out of Splunk, fast.

A Look Inside Atlas

Atlas is a subscription service that provides a clear path forward on your Splunk journey with revolutionary datacenter architectures, personal guidance and support, and a collection of applications and utilities that provide powerful insights, instantly. Let’s take a closer look into each component of Atlas.

1. Atlas Reference Designs

Atlas Reference Designs provide a blueprint for modern data architecture. They provide rapid performance increases and reduce indexer requirements, leading to a lower total cost of ownership for your environment. Atlas Reference Designs are built on Splunk validated designs, paired with top hardware and cloud environments, and powered by Kinney Group’s unique Splunk tuning and automation solutions.

Atlas Reference Designs are proven to reduce server counts, cut storage costs, and eliminate hidden OpEx expenses, all while enabling 10x improvements in your Splunk performance.

2. Atlas Applications and Utilities

Atlas’s Applications and Utilities include multiple tools built to monitor and easily manage your data in Splunk. With an interface that’s clean and easy to navigate, working in Splunk has never been easier.

“The current components in the Atlas platform were selected because they address horizontal pain points across many different Splunk instances and deployments that we’ve seen with Kinney Group customers,” Georges Brantley, Scrum Master at Kinney Group, emphasizes.“We want to help Splunk admins and users further curate their Splunk environment to produce what they need in order to accomplish their mission.”

3. Expertise on Demand

If those tools aren’t enough, how about we arm you with a team of expert Splunkers to help with everything else?

Expertise on Demand (EOD) is your team of dedicated Splunk professionals to help you with on-demand support and Splunk best practice training. Our EOD team can quickly assist with any Splunk question or fix in increments as small as 15 minutes.

Atlas gives you essential visibility into your environment, and Expertise on Demand makes sure you get the support you need.

Conclusion

Splunk is a journey, not a destination. And every component we’ve built into Atlas is specifically and thoughtfully designed to reduce the complexities of Splunk while removing the roadblocks on your journey to success.

There’s still plenty to come from the Atlas platform, and we can’t wait to share more with you. If you’re interested in learning more about the platform, fill out the form below.

No one likes mismatched data. Especially data that’s hard to filter and pair up with patterned data. A Regular Expression (regex) in Splunk is a way to search through text to find pattern matches in your data. Regex is a great filtering tool that allows you to conduct advanced pattern matching. In Splunk, regex also allows you to conduct field extractions on the fly.

Let’s get started on some of the basics of regex!

How to Use Regex

The erex command

When using regular expression in Splunk, use the erex command to extract data from a field when you do not know the regular expression to use.

Syntax for the command:

| erex <thefieldname> examples=“exampletext1,exampletext2”

Let’s take a look at an example.

In this screenshot, we are in my index of CVEs. I want to have Splunk learn a new regex for extracting all of the CVE names that populate in this index, like the example CVE number that I have highlighted here:

Next, by using the erex command, you can see in the job inspector that Splunk has ‘successfully learned regex’ for extracting the CVE numbers. I have sorted them into a table, to show that other CVE_Number fields were extracted:

The rex Commands

When using regular expression in Splunk, use the rex command to either extract fields using regular expression-named groups or replace or substitute characters in a field using those expressions.

Syntax for the command:

| rex field=field_to_rex_from “FrontAnchor(?<new_field_name>{characters}+)BackAnchor”

Let’s take a look at an example.

This SPL allows you to extract from the field of useragent and create a new field called WebVersion:

As you can see, a new field of WebVersion is extracted:

The Basics of Regex

The Main Rules

^ = match beginning of the line

$ = match end of the line

Regex Flags

/g = global matches (match all), don’t return after first match

/m = multi-line

/gm = global and multi-line are set

/i = case insensitive

Setting Characters

\w = word character

\W = not a word character

\s = white space

\S = not white space

\d = a digit

\D = not a digit

\. = the period key

Setting Options

* = zero or more

+ = 1 or more

? = optional, zero or 1

| = acts as an “or” expression

\ = escape special characters

( ) = allows for character groupings, wraps the regex sets

Some Examples

\d{4} = match 4 digits in a row of a digit equal to [0-9]

\d{4,5} = match 4 digits in a row or 5 digits in a row whose values are [0-9]

[a-z] = match between a-z

[A-Z] = match between A-Z

[0-9] = match between 0-9

(t|T) = match a lowercase “t” or uppercase “T”

(t|T)he = look for the word “the” or “The”

Regex Examples

If you’re looking for a phone number, try out this regex setup:

\d{10} = match 10 digits in a row

OR

\d {3}-?\d{3}-?\d{4} = match a number that may have been written with dashes 123-456-7890

OR

\d{3}[.-]?\d{3}[.-]?\d{4} = match a phone number that may have dashes or periods as separators

OR

(\d{3})[.-]?(\d{3})[.-]?(\d{4}) = using parentheses allows for character grouping. When you group, you can assign names to the groups and label one. For example, you can label the first group as “area code”.

If you’re looking for a IP address, try out this regex setup:

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} = searches for digits that are 1-3 in length, separated by periods.

Use regex101.com to practice your RegEx:

We’re Your Regex(pert)

Using regex can be a powerful tool for extracting specific strings. It is a skill set that’s quick to pick up and master, and learning it can take your Splunk skills to the next level. There are plenty of self-tutorials, classes, books, and videos available via open sources to help you learn to use regular expressions.

If you’d like more information about how to leverage regular expressions in your Splunk environment, reach out to our team of experts by filling out the form below. We’re here to help!