Installing Splunk

Getting started with Splunk is easy and straightforward (mostly) — especially if you’ve already made your architecture decisions. For the purpose of this tutorial, we’ll assume you’ve already checked Splunk’s documentation on system requirements. It’ll also be helpful to keep the full Splunk installation manual handy.

Note: If you’re using AWS for your Splunk deployment, Splunk offers a Splunk Enterprise Amazon Machine Image (AMI) that installs to AWS with one click. There are also containerized options for Splunk for Docker and Kubernetes.

Let’s dive into installing Splunk Enterprise

Installing Splunk Enterprise on Linux

You can download Splunk Enterprise for Linux from the Splunk website (you’ll need a free account).

Once you select your operating system from the tabs, and choose the package option you prefer (.deb, .tgz, or .rpm), you can simply click to download the file. Once you click, however, you’ll also be directed to a page with instructions for downloading directly from the command line using wget (filename below will be different depending on the version available at the time you click):

wget -O splunk-9.0.0-6818ac46f2ec-linux-2.6-x86_64.rpm

Why doesn’t Splunk put this on the page where you choose your download? A great question. Nobody knows. Maybe Buttercup? We’ll have to ask them next year at .conf23.

Once the .rpm has downloaded successfully, you can install it with this command:

rpm -i splunk-9.0.0-6818ac46f2ec-linux-2.6-x86_64.rpm

(Again, your file name may be different depending on the available version at the time of download.)

User Settings

First, we’ll want to make sure we can run Splunk as the splunk user — the install should have created that user and group, but you can verify with this command:

cut -d: -f1 /etc/passwd

This will display a list of local users. If you don’t see splunk in the list, create this user and group with the following:

adduser splunk
groupadd splunk

ulimits Settings

There are limits on the Linux platform known as ulimits that impact maximum file size, number of open files, user processes, and data segment sizes. On the command line, type:

ulimit -a

This will present a list of limits that you can verify against your settings. Need to adjust your settings to meet/exceed? Edit the /etc/system/system.conf file and adjust the following settings:


I like big pages and I cannot lie…

Some Linux distros enable the transparent huge pages feature by default. Splunk recommends this feature be disabled due to performance hits in Splunk Enterprise (30%+). A quick Google search will help you find the process for doing this for your Linux distribution and version.

Starting Splunk on Linux

Once you’ve installed and tweaked your settings, you’re ready to fire Splunk up for the first time! First, make sure you’re operating as the Splunk user:

su - splunk

Then, from the /opt/splunk/bin directory, type the following:

.splunk start

Want to skip the license agreement? You can also start Splunk by typing ./splunk start –accept-license to get to the good stuff without all the bothersome “reading” the kids are into these days.

Start on Reboot

Out of the box, Splunk doesn’t start when the server is rebooted. You can, however, have Splunk create a script that will enable this functionality by executing an “enable boot-start” command:

[root@ip-172-31-28-164 ~]# cd /opt/splunk/bin
[root@ip-172-31-28-164 bin]# ./splunk enable boot-start -user splunk
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

You’ll want to edit the /etc/init.d/splunk file and add USER=splunk after the RETVAL entry:

# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
# chkconfig: 2345 90 60
# description: Splunk indexer service

. /etc/init.d/functions

It’s important to specify -user splunk when you execute the enable boot-start command and implement this change to init.d or you’ll end up with file ownership headaches.

Stopping Splunk on Linux

Best practices dictate that you should stop Splunk from the command line before rebooting the server:

/opt/splunk/bin/splunk stop

Ready to Learn More?

Installing Splunk, of course, is just the beginning! Ready to learn more about getting the most from Splunk? Check out other entries in our Splunk 101 content. Want to take Splunk to the next level in your organization but need some help? We’d love to chat!

Reducing Costs with Splunk

As of the writing of this post, we are arguably in turbulent times. Publicly traded companies have recently entered a bear market, crypto currencies are down 70% (or more) from recent highs, and inflation is measured at a 40-year high. Leaders of companies big and small are rightfully concerned that the US and global economies are entering into a recession.

In preparation for a potential economic downturn, most organizations are looking internally to determine where costs can be reduced, what platforms are enablers for weathering an economic storm, and what should be cut.

Since 2013, our team has helped hundreds of commercial and public sector organizations with their implementation of Splunk, both on-prem and in the cloud. From many customers, we hear a recurring refrain of “Splunk is expensive.”

My first reaction to this comment is always “Splunk is expensive? Relative to what?”

Before Splunk, getting real-time analytics from disparate critical systems to address security, operations, and observability was really, really tough. Regardless of good times or bad, all organizations must be vigilant on security and optimal application performance — this is the new reality of a software-driven world. The ability to harness insights from “digital exhaust” produced by logs and machine data is invaluable in today’s modern, software-driven world. Splunk remains the best platform of its kind for gaining real-time intelligence from machine data organizations that have chosen Splunk have chosen wisely.

I understand the “Splunk is expensive” observation. If organizations are not getting enough tangible returns on their Splunk investments, then Splunk is expensive, regardless of how good the Splunk technology is. For that matter, any enterprise software or SaaS offering that does not provide measurable mission, financial, or human returns on investment should justifiably be viewed as “expensive.”

Optimize Splunk, and Turn It Into a Cost Reducer

We at Kinney Group view “reducing costs with Splunk” through two lenses:

  1. How can we reduce the costs associated with deploying, operating, and sustaining investments in Splunk technologies?
  2. How can we harness the power of Splunk to be a cost-reduction engine?

In 2021, our organization released Atlas — the Creator Empowerment Platform for Splunk. Purpose-built from the ground up to help customers in their Splunk journeys, Atlas accomplishes the two views of cost reduction referenced above.

Addressing lens #1 referenced above, we suggest pursuing a “1-2 punch” using the Atlas platform.

First, diagnose the health of a Splunk environment via the Atlas Assessment application, available free on Splunkbase. Using Atlas Assessment, customers can get visibility into areas of cost reduction and optimization for Splunk technologies, whether on-prem or in the cloud. Remarkably, Atlas Assessment returns actionable insights in less than 30 minutes.

The second punch is using the Atlas platform to address the identified areas of improvement that have been illuminated by the Atlas Assessment. Not sure if Atlas can help? We offer a full, 30-day trial of the Atlas platform absolutely free. Our experience is that Atlas Assessment, combined with the Atlas platform, provides tangible optimization and cost-reduction results for any Splunk implementation. And you can get started without spending a single dollar.

More specifically, customers find that Atlas reduces Splunk operating costs in the following manners:

  • License optimization: Whether the license is based on data ingest or workload, Atlas specifically identifies how any Splunk Enterprise or Splunk Cloud license can be optimized for maximum ROI.
  • Operational optimization: Atlas streamlines the daily operation and sustainment of Splunk implementations. These capabilities provide direct labor savings, while at the same time freeing valued personnel to spend more time creating analytics value from Splunk.
  • UX and adoption optimization: Splunk admins and users are the “creators” that drive organizational value from Splunk. Atlas helps drive adoption by making the use of Splunk much easier. More people using Splunk means more value for your organization.

Splunk as a Powerful Cost Reduction Engine

All systems and applications produce log data. And Splunk is the best platform on the planet for turning log data into insights for security and observability. Since we began using Splunk in 2013, we consistently find that Splunk can help organizations reduce the sprawl of siloed, single-use tools and monitors.

As organizations look to reduce costs, we encourage them to take a hard look at their entire landscape of software tools. If Splunk can deliver the outcome, why does an organization need another tool to deliver the same results?

When we optimize a Splunk environment using Atlas, we magically create additional Splunk capacity with existing license investments. This newfound added capacity can then be leveraged to help any organization reduce their footprint (and costs) associated with the sprawl of single-use tooling.

Reducing Costs Now for Weathering a Potential Storm

With Atlas and Atlas Assessment, we can deliver tangible cost savings immediately, and do so through the two lenses referenced above. Now is the time to prepare for the potential of an economic storm brought on by a recession. Atlas can help get you prepared.

Is Splunk expensive? Yes — it sure can be if it isn’t optimized and delivering tangible returns for the organization.

Is Splunk expensive when fully optimized with Atlas? NO! When running correctly, Splunk is the most powerful platform of its kind in the industry. Splunk customers have chosen wisely. We argue that once customers get Splunk optimized, it can be one of the most powerful cost-reduction weapons any organization can have.

Ready to take your next step?

Download the FREE Atlas Assessment application from Splunkbase for actionable (and no-cost) discoveries in your environment, or get started with a free 30-day trial of the Atlas Platform. Have questions? We’d love to answer them! Click here to schedule an introductory discover call.

Splunk Assist in Splunk 9

The wait is over and Splunk 9 is officially here! This release introduces a number of features and improvements aimed at making life easier for Splunk admins and users alike. Wondering which announcements and improvements really matter for you? Join us as we investigate and explore some of our favorite discoveries!

Splunk Assist in Splunk 9

Splunk 9 brings with it Splunk Assist, an exciting improvement to the Monitoring Console for On-Premises Deployments, helping Splunk Admins configure and secure their Splunk Environment faster than before!

Assist is a brand-new tab and suite of functionality for the Monitoring Console. It can be easily reached from the Monitoring Console’s navigation in Splunk 9.

So what does Splunk Assist bring to the table?

First, the primary Splunk Assist pane provides insights, grouped into three distinct areas:

  • Indicator Tabs: These are categories of indicators for which you can see additional information. Clicking on one of these tabs loads information about that indicator, including a graph with the number of instances in your deployment, and what their compliance status is (conforms, warning, or critical).
  • Overview Pane: Provides detailed information about the nodes in your Splunk environment. Status icons here also reflect their compliance status, and are grouped into search, indexing, and collection tiers.
  • Indicator Summary: This pane provides a list of all the indicators along with a summary of the information it collects, and why. You’ll see information relative to the indicator’s category, scope (where does the indicator apply), and results (compliance state).

Splunk Assist also tracks and visualizes SSL certificate best practices. This ensures your Splunk ecosystem is meeting security standards by clamping down on possible attack vectors, while also ensuring that expired SSL certs don’t creep up on a busy IT team.

What’s the catch?

This is incredible functionality, but it does come with a catch — Splunk Assist requires Splunk admins to leave Support Data Usage turned on in their environment. This will enable Splunk HQ to start investigating and tracking your Splunk environment data to find potential issues. While this won’t send your private event data upstream to Splunk, it does require outbound connections that many on-premises installations may lack.

Note: As with Splunk 8, Data Collection is switched ON by default when you install. You’ll need to manually opt-out if you don’t want to share your usage data with Splunk. If you opt-out, the Splunk Assist service won’t work.

Get Started with Monitoring Console

First, you’ll need to prepare your environment: You’ll need to enable support usage data, as previously mentioned, to get the insights that SA provides. ( You’ll also need to make sure you’ve configured Monitoring Console, if you haven’t already. If you’re utilizing a firewall, make sure you allow * on Port 443.

Next, you’ll activate Splunk Assist. From the system toolbar, choose Settings > Monitoring Console, and then choose Assist. You’ll need an activation code, which is tied to your Splunk license. If you’re not the license owner, you’ll need to reach out to get this code. If you’re the license owner and you’re thinking, “Uh… I didn’t get a code,” no worries — follow those same instructions, but choose the “Get an activation code” link.

The setup is very straightforward. Once you’ve activated Splunk Assist, it’s time to start using it! We’ll take a deeper dive into how to get the most value from Splunk Assist soon.

What’s next?

This is just one of the many incredible new features available in Splunk 9! Need to get up and running with Splunk 9 quickly? Our expertise (nearly 700+ Splunk engagements over the years) coupled with Atlas — The Creator Empowerment Platform for Splunk — means we can make your transition to Splunk 9 quick and effective!

We’d love to hear from you! Schedule a quick call to discuss your needs, or check out the Atlas overview video to learn more about empowering your team of Splunk Creators.


Ingest Actions in Splunk 9

The wait is over and Splunk 9 is officially here! This release introduces a number of features and improvements aimed at making life easier for Splunk admins and users alike. Wondering which announcements and improvements really matter for you? Join us as we investigate and explore some of our favorite discoveries!

Introducing the Ingest Actions UI in Splunk

It’s important in any Splunk deployment to ensure you’re ingesting the data you need, where you need it, when you need it, in the structure you need it in, and all without blowing out your license.

Splunk has long offered various methods by which to hone data, but those methods are far from user friendly. Data admins would be required to know the proper syntax and locations to manually edit configuration files, and — even then — would have no simple method to test their work and ensure its functioning as intended. This could result in the arduous process of having to manually delete buckets of sensitive data as admins literately test new configurations and syntax.

Splunk 9 has finally addressed this sore spot, with the new Ingest Actions user interface, found under Data Settings!

With Ingest Actions, it’s now easy to preview and define data masking, filtering, and even routing!


A ruleset consists of an event stream to which any number of rules are applied. The event stream can be either index data, a sample file, or even text from your clipboard.

Once the data has been sampled, rules may be previewed on it before being saved.

When using indexed data, you must select a sourcetype. Only sourcetypes defined in your props.conf file will be listed, but any undefined sourcetype may be added manually.

Additionally, the filter accepts wildcards:

Note, however, that using wildcards will result in invalid syntax for the props.conf stanza title:

Luckily, there is a workaround to make this stanza valid — simply prepend the title with the regular expression:


For example:


Data Masking

Often it may be important to mask certain parts of ingested data such as Social Security Numbers and passwords, for security or compliance reasons. This kind of information should never enter the Splunk environment to begin with, and therefore it should be erased or replaced at index time, rather than at search time.

With the new Ingest Actions interface, creating these rules is straightforward and intuitive. You even have the option to upload sample data for the rule creation, to avoid ever ingesting any sensitive data.

In the screenshot below I’ve used the “Paste from Clipboard” feature to create a single sample event in the same format as my sensitive data:

Next, I click Add Rule, and select Mask with Regular Expression.

Tip: If you would like to know more about writing Regular Expressions for Splunk, see our Beginner’s Guide to Regular Expressions post.

I know my passwords cannot contain spaces, so I use \S+ to match one or more non-whitespace characters. I put the preceeding string within parentheses to create a capture group, which can be referenced in the replacement expression using \1.

Note that the percentage and volume in KB of data masked is displayed in the rule header, as well as that of the total at the bottom.

A more detailed report is available from clicking the ruler icon at the top right.

Data Filtering

You may have a dataset which contains vital logs, but also a significant amount of superfluous spam. The last thing you want is to eat up your license on data which will only serve to slow down searches, complicate SPL day-to-day and increase resource utilization.

Ingest Actions allow for filtering using either regular expressions or eval expressions.

A regular expression filter can be applied to _raw, index, sourcetype, source, or host. Any event which matches will not be indexed.

An eval expression in this context is the same as a conditional statement you’d use in the first argument of an if function in an eval command. It can use any of the same eval functions such as true() or len(), and should evaluate to true or false. Events for which the expression evaluaties to true will not be indexed.

Data Routing

There may be times you wish to filter out certain data, but not drop it entirely. With the data routing ingest action, you can identify data to be routed elsewhere—either to a Splunk index, or to an external S3 compliant destination.

Data may be filtered by a regular expression, an eval expression, or not at all.

If you would like to route the data, but not prevent it from also being normally indexed, toggle the clone event option.

External S3 destinations can be defined once and used any number of times.

What’s next?

This is just one of the many incredible new features available in Splunk 9! Need to get up and running with Splunk 9 quickly? Our expertise (nearly 700+ Splunk engagements over the years) coupled with Atlas — The Creator Empowerment Platform for Splunk — means we can make your transition to Splunk 9 quick and effective!

We’d love to hear from you! Schedule a quick call to discuss your needs, or check out the Atlas overview video to learn more about empowering your team of Splunk Creators.

Solving Splunk Bundle Size Issues

Cluster Bundles are packages of knowledge objects that must be shared between indexers and search heads in clustered environments. Unfortunately, these can get too big and cause performance issues for your Splunk environment. We’ve discovered a trick that can drastically reduce bundle size, while maintaining operations, and improving performance! Strangely enough, this method is barely mentioned in Splunk Docs. It’s a hidden feature we need to share!

Bundles of Awesome!

The modern Splunk deployment has clustered Indexers and Search Heads that help share the load of reading, searching, and computing data for users and alerts, every second of the day. These separate instances communicate with each other to properly execute tasks and keep things running as smoothly as possible — but what happens when a user makes a change on one instance? It needs to waterfall down to the many other pieces in the Splunk architecture, and it does that using Splunk Bundle Replication.

This usually works great! Users edit items, the Search Heads and Indexers share information, and everything stays relatively up to date and actionable for users. However, when its functionality is pushed to its limits, both Splunk Admins and Users will experience a headache like none other.

A Bundle of Pain!

In mature Splunk ecosystems, this bundle system can start tripping over itself and quickly cause issues downstream. Having Knowledge Objects which are too big (or having too many) can cause replication errors, leading to search slowdowns for users, Search Heads spending precious CPU managing large files instead of search execution, and updates failing to be shared between Splunk instances. All these errors are the fast lane to Splunk instability (and a royal pain).

If only there was “one weird trick” to alleviate your bundle sized pain and prevent these issues!

One Weird Trick!

Surprise, surprise — there is! One cause of large bundle sizes is big lookup files your Splunk system creates and relies on for quick referencing. Unlike dashboards or other Knowledge Objects, however, lookups can get big and unwieldy, leading to your bundle size growing and growing.

Fortunately, this “hidden trick” we’re talking about can reduce the size of your lookups, and greatly reduce your bundle size. This trick? Compression!

Compress your Problems!

Splunk supports lookup compression, enabling Admins to convert their lookups to a much more reasonable size. If done right, there will be no usability difference! Follow the steps below to compress your largest lookups and fix your bundle size!

  1. Identify a large lookup file you would like to compress to reduce your bundle size
  2. Navigate to that file in the Command Line Interface of the system
  3. Gzip the lookup file (gzip largelookupfile.csv)
  4. Searches of these compressed lookups will now need to include a .gz, unless you create a lookup definition that uses the original lookup name, mapped to the new file.gz name!

That’s it! With this workflow, you can reduce the size of lookups by around 50%, and potentially reduce your bundle size by around 30% or more! All the while, your users’ searches and dashboards will operate the exact same, except for being error free. Compressed lookups can still be edited using outputlookup, and can of course be referenced using the lookup command.

How to Create a Splunk Pivot Dashboard

Creating Pivots in Splunk

Pivots are the perfect way to build personal a dashboard in Splunk without creating search queries manually. Whether you’re a beginner or an expert, learning how to build a Pivot dashboard can save you a ton of time (and headaches) when pulling data from your Splunk environment.  

Here’s a crash course on everything you’ll need to know about Pivots in Splunk. 

What is Pivot in Splunk? 

A Pivot is a dashboard panel in Splunk. The goal of Pivots is to make searching easier in Splunk by using existing data sets instead of SPL queries to populate the Pivot. 

Do I need to know SPL to build a Pivot or dashboard in Splunk? 

No, you don’t have to know SPL to build a Pivot dashboard in Splunk. By using data models and data sets, you can build a robust Pivot dashboard without using SPL or running queries manually. 

Who can build Pivots in Splunk? 

Anyone who uses Splunk to understand that data in their organization can build a Pivot dashboard in Splunk. Because it doesn’t require any SPL knowledge, anyone from a summer intern to the VP of Technology can build their own Pivot dashboards in Splunk. 

How to Create a Pivot in Splunk 

The drag and drop UI of Pivots makes it easy to build a Pivot dashboard in Splunk.  

Step 1: Create a New Data Model or Use an Existing Data Model 

To begin building a Pivot dashboard, you’ll need to start with an existing data model. If you don’t have an existing data model, you’ll want to create one before moving through the rest of this tutorial. 

Go to data models by navigating to Settings > Data Models. 

For this example, we’re using the standard data model Internal Audit Logs, but you can choose any data model in your environment. 

Splunk Tip: When your Splunk environment was created, it automatically came with the Internal Audit Logs data model. This data model includes all of your internal audit log data, so you can be sure that the Pivot table you’re creating will reflect real and accurate data. 

Step 2: Select Pivot 

In the top right corner of the screen select Pivot. 

Once you open your data model and select Pivot, you’ll see at least one (but likely more) data sets in the model. 

Step 3: Review the Data for Your Pivot 

It’s important to click on each data set and review the fields within it in order to find the data you want to include in your Pivot table. Once you find your desired fields, click on the name of the data set again to open your new Pivot.  

Step 4: Build Your Pivot 

Building your Pivot is both an art and a science. Here are the overarching elements you can manipulate to build your Pivot table. Play around here and see what data populates. Keep what you like, and remove what you don’t. 

  • Filter Your Data: You should filter your data so that it pulls information from the right time period. To do this click Filters and choose from Real Time, Relative, or All Time. In this example, we’re choosing a Relative time filter of Last 7 Days. 
  • Check the Count of Audit: After filtering your data, you’ll see how many audited events happened in the time frame you selected. In our case, we see 1,247 audits in the last 7 days. If you’re seeing zero audits, double-check that the data set you’re using actually has data, or try refiltering your data using a larger time frame. 
  • Add Fields to Your Pivot: Select Split Rows to reveal a dropdown of all the fields available in the rows of your Pivot. To start, we’re choosing the Action field. This will show us all the actions that happened in our audit and how many of each action occurred. You can continue to add rows to your Pivot for more details about the data. 
  • Add Fields to Your Pivot: Select Split Columns to reveal a dropdown of all the fields available in the columns of your Pivot. In this example, we’re choosing the Host field. This will show us all the hosts for each action in our audit. You can continue to add columns to your Pivot for more details about the data. 

Splunk Tip: The smaller and less complex your data set, the fewer fields you’ll have to choose from when splitting rows and columns. Don’t get discouraged if you have only a couple of fields to include in your Pivot. If there are additional fields you’d like to pull into your Pivot in the future, you can work with your Splunk team or ask the experts at Kinney Group to help you set them up. 

  • Add Visualizations: Although the default for building a Pivot in Splunk is to use a table, you can change the visualizations to display your data in different ways. On the right-hand side of your screen, you’ll see a vertical list of icons, each of which will display your data in a different graph or chart. In this example, we’re using the line graph visualization represented by the line graph icon. With any visualization, you can adjust the X-axis, y-axis, color, and other properties of your graph or chart. 

Splunk Tip: Visualizations represent what your data will look like in the finished Pivot dashboard. If you don’t choose a visualization, you’ll simply see the table and raw data in your Pivot. This makes it hard to see your data at a glance which is the point of building the Pivot dashboard. We highly recommend you choose a visualization for your data so that it reflects the information you want to see in your finished Pivot in an accurate and appealing way.  

Step 5: Save Your Pivot 

In the top right of the screen, select Save As > Dashboard Panel. 

Give your new Pivot a title and description. Then choose whether your Pivot will be private or public.  

Choose Save > View Dashboard to see your new Pivot. 

Step 6: Title your line chart.

Splunk Tip: We already named the Pivot dashboard, but you’ll still want to title your line chart so that you know what data is represented in it. As you add more visualizations of different data sets, you’ll find that naming each one makes your Pivot dashboard easier to use. 

Can Pivots be saved as reports panels in Splunk? 

Unfortunately, Pivots cannot be saved as reports panels in Splunk. If your team wants access to your Pivot dashboard for their own reporting needs, you can make the Pivot public and share it with them so they have access to it on demand. 

Key Takeaways for Creating Pivots in Splunk 

Pivots are an amazing tool for Splunk users who aren’t well-versed in SPL or building search queries. You don’t have to make decisions in the dark because you don’t understand Splunk as well as your engineers do. With Pivot dashboards, you’ll have the most important data at your fingertips when you need it, all without creating a single SPL search query. 

If you found this helpful… 

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate. 

Cue Expertise on Demand, a service that can help with those Splunk issues and improvements to scale. EOD is designed to answer your team’s daily questions and breakthrough stubborn roadblocks. Book a free consultation today, our team of experts is ready to help. 

Defining Data Sprawl in Splunk: Why it Matters, and What it’s Costing You

“Data Sprawl” isn’t really a technical term you’ll find in the Splexicon (Splunk’s glossary). Here at Kinney Group, however, we’ve been around Splunk long enough to identify and define this concept as a real problem in many Splunk environments.

What exactly is Data Sprawl? It’s not one, single thing you can point to, rather a combination of symptoms that generally contribute to poorly-performing and difficult-to-manage Splunk implementations. Let’s take a look at each of the three symptoms we use to define Data Sprawl, and break down the impacts to your organization:

  1. Ingesting unused or unneeded data in Splunk
  2. No understanding of why certain data is being collected by Splunk
  3. No visibility into how data is being utilized by Splunk

Ingesting unused or unneeded data in Splunk

When you ingest data you don’t need into Splunk, the obvious impact is on your license usage (if your Splunk license is ingest-based). This may not be terribly concerning if you aren’t pushing your ingest limits, but there are other impacts lurking behind the scenes.

For starters, your Splunk admins could be wasting time managing this data. They may or may not know why the data is being brought into Splunk, but it’s their responsibility to ensure this happens reliably. This is valuable time your Splunk admins could be using to achieve high-value outcomes for your organization rather than fighting fires with data you may not be using.

Additionally, you may be paying for data ingest you don’t need. If you’re still on Splunk’s ingest-based pricing model, and you’re ingesting data you don’t use, there’s a good chance you could lower Splunk license costs by reducing your ingest cap. In many cases, we find that customers have license sizes higher than they need to plan for future growth.

We commonly run into scenarios where data was being brought in for a specific purpose at one point in the past, but is no longer needed. The problem is that no one knows why it’s there, and they’re unsure of the consequences of not bringing this data into Splunk. Having knowledge and understanding of these facts provides control of the Splunk environment, and empowers educated decisions.

No understanding of why certain data is being collected by Splunk

Another common symptom of Data Sprawl is a lack of understanding around why certain data is being collected by Splunk in your environment. Having the ability to store and manage custom metadata about your index and sourcetype pairs — in a sane and logical way — is not a feature that Splunk gives you natively. Without this knowledge, your Splunk administrators may struggle to prioritize how they triage data issues when they arise. Additionally, they may not understand the impact to the organization if the data is no longer is coming in to Splunk.

The key is to empower your Splunk admins and users with the information they need to appropriately make decisions about their Splunk environment. This is much more difficult when we don’t understand why the data is there, who is using it, how frequently it is being used, and how it is being used. (We’ll cover that in more detail later.)

This becomes an even bigger issue with Splunk environments that have scaled fast. As time passes, it becomes easier to lose the context, purpose, and value the data is bringing to your Splunk mission.

Let’s consider a common example we encounter at Kinney Group.

Many organizations must adhere to compliance requirements related to data retention. These requirements may dictate the collection of specific logs and retaining them for a period of time. This means that many organizations have audit data coming in to Splunk regularly, but that data rarely gets used in searches or dashboards. It’s simply there to meet a compliance requirement.

Understanding the “why” is key for Splunk admins because that data is critical, but the importance of the data to end users is likely minimal.

(If this sounds like your situation, it might be time to consider putting that compliance data to work for you. See how we’re helping customers do this with their compliance data today with Atlas.)

The Atlas Data Management application allows you to add “Data Definitions,” providing clear understanding of what data is doing in your environment.

No visibility into how data is being utilized by Splunk

You’ve spent a lot of time and energy getting your data into Splunk but now you don’t really know a lot about how it’s being used. This is another common symptom of Data Sprawl. Making important decisions about how you spend your time managing Splunk is often based on who screams the loudest when a report doesn’t work. But do your Splunk admins really have the information they need to put their focus in the right place? When they know how often a sourcetype appears in a dashboard or a scheduled search, they have a much clearer picture about how data is being consumed.

Actively monitoring how data is utilized within Splunk is extremely important because you can understand how to effectively support your existing users and bring light to what Splunk calls “dark data” in your environment. Dark data is all of the unused, unknown, and untapped data generated by an organization that could be a tremendous asset if they knew it existed.

Kinney Group’s Atlas platform includes Data Utilization — an application designed to show you exactly what data you’re bringing in, how much of your license that data is using, and if it’s being utilized by your users and admins.


Most organizations may not realize that Data Sprawl is impacting their Splunk environment because it doesn’t usually appear until something bad has happened. While not all symptoms of Data Sprawl are necessarily urgent, they can be indicators that a Splunk environment is growing out of control. If these symptoms go unchecked over a period of time they could lead to bigger, more costly problems down the line.

Knowledge is power when it comes to managing your Splunk environment effectively. Kinney Group has years of experience helping customers keep Data Sprawl in check. In fact, we developed the Atlas platform for just this purpose. Atlas applications are purpose-built to keep Data Sprawl at bay (and a host of other admin headaches) by empowering Splunk admins with the tools they need.

Click here to learn more about the Atlas platform, to get a video preview, schedule a demo, or for a free 30-day trial of the platform.

Bridging the Splunk Usability Gap to Achieve Greater Adoption and Expansion

Splunk, the amazing “Data to everything” platform, provides some of the best tools and abilities available to really control, analyze, and take advantage of big data. But you don’t build such a powerful and expansive platform over a decade without it being a bit technical, and even difficult, to fully utilize.

This technical hurdle — that we lovingly call the “Usability Gap” — can stop Splunk adoption in its tracks or stall an existing deployment to its ruin. By clearing the Usability Gap, however, a Splunk environment can prosper and deliver a fantastic return on your investment.

So it begs a question — “what is the Usability Gap, and how do I get across?”

How to Recognize the Gap

What exactly makes up the steep cliff sides of the “Usability Gap?” Well, these symptoms can manifest themselves in any Splunk deployment or client ecosystem, and is caused just as much by human elements as technical blockers. 

The key to any good Splunk deployment is a properly focused admin. Many admins or admin teams were handed Splunk as an additional responsibility instead of a planned and scoped aspect of their job. This disconnect can lead to under-certified admins who lack the time and experience needed to quickly solve issues and incoming requests from Splunk users.

Splunk users can also be underequipped and undertrained. While formal training is available for users with Splunk Fundamentals certification and other online training, they may not meet the user where they are, and those solutions lack the benefits of in-person training with real, actionable data. These issues can be big blockers for learning Splunk and increase the time it takes for users to become confident with the system.

If you’re still not sure if you have a Usability Gap issue, check the activity found on the system itself. If your Splunk search heads are getting little action from users and admins, you know for a fact that something is coming between your users and their Splunk goals.

What a Gap Means for You

What are the consequences of a Usability Gap? They are wide ranging and impactful.

With a lack of focus and experience, admins are going to be severely hampered in achieving outcomes with Splunk. When technical issues arise with the complex Splunk ecosystem, or a unique data set requires attention, admins will have to carve out time to not only work on the issue at hand but learn Splunk on-the-fly as well. Without the proper support, progress slows and a lack of Splunk best practices is to be expected in these deployments.

Users without a watchful or knowledgeable eye will be left to their own devices. This can lead to poorly created searches and dashboards, bad RBAC implementation (if implemented at all), or worse — no movement at all. Without a guiding hand and training, the technical nature of Splunk will eventually cause users to misconfigure or slow down the platform, or just not adopt such an imposing tool. These issues together can lead to a peculiar event, where Splunk is labeled as an “IT tool for IT people.” This is far from the truth, but if users are not properly trained, and admins don’t have time to be proactive, only the technical savvy or previously experienced will be able to utilize the investment. While some outcomes will be achieved, many organizations will realize their significant investment isn’t aligned with their outcomes and will drop Splunk altogether, putting all the effort and time invested to waste.

Mind the (Usability) Gap

Fortunately, there’s an easy answer for solving these problems and bridging the Usability Gap in your environment — the Atlas™ Platform for Splunk. Atlas is geared towards increasing and speeding up Splunk adoption and enabling Splunk admins to do more with their investment. Let’s look at the elements of Atlas that help bridge the Usability Gap!

The Atlas Application Suite, which is a collection of applications and elements that reside on the search head, helps admins improve their deployment, and zero in on giving users a head start with achieving outcomes in Splunk. One such application is the Atlas Search Library.

Search Library gives users an expandable list of Splunk searches that are properly described and tagged for discoverability and learning. Using the Search Library, a Splunk User can create a library of knowledge and outcomes when it comes to the complex nature of Splunk’s Search Processing Language. This greatly accelerates skill sharing and education around SPL — one of Splunk’s biggest roadblocks.

Another element is the Atlas Request Manager. This application greatly increases the usability of Splunk by quickly linking admins and user with a request system built into the fabric of Splunk itself. Admins no longer need to spend time integrating other solutions, and users receive a robust system for asking for help with creating dashboards, Splunk searches, onboarding data, and more — all within Splunk!

Adding a data request is quick and painless thanks to Atlas Request Manager

Last, but certainly not least in bridging the Usability Gap, is Atlas Expertise on Demand. Expertise on Demand (EOD) is a lifeline to Kinney Group’s bench of trusted, Splunk-certified professionals when you need them most. EOD provides help and guidance for achieving outcomes in Splunk, and can lead the charge in educating your admins and users about all things Splunk. With EOD, your admins and users have all the help they need to maximize their Splunk investment.

Wrapping up

The Usability Gap is too big a problem to ignore. Frustrated users, overtaxed Splunk admins, and a clear lack of outcomes await any Splunk team that ignores the clear symptoms and issues presented by the Usability Gap. Hope is not lost, however! The Atlas platform is purpose-built to help you get over the hurdles of adopting and expanding Splunk. With incredible tooling to simplify searches, SPL gaps, and managing requests, not to mention Expertise on Demand, Atlas provides the support admins need and Splunk users with the attention they deserve for education and meeting their Splunk goals!

This just scratches the surface of what Atlas can do for your Splunk journey, so read more about our incredible platform and discover what you are missing!

Mastering Splunk Drilldowns With Conditions

Splunk Dashboards really start to shine after empowering Splunk Users with drilldowns. With drilldowns, users can click on any number or visual that suits their fancy and by default see the query powering it. Alternatively, specific drilldown actions can be defined, such as setting a token. Of course, there is the next level as well. By using the conditional elements <condition> and <eval> in their Splunk drilldowns, Splunk Admins can define multiple different possible sets of drilldown actions, creating a dynamic experience for users.

Splunk Admins can utilize Splunk’s XML syntax to conditionally populate tokens, link to new pages, and more! Imagine setting different tokens with specific values based on where users click on a table or based on the value they clicked compared to other search results. Join me as we dive into the deep end of Splunk Drilldowns!

The Basics

The first thing you need to know for conditionals and drilldowns is where they go. To use conditionals, Splunk Admins will need to get intimate with the XML view of their dashboards, since these features are not found on the Splunk Dashboard UI. So, create a dashboard, create a table with some good test data, and follow along!

This is the basic outline for where conditional elements live.

Conditional Expressions and the <condition> Element

This element wraps your drilldown actions, allowing Splunk Admins to define conditions using either the matchattribute to use an eval-like Boolean expression, or the field attribute to simply check the field that was clicked. If you have more than one condition, you can stack <condition> elements in the drilldown section. Only the first condition evaluating to true will perform its actions. Having a <condition> element at the end with no condition specified will let you define actions to perform when no condition has been met. Unfortunately, no, you cannot nest <condition> elements, but I like where your head is at!

Conditional attributes, like drilldown actions, can utilize tokens, including Splunk’s native ones. These come in two varieties, drilldown tokens and search job tokens. Mastering these both, and knowing where to go when you need guidance, will do wonders for making drilldowns work for you.

Here is an easy to reference picture that outlines how all these native tokens populate when a user clicks the circled “Login” in the following table:

If you are working with other visualizations, such as bar charts, check Splunk’s documentation for more direction! All these tokens will be extremely useful as we add condition statements to our drilldowns, so keep this knowledge handy!

Finally, Splunk XML has its own rules that may trip up newcomers. You cannot put down greater than (>) or less than (<) signs willy-nilly into the XML. Same goes for quotes (“) or ampersands (&). Check the table below for reference, but this will help you out later when writing out conditional expressions!

Character Splunk XML Approved Replacement
> &gt;
< &lt;
>= &gt;=
<= &lt;=
& &amp;

With that out of the way, let’s get cracking!

Conditional Drilldowns Based on Columns Clicked

Let’s set the scene, you have an amazing table that tracks latest actions on your website.

You want users to be able to click on a User in the User column to see more historical actions captured from that user, BUT if the user clicks on something in the Action column (like “Login”), you want to show all those actions regardless of users. Finally, if they click on an ID, don’t do anything! How can you make this easy to use for your users, so they are none the wiser? 

By using conditions!

This code snippet shows you 3 interesting things! 

  1. When using the match attribute to compare two strings, we can inject the field name using the $click.name2$ token, and we must use &quot; to wrap non-token strings. .
  2. You can use the field condition to easily do the above with built in functionality. We are using it to mimic the first condition but with comparing to the Action column instead.
  3. Finally, we have a condition match=”1=1”, this illuminates that conditions run in placement order, so when a user selects a cell, the first condition is performed, and if it fails, then the second field action tested, and if it fails, then finally this 1=1 would function like the “else” of the if statement. If any of the tests succeed, then no more conditions are executed.

With this code in place, if a User clicks on UserBravo in column User, then token user would be set to “UserBravo”, and nothing else. If a User clicks on Login in the Action column in the first row, then actiontracker would be set to “Login”, and nothing else. Finally, if a user selects 3333, then nothing happens! Amazing!

Condition Drilldowns Based on Cell Content

Wow, everyone was impressed with your snappy Splunk dashboard, they just can’t get enough! But Fred from IT has a request to turn it from awesome to legendary. You see, he can’t easily remember what those IDs in the far-right column mean. 

Crazy right? Everyone knows that if the ID is less than or equal to 4000 then its BAD and if its above 4000 its GOOD. Basic business Fred! But we are kind Splunk Rulers Admins and have gone mad with power want to help our colleague out. So, let’s see what conditions can do for comparing values!

Remember, as we previously discussed, conditions work in order from top to bottom, and if one conditional ‘passes’ then the others are ignored. Using this, we have created conditionals using match that achieves 3 great things!

  1. Ensures, as we did before, that we are clicking on something in the ID column
  2. Uses Boolean ‘AND’ to join the column name checker with the comparison $click.value2$ [>/<=] 4000. This compares the value we click with the number 4000.
  3. Ends with a conditional catch all of 1=1 that unsets the id token. 

With this code in place, when Fred clicks on 3333, it will fail the first condition, but succeed on the second since its both in the ID column and less than or equal to 4000, setting the id token to “BAD”. When Fred clicks on Login, then the id token gets cleared since the final conditional executes. Finally, when Fred clicks on 5555, then the id token will be populated with “GOOD” since he passes the first conditional. 

You can combine this code with the code from the previous example to have all features execute with no issues, giving your users the cool dashboards they deserve!

Evals, as a Treat

To close out, let’s unveil an underutilized condition option, the <eval> element. Now this won’t revolutionize your dashboards on the front end per se, but they can help Admins craft better drilldown logic. Unlike the <condition> element, the <eval> element is a type of drilldown action, meaning it can be used inside of, or instead of, <condition> blocks! Let’s just jump right to an example!

This code snippet shows two eval commands slipped right into your drilldown xml block. Let’s break down each one!

The ifTest block starts off with an if statement and works like any normal Splunk eval command. Just like an eval command, you can add eval functions such as tonumber and isint, and then we do a strict number comparison to see if we clicked on the magic “BOOM” number. This if statement is checking if the value being clicked is a number, and if it’s 5555. If it passes both, thanks to the AND, the token gets assigned “BOOM”. If not, then “UNBOOM”.

The second statement uses if’s partner in crime, case! With case logic, the system evaluates a logical statement, and if it evaluates to true, utilizes the next parameter as the return value. If the logical statement is false, then it moves on to the next statement. In our case here, it is validating the clicked value to see if it’s NOT a number. If it’s NOT a number, it sets the caseTest token to “WORD”. If it is a number, then the next logical statement checks to see if it’s 5555, and if it is, sets it to “BOOM”. Finally, if it’s neither of these things (like 3333), then nothing happens!

A neat thing about eval conditions is that all of them execute in parallel. Instead of the other conditions where they trigger one after another until one succeeds, all evals trigger with each click. This enables Admins to set multiple tokens easily with one click!

Let’s Wrap Up!

I hope with the previous examples and guidance you can see how pairing drilldowns with conditions can empower your Splunk dashboarding and give you more toys to play with! As a closing thought, a lot of examples given here use $click.value2$ and token manipulation. This was chosen for simplicity, but all these methods can utilize any of the tokens populated as described in The Basics above and could use drilldown links to other dashboards or websites as well! So, get creative, and go build something awesome!



Starting Small with Splunk: Reports and Dashboards for Beginners!

So, you’re new to Splunk. You have your data ingested up and running and you’re familiar with Splunk’s Search Processing Language (SPL for short). But now you’re wondering how to go from the massive sandbox that is the Splunk platform, to a tailored experience that gets you custom dashboards and reports.

We’ve got you covered with a complete beginner’s guide to reports and dashboards.  With these staple Splunk tools, you’ll be able to turn data into intelligence and intelligence into action! 

Try Atlas Free for 30 Days

Let’s get started with the basics.

How to Create a Report in Splunk

Scenario: A client wants to find the total number of successful purchases on their online store. They want to see how individual categories and products are selling.

Step 1: Start your search in the search and reporting app.

In this example, we’re opting to rename the counts so that “Total Purchases” is at the top of our results.

index=”splunk_test” sourcetype=”access_combined_wcookie” status=200 action=purchase

|stats count by productId, categoryId product_name

|rename count as “Total Purchases” productId as “Product ID” category Id as “Product Category” product_name as “Product Name”

|sort – “Total Purchases”

How to Create a Report in Splunk

Step 2: Save your report.

Select “Save As” then select “Report”.

Include your title, description, content type, and whether you’d like to include a  time range picker.

Select “Save”.

How to Save Your Report in Splunk

From here, you can “View” your dashboard and run it during your desired time period.

How to View Your Report in Splunk


How To Add a Splunk Report to a Dashboard

A dashboard is a collection of searches that you can view all at once. You can use dashboards to get greater insight into your data. For example, if you want a dashboard of all your sales reports, you can create that by adding each report visualization to a dashboard.

Method 1: Use the “Add to Dashboard” Button on the reports page.

How to Add a Splunk Report to Your Dashboard

Method 2: Use the “Save as New/Existing Dashboard” option on the search page.

No matter which method you choose to create your dashboard, the following steps will be the same.

Step 1: Configure your dashboard.

Add your dashboard title, dashboard ID, description, and permissions. Then choose whether you want to use classic or dashboard studio to build your dashboard. Finally, select your panel title, and visualization type, then save your dashboard.

Try Atlas Free for 30 Days

Step 2: Edit your dashboard.

You can choose from a number of options like editing the UI and source code of your dashboard or adding panels and inputs. Change the theme from light to dark if that’s your preferred style, and your dashboard is ready.

How to Edit Your Dashboard in Splunk

Splunk Reports and Dashboards Best Practices

1. Practice Makes Perfect

Your dashboard is extremely malleable, so take your time crafting it. You can start with simple visualizations for the time being and optimize them later using the “open search” button or “change visualization” button.

2. Save Often

This simple, yet often overlooked step can save you a lot of headaches down the road. If you leave the dashboard mid-edit, your work won’t be saved, so be sure to hit the save button before you go.

If you found this helpful…

You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.

Cue Atlas Assessment 30-day free trial: a customized report to show you where your Splunk environment is excelling and opportunities for improvement. You’ll get your report in just 30 minutes.

Atlas 30-day free trial from Kinney Group