The Ten Command(ment)s of Splunk

There are many do’s and don’t’s when it comes to Splunk. In our time supporting Splunk customers through Expertise on Demand, Team Tech Ops has seen the good, the bad, and the ugly situations customers can fall into with Splunk.

We’re happy to present the Tech Ops Ten (Command)ments of Splunk best practices.

1) Thou shalt NEVER search index=*

This one is pretty self-explanatory. Splunk has A LOT of data.

Figure 1: "Splunk Slaps" meme
Figure 1: “Splunk Slaps” meme

In most cases, hundreds of gigabytes, maybe even terabytes of data. I’m sure you tried running a search that looks in one index across millions of events and found it took a very long time to complete.

Now, imagine that across all of your indexes. Not many Splunkers can see this full picture because there’s always a search that will not complete (unless you have a tiny environment or use something like tstats).

Searching “index=*” goes into what I like to call the worst practices box.

2) Thou shalt remove real-time & All Time as an option for basic users

Right in line with never looking at every index humanly possible, we also want to avoid looking at every event (that does exist or will exist).

Running a search in real-time or across all-time causes a resource strain on the environment and may even cause disruption for your fellow Splunk users.

3) Thou shalt not ingest data into the main index

Main index is a default index for Splunk Enterprise. Without specifying an index for your inputs, all your data will default to the main index.

Typically, it is the best practice if you never send information to the main index. Ever. If you thought it was confusing to find data when it nicely organized in your indexes and source types, try finding anything when it’s completely jumbled up in one place.

4) Thou shalt leave on ALL search formatting settings

This one is a “You v.s. The Guy she tells you not to worry about” situation.


Figure 2 - Splunk without search formatting settings
Figure 2 – Splunk without search formatting settings

The Guy she tells you not to worry about:

Figure 2 - Splunk with ALL search formatting settings
Figure 3 – Splunk with ALL search formatting settings

5) Thou shalt view the monitoring console before requesting a performance dashboard be built

Most of the information is already there, you don’t need to reinvent the wheel or in this case… the monitoring console.

Figure 3 - Splunk monitoring console meme
Figure 4 – Splunk monitoring console meme

6) Thou shalt look for an add-on first before onboarding new data

If you’re onboarding data, there is probably an app or add-on that can help. You’re going to save a lot of time (and aspirin) utilizing one of Splunk’s app or add-on tools. Note: this does not apply if you’re ingesting something completely unique to you or your company.

Figure 5 - Save time with a Splunk app or add-on
Figure 5 – Save time with a Splunk app or add-on

7) Thou shalt follow correct directory precedence

NEVER save a .conf file in /default. It’s that simple. Just don’t do it.

Figure 6 - Don't use .conf file in /default in Splunk
Figure 6 – Don’t use .conf file in /default in Splunk

8) Thou shalt have all instances of Splunk on a supported version

If you wouldn’t allow an unsupported version of Windows Server in your environment, then why would you allow an unsupported version of Splunk in?

Figure 7 - update your version on Splunk
Figure 7 – Update your Splunk version

9) Thou shalt use forwarder management

Think smarter not harder. Forwarder management makes it easier to keep all your forwarders buttoned up and working properly. The alternative is to make changes and updates manually and individually, and depending on how many clients you have…that might take a while. Splunk’s native forwarder management tool is cool, but Kinney Group’s Forwarder Awareness application (through Atlas) is cooler. Check out this incredible tool that will save you a TON of time in Splunk.

Figure 8- Utilize forwarder awareness so you don't look like this guy
Figure 8- Utilize forwarder awareness so you don’t look like this guy

10) Thou shalt not use join/subsearches unless absolutely necessary

I want to start off by saying that sub-searches aren’t bad, they’re just not as efficient as other solutions. There’s more to come on this rule, but trust our advice, for now, avoid this at all causes.


Your data is important, so how you work with it in Splunk makes all the difference in the value you’ll get out of the platform. The Tech Ops team has worked with hundreds of Splunk customers, from our experience, these tips are a great place to start in adopting Splunk best practices. If you’d like to work directly with us, the experts, please fill out the form below!

Meet Atlas’s Data Management

Splunk is the data to everything platform, capturing massive volumes of data every day. Users will know, though, that without visibility, it can be difficult to extract the maximum value from Splunk. Too often, insufficient monitoring can lead to serious issues in a Splunk instance: platform underutilization, license overage, and even missing data. Each of these problems translates into a serious cost in financial resources, not to mention the hours of human intervention spent on troubleshooting a Splunk environment.

Atlas makes data management easy.

Figure 1 – the Data Management icon on the Atlas Core homepage

Atlas, Kinney Group’s revolutionary new platform for Splunk, includes the Data Management application, a tool that displays all data requests and definitions for accessible monitoring and management. Gone are the days of mysterious license overages and redundant data requests—Atlas ensures unparalleled visibility to guarantee efficient use of data resources.

Data Management

The Data Management tab, built for Splunk admins, is a centralized hub for all data requests and definitions. The data requests section shows current and past requests, including status and sourcetype, so users can easily view important information in one place.

Figure 2 – the Data Management tab dashboard

Expandable metadata reveals the “why” of each entry in the form of details and customizable notes. From this section, Atlas users can easily edit the request and create a definition from it. New requests can also be created directly from this section, and any request can be edited or deleted at any point. This feature empowers any Splunk admin to get the information they need from their data, fast.

Figure 3 – the New Data Request pop-up window
Figure 4 – the Data Definitions section within the Data Management tab

The data definitions section displays active and inactive definitions, providing a comprehensive view of details for all entries. This section also includes expandable metadata so users can see descriptions and notes. This degree of transparency is key to taking advantage of your Splunk license by ensuring that your instance is ingesting the right data. Each definition can be easily edited, and users can create a new definition directly within the section. Like a data request, the definitions can also be easily edited or deleted.

Figure 5 – the New Data Definition pop-up window

Data Inventory

The Data Inventory tab is a dashboard of existing data organized by sourcetype and index built for admins. With access to sourcetype details, including the capability to edit definitions, you’ll never lose sight of what your instance is ingesting and why. To help users monitor the volume of their data usage, this dashboard is purpose-built to include a measure of license usage for each entry.

This is a fantastic place to start adding definitions to your high-volume data sets, which will then appear on your Data Management dashboard.

Figure 6 – the Data Inventory tab dashboard

Request Data

Users can request new data and monitor current data requests from the Request Data tab. Administrators can then address those requests directly in Atlas. This dashboard includes the detailed visibility of Data Inventory and Data Management, as well as the ability to edit, add, and delete requests directly within the tab. This is a dashboard that any Splunk user can visit, and it enables them to create and track their own requests. Admins can view all requests and can approve, reject, and turn them into definitions.

Figure 7 – the Request Data tab dashboard



The “data to everything” platform promises incredible results—but you need a high degree of visibility within a Splunk environment to make that happen. Atlas’s Data Management application provides the transparency you need to ensure your data requests are being collected and addressed efficiently, eliminating costly data sprawl. Teams can now collaborate seamlessly with the knowledge that their data requests won’t be hidden or lost, bringing your organization one step closer to getting every insight you can out of your data.

There’s more to come from Atlas! Fill out the form below to stay in touch with Kinney Group.


Contact Us!

Meet Atlas’s Scheduling Assistant

Searches are at the heart of Splunk. They power the insights that turn data into business value—and Atlas has plenty of them collected in the Search Library. Simple dashboards and ad-hoc searches, though, are only the first step: the real magic happens with the Splunk scheduler. However, as Splunkers will know, it’s all too easy to bog down an environment with poorly-planned search schedules, redundancies, and heavy jobs. Soon, this leads to skipped jobs, inaccurate results, and a slow and frustrating user experience.

Atlas has a first-of-its-kind solution.

The Scheduling Assistant application provides a real-time health check on the use of Splunk’s scheduler and scheduled searches. In addition, it includes a built-in mechanism to fix any issues it finds. Atlas’s powerful Scheduling Assistant ensures that your scheduled searches in Splunk are running efficiently by providing the visibility you need to make the most of your data resources.

Scheduler Activity

In Atlas’s Scheduling Assistant, you’ll find the Scheduler Activity resource. The Scheduler Activity tab is your starting point for assessing how efficiently your environment is currently executing scheduled Splunk searches. Then, the Scheduler Health Snapshot section offers a health score based largely on historic findings like skipped ratio and search latency, as well as a glimpse forward at future schedule concurrency.

Figure 1 - Scheduled Activity tab in Splunk
Figure 1 – Scheduled Activity tab in Splunk

Below the Health Snapshot, the Concurrency Investigation section lets users view and sort their scheduled searches with a helpful translation of the scheduled run times. These dashboards display Atlas’s computed concurrency limits for a Splunk environment, which dictate the maximum number of searches that can be run at any given time.

These real-time insights inform how users can schedule searches for the fastest, most efficient results.

Figure 2 - Concurrency Investigation tab in Scheduling Assistant
Figure 2 – Concurrency Investigation tab in Scheduling Assistant
Figure 3 - Scheduling Assistant preview for Splunk
Figure 3 – Scheduling Assistant preview for Splunk

Next up is Historical Performance, which interprets how scheduled searches are running. This dashboard and graph display average CPU and physical memory used. Also included are search metrics like run time and latency, for example.

Figure 4 - Historical performance of scheduled searches in Splunk
Figure 4 – Historical performance of scheduled searches in Splunk

After Historical Performance, the Scheduled Search Inventory section provides details on all manually scheduled searches. It also allows users to quickly drill down to the Scheduling Assistant tool for any given search.

Figure 5 - Search Inventory of all searches in Splunk
Figure 5 – Search Inventory of all searches in Splunk

Scheduling Assistant

The Scheduling Assistant dashboard allows users to select a single scheduled search to investigate and modify.

Figure 6 - Snapshot of Scheduling Assistant dashboard
Figure 6 – Snapshot of Scheduling Assistant dashboard
Figure 7 - Key metrics on search activity in Splunk
Figure 7 – Key metrics on search activity in Splunk

This section provides key metrics for the search’s activity to highlight any issues. Atlas users can experiment by changing the selected search’s scheduling setting. By editing the Cron setting and submitting a preview, users can compare the Concurrent Scheduling and Limit Breech Ratio to see if their tested Cron setting improves overall outcomes.

If the modified schedule is satisfactory, the user can then save changes and update the saved search—all within the Atlas platform.

Cron Helper

Splunk uses Cron expressions to define schedules, and Atlas’s Cron Helper tab provides a quick and easy way to test them. Not only does this tool enable fast, direct translations, it also acts as a learning tool for those new to Cron.

The syntax key below the Cron bar displays the definitions of each character, allowing users to try their hand at creating and interpreting their own Cron expressions.

Figure 8 - Atlas Cron Helper
Figure 8 – Preview of Atlas Cron Helper

Scheduler Information

The Scheduler Information dashboard is a knowledge base for the complex definitions and functions that power Splunk’s scheduled searches. The environment’s limits.conf is present for review, and the current statistics on currency limits are provided for clarity.

These relatively static values are vital to understanding the scheduler and taking full advantage of its potential.

Figure 9 - Preview of Scheduler Information dashboard
Figure 9 – Preview of Scheduler Information dashboard

In Conclusion

Powered by these four revolutionary features, Atlas’s Scheduling Assistant provides unprecedented insight into Splunk searches. The power to survey, schedule, and change searches is in the user’s hands, saving your team time and resources.

There’s more to come from Atlas! Stay informed by filling out the form below for more information from KGI.

Contact Us!

Meet Atlas’s Forwarder Awareness

If there was a secret sauce for Splunk, the key ingredient would be the platform’s universal forwarders. Providing users with the ability to automatically send data for indexing, Splunk forwarders are essential to data delivery in Splunk Enterprise and Splunk Cloud environments.

In most Splunk instances, you have multiple forwarders. These forwarders throw data at your search heads and indexers in order to read and store your data. However, there has historically been an issue with forwarders: they go missing and they fail.

If you’re looking at your data pipeline in Splunk, your forwarders are on the front line. Forwarders play a pivotal role in ingesting your data; however, they can disappear or unexpectedly fail (without you knowing). A missing forwarder may result in an issue as small as temporarily not ingesting data. However, a missing forwarder could also be an indication of a much larger issue, like an entire server going down.


To solve this time-old problem in Splunk, we’ve built an application within Atlas, our new platform for Splunk, that allows you to have eyes on all of your forwarders in one place.

Atlas’s Forwarder Awareness Application

Atlas Forwarder Awareness is an application that provides visibility into all of your forwarders, their statuses, and any misconfigurations or failures within your environment. Built within the Atlas Application Suite, the Forwarder Awareness tool enables teams to have constant visibility into their forwarders’ health and statuses.

Now teams can quickly determine if a forwarder is missing and take action—immediately.

In Splunk’s own Forwarder Management interface, users are alerted when forwarders go missing with limited information or guidance on the issue. When this happens, Splunk teams have to dig through alerts in their Splunk monitoring console to try and identify an issue with their forwarders.

Figure 1 - Forwarder Awareness Interface
Figure 1 – Forwarder Awareness Interface

The Atlas Forwarder Awareness tool sends you a list of the forwarders that are missing and which data sources are impacted. Instead of requiring users to log into their Splunk monitoring console, users can now access this critical information on their forwarders directly through their Search Head Cluster. This application offers real-time visibility and awareness into your forwarders’ health and status.

The Value in Visibility

Without requiring admin access, you (and your team) have full visibility into the status and health of your forwarders in one view. From this view, users can view visual graphs representing forwarder statuses by operating system, forwarder types in use, and forwarders’ SSL status. On the application, users also have visibility into top-performing forwarders (by total WB) and missing forwarders.

Figure 2 - Forwarder Awareness Dashboards
Figure 2 – Forwarder Awareness Dashboards

In the example below, you’ll see a really powerful element of Forwarder Awareness. You’ll notice on the screen below that a forwarder is offline (with no contact in 15 minutes or longer), the last time Splunk saw that forwarder, and what sourcetypes may be affected. That view does not require admin access.

Figure 3 - Example of "Missing Forwarder" feature
Figure 3 – Example of “Missing Forwarder” feature

These insights are invaluable and critical information for you and your team to identify — and these immediate insights are only available to users through Atlas’s Forwarder Awareness application. In any other situation, teams would spend hours of their time and resources to identify this same information.

To put it simply, a missing forwarder means missing data, failed compliance standards, inactive SSL certificates, and many more detrimental losses for Splunk teams. Ultimately, a missing forwarder can be extremely costly to an organization in both data loss and spent resources.


Every Splunk instance is at risk of a failed or missing forwarder. With your forwarders being at the front line of your data pipeline, it’s essential to have eyes on them at all times. With Atlas’s Forwarder Awareness Application, you have the visibility you need.

This is just a glimpse into the power of the Atlas platform. Paired with more applications, reference designs, and support services, Atlas enables all Splunk teams to be successful. If you’d like to learn more about the Atlas Platform, let us know in the form below.

Schedule a Meeting

Tips for Tech Recruiters: Learning the Lingo

Throughout my recruiting career, I’ve been primarily sourcing, screening, and networking with candidates in the technical field.  I’ve worked with .NET developers, web developers, systems administrators, desktop and helpdesk support, network administrators, and more.  All of these individuals have a very unique language deriving from their respective careers that were learned over years.  Through different conversations, interviews, and networking events you really start to pick up on the lingo and what different technical terms mean; which helps me determine which candidates are knowledgeable and which candidates are just giving me surface-level answers.

Given all that has gone on in the world currently, I’ve had to learn a number of different “languages” in different recruiting positions within the last 12-18 months.  I’ve learned and used many tools that helped progress my expertise in the recruiting space. I’d love to share these tips with you.

Teaching an Old Dog New Tricks

The best way to learn different technical languages, specific positions and different industry lingo is to talk to as many people as possible.  I have to be frank here for a second – all of the new technical languages were a little overwhelming when I started here at KGI. Everyone has to ramp into these new environments, so here are a couple of practices that can help:

Interviews are Learning Opportunities

Completing a lot of interviews with candidates who may or may not be qualified is good. But you should also use those conversations to learn more about their specific experience, details of job duties, and language that may differ from other environments.

Learn From Your Technical Talent

A great resource is utilizing our own colleagues and technical consultants as learning tools.  These individuals make themselves readily available to talk to and give more detail about their past experiences, current skill set, and job details they execute for Kinney Group.  This has been a HUGE help for me in terms of learning about Splunk, data analytics, and the different certifications and security clearances that are required.


The Secret to Security Clearances

Speaking of security clearances… this has been another great learning curve moving into a governmental talent acquisition opportunity.  From the outside looking in, you don’t ever think about how high-security clearance levels go.

Lean on Your Experts

A great aspect about Kinney Group is that our very own Facility Security Officer, Casie Nolan. Having a person whose specific job is to know each security clearance, how much it takes that security clearance, and how long the process will be to obtain that security clearance is incredibly helpful.  She talks to candidates and knows what qualifications a potential candidate should have – and when a candidate gives her their current security clearance, she’s able to verify the accuracy, quickly.

This is a tricky area for most recruiters, as recruiting cleared resources can be complicated. If you don’t have a direct work resource to lean on, reach out to your network for guidance.


To Sum It Up: Be a Sponge

When working with different companies and industries, be a sponge.  Soak up every bit of information and talk to as many expert level people in that field as possible.  Lastly, the best way to learn is to get uncomfortable and just jump in the pool with both feet, survive initially, and then excel!

Being in the recruiting world within tech comes with incredible opportunities for growth and learning. I hope you could take some practical tips back to your own organizations.

Meet Atlas’s Search Library

One key pain point for Splunk admins and users is the inability to track, store, and view searches in one place. On top of keeping tabs on a dizzying amount of searches, users must write queries in Splunk Processing Language (SPL), which is complex and difficult to learn. Writing efficient searches in SPL takes abundant time and resources that many teams can’t afford to spare. Coordinating searches between users and admins eats up further time and can produce confusion for any team—and that’s not to mention the major obstacles that slow or failed searches can introduce.  

Optimizing and keeping track of searches is just one of the issues facing IT teams today—thankfully, we’ve got a solution. Atlas, a platform developed by Kinney Group to help users navigate Splunk, includes a comprehensive and customizable Search Library to aid users in creating and using searches.  

Figure 1 – The Search Library icon from the Atlas Core homepage

The Atlas Search Library

Collected Searches

The Search Library contains a collection of helpful, accessible searches pre-built by KGI engineers. Users also have the ability to save their own custom searches, which can be edited or deleted at any time. These are listed by name and use case, making it easy to identify the purpose of each search. All searches in the library include expandable metadata so that users can see additional information, including the SPL query, within the table. This insight into the SPL enables faster, easier education for those looking to write their own queries. Users can also filter searches to quickly and easily find all applicable listings, giving users and admins an unprecedented degree of visibility.  

Figure 2 – Atlas’s Search Library tab 

Using the Searches

Performing one of these searches couldn’t be easier. Clicking “Launch Search” will open a separate tab where you can view details of the search’s results and tweak the SPL query—all without changing the originally saved search. This capability enables those without a knowledge of SPL to learn and use powerful, intricate searches.  

Figure 3 – The launched search, open in a separate tab

Search Activity

The Search Library component also includes a Search Activity tab, which can be used to monitor which searches are run when, how frequently, and by whom. Having this visibility on one page allows users to see redundancies and overall usage of a search. The Search Activity tab includes the same level of detail as the Search Library, meaning users can dive into the specifics of each search. The tab is also filterable so users can identify exactly which searches they’re shown. You can also add any search in the Search Activity tab to the Search Library, making it easier than ever to keep track of what you need in Splunk.  

Figure 4 – The Search Activity tab of the Search Library


Any user is liable to hit a few roadblocks on their Splunk journey. With Atlas’s Search Library application, your team can be sure that searches won’t be one of them.  

The Search Library is only one of Atlas’s innovative features, and we’re looking forward to sharing so much more from the platform with you. If you’re eager to learn more about Atlas in the meantime, fill out the form below.

Schedule a Meeting