Installing Splunk

Getting started with Splunk is easy and straightforward (mostly) — especially if you’ve already made your architecture decisions. For the purpose of this tutorial, we’ll assume you’ve already checked Splunk’s documentation on system requirements. It’ll also be helpful to keep the full Splunk installation manual handy.

Note: If you’re using AWS for your Splunk deployment, Splunk offers a Splunk Enterprise Amazon Machine Image (AMI) that installs to AWS with one click. There are also containerized options for Splunk for Docker and Kubernetes.

Let’s dive into installing Splunk Enterprise

Installing Splunk Enterprise on Linux

You can download Splunk Enterprise for Linux from the Splunk website (you’ll need a free account).

Once you select your operating system from the tabs, and choose the package option you prefer (.deb, .tgz, or .rpm), you can simply click to download the file. Once you click, however, you’ll also be directed to a page with instructions for downloading directly from the command line using wget (filename below will be different depending on the version available at the time you click):

wget -O splunk-9.0.0-6818ac46f2ec-linux-2.6-x86_64.rpm https://download.splunk.com/products/splunk/releases/9.0.0/linux/splunk-9.0.0-6818ac46f2ec-linux-2.6-x86_64.rpm

Why doesn’t Splunk put this on the page where you choose your download? A great question. Nobody knows. Maybe Buttercup? We’ll have to ask them next year at .conf23.

Once the .rpm has downloaded successfully, you can install it with this command:

rpm -i splunk-9.0.0-6818ac46f2ec-linux-2.6-x86_64.rpm

(Again, your file name may be different depending on the available version at the time of download.)

User Settings

First, we’ll want to make sure we can run Splunk as the splunk user — the install should have created that user and group, but you can verify with this command:

cut -d: -f1 /etc/passwd

This will display a list of local users. If you don’t see splunk in the list, create this user and group with the following:

adduser splunk
groupadd splunk

ulimits Settings

There are limits on the Linux platform known as ulimits that impact maximum file size, number of open files, user processes, and data segment sizes. On the command line, type:

ulimit -a

This will present a list of limits that you can verify against your settings. Need to adjust your settings to meet/exceed? Edit the /etc/system/system.conf file and adjust the following settings:

[Manager]
DefaultLimitFSIZE=-1
DefaultLimitNOFILE=64000
DefaultLimitNPROC=16000

I like big pages and I cannot lie…

Some Linux distros enable the transparent huge pages feature by default. Splunk recommends this feature be disabled due to performance hits in Splunk Enterprise (30%+). A quick Google search will help you find the process for doing this for your Linux distribution and version.

Starting Splunk on Linux

Once you’ve installed and tweaked your settings, you’re ready to fire Splunk up for the first time! First, make sure you’re operating as the Splunk user:

su - splunk

Then, from the /opt/splunk/bin directory, type the following:

.splunk start

Want to skip the license agreement? You can also start Splunk by typing ./splunk start –accept-license to get to the good stuff without all the bothersome “reading” the kids are into these days.

Start on Reboot

Out of the box, Splunk doesn’t start when the server is rebooted. You can, however, have Splunk create a script that will enable this functionality by executing an “enable boot-start” command:

[root@ip-172-31-28-164 ~]# cd /opt/splunk/bin
[root@ip-172-31-28-164 bin]# ./splunk enable boot-start -user splunk
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

You’ll want to edit the /etc/init.d/splunk file and add USER=splunk after the RETVAL entry:

#!/bin/sh
#
# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
#
# chkconfig: 2345 90 60
# description: Splunk indexer service
#
RETVAL=0
USER=splunk

. /etc/init.d/functions
…

It’s important to specify -user splunk when you execute the enable boot-start command and implement this change to init.d or you’ll end up with file ownership headaches.

Stopping Splunk on Linux

Best practices dictate that you should stop Splunk from the command line before rebooting the server:

/opt/splunk/bin/splunk stop

Ready to Learn More?

Installing Splunk, of course, is just the beginning! Ready to learn more about getting the most from Splunk? Check out other entries in our Splunk 101 content. Want to take Splunk to the next level in your organization but need some help? We’d love to chat!

Architecting Splunk Primer

If you’re just starting out with Splunk, you most like won’t be expected to architect or implement your Splunk environment from scratch. (That type of project is usually — and highly recommended to be — led by or assisted by Splunk-certified professionals.) That said, maybe you’re trying to spin up a Splunk sandbox, joining an existing team and need to come up the curve, or you’re looking to improve your existing architecture.

Regardless of your situation, there are a few considerations when taking a look at your Splunk environment’s architecture:

Splunk On-prem vs. Splunk Cloud

While on-prem deployments of Splunk have a variety of infrastructure considerations, Splunk Cloud presents some compelling benefits — simply forward your data to Splunk Cloud, and it will “automagically” make sure you have the resources you need to handle the data, and data is managed securely and efficiently.

Splunk Cloud also introduces Workload Pricing rather than ingest-based pricing. Meaning you can ingest all the data you want, and only pay for what you actively use (workload).


What’s the best choice for you?

Do you prefer to have Splunk running locally and have control over your hardware and infrastructure components? Or do you prefer to let a third-party manage the infrastructure and only concern yourself with the results you’re getting from the data? (That’s not a trick question, by the way — there are pros and cons with each approach that are entirely dependent on your organization’s unique needs and requirements.)

Splunk Validated Architectures

If you choose an on-prem approach for your Splunk deployment, there are a variety of solutions that can help you get started. One such solution is leaning on Splunk’s catalog of Splunk Validated Architectures (SVAs).

Splunk’s product documentation is excellent, but there are gaps relative to architecture, best-practices, and — frankly — what works. And it makes sense. Everyone has different needs, so documentation couldn’t realistically cover every possible scenario. SVAs provide standardized “blueprints” for deployment you can leverage that Splunk has vetted.
Check out the “Splunk Validated Architecture” white paper from Splunk for more information.

Of course, SVAs are just a starting point. Kinney Group’s team of Splunk-certified experts would love to help you figure out what would work best for your specific needs.

Use Cases

Determining your approach to architecture has a lot to do with the data you need to bring in. If you find yourself stuck on architecture design, it may be helpful to start with your use case and work out from there.

If your primary use case is compliance, for example, you’ll need an architecture and environment that allows you to keep data ingested for a period of time, makes it accessible for another period of time, makes audits easy and as pain-free as possible, etc. If you’re a system administrator, you’d be bringing in different data sets and have different expectations of how to work with that data. Security your main focus? Insider threats? Application Management? You’d have an entirely different set of expectations and needs.

We recommend taking a look at Splunk’s Use Case Definitions and Use Case video library for more details (particularly helpful for beginner and intermediate Splunk users).

Kinney Group Reference Designs

Whatever your use case and needs, the bottom line is that there’s not a “push-button” type solution for Splunk architecture available from Splunk directly. And Splunk Validated Architectures, while a great starting point, don’t always utilize the most modern techniques and available infrastructure.

Kinney Group is leading the way with Reference Designs for Splunk that take the fundamentals and best practices of Splunk’s Validated Architectures and modernizes them for incredible performance gains. Our FlashStack and MSP Reference Designs, for example, provide a 10x boost in performance while utilizing 75% fewer physical indexers.

We’ve published four white papers to date that provide an understanding of our approach and associated benefits — all of which can be downloaded from our website — that are worth a look as you consider your next steps for planning your environment.

DIY vs Professional Services vs MSP…

While it’s possible to architect a ground-up solution yourself (if you have the right team in place), you may be better served to engage with Splunk architecture experts that know the right questions to ask, the best way to meet your unique needs, and have the expertise to mitigate risk and create opportunities for success with the platform.

One word of caution, however — traditional professional service providers tend to “blow in and blow out.” They may answer the mail for the immediate need, but often leave the internal team without the tools and knowledge they need to be successful and enjoy continued success.

With nearly 700 Splunk engagements under our belt, we’ve learned a lot about providing incredible solutions that are sustainable. Our approach is to empower the Splunk Creators who will be tasked with making the environment produce results by bringing them alongside each step of the journey, providing knowledge transfer throughout the process, and leaving them with what they’ll need to be successful long after our engagement has ended.

We’d love the opportunity to talk to you about your Splunk environment and architecture needs. Click here to schedule a quick meeting with a member of our team.

Preparing for Splunk Certifications

When it comes to preparing for Splunk Certification exams, there are two questions I see in the Splunk community this post will address:

  1. “I’m going to take the ____ certification test. How should I study?”
  2. “What is the ‘secret’ to passing the cert exams?”

In the post, we’ll advise studying techniques and provide the “secret” for passing Splunk Certifications… and, along the way, you’ll get better at using Splunk.

Note: This information is current as of March 2021. Please check the Splunk Training website for potential changes.

Step 1: Determine Splunk Certification Course Prerequisites

First, review the requirements for the certification. Namely, do you have to take any Splunk Education courses? I recommend the education courses for all certifications, but I understand if experienced Splunkers want to focus their education budgets on new topics or advanced classes.

Head to Splunk’s Training and Certification Page and select Certification Tracks on the left menu. The details for each certification list if the classes are required or strongly recommended (coursework will increase understanding of the concepts and make a pass more likely).

For example, select Splunk Enterprise Certified Admin to open the details and then select the top link. In the description, it states: “The prerequisite courses listed below are highly recommended, but not required for candidates to register for the certification exam.” Ergo, you do not have to take the classes (though you probably should).  

The Splunk Enterprise Certified Architect lists that the prerequisite courses through the Data and System Admin courses are not required. This means the only courses required for Certified Architect are: Troubleshooting Splunk Enterprise, Splunk Enterprise Cluster Administration, Architecting Splunk Enterprise Deployments, and the Splunk Enterprise Practical Lab.

Step 2: Determine Required Splunk Certifications

The same website, Splunk’s Training and Certification Page will also list any certification requirements for taking the certification you wish. For example, to obtain Splunk Enterprise Certified Architect, you must be a current Splunk Enterprise Certified Admin and a current Splunk Core Certified Power User.

To find which certifications are prerequisites for the cert you wish to take, on Splunk’s Training and Certification Page, click on Certification Track and then navigate to the particular certification you want to review.

Step 3: Review What Topics the Exams Cover

One of the most common questions I see and hear is, “What is on the Test?” Fortunately, Splunk publishes an exam blueprint for each of its certification tests. Splunk’s Training site lists these blueprints in the Splunk Certification Exams Study Guide, along with sample questions for most of the tests.

Let’s investigate the Splunk Core Certified Power User:

Splunk’s Test Blueprint states that this is a 57-minute, 65-question assessment evaluating field aliases, calculated fields, creating tags, event types, macros, creating workflow actions, data models, and CIM. Whew, so it spells out the main topics and explains them in more detail before giving out the critical information: exactly what topics are on the exam and the percentage of those topics on the typical exam.

We learn from the document that 5% of the exam deals with the topic “Using Transforming Commands for Visualizations” and further shows two elements: 

The topic “Filtering and Formatting Results” makes up 10% and has elements:

  • Using the eval command.
  • Using search and where commands to filter results.
  • Using the fillnull command.

The exam continues by listing out the ten topics of the exam and their elements. If a candidate is going to pass this exam, they should be knowledgeable on the topics listed. Bonus: if the candidate is good with these topics, they likely can perform the job as a Splunk Power User/Knowledge Manager.

Step 4: Review Material, Focusing on Unfamiliar Topics

In Step 3, we found what topics are on the different exams. Now comes the big question: how do I prepare for the exams?

1. Gather your study material: 

If you took the Splunk Education Classes, get the class docs. Those are great at taking cumbersome topics and presenting them in an accessible method.

Splunk Docs has exhaustive details on the variety of exam topics.

2. Practice on Splunk Instance(s):

We can read until we’re bleary-eyed, and that may be enough for you, but I find people learn better using a combination of reading and practice. If you have a laptop/desktop (windows, Linux, or Mac), then you can download Splunk—for free—install it on your system, and use that for practice. The free install works great for User, Power User, Admin, and Advanced Power User. For ITSI or ES, the best approach is to use a dev instance (if you are lucky enough to have access to one) or the Free Trials from Splunk Cloud. Other exams work best in a private cloud or container system (after all, it’s hard to learn how to use a cluster if you don’t have a cluster). 

Back to our example for Splunk Core Power User: 

Grab the Fundamentals 1 and Fundamentals 2 course material, have a Splunk instance installed, and open a web browser. Then, go through the exam blueprint one topic at a time. In this example, we’ll look at “Describe, create, and use field aliases.” The Fundamentals 2 course material explains what a field alias is and provides examples of its use. You can also supplement that material with the Splunk Knowledge Manager Manual section on Field Aliases. Run through creating field aliases in your Splunk instance until you have the topic down.

Then you can move on to the next section, find the relevant course material/documentation, and practice.

The Non-Step: Or, The Elephant in the Phone Booth

I need to address a question that gets asked far too often…

Q: “Dumps. Where do we find them?”

A: “Don’t do that.” (though sometimes the language is much more colorful)

Q: “Why not?”

Answer 1: Splunk Certification strictly prohibits using dumps, and their use is grounds for being banned from taking Splunk Certs. That’d suck for someone making Splunk their focus to limit their career by never earning any certifications.

Answer 2: The goal of certification is to prove the ability to use the product, not the ability to memorize test questions. If you tell an employer that you have the Power User Cert, it comes with a promise that you have the skills. Don’t be the person faking it. 

The Cert Secret

Finally, the “secret” method for passing Splunk certs: Find the topics and study those. Sometimes the best secrets are the obvious ones.

Best of luck in your testing!