In many environments, services and dependency hierarchies may be constantly changing. Maybe you are working with a containerized application and your web, application, and database servers are ephemeral. Maybe you are managing a large Citrix environment where resource pools are constantly being created depending on demand. Whatever your situation, if your service topology is dynamic, manually creating new services in Splunk ITSI can be laborious.

By crafting a search that monitors your data and identifies newly created services as well as their relationship to other new or already existing services you can automate the process of ITSI service creation. Follow along on this path to automating the creation of ITSI services in an environment that is dynamic.

Identify Your Entities

As a pre-requisite, you will want to ensure that you have all existing entities imported in ITSI. To guarantee that entities are added as they come into existence following the Splunk guidance for setting up a recurring import of entities in ITSI.

First, navigate to the ITSI Services View.

Then, select: Create Service > Import from Search.

Now we craft a search that will accomplish four main goals:

1. 1. Identify newly created services
   2. Map out service dependencies
   3. Associate entities with newly found services
   4. Associate services with any applicable service templates

Let’s walk through this search to explain what’s happening…

index=vmware sourcetype=vmware:perf

| stats values(host) as entities by env

| eval entities=mvjoin(entities,",")

| rename env as service_title

| eval service_template="vcenter_health"

This first part of the search is identifying new services, which in this case are representative of VMs that exist within a certain environment. In order to identify these services, we must use fields that currently exist. We use the information within the events to identify entities associated with this new service, and we use the eval command’s mvjoin function to create a comma-separated list of these entities

| append [ search index=vmware sourcetype=vmware:perf

| stats values(env) as service_dependencies by vc

| eval service_dependencies=mvjoin(service_dependencies,",")

| rename vc as service_title  ]

Find Your High-Level Services

This next portion of the search will work to identify higher-level services, for example, services that might include the services created in the first step as service dependencies. In the case that these higher-level services already exist, this portion of the search will serve only to map these to the newly identified services in the first step. We again use the eval command’s mvjoin function to create a comma-separated list of these service dependencies.

On the next page, we map the fields from the results of our search to the Service Title, Dependent Services and Service Template Link. You’ll notice that there isn’t an option here to identify the entities associated with each service, that happens on the next page. At the bottom of the page, you can get a preview of the services that will be imported as well as any dependency relationships that exist between the services that are being created/updated. Once everything looks good, click “Next.”

Define Entity Rules

Next, we define any entity rules that are to be associated with a created/updated service. In order for this step to work correctly, you will need to have associated the created service with a service template that includes the Entity Rule (‘matches a value to be defined in the service’). In the dropdown on the right, select the field in your search results that include the comma-separated list of entities. The preview pane will show you the entities that will be associated with each created/updated service. When everything looks good, click “Import.”

When the import has completed you will see the following page:

Here are a few more tips to follow once you’ve imported…

To schedule this import to occur on a recurring basis, select ‘Set Up Recurring Import.’ Then, you can view your newly created import, navigate to Settings > Data inputs and select “IT Service Intelligence CSV Import” from the list.

By default, your newly created import uses an update type of UPSERT. This will ensure that existing services are updated rather than being overwritten. You can also adjust the interval on which the import search will run from this page. Start by selecting your created import and checking the “More settings” checkbox on the following page. In this example, we’ve created all services from a single search. If you would prefer to break this down into several different searches, you’ll want to make sure that the searches to create lower-level services run before the searches to create the higher-level services that depend on those lower-level services.

Mission Accomplished

Please use these tips and apply them to your ITSI environment. We want to make your day job easier with Splunk, and automating ITSI service creation is just one way to do it. With experience in all things ITSI and Splunk, we’re packed with expertise in the platform. If you’re interested in speaking with one of our technical experts, let us know below.

If you’re reading this, the words “Splunk” and “deployment” may strike some level of interest. And not just the idea of deploying, but deploying well. Taking on a Splunk deployment and integrating the platform with your team, your data, and your company is a large undertaking. As a company that’s driven mission success on Splunk deployments for years, we have a few thoughts on delivering a successful implementation.

As a reminder, the two primary components of this framework are:

Part 1: The Operational Framework of a Splunk Implementation

Part 2: Best Practices for the “Four Functional Areas” of Splunk Implementation

Let’s dive in to Part Two…

The Four Areas of Splunk Implementation

A best practice is a set of commercial or professional procedures that are accepted or prescribed as being correct or most effective. They’re a standard of how we do things well, such as requesting new data or using Splunk Validated Architectures for your deployment.

There are four functional areas in every Splunk implementation. Here are the best practices for each:

Platform

Best practices in this area help with the availability, scalability, and maintainability of the Splunk deployment:

• Use the best Splunk Validated Architecture for your deployment.
• Set up clustering and data replication for redundancy and high availability
• Establish configurations management policies for knowledge objects
• Set up a sandbox environment to test changes and research new features introduced via platform updates

Program

These include business alignment, operations, collaboration, use cases, and staffing; which enable you to realize maximum value from your Splunk deployment:

• Establish robust change management policies and procedures and implement a source control system
• Develop standard naming convention, RBAC policies, standards for reports, and dashboards. Test thoroughly for adherence to these standards
• Implement configuration management software, like Puppet, to monitor drift
• Share success stories between teams, inspire new ideas, promote knowledge sharing, and participate in Splunk user activities such as user groups, Splunk Live, conferences, etc.
• Hold regular stakeholder meetings to provide updates on progress and demonstrate the value.

Data

These are to help generate well-designed and practical use cases that achieve the desired business outcome from the right set of data:

• Implement a request process and tools – use ticketing systems for new data request submissions and tracking
• Develop and maintain a data dictionary to capture meta-information about the data ingested
• Create processes to document and understand the use cases for every data request and follow typical data management lifecycles to retire data no longer in use
• Test, test, test. Use the sandbox environment to validate everything before deploying to production.

People

Having the right people trained, empowered, and motivated to do what they do best, the most essential best practice to adopt. Get smart people with the right mission plans and provide the necessary training and tools to be successful.

Your Splunk Experts

Take these tips and put them into action. If you’re looking ahead to a Splunk implementation in your future and need a helping hand, check out our Splunk services to see if we can be of assistance. If you’re looking more for best practice tips on-demand while you work through Splunk, Expertise on Demand may be a great resource for you. Regardless, all the best in your Splunk deployment efforts – we hope this two-part series got you closer to Splunk deployment success.

In many organizations, Splunk users and admins spend their time in Splunk onboarding new data, writing new searches (trying to find a needle in a haystack), or creating reports and dashboards for others to consume. The more friction that can be removed for users, the more value they recognize from the investment. Let’s walk through some Splunk implementation best practices that can help you in your deployment.

What we see as the Splunk implementation grows — in terms of data ingest volume and active users — the overall experience starts to deteriorate for everyone. The issues follow very similar patterns, and you want to mitigate some of them before too late. (You don’t want your Splunk Admins to want to write letters like this, do you?)

Splunk has published a success framework handbook to help you get started. This handbook provides reference materials, templates, and expert guidance for every aspect of a Splunk implementation, from data onboarding and platform management to suggestions for user education.

The two main components of this framework are:

1.) The Operational Framework of a Splunk Implementation

2.) Best Practices for the Four Functional Areas of Splunk Implementation

We’ll cover the first part here today (stayed tuned for the second part, coming soon).

The Operational Framework

Often, the Splunk implementation starts as a small “skunk-works” project within a team. Soon after the project kicks off, however, it grows into a large, poorly managed enterprise platform, not providing the desired value or outcome. Lack of a practical operational framework is one of the most common causes of poor user experience and adoption in a Splunk deployment. Like any enterprise platform, it is imperative to start with strong fundamentals. Establishing the purpose, goals, and ownership of the Splunk platform will increase the likelihood of success and wide-spread adoption.

1. Start with the Why

Ask yourself and your team important questions before diving into implementation. Here are a few to start with:

• Why is Splunk a critical tool in the success of the organization’s mission?
• Why does leveraging this platform to analyze data to assist with strategic and tactical decisions help increase the probability of attaining the organization’s financial targets?
• Why does uniform and wide-spread adoption help colleagues and teammates better achieve work-life balance and maintain a healthy lifestyle?

2. Find a Champion

Find a leader who is most passionate about the “why.” Enable them to provide the resources and support the success of the implementation. To help the leader, find a champion within the team who will share with all the groups the “art of the possible.”

3. Define Success

What does the implementation look like in an ideal state? Establish success criteria early, define success metrics, and hold everyone accountable for achieving these measurements.

4. Operations Framework

Choose a centralized, federated, or hybrid model for operational success and empower the teams to deliver using the framework. Make sure you review the advantages and challenges of each type of framework before defining your framework.

What’s Next?

Now that you’ve defined your operational framework, let’s take this “base coat” and apply it to your Splunk implementation. Kinney Group has years of Splunk experience and experience backing Splunk implementations from start to end. You’ll read about the best practices we’ve learned about the implementation process in the second part of this series. If this strikes a chord with an upcoming Splunk project your team may be developing, let’s chat. Fill out the form below to talk with one of our expert team members.

In Dude, Where’s My Data? (Part One), you learned how to configure your data source to send your data to Splunk. In Dude, Where’s My Data? (Part Two), you learned how to configure Splunk to receive and parse that data once it gets there. However, you still aren’t seeing your data in Splunk. You are pretty sure you configured everything correctly, but how can you tell?

Check your Splunk configuration for any errors. In these instances, there are three troubleshooting steps that I like to look at in order to ascertain what the problem could be.

They are as follows:

1. Check for typos

2. Check the permissions

3. Check the logs

Check for typos

There is always the possibility that even though the inputs look correct, there may be a typo that you originally missed. There may also be a configuration that is taking precedence over the one you just wrote. The best way to check is to use btool on the Splunk server configured to receive the data. This command-line interface (CLI) command checks the configuration files, merges the settings that apply to the same stanza heading, and returns them in order of precedence.

When looking for settings that relate to the inputs configured for a data source, this simple command can be run:

./splunk btool <conf_file_prefix> list -app=<app> --debug | grep <string>

Where <string> is a keyword in the input that you are looking for, will help quickly locate the settings that apply to that particular input.

Check the permissions

More times than not, the issue preventing Splunk from reading the log data is that the user running Splunk doesn’t have the correct permissions to read the file or folder where the log data is stored. This can be fixed by adding the user running Splunk to the group assigned to the file on the server that is configured to send data to Splunk. You should then, make sure that the group has the ability to read the file. On a Linux host, if you wanted Splunk to read, for example, /var/log/secure/readthisfile.log, you would navigate to the /var/log/secure folder from the command line interface using the following command:

cd /var/log/secure

Once there, you would run this command:

ls -l

This will return results that looks similar to the line below:

rwxr----- creator reader   /var/log/secure/readthisfile.log

Where creator, who is the user that owns that file, has the ability to read, write, and execute the file; reader, who is the group that owns the file, has the ability to read the file, and all other users cannot read, write, or execute the file.

Now, in this example, if the user running Splunk is Splunk, then you can check which groups that Splunk belongs to by running the following command:

id splunk OR groups splunk

If the results show that the Splunk user is not a member of the reader group, a user with sudo access or root, can add Splunk to the reader group using the following command:

sudo usermod -a -G reader splunk_reader

Check the logs

If the Splunk platform’s internal logs are accessible from the Splunk GUI, an admin user can use the following command to check for errors or warnings:

index=_internal (log_level=error OR log_level=warn*)

As a bonus, if your firewall or proxy logs are configured to send data to Splunk, and those logs are capturing data about network traffic between the data source and the Splunk server configured to receive data from your data source, searching these logs for errors by specifying the IP address and/or hostname of the sending or receiving server will help you find out if data is being blocked in transit. On a Linux host, the following commands can also tell you which ports are open:

sudo lsof -i -P -n | grep LISTEN

sudo netstat -tulpn | grep LISTEN

sudo lsof -i:22 ## see a specific port such as 22 ##

sudo nmap -sTU -O localhost  

Data Dreams Do Come True!

One, Two, Three strikes… and you’re out of problems with Splunk. Ha, if only these three blog posts could fix all of your Splunk issues, but we hope it helps. If you’re still having Splunk data configuration issues, or have any other troubleshooting needs, see how our Splunk services can help!