Join Kinney Group Splunk Enterprise Security Certified Admin Hailie Shaw as she walks through the process of data model mapping in Splunk for CIM compliance. Catch the video tutorial on our YouTube channel here.

Note: the data visible in this video and blog post are part of a test environment and represent no real insights. 

Starting Out with Data Model Mapping

As Splunk engineers, we constantly deal with the question: How do I make my data, events, and sourcetypes Common Information Model (CIM) compliant? This is especially crucial with Enterprise Security use cases, when data will need to map to a CIM compliant data model.  When we search the environment pre-CIM compliance, our query will return no results. This is what we aim to change with data model mapping. 

Begin by downloading the Splunk Add-on Builder app, which you’ll use later. Let’s transition to Splunk docs. The left panel of the webpage lists all data models. For any option, Splunk docs will list out all fields to which users can map. This should be the first step when choosing the data model that matches your environment’s existing fields as closely as possible.  

Creating the Add-on

Turn to Splunk’s Search & Reporting app and navigate to the data that you’re looking to map; the fields will populate on the left side of the screen. Pay special attention to the sourcetype that you wish to match—you’ll need to supply this to the Splunk Add-on Builder app that will map the fields for CIM compliance. Within Splunk Add-on Builder, select “New Add-on.” The default settings will be sufficient with the exception of the required Add-on name. Click “Create.”  

Click Manage Source Types on the top banner menu. Click “Add,” and then “Import from Splunk” from the dropdown menu. Select your sourcetype, which should populate within the menu after you import data from Splunk. Click Save, and the events will be uploaded. Next, click Map to Data Models on the top banner menu.  

Mapping the Data Model

From here, select “New Data Model Mapping.” You’ll be prompted to enter a name for the event type and select the sourcetype you’re using. The search will populate below, automatically formatted with the correct name. Click “Save.” The resultant data model will include, on the left-hand side of the screen, a list of the event type fields. Hovering over each entry within this list will reveal a preview of the data that exists within the field. 

Essentially, the app takes the fields from your initial data and transfers them onto the data model. On the right of the screen is where you’ll select a data model for the fields to map to. Each data model is filtered through Splunk’s CIM, and you can select which is most appropriate based on the Splunk documentation with which we began.  

When you select a data model, the Add-on Builder will provide supported fields, which you can cross-reference with Splunk Docs; the app is a field-for-field match. This step will give you a list of CIM-approved fields on the right to complement the original fields on the left. To map them together, click “New Knowledge Object” and select “FIELDALIAS.” This is where you’ll need to select which data model field most closely matches the initial event type or expression.

Once you’ve made a match, select OK, and the app will provide a field alias. Repeat this process for each field you wish to include. Once you’re satisfied, click “Done.” As you can see, the data has now populated with the sourcetype listed. 

Validate and Package 

It’s important to validate your package to ensure that it follows best practices. To do so, click “Validate & Package” from the top banner menu. Click the green “Validate” button. When the validation reaches 100%, you can download the package. This page will also generate an Overall Health Report detailing which elements, if any, should be addressed. Once downloaded, change the name of the file to a .zip. By double-clicking, the file will extract, and you can open it to view details of the sourcetype and events within.  

Return to the Splunk instance through Search and Reporting and run the data model command again as a search. Now, your events will populate in the data model! You can also view the CIM compliant aliases in Settings > All Configurations.  

Compliance, Achieved

The resultant data model is good to go—CIM compliant and ready to be exported. Check out more Splunk tips on our blog and our YouTube channel. To get in touch with engineers like Hailie, fill out the form below: 

When users import a data file into Splunk, they’re faced with a dense, confusing block of characters in the data preview. What you really need is to make your data more understandable and more accessible. That’s where data parsing and event breaking come in.

In this brief video tutorial, TechOps Analyst, Hailie Shaw, walks you through an easy way to optimize and configure event breaking— spoiler alert: it boils down to two settings in props.conf.

This function within Splunk can greatly simplify the way you interact with your data. After learning it, you’ll get the insights you need, faster.

Event Breaking

When importing data into your Splunk instance, you’ll want to be able to separate it based on events, which enables legibility and ease of interpretation. Because the imported data file isn’t pre-separated, event breaking is an essential skill. The most important part in separating data is the line breaker within the event boundary, which is how Splunk decides how to group and separate certain events.

LINE_BREAKER is how Splunk separates events into different lines. The default = a sequence of new lines, followed by all carriage returns within parentheses.

SHOULD_LINEMERGE is how Splunk merges separate lines into individual events. The default value = true, but should always be set to false as a best practice. If set to true, the value will automatically set to false once Splunk is supplied with your new regex, which provides a specific guide for breaking up new lines and preventing them from merging.

When you import the data file, the term “<event>” appears several times intermittently within the block of data, which is listed as one large event. By pasting the original block of data into regex101.com, users can identify the <event>s that should ideally begin new lines. Then enter the regex resulting from this step into Splunk in props.conf, which will populate the desired fields.

By selecting regex as the event-breaking policy and entering the pattern from regex101.com, the data preview will display your data separated into events (each beginning with “<event>”). The regex in props.conf defines both terms listed above. This data preview can now be saved as a new sourcetype.

Learn More!

Learning event breaking can help make your data more organized and legible for members of your team. Kinney Group’s Expertise on Demand service offers a wealth of Splunk tips and knowledge, tailored by analysts like Hailie to your team’s specific needs. Check out the blog for additional tutorials from our engineers. To learn more about Expertise on Demand, fill out the form below!

If there was a secret sauce for Splunk, the key ingredient would be the platform’s universal forwarders. Providing users with the ability to automatically send data for indexing, Splunk forwarders are essential to data delivery in Splunk Enterprise and Splunk Cloud environments.

In most Splunk instances, you have multiple forwarders. These forwarders throw data at your search heads and indexers in order to read and store your data. However, there has historically been an issue with forwarders: they go missing and they fail.

If you’re looking at your data pipeline in Splunk, your forwarders are on the front line. Forwarders play a pivotal role in ingesting your data; however, they can disappear or unexpectedly fail (without you knowing). A missing forwarder may result in an issue as small as temporarily not ingesting data. However, a missing forwarder could also be an indication of a much larger issue, like an entire server going down.

To solve this time-old problem in Splunk, we’ve built an application within Atlas, our new platform for Splunk, that allows you to have eyes on all of your forwarders in one place.

Atlas’s Forwarder Awareness Application

Atlas Forwarder Awareness is an application that provides visibility into all of your forwarders, their statuses, and any misconfigurations or failures within your environment. Built within the Atlas Application Suite, the Forwarder Awareness tool enables teams to have constant visibility into their forwarders’ health and statuses.

Now teams can quickly determine if a forwarder is missing and take action—immediately.

In Splunk’s own Forwarder Management interface, users are alerted when forwarders go missing with limited information or guidance on the issue. When this happens, Splunk teams have to dig through alerts in their Splunk monitoring console to try and identify an issue with their forwarders.

The Atlas Forwarder Awareness tool sends you a list of the forwarders that are missing and which data sources are impacted. Instead of requiring users to log into their Splunk monitoring console, users can now access this critical information on their forwarders directly through their Search Head Cluster. This application offers real-time visibility and awareness into your forwarders’ health and status.

The Value in Visibility

Without requiring admin access, you (and your team) have full visibility into the status and health of your forwarders in one view. From this view, users can view visual graphs representing forwarder statuses by operating system, forwarder types in use, and forwarders’ SSL status. On the application, users also have visibility into top-performing forwarders (by total WB) and missing forwarders.

In the example below, you’ll see a really powerful element of Forwarder Awareness. You’ll notice on the screen below that a forwarder is offline (with no contact in 15 minutes or longer), the last time Splunk saw that forwarder, and what sourcetypes may be affected. That view does not require admin access.

These insights are invaluable and critical information for you and your team to identify — and these immediate insights are only available to users through Atlas’s Forwarder Awareness application. In any other situation, teams would spend hours of their time and resources to identify this same information.

To put it simply, a missing forwarder means missing data, failed compliance standards, inactive SSL certificates, and many more detrimental losses for Splunk teams. Ultimately, a missing forwarder can be extremely costly to an organization in both data loss and spent resources.

Conclusion

Every Splunk instance is at risk of a failed or missing forwarder. With your forwarders being at the front line of your data pipeline, it’s essential to have eyes on them at all times. With Atlas’s Forwarder Awareness Application, you have the visibility you need.

This is just a glimpse into the power of the Atlas platform. Paired with more applications, reference designs, and support services, Atlas enables all Splunk teams to be successful. If you’d like to learn more about the Atlas Platform, let us know in the form below.

With Splunk, getting data in is hard enough. After uploading a CSV, monitoring a log file, or forwarding data for indexing, more often than not the data does not look as expected. These large blocks of unseparated data are hard to read and unable to be searched. If the data is not separated into events, you may be wondering how to correctly parse, and perform advanced search commands using fields.

This is where field extraction comes in handy.

Field Extraction via the GUI

Field extractions in Splunk are the function and result of extracting fields from your event data for both default and custom fields. Basically, organize your data with field extractions in order to see the results you’re looking for.

Pictured above is one of Splunk’s solutions to extracting searchable fields out of your data via Splunk Web. Within the Search and Reporting App, users will see this button available upon search. After clicking, a sample of the file is presented for you to define from events the data. The image below demonstrates this feature of Splunk’s Field Extractor in the GUI, after selecting an event from the sample data.

From here, you have two options: use a regular expression to separate patterns in your event data into fields, and the ability to separate fields by delimiter. Delimiters are characters used to separate values such as commas, pipes, tabs and colons.

If you have selected a delimiter to separate your fields, Splunk will automatically create a tabular view in order to allow you to see what all events properly parsed would look like compared to its _raw data pictured above.

Functionality is provided to rename all fields parsed by the selected delimiter. After saving, you will be able to search upon these fields, perform mathematical operations, and advanced SPL commands.

What’s Next? Rex and Erex Commands 

After extracting fields, you may find that some fields contain specific data you would like to manipulate, use for calculations, or display by itself. You can use the Rex and Erex commands to help you out.

Rex

The Rex command is perfect for these situations. With a working knowledge of regex, you can utilize the Rex command to create a new field out of any existing field which you have previously defined. This new field will appear in the field sidebar on the Search and Reporting app to be utilized like any other extracted field.

Syntax

| rex [field=<field>] (<regex-expression>)

For those who would like to use the Rex command, and would like resources to learn, please utilize websites such as https://regex101.com/ to further your development.

In order to define what your new field name will be called in Splunk, use the following syntax:

| rex [field=<field>] (?<field_name>”regex”)

Erex

Many Splunk users have found the benefit of implementing Regex for field extraction, masking values, and the ability to narrow results. Rather than learning the “ins and outs” of Regex, Splunk provides the erex command, which allows users to generate regular expressions. Unlike Splunk’s rex and regex commands, erex does not require knowledge of Regex, and instead allows a user to define examples and counterexamples of the data to be matched.

Syntax

 | erex <field_name> examples="<example, <example>" counterexamples="<example,

<example>"

Here’s an example of the syntax in action:

 | erex Port_Used examples=”Port 8000, Port 3182”

That’s a Wrap

There is a ton of incredible work that can be done with your data in Splunk. When it comes to extracting and manipulating fields in your data, I hope you found this information useful. We have plenty of Splunk tips to share with you. Fill out the form below if you’d like to talk with us about how to make your Splunk environment the best it can be.

There is immense power built within the Splunk data center model. However, with great power comes even greater complexity. The traditional model is difficult to scale for performance, and Splunk teams already working against license costs are constantly faced with performance issues.

We know your teams have more to do with their time, and your organization has better ways to spend their budget.

With Atlas’s Reference Designs, we’ve found a better path forward.

Background

The Atlas Reference Designs provide a blueprint for modern data center architecture. Each provides blazing-fast performance increases and reduces indexer requirements, leading to a lower cost of ownership for your environment. Our Reference Designs are built to remove any friction caused by building and deploying your Splunk environment, serving all Splunk customers. Whether you’re running Splunk on-prem, in a hybrid fashion, or in the cloud, Atlas’s Reference Designs provide you with a simplified and proven solution to scaling your environment. 

There is a common problem that every organization faces with Splunk: ingesting data effectively.

The Atlas Reference Designs present you with a solution that pairs the Splunk Validated Architectures with the needed hardware or environment in order to deliver proven performance results.

Our first set of Reference Designs were built for on-prem Splunk instances in partnership with Pure Storage, a leader in the enterprise platform and primary storage world. Check out Pure Storage’s many nods in the Gartner Magic Quadrant for their innovative solutions.

With the matched power of Pure Storage with Kinney Group’s Splunk solutions, we’ve been able to decrease the indexers needed through Pure’s speed and increased IO on the back end—and we have the tested results to prove it.

Proven Results

Splunk recommends that for every 100 GB you ingest, you create at least one indexer. Let’s look at a real-world example delivered by the Atlas team: a company is ingesting 2 TB/day in Splunk—based on Splunk’s recommendations, this company relied on 20 indexers.

By applying the Atlas Reference Design, we were able to reduce that physical indexer count to just five indexers. This significantly reduces the cost of owning Splunk while increasing performance by ten times.

For those who invest in Splunk infrastructure on-prem, this means huge savings. Think of the costs that 20 indexers entail versus five—the cost savings are huge. Then factor in the impact of reducing the exhausting manual hours spent on Splunk to just a few minutes of human interactions.

Conclusion

To sum it up, we took an incredibly complex design and made it shockingly easy. Through automation and the great minds behind Atlas, your Splunk deployments can now launch with the push of a button, less time and fewer resources, and guaranteed results. The Atlas Reference Designs are built for all customers, with many releases to come; get in touch using the form below to inquire about our other Reference Designs.

Last week, we shared some big news! Kinney Group launched Atlas — an exciting new platform for navigating and achieving success with Splunk. We’re thrilled to reveal this revolutionary platform to the world and share more about its vision and technical components.

The mission of Atlas is to help customers derive meaningful value from Splunk for their organization and their colleagues.

Background

What’s stopped organizations so far from reaching their goals in Splunk? The answer is all too real for our friends in the industry: IT teams are under siege.

Splunk is and should always be a key enabler for organizations to reach business, security, and operational objectives. Splunk gives IT teams meaningful insights into security posture, IT Ops, and other analytics. Here’s the issue: these teams are buried in work. IT teams are already charged with an ever-growing list of IT demands. And to top it off, they’re now tasked with managing, implementing, and learning Splunk.

Atlas enables these teams to derive value out of Splunk, fast.

A Look Inside Atlas

Atlas is a subscription service that provides a clear path forward on your Splunk journey with revolutionary datacenter architectures, personal guidance and support, and a collection of applications and utilities that provide powerful insights, instantly. Let’s take a closer look into each component of Atlas.

1. Atlas Reference Designs

Atlas Reference Designs provide a blueprint for modern data architecture. They provide rapid performance increases and reduce indexer requirements, leading to a lower total cost of ownership for your environment. Atlas Reference Designs are built on Splunk validated designs, paired with top hardware and cloud environments, and powered by Kinney Group’s unique Splunk tuning and automation solutions.

Atlas Reference Designs are proven to reduce server counts, cut storage costs, and eliminate hidden OpEx expenses, all while enabling 10x improvements in your Splunk performance.

2. Atlas Applications and Utilities

Atlas’s Applications and Utilities include multiple tools built to monitor and easily manage your data in Splunk. With an interface that’s clean and easy to navigate, working in Splunk has never been easier.

“The current components in the Atlas platform were selected because they address horizontal pain points across many different Splunk instances and deployments that we’ve seen with Kinney Group customers,” Georges Brantley, Scrum Master at Kinney Group, emphasizes.“We want to help Splunk admins and users further curate their Splunk environment to produce what they need in order to accomplish their mission.”

3. Expertise on Demand

If those tools aren’t enough, how about we arm you with a team of expert Splunkers to help with everything else?

Expertise on Demand (EOD) is your team of dedicated Splunk professionals to help you with on-demand support and Splunk best practice training. Our EOD team can quickly assist with any Splunk question or fix in increments as small as 15 minutes.

Atlas gives you essential visibility into your environment, and Expertise on Demand makes sure you get the support you need.

Conclusion

Splunk is a journey, not a destination. And every component we’ve built into Atlas is specifically and thoughtfully designed to reduce the complexities of Splunk while removing the roadblocks on your journey to success.

There’s still plenty to come from the Atlas platform, and we can’t wait to share more with you. If you’re interested in learning more about the platform, fill out the form below.

No one likes mismatched data. Especially data that’s hard to filter and pair up with patterned data. A Regular Expression (regex) in Splunk is a way to search through text to find pattern matches in your data. Regex is a great filtering tool that allows you to conduct advanced pattern matching. In Splunk, regex also allows you to conduct field extractions on the fly.

Let’s get started on some of the basics of regex!

How to Use Regex

The erex command

When using regular expression in Splunk, use the erex command to extract data from a field when you do not know the regular expression to use.

Syntax for the command:

| erex <thefieldname> examples=“exampletext1,exampletext2”

Let’s take a look at an example.

In this screenshot, we are in my index of CVEs. I want to have Splunk learn a new regex for extracting all of the CVE names that populate in this index, like the example CVE number that I have highlighted here:

Next, by using the erex command, you can see in the job inspector that Splunk has ‘successfully learned regex’ for extracting the CVE numbers. I have sorted them into a table, to show that other CVE_Number fields were extracted:

The rex Commands

When using regular expression in Splunk, use the rex command to either extract fields using regular expression-named groups or replace or substitute characters in a field using those expressions.

Syntax for the command:

| rex field=field_to_rex_from “FrontAnchor(?<new_field_name>{characters}+)BackAnchor”

Let’s take a look at an example.

This SPL allows you to extract from the field of useragent and create a new field called WebVersion:

As you can see, a new field of WebVersion is extracted:

The Basics of Regex

The Main Rules

^ = match beginning of the line

$ = match end of the line

Regex Flags

/g = global matches (match all), don’t return after first match

/m = multi-line

/gm = global and multi-line are set

/i = case insensitive

Setting Characters

\w = word character

\W = not a word character

\s = white space

\S = not white space

\d = a digit

\D = not a digit

\. = the period key

Setting Options

* = zero or more

+ = 1 or more

? = optional, zero or 1

| = acts as an “or” expression

\ = escape special characters

( ) = allows for character groupings, wraps the regex sets

Some Examples

\d{4} = match 4 digits in a row of a digit equal to [0-9]

\d{4,5} = match 4 digits in a row or 5 digits in a row whose values are [0-9]

[a-z] = match between a-z

[A-Z] = match between A-Z

[0-9] = match between 0-9

(t|T) = match a lowercase “t” or uppercase “T”

(t|T)he = look for the word “the” or “The”

Regex Examples

If you’re looking for a phone number, try out this regex setup:

\d{10} = match 10 digits in a row

OR

\d {3}-?\d{3}-?\d{4} = match a number that may have been written with dashes 123-456-7890

OR

\d{3}[.-]?\d{3}[.-]?\d{4} = match a phone number that may have dashes or periods as separators

OR

(\d{3})[.-]?(\d{3})[.-]?(\d{4}) = using parentheses allows for character grouping. When you group, you can assign names to the groups and label one. For example, you can label the first group as “area code”.

If you’re looking for a IP address, try out this regex setup:

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} = searches for digits that are 1-3 in length, separated by periods.

Use regex101.com to practice your RegEx:

We’re Your Regex(pert)

Using regex can be a powerful tool for extracting specific strings. It is a skill set that’s quick to pick up and master, and learning it can take your Splunk skills to the next level. There are plenty of self-tutorials, classes, books, and videos available via open sources to help you learn to use regular expressions.

If you’d like more information about how to leverage regular expressions in your Splunk environment, reach out to our team of experts by filling out the form below. We’re here to help!