Every Question You Should Ask About Assets in Splunk

In Splunk Enterprise Security, asset and identity data management is essential to fully utilize the platform. An asset is a networked system in a customer organization. And the identity is a set of names that belong to or identify an individual user or user account. Having an accurate, complete list of your organization’s assets and identities is key to any security posture. Without it, you will not be able to answer basic, but important questions surrounding normal activity for your organization.

Having this list will allow you to assess the criticality and legitimacy of an entity on your network.

Ask Yourself

Ask yourself each of the following questions to identify every asset needed within your organization.

Does that system belong to the organization?

Who owns that system?

Is the system owner different from the application and/or data owner?

What other systems, applications, and network segments should that system be able to communicate with?

Which applications are running on that system?

What applications are supposed to be running on that system?

Have any new applications been installed on that system recently and if so, who installed them?

Has an application recently begun communicating on a new port?

Who is supposed to have access to that system?

Who is supposed to have access to the applications on that system?

Does that user’s activity correspond to the level of access they have been granted?

Is the frequency by which that user accesses that system consistent with how often that user normally accesses it?

Have a user’s privileges recently been elevated?

If so, who elevated those privileges?

Has a system recently downloaded or uploaded a large amount of data outside of the organization?

Is the amount of traffic generated by that system consistent with the amount of traffic generated by that system on previous days and with other systems running similar or the same application?

View this as a checklist. When considering the assets and identities within your organization, these questions should help you identify the right players. Documentation of these questions is important.

Look at Your Logs

Once the critical systems are identified, the answers to these questions will help you to monitor your assets. You can build your reporting to identify data that differs from the normal usage and activity of the systems. When you’re looking to monitor your assets, refer to these logs:

Network traffic logs

Authentication logs

Application logs

Change management logs

Endpoint protection logs

Web logs

A Quick Guide to Asset Management in Splunk

There are your quick and easy steps to asset and identity management within Splunk. Sometimes, you need to ask yourself a full slate of questions to fully understand the system information around your security posture in Splunk.  Kinney Group has years of experience working in Splunk and Splunk Enterprise Security. If you’re looking for help identifying or managing your assets and identities, our services can help.

Splunk Field(s) of Dreams: How-to Create Calculated Fields and Aliases

Providing your organization with a user-friendly search and analytics experience is critical to improving the usability of data in Splunk. By creating field aliases and calculated fields in Splunk, users can query new fields with or without altering the original field. In this way, users can choose to:

  • Correct an original field name that is truncated, misspelled, or abbreviated
  • Correlate or aggregate a field with a similar field from a different sourcetype
  • Better describe the data in the field
  • Create a field to filter data
  • Confirm with the Common Information Model (CIM)

Define Field Alias vs Calculated Field

A field alias is an alternate name that can be assigned to a field. Multiple field aliases can be created for one field. A calculated field is a way to perform repetitive, long, or complex derivations from the calculation of one or more other fields.

Though both are search-time operations that make it easier to interact with your original data, the field alias takes precedence over the calculated field. Thus, a field alias cannot be created for fields that were created as a calculated field. Both can override an existing field with the new field. To create the field, the user can either add the field to the configuration file, props.conf, or add it from the Splunk Web GUI.

Create a Field Alias from Splunk Web

To create a field alias from Splunk Web, follow these steps:

  1. Locate a field within your search that you would like to alias.
  2. Select Settings > Fields.
  3. Select Field aliases > + Add New.
  4. Then, select the app that will use the field alias.
  5. Select host, source, or sourcetype to apply to the field alias and specify a name.
    1. Note: Enter a wildcard to apply the field to all hosts, sources, or sourcetypes.
  6. Enter the name for the existing field and the new alias.
    1. Note: The existing field should be on the left side, and the new alias should be on the right side.
    2. Note: Multiple field aliases can be added at one time.
  7. (Optional) Select Overwrite field values if you want your field alias to remove the field alias name when the original field does not exist or has no value, or replace the field alias name with the original field name when the field alias name already exists.
  8. Click Save.
Figure 1 - Field Alias from Splunk Web
Figure 1 – Field Alias from Splunk Web

Create a Calculated Field from Splunk Web

To create a calculated field from Splunk Web, follow these steps:

  1. Select Settings > Fields.
  2. Select Calculated Fields > + Add New.
  3. Then, select the app that will use the calculated field.
  4. Select host, source, or sourcetype to apply to the calculated field and specify a name.
    1. Note: Enter a wildcard to apply the field to all hosts, sources, or sourcetypes.
  5. Enter the name for the resultant calculated field.
  6. Define the eval expression.
Figure 2 - Calculated Field from Splunk Web
Figure 2 – Calculated Field from Splunk Web

However, one of the things to note is that when you create the field alias or calculated alias in the Splunk Web GUI, the field is saved in the /etc/system/local/props.conf configuration file. If you want the configuration file to live in the app associated with the data you are defining the field for, you have to save the field in the /etc/apps/<app_name_here>/local/props.conf configuration file.

Create a Field Alias or Calculated Field in props.conf

To create a field alias or a calculated field in props.conf:

  1. Navigate to /etc/apps/<app_name_here>/local/props.conf
  2. Open the file using an editor
  3. Locate or create the stanza associated with the host, source, or sourcetype to apply to the field alias or calculated field.
  4. Next, add the following line to a stanza:

FIELDALIAS-<class> = <orig_field_name> AS <new_field_name>

EVAL-<field_name> = <eval_statement>
    • <stanza> can be:
      1. host::<host>, where <host> is the host for an event.
      2. source::<source>, where <source> is the source for an event.
      3. <source type>, the source type of an event.
    • Field aliases must be defined with FIELDALIAS.
      1. Note: The term is not case sensitive and the hyphen is mandatory.
      2. <orig_field_name> is the original name of the field. It is case sensitive.
      3. <new_field_name> is the alias to assign to the field. It is case sensitive.
      4. Note: AS must be between the two names and multiple field aliases can be added to the same class.
    • Calculated fields must be defined with EVAL.
      1. Note: The term is not case sensitive and the hyphen is mandatory.
      2. <field_name> is the name of the calculated field. It is case sensitive.
      3. <eval_statement> is the expression that defines the calculated field. Much like the eval search command, it can be evaluated to any value type, including multi-value, boolean, or null.

Creating field aliases and calculated fields help make the data more versatile. By using both the original fields and the new fields, users can create knowledge objects that craft a visual story about what the data represents. A well-crafted data visualization can help users understand trends, patterns, and relationships. Making meaningful correlations will ultimately lead to making better decisions.

Need more Splunk Tips?

As a dedicated Splunk partner with a bench full of experts, we’ve gained valuable insights and understanding of the Splunk platform that can excel your business forward. When it comes to best practice methods, training, and solution delivery, we’ve developed service offerings that can help any organization exceed its Splunk goals. For Splunk tips like this post, check out our Expertise on Demand service offering. If you’re working on projects that require a larger scope and Splunk skills, see what our professional service offerings can deliver for you.

Dude, Where’s My Data (Part 3)

In Dude, Where’s My Data? (Part One), you learned how to configure your data source to send your data to Splunk. In Dude, Where’s My Data? (Part Two), you learned how to configure Splunk to receive and parse that data once it gets there. However, you still aren’t seeing your data in Splunk. You are pretty sure you configured everything correctly, but how can you tell?

Check your Splunk configuration for any errors. In these instances, there are three troubleshooting steps that I like to look at in order to ascertain what the problem could be.

They are as follows:

1. Check for typos

2. Check the permissions

3. Check the logs

Check for typos

There is always the possibility that even though the inputs look correct, there may be a typo that you originally missed. There may also be a configuration that is taking precedence over the one you just wrote. The best way to check is to use btool on the Splunk server configured to receive the data. This command-line interface (CLI) command checks the configuration files, merges the settings that apply to the same stanza heading, and returns them in order of precedence.

When looking for settings that relate to the inputs configured for a data source, this simple command can be run:

./splunk btool <conf_file_prefix> list -app=<app> --debug | grep <string>

Where <string> is a keyword in the input that you are looking for, will help quickly locate the settings that apply to that particular input.

Check the permissions

More times than not, the issue preventing Splunk from reading the log data is that the user running Splunk doesn’t have the correct permissions to read the file or folder where the log data is stored. This can be fixed by adding the user running Splunk to the group assigned to the file on the server that is configured to send data to Splunk. You should then, make sure that the group has the ability to read the file. On a Linux host, if you wanted Splunk to read, for example, /var/log/secure/readthisfile.log, you would navigate to the /var/log/secure folder from the command line interface using the following command:

cd /var/log/secure

Once there, you would run this command:

ls -l

This will return results that looks similar to the line below:

rwxr----- creator reader   /var/log/secure/readthisfile.log

Where creator, who is the user that owns that file, has the ability to read, write, and execute the file; reader, who is the group that owns the file, has the ability to read the file, and all other users cannot read, write, or execute the file.

Now, in this example, if the user running Splunk is Splunk, then you can check which groups that Splunk belongs to by running the following command:

id splunk OR groups splunk

If the results show that the Splunk user is not a member of the reader group, a user with sudo access or root, can add Splunk to the reader group using the following command:

sudo usermod -a -G reader splunk_reader

Check the logs

If the Splunk platform’s internal logs are accessible from the Splunk GUI, an admin user can use the following command to check for errors or warnings:

index=_internal (log_level=error OR log_level=warn*)

As a bonus, if your firewall or proxy logs are configured to send data to Splunk, and those logs are capturing data about network traffic between the data source and the Splunk server configured to receive data from your data source, searching these logs for errors by specifying the IP address and/or hostname of the sending or receiving server will help you find out if data is being blocked in transit. On a Linux host, the following commands can also tell you which ports are open:

sudo lsof -i -P -n | grep LISTEN

sudo netstat -tulpn | grep LISTEN

sudo lsof -i:22 ## see a specific port such as 22 ##

sudo nmap -sTU -O localhost  

Data Dreams Do Come True!

One, Two, Three strikes… and you’re out of problems with Splunk. Ha, if only these three blog posts could fix all of your Splunk issues, but we hope it helps. If you’re still having Splunk data configuration issues, or have any other troubleshooting needs, see how our Splunk services can help!

Dude, Where’s My Data? (Part Two)

In Dude, Where’s My Data – Part One, we covered how to identify the right data to bring in. Now, let’s look at how you can ensure you’re getting that data into Splunk in the right way.

One of the easiest ways to ensure that your data is coming in correctly is to create a Technical Add-on (TA) for each data source you are sending to Splunk. By putting all the settings in one central location, you, your team, and any support representative can quickly create, read, update or delete configurations that tell Splunk how to process your data. This information can include:

  • Field extractions
  • Lookups
  • Dashboards
  • Eventtypes
  • Tags
  • Inputs (where the data is coming from)
  • And who has access to the view this data

Technical Add-ons are the lifeblood of any well-tuned Splunk environment and can mean the difference between spending hours and spending minutes troubleshooting simple problems.

Getting the Data In

There are several ways to bring data in, including uploading a log file from the Splunk Web GUI, specifying a Universal Forwarder (UF) using CLI or modifying the configuration files directly. Customers often don’t realize that using more than one of these methods can cause configurations to be stored in several places. You can find these configurations commonly stored in the following folders:

  • /etc/system/local
  • /etc/apps/<appname>/local
  • /etc/apps/<username>local

Having log files stored in that many places can make it difficult to determine which configurations take precedence. By storing configuration files related to a single data source in one central location, there is no need to wonder which configuration is the one that is active. It also allows you to quickly expand your architecture by sharing your TA with other Splunk servers in your deployment.

Call the Experts

That closes up our two-part walk-through on getting data into Splunk the right way. Now let’s get these Splunk roadblocks removed. Check out  Kinney Group’s service offerings to find the specialized work we can deliver for you.

Want to learn more? Fill out your contact information below!

Dude, Where’s My Data? (Part One)

Trying to find the proverbial “needle in a haystack” can be overwhelming when it comes to getting data into Splunk. Customers are often in such a hurry to provide value from their new Splunk deployment that they start bringing in all their data at once. This can lead to data uncertainty. How do you find what is truly important when all of it seems important? It’s as if your organization is having an existential crisis. So, what do you do?

1. Identify your use cases

Here are some questions and common use case areas you’ll need to get answered to kick things off…

Save time

  • Where are your employees spending most of their time?
  • What reports do they have to create manually every month?
  • What can be automated using Splunk?

Find the blind spots

  • Where are your organizational blind spots?
  • Do you know which servers are experiencing the most activity?
  • Are the most active servers the ones you thought it would be?

Clarity on systems

  • Are you planning for a major expansion or system adoption?
  • Do you have enough resources to accommodate the number of users?
  • Is access limited to only those users who need it?
  • Do we have an effective means of capacity planning?

Look at the ROI

  • Can we cut costs?
  • Which systems or over or undersized?
  • Do we need more bandwidth?

These and other questions are a good place to start to help you categorize your data needs quickly. Though you will probably not identify all your use cases at once, you will most likely uncover the most pressing issues on the first pass.

2. Prioritize your use cases

Once you have identified and the questions you would like to answer, you must arrange your data into categories based on their priority. The easiest grouping is:

  • Needs
  • Wants
  • Luxuries

These categories will help you segment the use cases into tasks that you should focus on immediately. Needs are things that will benefit the largest group of people and/or will potentially save your organization money in the long run. The needs are really what brings value to the way the business is run. Wants are things that will make a subset of users very happy to have but they could continue to function, albeit not as efficiently, if they had to wait a little while longer. Luxuries are cool to have, but probably satisfy a very specific niche request.

3. Identify your data sources

Once you have identified and prioritized the questions you would like to answer, you must identify which data will help you answer those questions. Make sure to consider which data sources will help you satisfy several use cases at once. This will help you correctly determine the size of your daily license and make sure you only focus on the data sources you need to address the needs and wants of your organization.

4. Identify your heaviest users

By creating a list of people who need access to each data source, you can correctly determine how large an environment is needed to support all the data sources you plan to bring in. It also helps when determining each user’s level of access. If a data source is widely popular, it may behoove you to create a dashboard and/or report to quickly disseminate important information that the users may need. It will also help size expansion of the environment.

By taking these four steps, users will not only feel like their needs are being heard, it will help them feel empowered to identify further use cases for future expansion. It will free up their time to focus on more complicated tasks and can mean the difference between them being proactive as opposed to reactionary. By taking the organization’s greatest needs into account, it can mean the difference between users adopting a Splunk implementation as their own and it being discarded as just another tool.

What’s Next?

Stay tuned for Part Two to learn more on how to get your data into Splunk in the right way. Until then, take a look at Kinney Group’s specialized service offerings. Whether we can help you clean up your existing data or in getting data into your environment correctly, Kinney Group has the expert Splunk consultants to help you with just that.

Want to learn more? Fill out your contact information below!