It’s Finally Here! Splunk Data Stream Processor

Splunk Data Stream Processor is finally here! The long-awaited Splunk Data Stream Processor is no longer in beta and is now released for public consumption. We’ve been anticipating the DSP service for quite some time. Who hasn’t been craving the real-time data processing and insights that DSP provides?

The Splunk Data Stream Processor (DSP) is a data stream processing service that manipulates data in real-time and shoots that data over to your preferred platform. DSP provides the ability to continuously collect high-velocity, high-volume data from diverse data sources, and distribute it to multiple destinations in milliseconds.

What’s the buzz about Splunk DSP?

Stream processing is the processing of data in motion, it is designed to analyze and compute on data instantaneously as it is continuously received. The majority of data sources are born in continuous streams, so being able to process them as such provides almost real-time insight into events for your analysts.

This is different from the “standard” data processing called batch processing. Batch processing collects the data (in batches) and then processes that data. The benefit to Stream processing is that you will have immediate insight into your critical events and can act on notable events more quickly.

How can I use Splunk DSP?

Use Case #1: Data Filtering/Noise Removal

With DSP, you can… filter or route non-useful and noisy logs to a destination of your choice. This use case allows you to route these logs to a separate syslog or storage solution for aggregation, but it is outside of Splunk, so it does not affect your Splunk license and it doesn’t fill your indexes with unwanted data. 

Use Case #2: Data Routing

With DSP, you can… ensure reception of high-velocity, high-volume data to multiple destinations. This use case allows you to send your data to Splunk, containers, S3, syslog aggregate and more at a rapid pace. This allows you to split the did to send to multiple destinations at the source without first having to index the data into Splunk then sending it off. Allowing for more efficient data flow.  

Use Case #3: Data Formatting

With DSP, you can… format your data using provided functions based on your configured conditions. This is a fairly straightforward use case allowing you to format your events to make your raw logs human-readable and informative without having to first index the data into Splunk. This can be combined with any of the use cases in this list to achieve maximum value with DSP.  

Use Case #4: Data Aggregation

With DSP, you can… aggregate data based on configured conditions and identify abnormal patterns in your data. You can pre-configure rules or conditions that will send data to different aggregate points based on the patterns within the data, that pertain to the rules configured. If you have a data source with a mixture of different kinds of logs, you can now pick up all the logs and forward them to different destinations with ease. 

What do I need with DSP?

First, look into what data sources are supported by Splunk DSP. Here are the data sources that are currently supported by the current version. Be on the lookout for more data sources that to be added in future releases.

Figure 1 - Splunk DSP supported data sources

Figure 1 – Splunk DSP supported data sources

Here are the system requirements that come with Splunk DSP. I’ve listed the more info on those below…

Figure 2 - Splunk DSP system requirements

Figure 2 – Splunk DSP system requirements

We’ve been more than excited for the release of this data stream processing service… we hope you are too. If you’re interested in learning more about Splunk Data Stream Processing, fill out the form below.

Author

Start typing and press Enter to search