Understanding Splunk's MLTK
Splunk’s Machine Learning Toolkit (MLTK) is a powerful app that brings machine learning capabilities directly into the Splunk search interface. It enables users to apply statistical models and machine learning algorithms to their data with minimal coding knowledge. If you’re new to MLTK or want a deeper dive, check out our earlier post: “Unlocking Advanced Analytics: The Machine Learning Toolkit in Splunk“.
What is MLTK Really Doing?
At its core, the Machine Learning Toolkit is a collection of mathematical models. Each model attempts to learn patterns from your data using different statistical techniques. When trained on a set of historical data, the model can flag values that deviate significantly from expected patterns.
The DensityFunction algorithm is especially useful for anomaly detection. It uses Kernel Density Estimation (KDE) to understand the normal distribution of numeric values in a dataset. Then it scores each data point based on how likely it is to occur. A low score indicates an anomaly. This makes it ideal for identifying unexpected dips or spikes in values such as event counts, latencies, or usage metrics.
Think of DensityFunction like drawing a curve over a pile of data points that represent normal behavior. It learns the “shape” of what normal looks like. Then, when new data arrives, it checks how well that new point fits under the curve. If it falls too far outside, the model considers it unusual or abnormal. This approach allows the system to catch strange behavior without needing any prior knowledge of what an anomaly might look like.
Why Monitor Data Ingest?
Ensuring complete data ingestion is critical for organizations that rely on real-time analytics, security monitoring, or compliance reporting. Without reliable ingest, dashboards may present incomplete information. Alerts could be missed. Trends may be misinterpreted.
There is no built-in mechanism in Splunk that guarantees all data is arriving as expected. Often, identifying ingestion drops requires manual inspection.
Consider these examples:
- Security teams may miss signs of an attack if logs from firewalls or endpoint protection tools fail to ingest.
- Healthcare providers relying on real-time monitoring may suffer delays in patient alerting if device logs are incomplete.
- Retail operations using logs for point-of-sale or supply chain tracking might lose transactional visibility during ingest outages.
In all cases, loss of data can lead to faulty decision-making, compliance risks, and reduced operational awareness.
Implementing DensityFunction for Ingest Monitoring
To monitor data ingest volume, we’ll use the tstats command to count events over time for the wineventlog index. Then, we’ll train a model using the DensityFunction algorithm to detect drops.
1. Model Creation Search
| tstats count where index=wineventlog by _time span=1h
| eval HourOfDay = strftime(_time, "%H")
| fit DensityFunction count by "HourOfDay" into ingest_monitor_model
This search counts the number of events in the wineventlog index every hour. It also creates a new field called HourOfDay so that the model can learn expected behavior for each hour independently. Then, it trains a DensityFunction model using the hourly event counts grouped by hour of day. This helps the model detect drops in ingest volume that may vary based on the time of day. This approach can be adjusted to account for daily, weekly, or even monthly ingest volume.
2. Applying the Model for Alerting
In your alert search, use the model to identify anomalies. This step is where the model gets applied to new incoming data, helping you detect deviations in real time:
| tstats count where index=wineventlog by _time span=1h
| eval HourOfDay = strftime(_time, "%H")
| apply ingest_monitor_model
| where ingest_monitor_model.predicted=0
This search applies the trained model and filters for anomalies (where the predicted field equals 0). The ingest_monitor_model.predicted field is automatically created by the model and indicates whether the data point falls within the expected density range. A value of 1 means the point is considered normal; 0 means it’s an outlier — either too high or too low. If any result is returned, that’s your signal of a potential spike or drop in data ingest.
Other Use Cases for DensityFunction
While this post focuses on data ingest, the DensityFunction algorithm is highly versatile. Here are four other practical applications:
1. Login Success/Failure Monitoring
Monitor daily login activity to flag spikes in failures or unusual authentication patterns.
index=auth sourcetype=login_logs
| timechart span=1d count by status
| eval DayOfWeek = strftime(_time, "%A")
| fit DensityFunction failure_count by "DayOfWeek" into login_anomaly_model
2. Firewall Deny/Block Anomalies
Detect unusual changes in denied connection attempts that could point to network scanning or configuration issues.
index=firewall sourcetype=pan_logs action=deny
| timechart span=1h count
| eval HourOfDay = strftime(_time, "%H")
| fit DensityFunction count by "HourOfDay" into firewall_anomaly_model
3. Scheduled Job Execution Monitoring
Track successful job runs weekly and detect anomalies caused by missed or repeated executions.
index=jobs sourcetype=batch_logs status=success
| timechart span=1w count
| eval WeekOfYear = strftime(_time, "%U")
| fit DensityFunction count by "WeekOfYear" into job_execution_model
4. Alert or Email Volume Spikes
Ensure the alerting systems are functioning normally by monitoring the volume of outbound alert messages each month.
index=alerts sourcetype=email_logs
| timechart span=1mon count
| eval Month = strftime(_time, "%B")
| fit DensityFunction count by "Month" into alert_monitor_model
These examples demonstrate that DensityFunction is not limited to one type of metric. Wherever there’s numeric time-series data with a reasonably predictable baseline, it can help identify unexpected changes.
From Manual to Automated: Using Atlas to Monitor Ingest
While it’s completely possible to build ingestion monitoring workflows manually using MLTK, it can become time-consuming—especially as the number of datasets grows.
If you’re looking for a more streamlined, scalable approach, our Splunk app Atlas includes a feature called Monitor that simplifies this process. With Monitor, you can configure ingestion monitoring for any dataset by index, sourcetype, and/or source—all through a single user-friendly screen. No SPL required.
Monitor allows you to register data owners and set thresholds, so when ingest volumes fall below expected levels, built-in alerts notify the right people automatically. This reduces operational overhead while providing reliable, proactive oversight of your ingestion pipeline.
Atlas empowers teams to monitor ingest at scale, combining convenience with the reliability of tested practices.
Conclusion
Using the Machine Learning Toolkit, particularly the DensityFunction algorithm, enables teams to detect ingestion anomalies that would otherwise go unnoticed. It strengthens monitoring posture and helps automate visibility into critical data pipelines.
The more you integrate models like these into operational workflows, the more resilient and intelligent your Splunk environment becomes. If you’re interested in learning more about how to apply MLTK in real-world scenarios, stay tuned for future posts in this series.
To access more Splunk searches, check out Atlas Search Library, which is part of the Atlas Platform. Specifically, Atlas Search Library offers a curated list of optimized searches. These searches empower Splunk users without requiring SPL knowledge. Furthermore, you can create, customize, and maintain your own search library. By doing so, you ensure your users get the most from using Splunk.
