This blog article explains how Splunk receives Syslog events, indexes them, and makes it available for users to view.
First, let’s go over the basics of Syslog.
What is Syslog?
Syslog stands for System Logging Protocol and is a standard protocol used to send system or event data logs in a particular message format to a central location for storage. This logging protocol is a crucial part of network monitoring as it helps you track the overall health of network devices by simplifying log message management.
Syslog messages are transferred using UDP on port 514. However, there’s no guarantee of message acknowledgment and delivery on the receiver/server-side due to the connectionless nature of UDP. Some network devices use TCP for confirmed message delivery to overcome this issue.
What is Syslog Server?
Syslog server, also known as the syslog collector or receiver, centrally stores the syslog messages from various network and security devices. Syslog-ng is an open source implementation of the syslog protocol for Linux and Unix-like systems. Windows OS doesn’t provide native support for this logging protocol, but you can still use third-party tools to collect event logs for Windows and transfer them to a syslog service.
To implement a syslog management tool a user can simply install syslog-ng/rsyslog using the included packages in the operating system. However, syslog-ng is highly preferred for Splunk.
The Benefits of Using Syslog-ng Server
A Splunk instance can listen on any port for incoming syslog messages. While this is easy to configure, it’s not considered best practice for getting syslog messages into Splunk. If the splunkd process stops, all syslog messages sent during the downtime would be lost. Additionally, all syslog traffic would stream to a single Splunk instance, which is not always wanted if it can be configured to spread syslog data amongst Heavy forwarders or indexers.
Syslog-ng receives log messages and lets you write it to a file. It has content based filtering and flexible configuration capabilities which will reduce data load on Splunk, network capacity requirements thereby reducing license costs and improving overall performance.
Collecting data using a universal forwarder from a syslog server allows you to send the data to multiple Splunk indexers in a load balanced fashion. Data can also be archived and stored for a longer duration on a syslog-ng server. You can configure each unique data source to have a separate directory.
When you have a network issue between syslog servers and Indexers, a universal forwarder can buffer the data for some time. If the forwarder goes down, when it comes back up it knows where it left off and starts reading from there. Therefore, ensuring there’s no data loss.
How To Get Syslog Data Into Splunk
Step 1: Install Syslog-ng which is pre-packaged with some versions of Linux and enable it to start at boot.
Check iptables to determine which ports are open.
# iptables –L –n
We need port 514 (which is the default syslog port for root) to be added to iptables. To add UDP port 514 to /etc/sysconfig/iptables, use the following command below.
# iptables -A INPUT -p udp -m udp –dport 514 -j ACCEPT
Step 2: Now, modify syslog-ng.conf with appropriate settings to write data to disk.
Use native capabilities of syslog-ng to suppress non-critical information and route them to desired destination file. Having separate directories for each unique data source is a best practice.
# syslog-ng.conf file # options { chain_hostnames(no); create_dirs (yes); dir_perm(0755); dns_cache(yes); keep_hostname(yes); log_fifo_size(2048); log_msg_size(8192); perm(0644); time_reopen (10); use_dns(yes); use_fqdn(yes); }; source s_network { udp(port(514)); }; #Destinations destination d_palo_alto { file(“/data/syslog/paloalto/$HOST/$MONTH-$DAY-$HOUR-palo.log” create_dirs(yes)); }; destination d_all { file(“/data/syslog/catch_all/$HOST/$MONTH-$DAY-$HOUR-catch_all.log” create_dirs(yes)); }; # Filters filter f_palo_alto { match(“009401000570” value(“PROGRAM”)) or match(“009401000570” value(“MESSAGE”)); }; filter f_all { not ( filter(f_palo_alto) ); }; # Log log { source(s_network); filter(f_palo_alto); destination(d_palo_alto); }; log { source(s_network); filter(f_all); destination(d_all); };
Save the file and reload syslog-ng service for new changes to take effect
# Reloading configuration file without restarting syslog-ng systemctl reload syslog-ng # restarting syslog-ng systemctl restart syslog-ng
Step 3: Then, install the latest version of Splunk Universal forwarder on the machines where syslog-ng is installed.
Start Splunk service and enable splunkd to start at boot. Ensure log files are readable by ‘splunk’ users. Deploy inputs.conf and outputs.conf files to monitor the files created by syslog-ng service. Splunk forwarder monitors these log files and sends data to Indexers near real time.
# inputs.conf [monitor:///data/syslog/paloalto/*/*.log] sourcetype = pan:log index = paloalto_logs disabled = false host_segment = 4
Depending on volume, having multiple syslog-ng servers (at least 2) behind a load balancer is best practice. This provides redundancy and fail over. To ensure syslog-ng doesn’t fill the filesystem up with log files, create a cron job which runs once every day to remove old log files after “x” days.
# remove files older than 10 days 0 1 * * * /bin/find /data/syslog/ -name \*.log -type f -mtime +10 -exec rm {} \;
Syslog-ng provides a great level of flexibility in filtering and routing logs which in turn reduces license costs and load on Splunk. It can be horizontally scaled to support increase in log volumes. Install syslog-ng in your environment and handle terabytes of syslog data with ease.
If you found this helpful…
You don’t have to master Splunk by yourself in order to get the most value out of it. Small, day-to-day optimizations of your environment can make all the difference in how you understand and use the data in your Splunk environment to manage all the work on your plate.
Cue Atlas Assessment: Instantly see where your Splunk environment is excelling and opportunities for improvement. From download to results, the whole process takes less than 30 minutes using the button below:
