Splunk is a versatile app that performs numerous tasks ranging from indexing events, searching events, processing your data into events, and so forth. The data for most of these tasks and events are kept in logs that can be retrieved.

Below, we detail the locations for the primary Splunk log types.

1. Software Logs

Splunk records numerous events such as audits, configuration changes, exports, and so forth. These actions are filed under the internal logs.

The default path for the internal logs in your Splunk app are located at
$SPLUNK_HOME/var/log/splunk

The contents from this path are typically sent to the _internal index file by default as well. If you've set up your app as a Forwarder, you can expect that a certain subset of your logs will be monitored and forwarded to the indexing tier.

2. Introspection Logs

These logs cover different data than the logs above. The introspection logs typically record events from the app on your hosting system.

When seeking your Splunk Introspection logs, they are located at a different path:
$SPLUNK_HOME/var/log/introspection

Contents from here will be sent by default to the _introspection index.
Similarly, if you've set up your Splunk as a Forwarder, you'll find these logs being forwarded to your indexing tier as well.

3. Search Logs

Finally, if you're looking for logs about a particular search, run times, or other general performance metrics, you'll want to view your Splunk Search Logs.

To retrieve your Splunk search logs, these can be found in subfolders within the following path:
$SPLUNK_HOME/var/run/splunk/dispatch/

Searching for Logs

In the default setting for your app, only users with the Admin role have the ability to search through the _internal index.

Admins can search through internal logs with this command:
index=_internal

It can also be filtered to search for errors and warnings with this:
index=_internal (log_level=error OR log_level=warn*)

Log Management

The historical internal software logs are rolled based on their file size, typically being a set of five logs at around 25 MB each.

You can review these settings under this path:
$SPLUNK_HOME/etc/log.cfg

If you wish to modify your log management process or increase the size of your historical logs, you can create a $SPLUNK_HOME/etc/log-local.cfg file to host your changes. This file takes precedence over the log.cfg file as it won't be overwritten when you update your application.

If you are considering any custom cybersecurity solution that focuses on the resources and needs of your organization, please send a note to connect@metronlabs.com.