Tag Archives: Splunk

Splunk Field Extractions for Symantec Messaging Gateway A.K.A Brightmail Syslogs

The Symantec Messaging Gateway formally known as Brightmail is a spam filtering appliance, you can read more about it from Symantec here. The appliance appears to run on Linux and it has both a web-interface and a command line interface accessible via SSH. It also has the ability to log system and application level logs via syslog.

The system level logs include processes such as sshd, crond and sudo; the application application level mail logs consist of two processes: ecelerity and bmserver. In this post I focus on the application level logs, those beginning with the <142> prefix. Symantec has some not so helpful documentation on this appliance’s log formats here: https://support.symantec.com/en_US/article.HOWTO15282.html

From what I see in Splunk the logs are in the format: <identifier>date time server-name process[process-number]: process-id|message-id|event|variable-log-format. There appears to be 18 different application level log events all with a different format. Those events are: IRCPTACTION, ACCEPT, VERDICT, TRACKERID, UNTESTED, FIRED, SENDER, LOGICAL_IP, EHLO, MSG_SIZE, MSGID, SOURCE, SUBJECT, ORCPTS, DELIVER, ATTACH, UNSCANNABLE and VIRUS. These different formats make it impossible to use Splunk’s built in field extraction interface. An alternative solution is to write a custom regex extraction.

Lacking complete documentation I had to reverse engineer a regex extraction from the logs being sent to my Splunk server. With this in mind be warned that my final regex extraction may contain errors. Also be aware that the fields in the extraction are names that I assigned and are not the official field names as I could not find complete documentation on this system’s log format.

I used Regex101.com to help me craft and test my regular expression extraction. You can view my saved regex and test string with anonymized syslog entries at: https://regex101.com/r/kR0iS8/1. If you visit this link, and are familiar with regular expressions, you will notice that I used multiple positive look-behinds. This is the best solution I could come up with to deal with the variable log formats produced by the SMG. My skill with regular expressions is intermediate at best, so there may very well be better solutions out there. If someone with more knowledge of regular expressions reads this article and cares to correct me, feel free to leave a comment.

Below is the final regular expression that I came up with. This seems to work for the majority of the logs that Splunk processes from Symantec Messaging Gateway, but may need further tweaking.

^<142>(?P<date>\w+\s+\d+)\s+(?P<time>[^ ]+)\s+(?P<server>\w+)\s+(?P<process_name>[a-z]+)\[(?P<process_number>\d+)[^ \n]* (?P<process_id>[^\|]+)\|(?P<message_id>[^\|]+)\|(?P<action>IRCPTACTION|VERDICT|UNTESTED|FIRED|SENDER|LOGICAL_IP|EHLO|MSG_SIZE|MSGID|SOURCE|SUBJECT|ORCPTS|TRACKERID|ATTACH|UNSCANNABLE|VIRUS|DELIVER|ACCEPT)(?:(?:(?<=ACCEPT|DELIVER|LOGICAL_IP)\|(?P<src>[^:\s]+)(?::(?P<port>[0-9]+))?(?:\|(?P<to>[^\s]+))?)|(?:(?<=FIRED|IRCPTACTION|ORCPTS|TRACKERID|UNTESTED|VERDICT)\|(?P<recipient>[^\s\|]+)(?:\|)?(?P<result>[a-z][^\|\s]+)?(?:\|(?P<result_2>[a-z][^\|]+))?(?:\|(?P<result_3>.+))?)|(?:(?<=SENDER)\|(?P<from>[^\s]+))|(?:(?<=MSG_SIZE)\|(?P<msg_size>\w+))|(?:(?<=SUBJECT)\|(?P<subject>.*))|(?:(?<=ATTACH)\|(?P<attachment>.+))|(?:(?<=UNSCANNABLE)\|(?P<reason>.+))|(?:(?<=VIRUS)\|(?P<virus_name>.+))|(?:(?<=EHLO)\|(?P<fqdn>.+)))?

If readers have any questions of comments about this extraction, feel free to leave a comment and I will try to respond in a timely manner.

Splunk HTTP Event Collector Python 3 Example

With Splunk’s latest release of version 6.3 a new feature called HTTP Event Collector has been added. It allows for sending JSON formatted data to Splunk via an HTTP call. I won’t go into all the details of this feature in this post, but for the curious more information can be found here.

This feature is great for anyone who wants to easily get data into Splunk using their own scripts. With this being a new feature there is not yet many examples of how to use this on the scripting side. In this post I want to provide an example in Python that others can use to build upon in their own code.

Below is a short and documented example using the urllib library to craft an HTTP request that Splunk’s HTTP Event Collector will accept.

import urllib.request
import json

def send_event(splunk_host, auth_token, log_data):
   """Sends an event to the HTTP Event collector of a Splunk Instance"""
      # Integer value representing epoch time format
      event_time = 0
      # String representing the host name or IP
      host_id = "localhost"
      # String representing the Splunk sourcetype, see:
      # docs.splunk.com/Documentation/Splunk/6.3.2/Data/Listofpretrainedsourcetypes
      source_type = "access_combined"
      # Create request URL
      request_url = "http://%s:8088/services/collector" % splunk_host
      post_data = {
         "time": event_time, 
         "host": host_id,
         "sourcetype": source_type,
         "event": log_data
      # Encode data in JSON utf-8 format
      data = json.dumps(post_data).encode('utf8')
      # Create auth header
      auth_header = "Splunk %s" % auth_token
      headers = {'Authorization' : auth_header}
      # Create request
      req = urllib.request.Request(request_url, data, headers)
      response = urllib.request.urlopen(req)
      # read response, should be in JSON format
      read_response = response.read()
         response_json = json.loads(str(read_response)[2:-1])
         if "text" in response_json:
            if response_json["text"] == "Success":
               post_success = True
               post_success = False
         post_success = False
      if post_success == True:
         # Event was recieved successfully
         print ("Event was recieved successfully")
         # Event returned an error
         print ("Error sending request.")
   except Exception as err:
      # Network or connection error
      post_success = False
      print ("Error sending request")
      print (str(err))

   return post_success

def main():
   splunk_auth_token = "00000000-0000-0000-0000-000000000000"
   splunk_host = ""
   log_data = {
      "data_point_1": 50,
      "data_point_2": 20,
   result = send_event(splunk_host, splunk_auth_token, log_data)
   print (result)


A few things to note: this example is not using SSL, so the Enable SSL check box in the HTTP Event Collector global settings must be unchecked. Also Splunk is picky about the top level JSON keys, only a few specific keys can be used. Those keys are: time, host, source, sourcetype, index and event. All custom data should be under the event key. Finally this code should work in all versions of Python after 3.0.