HOWTO: ARCSIGHT SYSLOG CONNECTOR

By Amir Alsbih
Version 0.1 (Draft)

Basics

The first thing you need to do is enable Preserve Raw Event at the connector. So you will see what
the ArcSight will receive from the syslog sender.

Now copy the different categories of the syslog messages that the ArcSight seas for example:

<14>Feb 17 07:19:53 sbcptweb2 surf2jail[17418]: surftojail: (success) 127.0.0.1 2010-02-
06T20:32:17+00:00 00:15:ab:de:7e:17 6.34 test.user
<12>Feb 16 15:36:00 sbcptweb2 surf2jail[28175]: surftojail: (failure) 127.0.0.1 2010-02-
09T14:36:00+00:00 - 3.51 test.user
in a new file and comment them out by using #.

Name this file <name>.subagent.sdkrfilereader.properties. In our case that would be
surf2jail.subagent.sdkrfilereader.properties
Match the device that you need
We now have a file named surf2jail.subagent.sdkrfilereader.properties that contains

#<14>Feb 17 07:19:53 sbcptweb2 surf2jail[17418]: surftojail: (success) 127.0.0.1 2010-02-
06T20:32:17+00:00 00:15:ab:de:7e:17 6.34 test.user
#<12>Feb 16 15:36:00 sbcptweb2 surf2jail[28175]: surftojail: (failure) 127.0.0.1 2010-02-
09T14:36:00+00:00 - 3.51 test.user

First we need to write an expression that wills fulfil the following:
1. match the device
2. find the token that will categorize the event
3. the payload of that category
It is important to understand that the red marked part is the syslog header, so the regular expression
that has to be defined only will see the blue part of the message.
It is important to understand that you have to escape everything twice otherwise your connector wont
work and that everything that is between () is tokenized so that you can access it.

So in our case that regular express will be surf2jail.* \\((\\w+)\\) (.*).

In our case we will have a token (\\w+) that will categorize the events {success, failure, status}. And
the payload of this category (.*) which will be the rest of the syslog message in our case.

Now you have to add the regular expression to the file. In our case that will be:

regex=surf2jail.* \\((\\w+)\\) (.*)

No you have to define the token count and which token (the thing between ()) is the part where you
categorize your message and which part is the message itself. In our case that will be:

token.count=2
token[0].name=SubmessageIdToken
token[0].type=String
token[1].name=SubmessageToken
token[1].type=String

As you see the numbering begins as in every programming language at 0!

So we now have the part that will categorize the events as token[0] and name it SubmessageIdToken.
The payload of the category will be saved in token[1] which will gets the name SubmessageToken.

After that you can add the default mappings for this device like the device Product, the Severity and so
on. In our case that will be:
event.deviceReceiptTime=__useCurrentYear(_SYSLOG_TIMESTAMP)
event.deviceSeverity=_SYSLOG_PRIORITY
event.sourceAddress=_SYSLOG_SENDER
event.deviceVendor=__getVendor("My Company")
event.deviceProduct=__stringConstant("Surf2Jail")
event.deviceSeverity=__stringConstant("Information")
event.deviceEventClassId=__stringConstant("Surf2Jail")
Generating the sub messages
Now it is the time to generate the sub messages (e.g. the category expressions).

For this you have to define where the connector can find the part that will categorize the event and the
category payload. Since we have named them before SubmessageIdToken and SubmessageToken
we now can use the name we have defined before. So in our case we will add:

#submessages
submessage.messageid.token=SubmessageIdToken
submessage.token=SubmessageToken

Now the interesting part begins.
The submessage.messageid.token now will contains one of {success, failure, status} so we have three
different categories (and one default catch all) so we have to add that information to our file:

submessage.count=4

After that we have to define the submessages. Lets take the syslog message:

#<14>Feb 17 07:19:53 sbcptweb2 surf2jail[17418]: surftojail: (success) 127.0.0.1 2010-02-
06T20:32:17+00:00 00:15:ab:de:7e:17 6.34 test.user
as an example.
The goal is to match the success events so messageid will be success. So in our case we have to
add:

submessage[0].messageid=success

to the file.

After that we have to define the number of regular expressions that will match possible payloads. In
our case they all build up the same way so it will be:

submessage[0].pattern.count=1

After that we have to write a regular expression that will match the payload of this category and
extracts the necessary informations by tokenizing them via ().So in our case that would be:

submessage[0].pattern[0].regex=(\\d+\\.\\d+\\.\\d+\\.\\d+) (\\d+-\\d+-
\\d+T\\d+:\\d+:\\d+\\+\\d+:\\d+) (\\S+:\\S+:\\S+:\\S+:\\S+:\\S+) (\\d+\\.\\d+) (\\S+\\.\\S+)

Remember to escape things twice!

After that we have to map the tokenized fields to ArcSight fields. In our case that will be:

submessage[0].pattern[0].fields=event.deviceCustomString1,event.deviceCustomString2,even
t.deviceCustomString3,event.deviceCustomString4,event.sourceUserName

After that you can add the type of the tokens (it is not a must, but should be done to get a clean
connector). In our case that will be:

submessage[0].pattern[0].type=String,String,String,String,String

After that you should add mappings for that message like category object, category behaviour,
category Significance, As well as the event.name, event.message and so on. It is important to
understand that you have to escape the = by using \=! So in our case a cleaned view will be:

submessage[0].pattern[0].extramappings=event.name\=__stringConstant("Datenbankabfrage"
)|event.message\=__stringConstant("Erfolgreiche Anfrage auf die
Datenbank"|event.deviceCustomString4Label\=__stringConstant("Abfragedauer")|\
event.categoryObject\=__stringConstant("/Host/Application/Service")|\
event.categoryBehavior\=__stringConstant("/Communicate")|\
event.categoryDeviceGroup\=__stringConstant("/Application")|\
event.categorySignificance\=__stringConstant("/Normal")|\
event.categoryOutcome\=__stringConstant("/Success")

Repeat that step for all the categories you have.
After that include a default category that will match all the unmatched cases like the following:

# Default Message
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=(.*)
submessage[3].pattern[0].fields=event.message
submessage[3].pattern[0].extramappings=event.name\=__stringConstant("Unparsed Event")
Uploading the connector
The syslog connector you know have written must be placed inside the dir:

current/user/agent/flexagent/syslog/

If you want to upload it by the connector appliance assume that you are at the base dir
current/user/agent.