Escolar Documentos
Profissional Documentos
Cultura Documentos
Table of Content
1 Overview..................................................................................................................................................... 4 1.1 OSSIM Agent Role ............................................................................................................................... 4 Event Collection .......................................................................................................................... 4 Event Normalization ................................................................................................................... 4
OSSIM Server Role .............................................................................................................................. 6 Event Enrichment ....................................................................................................................... 6 Policies and Actions .................................................................................................................... 7
Configuring Detector Plugins ....................................................................................................................10 2.1 Rsyslog ..............................................................................................................................................10 Configuration File .....................................................................................................................10 Listener Configuration ..............................................................................................................10 Filters ........................................................................................................................................10
Detector Plugin Configuration ..........................................................................................................13 Configuration Files ....................................................................................................................13 Common Event Types ...............................................................................................................13 Parameters ...............................................................................................................................13 Using Local (Plugin) Variables...................................................................................................15 Using Global (Agent) Variables .................................................................................................15
Functions .......................................................................................................................................... 16 Path...........................................................................................................................................16 Conversions ..............................................................................................................................16 Application Specific Translations ..............................................................................................17 User Defined Translations ........................................................................................................17
2.7.1 Page 2
Loading Plugins .................................................................................................................................21 Priority and Reliability values ...................................................................................................21 SQL Statement ..........................................................................................................................21
Plugin Activation ...............................................................................................................................22 Activate the Plugin on the Server Side .....................................................................................22 Activate the Plugin on the Agent Side ......................................................................................22
2.9.1 2.9.2 3 4 5
Log files .....................................................................................................................................................22 Debugging .................................................................................................................................................22 Appendix...................................................................................................................................................23 5.1 5.2 Regular Expressions ..........................................................................................................................23 Configuration Example .....................................................................................................................25 Scenario ....................................................................................................................................25 Write a script to monitor the last status ...............................................................................25 Log sample ................................................................................................................................25 Collect the logs in a new log file ...............................................................................................25 Restart rsyslog .......................................................................................................................26 Check whether the new entries are written in the new log file...............................................26 Create a plugin file....................................................................................................................26 Register the Plugin with the OSSIM Agent ...............................................................................29 Register the Plugin with the OSSIM Server ..............................................................................30 Check whether the plugin was successfully registered ............................................................31 Restart the OSSIM Server .........................................................................................................31 Restart the OSSIM Agent ..........................................................................................................31 Check whether Events and Alarms are received ......................................................................32
5.2.1 5.2.2 5.2.3 5.2.4 5.2.5 5.2.6 5.2.7 5.2.8 5.2.9 5.2.10 5.2.11 5.2.12 5.2.13
Page 3
1 Overview
1.1 OSSIM Agent Role
1.1.1 Event Collection The collection process involves extracting the data logs from the source systems (Security, OS, RDBMS, etc.) and allows first steps for event log filtering. At this stage can be decided what is going to be read by the OSSIM Agent and what is going to be discarded before having an impact on the system performance. Before starting to write a plugin some actions to reduce the amount of events could be considered: Manage the log level settings at the application and managed device level Fix the problem that is generating events Use Pcap filters to ignore certain hosts or networks (Snort, Tcpdump...) In deployments with a big amount of analysed data, filtering at the application level should be done whenever possible
Log Files Good practice is to use one log file per plugin in order to increase performance. Having just one generic log file, all the plugins would have to read the same extensive content in order to catch the few relevant entries. Using rsyslog it is possible to filter the collected logs based on the syslog tags.
1.1.2 Event Normalization In the normalization stage a series of rules or functions applies to the data extracted from the source system in order to transform it in a common OSSIM format. o Raw Event The raw event might be a generic syslog message, an application log, an SNMP trap, the result of an SNMP or SQL Query or some other kind of information in a more or less structured form that is appended to a log file. Example: dmz01:/var/log/auth.log: May 30 13:15:52 dmz01 sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2
Page 4
Building Collector Plugins - Admin Guide o Normalized Event There is a certain set of fields which are required in order to ensure a consistent evaluation and correlation of the events by the OSSIM server. These fields can be populated with information from the log message or statically through the plug-in. Example: ossim-sensor:/var/log/ossim/agent.log: 2010-05-30 13:15:49,441 Output [INFO]: event type="detector" date="1275239752" sensor="192.168.178.201" interface="eth0" plugin_id="4003" plugin_sid="7" src_ip="192.168.178.20" src_port="4445" dst_ip="192.168.178.200" dst_port="22" username="root" log="May 30 13:15:52 dmz01 sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2" fdate="2010-05-30 13:15:52" tzone="0"
Page 5
Building Collector Plugins - Admin Guide suspicious, with 500 connections to different hosts in less than an hour the attack would get more and more reliable. Asset Value It is assigned to both the Source and the Destination Hosts and represents the importance the host has to the enterprise. Range: 0 - 5 Default value: 1 (also used for hosts not being defined in the asset database) Example: A database server can have an asset value of 5, a development test server an asset value of 2 and an unknown host in the Internet causing a portscan event would just have an asset value of 1. o Alarm Based on the Event Priority (0-5), Event Reliability (0-10) and the Asset Value (0-5), a Risk Value (0-10) is calculated and for values equal or greater than 1 Alerts are generated. The Risk is calculated based on the following formula: Risk = (Priority * Reliability * Asset) / 25 1.2.2 Policies and Actions Policies are defined in order to define what has to be done with the events as they reach the OSSIM Server: Correlation (i.e. checked against the correlation directives) Forwarding (i.e. one copy is sent to the forensic storage) Actions (i.e. send an email) Discard - the last filter possibility before saving the event in the database, although it is recommended to filter the events as close to the source as possible. Policies can make decisions on which events are going to be filtered based on: Source and Destination Assets (Hosts, Networks, ANY...) Ports Plugin Group Time Range
Page 7
First thing to start with is checking which log messages the application generates and eventually identify sets of logs having a similar structure. Those logs having a similar structure will be where possible covered by a single collector rule. o Create a Plugin File
Best is to copy one existing file and modify its content to match the new application. Should a plugin exist for a similar application, it is recommended to copy such a file, as there is a good chance that rules have a similar content and are grouped in a similar way - a generic HTTP-Proxy log will always contain a URL, a generic Firewall log will contain a Source IP Address and Source Port as well as a Destination IP Address and Destination Port. Some user defined fields might be defined for a specific application and the correlation at the server level can be simplified if similar applications use the same user defined fields. o Define a Generic Rule
This is the last Rule to evaluate, which catches all the events that cannot be grouped under specific rules. o Define Specific Rules
The Specific rules are defined for specific error conditions or categories of events. There might also be that one single rule is used to generate different types or subtypes of events. o Discard Noise
Events that are considered noise can be discarded by OSSIM by excluding certain event subtypes (Plugin_SIDs) in the plugin file, by the way the regular expressions are defined or by using policies. However, the best way to discard events is by filtering them on the monitored device or at syslog level on the host running the OSSIM Agent. o Review the Evaluation Order
The rules are evaluated alphabetically, which means that all it counts is the name of a rule and not the position in the plug-in file. The Generic Rule might even be on the first position if the name is properly chosen. Having rules alphabetically placed after the Generic Rule will have as effect that the corresponding logs will be evaluated as generic events instead of having the proper event type and subtype assigned. o Register the Plugin with the OSSIM Agent
In order to have a Plugin activated and sending events to the OSSIM server, the path to the plugin file has to be specified in the Agent configuration file.
Page 8
Building Collector Plugins - Admin Guide o Register the Plugin with the OSSIM Server
This is required in order to let the server know which events should be expected and which priority and reliability values the events should get assigned. o Activate the Plugin on the Server Side
Restart the OSSIM Server process. o Activate the Plugin on the Agent Side
Using the logger command sample logs can be replayed in order to test the operation of the OSSIM Agent or Server.
Page 9
Page 10
Page 11
Building Collector Plugins - Admin Guide [plugin-defaults] In this category variables can be defined to be used in the plugins configuration. Example: [plugin-defaults] date_format=%Y-%m-%d %H:%M:%S interface=eth0 sensor=192.168.178.201 [plugins] Defines which plugins (detectors and monitors) are enabled name_of_the_plugin=path_to_the_plugin_config_file Example: [plugins] postfix=/etc/ossim/agent/plugins/postfix.cfg ssh=/etc/ossim/agent/plugins/ssh.cfg [watchdog] Monitor the process associated to each plugin (In case it is running in the same machine) enable: interval: restart_interval: Enable or disable (True or False) Wait X seconds between checks Restart the process every X seconds (This has to be enabled in each plugin)
Page 12
Plugin statement: source=wmi 2.3.3 Parameters [DEFAULT] Any variable defined inside this category will be sent to the OSSIM Server if not modified by a plugin rule. User reserved range is between 9000 and 10000. plugin_id: Example: plugin_id=4003 Numerical identifier of the plugin within the OSSIM system
Page 13
Building Collector Plugins - Admin Guide [config] type: enable: source: location: create_file: process: start: stop: startup: shutdown: exclude_sids=SID List Example (hp-eva): process=snmptrapd start=yes stop=yes startup=/etc/init.d/snmpd start shutdown=/etc/init.d/snmpd stop exclude_sids=404,200,403 [translation] string=value Example (Postfix): [translation] sent=10 bounced=11 [Rule IDs Specific Rules] Here are the events collected and normalized. event_type=event regexp=Regular Expression plugin_sid=Plugin SID Event_Field=Value Example(ssh): [01 - Failed password] event_type=event regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for inval user (?P<user>\S+)\s+from\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)" plugin_sid=1 date={normalize_date($1)} Page 14 detector Enable or Disable the plugin (It must be enabled in config.cfg) Source of the events (log, mssql, mysql, wmi) The file(s) where the logs can be found - can contain multiple comma-separated files Create the log file in case it does not exist Name of the process generating logs (If on the same system) Start the process when the agent starts (yes/no) Stop the process when the agent stops (yes/no) Command that starts the process Command that stops the process Use this option to exclude SIDs
Building Collector Plugins - Admin Guide src_ip={$src} dst_ip={resolv($sensor)} src_port={$sport} username={$user} [Rule IDs Specific Rules] [Rule ID Generic Rule] Example (ssh): [99 - Generic rule] # Nov 15 11:55:35 11.1.4.9 sshd[1769702]: ********** event_type=event regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*" plugin_sid=99 date={normalize_date($1)} dst_ip={resolv($sensor)} Note: As rules are ordered alphabetically the Generic Rule has to have the highest Rule ID . 2.3.4 Using Local (Plugin) Variables The different configuration variables defined in the plugin configuration file can be used with the following syntax: %()s Example: process=pads shutdown=killall -9 %(process)s 2.3.5 Using Global (Agent) Variables \_CFG() Example: In the agent configuration file (/etc/ossim/agent/config.cfg): [watchdog] restart_interval=3600 ; seconds between plugin process restart In the plugin configuration file(/etc/ossim/agent/plugins/*.cfg): restart_interval=\_CFG(watchdog,restart_interval)
Page 15
2.4 Aliases
2.4.1 Path /etc/ossim/agent/aliases.cfg 2.4.2 Predefined Regular Expressions The predefined regular expressions can be used when creating new plugins. IPV4= \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} MAC= \w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2}:\w{1,2} PORT= \d{1,5} TIME= \d\d:\d\d:\d\d
SYSLOG_DATE= \w{3}\s+\d{1,2}\s\d\d:\d\d:\d\d SYSLOG_WY_DATE= \w+\s+\d{1,2}\s\d{4}\s\d\d:\d\d:\d\d To use an Alias in the regular expression use the \IPV4, \MAC, \SYSLOG_DATE, etc.
2.5 Functions
2.5.1 Path /usr/share/ossim-agent/ossim_agent/ParserUtil.py 2.5.2 Conversions resolv(host): resolv_ip(addr): resolv_port(port): normalize_date(date): translates a host name to an IPv4 address translates an IPv4 address to a host name translate a port name into its number convert date strings to isoformat (must tag the regular expressions with the following: <year>, <month>, <minute>, <hour>, <minute>, <second> or <timestamp> for timestamps. To define new date formats add a new regexp to the DATE_REGEXPS array. translates the protocols to the protocol numbers, based on the PROTO_TABLE calculates the md5 checksum all upper case get the integer value of a hexadecimal number
Page 16
Building Collector Plugins - Admin Guide 2.5.3 Application Specific Translations snort_id(id): intrushield_sid(sid,name): adds 1000 to the Snort ID all McAfee Intrushield IDs are divisible by 256, and this length doesn't fit in the OSSIM table ( mcafee_sid = hextoint(mcafee_sid)/256) translates the Netscreen messages based on the NETSCREEN_IDP_SID_TRANSLATION_TABLE translation table (defined in ParserUtil.py) translates the ISS_SiteProtector messages based on the ISS_SITEPROTECTOR_SID_TRANSLATION_MAP translation table (defined in ParserUtil.py) normalize interface name to either ext or int
netscreen_idp_sid(msg):
iss_siteprotector_sid(msg):
resolv_iface(iface):
2.5.4 User Defined Translations translate(string): translates strings based on the entries defined in the [translation] section of the plugin.
Example (from the iptables plugin): # The translation section in the plugin configuration file [translation] ACCEPT=1 REJECT=2 DROP=3 DENY=3 Inbound=4 Outbound=5 # Rule ID [0 - iptables] # Log sample # Oct 31 08:59:25 M2600001 kernel: RULE 0 -- ACCEPT IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 # TOS=0x00 PREC=0x00 TTL=64 ID=8437 DF PROTO=TCP SPT=57275 DPT=836 SEQ=2806649400 # ACK=0 WINDOW=32767 RES=0x00 SYN URGP=0 # Log Parsing regexp=(\S+\s+\d+\s+\d\d:\d\d:\d\d)\s+(\S*) (\S*):.*?(\S+)\s+IN=(\S*) OUT=(\S*) SRC=(\S+) DST=(\S+) LEN=(\d+) \S+ \S+ TTL=(\d+) .*? PROTO=(\S*) SPT=(\d*) DPT=(\d*) # plugin_sid is set to 1, the translated value for ACCEPT plugin_sid={translate($4)}
Page 17
userdata1 userdata9 User defined fields that could be used in custom reports, correlation directives, etc. Special types of events and the list of fields that can be used in each event type: Host-os-event host os sensor interface date Host-mac-event host mac vendor sensor interface date Host-service-event host sensor interface port protocol service application date
Page 18
2.7 Rules
The Rules define the format of each event and how they are normalized. It is composed by a regular expression and the list of fields that the event will include once it is sent to the OSSIM Server. In some cases only one regular expression will collect every event coming from one application, in some other cases more than one rule will be required. 2.7.1 Evaluation Order Rules are loading in alphabetical order based on the name given to each rule (Rule ID). Once the log matches the regex of one rule the ossim agent stops processing the event, therefore generic rules must be the last to be evaluated. 2.7.2 Structure o Name / Rule ID The name of the rule is mandatory o Regular Expression The regexp field contains the regular expression that defines the format of the events, and extracts the information to normalize the event. The regular expression has to be written following Python regular expression syntax: http://docs.python.org/library/re.html The information extracted by the regular expression from the log can be accessed by: Position: (\d\d):(\d\d):(\d\d) hour={$1} minutes ={$2} seconds={$3} Tags: (?P<hour>\d\d):(?P<minutes>\d\d)(?P<seconds>\d\d) hour={$hour} minutes ={$minutes} seconds={$seconds} o Normalized Fields As the server must receive normalized events, where IP addresses for instance are using the IPV4 format and the date uses the format YYYY-MM-DD HH:MM:SS (2010-12-31 22:57:00) To simplify the process of normalizing events functions are defined (more details on functions can be found in the Functions section of this document): resolv() Translates hostnames into IPV4 addresses (DNS queries)
Page 19
Building Collector Plugins - Admin Guide normalize_date() The normalize_date function translates many date formats into the format accepted by the OSSIM Server. o Translations Used for instance when the Event ID is not numeric, but plugin_sid has to be numeric. Translations have to be defined inside the [translation] section. The actual translation is triggered by using the translate() function. o Exclusions Some events can be filtered during the collection process editing the configuration file for each plugin: Using the option exclude_sids Modifying the regular expressions to avoid matching certain events
Page 20
To run the script use the following command (please double-check the content of the SQL scripts and the command line syntax before applying the changes to the database): ossim-server:/usr/share/doc/ossim-mysql/contrib/plugins# ossim-db < ssh.sql Example (/usr/share/doc/ossim-mysql/contrib/plugins/ssh.sql): -- SSHd -- plugin_id: 4003 DELETE FROM plugin WHERE id = "4003"; DELETE FROM plugin_sid where plugin_id = "4003"; INSERT INTO plugin (id, type, name, description) VALUES (4003, 1, 'sshd', 'SSHd: Secure Shell daemon'); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (4003, 1, NULL, NULL, 'SSHd: Failed password', 3, 2); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (4003, 2, NULL, NULL, 'SSHd: Failed publickey', 2, 2); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority,reliability) VALUES (4003, 99, NULL, NULL, 'SSHd: Generic SSH Event', 1, 1); Page 21
3 Log files
Generic Syslog /var/log/syslog (Unix) /var/adm/messages (Solaris) To identify where the logs for specific applications or certain logging levels are saved, check the /etc/syslog.conf or /etc/rsyslog.conf files. OSSIM Agent /var/log/ossim/agent.log OSSIM Server /var/log/ossim/server.log
4 Debugging
Note: Do never leave an application running in Debug mode in a production environment OSSIM Agent ossim-agent vv OSSIM Server ossim-server D6
Page 22
5 Appendix
5.1 Regular Expressions
Operator c \c ^ $ . [] [^] Meaning A non-special character matches with itself Removes the special meaning of the character c; The RE \$ matches with $ Indicates the beginning of the line Indicates the end of the line Any individual character One or any of the characters ; accepts intervals of the type a-z, 0-9, A-Z A char different from ; Accepts intervals of the type a-z, 0-9, A-Z
Regular Expression a.b a..b [abc] [aA] [aA][bB] [0123456789] [0-9] [A-Za-z] [0-9][0-9][0-9] [0-9]* [0-9][0-9]* ^.*$
Matches with axb aab abb aSb a#b ... axxb aaab abbb a4$b ... a b c (one character srtings) a A (one character srtings) ab Ab aB AB (two character srtings) 0123456789 0123456789 A B C ... Z a b c ... Z 000 001 .. 009 010 .. 019 100 .. 999 empty_chain 0 1 9 00 99 123 456 999 9999 ... 0 1 9 00 99 123 456 999 9999 99999 99999999 ... A full line
Page 23
Building Collector Plugins - Admin Guide Operator r* r+ r? r{n} r{,m} r{n,m} r1|r2 Meaning 0 or more occurrences of the RE r 1 or more occurrences of the RE r 0 or an occurrence of the RE r, and no more n occurrences of the RE r 0 or at most m occurrences of the RE r N or more occurrences of the RE r, but at most m The RE r1 or the RE r2
Matches with 0 1 9 00 99 123 456 999 9999 99999 99999999 .. empty_string 0 1 2 .. 9 empty_string ab ababab abababababab empty_string 1234ab 9ab9ab9ab 9876543210ab 99ab99ab ...
Regular expression \d \D \s \S \w
Matches with Any decimal character Any non decimal character Any space character Any non space character Any alphanumeric character and _
\W \Z
[^a-zA-Z0-9_]
Page 24
5.2.4 Collect the logs in a new log file Add the following to the rsyslog.conf on the system running the OSSIM Agent:
# # LOGON_EXAMPLE # local2.info /var/log/last_logon.log
Page 25
5.2.6 Check whether the new entries are written in the new log file
opensourcesim:/etc/ossim/agent/plugins# tail -f /var/log/last_logon.log Jul 14 19:38:49 dmz01 LOGON_EXAMPLE: > root pts/2 localhost Wed Jul 14 19:38 still logged in Jul 14 19:38:54 dmz01 LOGON_EXAMPLE: > root pts/2 localhost Wed Jul 14 19:38 - 19:38 (00:00) Jul 14 19:38:59 dmz01 LOGON_EXAMPLE: > ossim pts/2 localhost Wed Jul 14 19:38 still logged in Jul 14 19:40:51 dmz01 LOGON_EXAMPLE: > ossim pts/2 localhost Wed Jul 14 19:38 - 19:40 (00:01) Jul 14 20:15:09 dmz01 LOGON_EXAMPLE: > reboot system boot 2.6.31.6 Wed Jul 14 17:39 - 20:15 (02:35)
5.2.7 Create a plugin file Copy an existing plugin to build the new one on the existing structure
opensourcesim:/etc/ossim/agent/plugins# cp syslog.cfg example.cfg
Page 26
Page 27
Page 28
Building Collector Plugins - Admin Guide 5.2.8 Register the Plugin with the OSSIM Agent Add the example.cfg plugin to the agent configuration file
opensourcesim:~# more /etc/ossim/agent/config.cfg [plugins] syslog=/etc/ossim/agent/plugins/syslog.cfg example=/etc/ossim/agent/plugins/example.cfg wmi-system-logger=/etc/ossim/agent/plugins/wmi-system-logger.cfg Alternatively the plugin can be activated with the ossim-setup utility: opensourcesim:~# ossim-setup 1) Change Sensor Settings
2)
3)
Page 29
5.2.9 Register the Plugin with the OSSIM Server Copy an existing SQL script to build the new one on the existing structure
opensourcesim:/usr/share/doc/ossim-mysql/contrib/plugins# cp syslog.sql example.sql
Rules having the same plugin_sid will only require one SQL statement and plugin_sid defined on the OSSIM server. Different rules where used just because both IP addresses and hostnames are returned as sources by the last command.
-- plugin_id: 9001 DELETE FROM plugin WHERE id = "9001"; DELETE FROM plugin_sid where plugin_id = "9001"; INSERT INTO plugin (id, type, name, description) VALUES (9001, 1, 'Example', 'User logons based on the last output'); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 1, NULL, NULL, 'Login: System console' , 5, 5); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 2, NULL, NULL, 'Logout: System console' , 5, 5); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 3, NULL, NULL, 'Login: Pseudo terminal' , 3, 5); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 4, NULL, NULL, 'Logout: Pseudo terminal' , 3, 5); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 5, NULL, NULL, 'System reboot: Restarted' , 5, 5); INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 99, NULL, NULL, 'Last: Generic messages' , 1, 1);
After changing the script to reflect the Plugin IDs and SIDs, load the changes with the command:
opensourcesim:/usr/share/doc/ossim-mysql/contrib/plugins# cat example.sql | ossim-db
Page 30
Building Collector Plugins - Admin Guide 5.2.10 Check whether the plugin was successfully registered
Plugin ID
Plugin SIDs
5.2.11
5.2.12
Page 31
Building Collector Plugins - Admin Guide 5.2.13 Check whether Events and Alarms are received
Events
Alarms
Page 32