Statistical Anomaly Detection With ESM

Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Anurag Singla
Sr. Manager, R & D

Statistical anomaly detection
with ESM Correlation
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
Agenda
Anomaly detection primer
Use case 1: data monitors
Review of recent ESM correlation features
Use case 2: financial fraud detection
Use case 3: time-sequence anomalies
Caveats and conclusions
Baselining
Record statistics of a certain behavior/event flow
Typically involves trending over long time periods
Detect deviations from this baseline in order to discover anomalies

Threat (anomaly) score
Amount of statistical deviation from baseline for a new event / pattern
User logged in 5x longer than usual
Unusually high volume data transfers from application host
Large bank account transfers compared to monthly average

Some types of anomalies are well understood and can be captured using static rules
Login from multiple geo regions in short time period
Large # sources connecting to same target
Known virus/worm definitions

Others are more dynamic and have too much variance across individual actors.
Use case 1: data monitors
Data Monitors
Track real-time event streams.
Perform automatic short term baselining.
Audit events if some statistic (eg. Moving Average) deviates from recent history by a
certain threshold. Rule can fire and take action when threshold exceeded.

Not suited to analysis over long periods.
Use case 2:
financial fraud detection
Behaviour-based fraud detection
Methodology: compare an account holders recent transactions with the historical behavior
patterns of the account holder, looking for anomalies.

Fire an alert if the total amount of money a customer has transferred out of his account today is
greater than the accounts historical cumulative monthly average.
If the customer transfers out on average $1200 per month, then as soon as a transaction is received that
puts his daily transfer total (since 12am this morning) above $1200, we should fire an alert and add the
account to a watch list for investigation.
Conditions for detecting fraud can be fine tuned to avoid false positives.

ESM enhancements for anomaly detection
Baselining:
Interval queries on active lists
Lightweight rules
Timestamp granularity variables

Real-time detection:
Cumulative active list fields
Lightweight rules
Time-partitioned active lists

Problem: atomic column increment
Solution: cumulative numeric columns
For numeric column types (Integer, Long, Double),
subtypes SUM, MIN, MAX.
The value from AddToAL action is combined
atomically with existing value.
To implement counter, use Integer(SUM) field and
add value 1 each time
Can obtain mean value using counter in
conjunction with value sum field
Num
transactions
Total
amount
Max Amount Min Amount
1 200.00 200.00 200.00
Num
transactions
Total
amount
Max
Amount
Min
Amount
1 200.00 200.00 200.00
Values inserted by AddToList action
(CustomerId and TimeKey same for all)
Resulting values in AL entry
Num
transactions
Total
amount
1 50.00 50.00 50.00
Num
transactions
Total
amount
Max
Amount
Min
Amount
1 200.00 200.00 200.00
2 250.00 200.00 50.00
Num
transactions
Total
amount
1 875.00 875.00 875.00
Num
transactions
Total
amount
Max
Amount
Min
Amount
1 200.00 200.00 200.00
2 250.00 200.00 50.00
3 1125.00 875.00 50.00
Lightweight rules

Designed for data list maintenance
No correlation or audit event when rule fires
No aggregation (stateless)
Can match large # events and not get disabled

Allow separation of data maintenance and risk
analysis logic.
Processed earlier than regular rules.
Lightweight rule actions executed before regular
rule conditions are evaluated.

Timestamp granularity variables
Convert timestamp to beginning of time period
(hour/day/month, etc.)
Use result in AL key field
Input arguments are timestamp field (eg. EndTime,
DeviceCustomDate) plus granularity selection.
Output is timestamp value shifted to desired
boundary.
Example: transaction time = 2012 -09-11-14:32
Start of hour: 2012-09-11-14:00
Start of day: 2012-09-11-00:00
Start of month: 2012-09-01-00:00

Time-partitioned partially cached active lists

Keep most recent entries (latest timestamp key
value) in memory for fast correlation
Evict entries with older timestamp value
Partition the cache into buckets based on time
values.
PLEASE do not insert random time values (like
EndTime) into time field!
Interval query on active list
Problem: query the entries of an
active list based on a time interval.
Query on active list was snapshot based
all the entries in the AL were
considered.

Solution: interval queries on AL
Make the query type interval.
Enter the start time and end time of the
query.
Select the field on which the time interval
will be evaluated.
Workflow: baselining and detection
Lightweight
rules
Daliy
active list
Monthly
trend
Monthly
active list
Stats
trend
Historical
stats
active list
Rules
Transaction
events
Anomaly?
Update daily
transaction
stats
Runs every 30
days
Runs every 30
days over 180
days
Read daily
values
Read historical
values
Daily transactions active list
Cumulative Fields
TimePartitioned
Historical stats active list
Active list roll up using trends
CustId TimeKey Total Max
46201 Aug 5
th
$250 $150
46201 Aug 14
th
$700 $700
50532 Aug 7
th
$100 $100
50532 Aug 28
th
$3290 $3000
Cust TimeKey Total Max
daily
46201 Aug $950 $700
50532 Aug $3390 $3290
Daily transactions Monthly transactions
Trend Query
Interval: 1 mo
Freq: 1 mo
Active list roll up using trends
CustId TimeKey Total Max
daily
46201 May $2500 $1800
46201 Jun $800 $300
46201 Aug $950 $700
50532 Apr $1600 $1000
50532 Jul $810 $600
50532 Aug $3390 $3290
Cust Max
daily
Max
monthly
Mean
monthly
46201 $1800 $2500 $1417
50532 $3290 $3390 $1933
Monthly transactions Historical stats
Trend Query
Interval: 6 mos
Frequency: 1 mo
Data maintenance rule
Fraud detection rule
Fraud detection rule in action

Snapshot in time for CustomerId 46201:
Historical Stats AL: mean monthly total = $1200
Daily Transactions: cumulative total = $1150

Transaction event received: CustomerId = 46201, transaction value = $100
1. Data Maintenance rule (lightweight): updates daily total to $1150 + $100 = $1250
2. Fraud Detection rule (standard): looks up Daily AL, finds updated cumulative value ($1250)
3. Condition is matched, rule fires and adds customer account to Suspicious Accounts AL
Use case 3:
sequence anomaly detection
Use case 3: sequence anomaly detection
What do users normally do?
Baselining process
Vertica /R
Historical
User Data
Profile 1 Profile 2
Profile 3
Transition Probability Matrix
Hadoop /
Mahout
User-Profile Mapping
Transition diagram online banking
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
1.0
Help
0.4
0.4
0.2
0.12
0.72
0.06
0.08
1.0
0.02
0.80
0.05
0.15
0.80
0.05
1.0
0.15
0.55
0.18
0.27
0.52
0.46
0.02
0.1
0.7
0.2
0.36
0.64
State transition probabilities for specific user profile (101)
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.4
0.4
0.2
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.12
0.72
0.06
0.08
0.02
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.80
0.05
0.15
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.80
0.05
0.15
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.52
0.46
0.02
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.55
0.18
0.27
Transition probability (baseline) active list
Real-time anomaly detection process

Online session in progress
Receive incoming event containing: {UserId, SessionId, OldState, NewState}
Lightweight (maintenance) rule does:
1. Lookup Profile for User
2. Look up transition probability between old and new state
3. Replace missing transition with low prob value (e.g. 0.000001)
4. Compute negative log (JME variable), add to session anomaly score (SUM field)

Standard rule checks anomaly score against defined threshold
Normal user session
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.4 (0.40)
1.0 (0)
0.72 (0.14)
0.80 (0.097)

0.80 (0.097)
0.52 (0.28)
Anomaly Score (Sum of reds) = 1.02
Fraudulent user session, part 1
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.4 (0.40)
1.0 (0)
0.80 (0.097)

Anomaly Score (part 1) = 5.20
1
.
0

(
0
)

Fraudulent user session, part 2
Login
Main
Page
Add
Account
Check
Balance
Transfer
Page
Transfer
Init
Transfer
Commit
Transfer
Failed
Transfer
Success
Remove
Account
Help
0.80 (0.097)

Anomaly Score (part 2) = 3.27, total = 5.20 + 3.27 = 8.47 (vs 1.02)
0.18 (0.74)
1.0
0.80 (0.097)
Anomaly detection - caveats
Statistical baseline issues (hat tip: John Petropoulos)
Prone to being skewed.
During an attack, short-term stats are prone to being skewed quickly.
How do we deal with this? Remove the offending entries?

False positives can be reduced, never eliminated
Users sometimes behave strangely
Seasoned fraudsters can appear eerily natural
Thank you
Security for the new reality

Statistical Anomaly Detection With ESM

Enviado por

Dados do documento

Descrição original:

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Statistical Anomaly Detection With ESM

Enviado por

Direitos autorais:

Formatos disponíveis

Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Você também pode gostar