Night Time Video

Enhancement And
Unusual Activity Detection

Introduction
Due to the development of moving object detection and tracking algorithms for
surveillance applications, we can easily obtain the relevant contextual
information and further analyse the common patterns in order to detect
potential anomalies. Object detecting and tracking are important in any vision
based surveillance system. Object detection and tracking in night surveillance
is an important part of visual surveillance, since unusual events usually happen
at night.

Motivation

In the field of surveillance most of the work has been focused on day time
video surveillance. Very few approaches has been presented for night time
video surveillance.
Work has been done on night surveillance using thermal cameras. But they
are very costly which cant be feasible for all scenarios.
Since human vision is restricted to function well at night due to low
visibility as well as humans are prone to more errors so it is necessary to
design a system which is robust and can detect unusual activity

Scope
A good and efficient surveillance system is needed in many areas in Private
sectors, Military Areas, Industries, Home, etc.
Scope includes:

Automatic warning system for abnormal activity.

Digital enhancement of nighttime video for better review.

Security and Surveillance in Military Areas, Private Sectors, Industries,

Homes, etc. at night

Challenges

Images captured at night for visual surveillance have low SNR, low
contrast.
Image captured at night has nearly no color information which leads to the
failure of traditional day time approaches.
At night there is a poor distinction between the object and the background,
that pose more challenges to moving object detection methods based on
analysis the difference between the successive frames.

Challenges (Continued)

Uncommon behavior is not subjective. It is, however, context sensitive. For

example, running could be uncommon in one scenario, while it is common
in other scenarios. So, we need contextual information to distinguish these
situations.

To handle completely dark scenes, thermal infrared camera is required.

But, such cameras are too costly and are thus we need to develop
approaches which work with ordinary surveillance camera.

Objectives

The system should be able to perform in low-light situations during night.

It should detect moving objects and mark it with a bounding box.

It should track the moving objects and provide trajectory information of the
object.

It should detect any unusual activity that happens within the scope of the
camera.

Design

Main Components
Night Video Enhancement
Object detection and tracking
Unusual Activity Detection

Night Video Enhancement

This process consist of 5 steps :
1)
2)
3)
4)
5)

Motion Based Background Estimation

Illumination based segmentation
Illumination histogram equalization
Moving object segmentation
Fusion and enhancement

Block Diagram for the Enhancement Module

Algorithmic Details

Motion Based Background Estimation

A pixel level motion detection method which could identifies each pixel's
changing character over a period of time by frame to frame difference and
analyzes a dynamic matrix D(k) presented in this paper.

Illumination Based segmentation

Here the background image of day and night ( DB and NB ) are
transformed from RGB color space to HSV (Hue Saturation Value) color
space.

Algorithm Details (Continued)

Moving Object Segmentation

First illumination histogram equalization in the night video N (i,j) (V) is
used . Pixels will be classified to M levels according to their luminance.
After that, different thresholds will be assigned for different classes in the
background subtraction.

Image Fusion
Choosing the maximum value of the coefficients of the night input image
and daytime reference background image for the high frequency band. For
the low frequency band, the coefficients of the images are weighted
according to the motion and illumination map.

Object Detection and Tracking

Objects are detected using contrast changes measured by taking

subimage inter frame difference.

Motion prediction and spatial nearest neighbor data association are used
to track objects and give feedback to the first stage

Block Diagram for the Object Detection and Tracking Module

In the figure, I is the image frame, R is the interframe relation which describes the
similarity between two frames, T1 is the contrast threshold and T is the contrast change
threshold.

Algorithmic Details

Visible image content detection based on local contrast computation

The object is detected using local contrast computed over the entire
image.

Moving object detection based on changing contrast saliency

In the second step of the algorithm, changes in contrast
saliency are used to filter the results of the first step and obtain
the moving objects.

Algorithm Details (Continued)

Object tracking
The tracking algorithm used in this work is based on an objects position
and velocity. The data association is realized by Mahalanobis distance.

Unusual Activity Detection

An unsupervised feature-based abnormality detection algorithm suited for
online video surveillance applications is used. The features used in this method
include trajectories, object sizes, and velocities. Unlike the traditional
trajectory-based abnormality detection, we consider both the trajectory-based
information and region-based information.The proposed method has the
advantage of detecting abnormal events that cannot be detected by
trajectory-based algorithms alone.

Block Diagram for the Unusual Activity Detection Module

Algorithmic Details

Automatic Feature Clustering

This feature clustering algorithm is used for extracting the common patterns from
the observed object features. It includes:

Feature Acquisition and Trajectory Preprocessing

PCA Based Spectral Clustering
Automatic Selection of Number of Clusters

Algorithm Details (Continued)

Modelling the Trajectories

Trajectory-Based Abnormality Detection
An online trajectory-based abnormality detection method is used, which
can automatically detect abnormal trajectories that disobey the common
patterns.

Region-Based Abnormality Detection

It includes:
Region-Based Object Information
Region-Based Abnormality Detection

Algorithm Details (Continued)

Combined Algorithm For Abnormality Detection

At each current object position, not only the object trajectory will be
compared with the trajectory model but also the size and velocity will be
compared with the region-based model. Only if all features are matched, is
the object classified as normal.

Hardware and Software Requirements

Hardware
Standard Camera

Software
1.
2.
3.
4.
5.
6.
7.

Python
Anaconda
OpenCV
Jupyter Notebook
Libraries :Numpy,
Scipy
Scikit-learn

Activity Time Chart

Future Work
Improved the Detection and Tracking Module

It is based on salient contrast analysis.

It increases adaptability of the algorithm by applying learning and

improves the effectiveness of detection by analyzing objects trajectories.

Block Diagram for the Improved SCC Model

END