Proceedings of the 9th International Conference on n f o ~ t i o n Technology and

Applications in Biomedicine, ITAB 2009, Larnaca, cyprus, 5-7 November 2009

A Simple Algorithm to Monitor DR for Real Time Treatment
Applications
Banitsas K., Pelegris P., Orbach T., Cavouras D., Sidiropoulos K. and Kostopoulos S.
Abstract- As the demand for effective and reliable telecare
systems increases rapidly over the last years, novel ideas
applied on existing consumer products enables the development
of innovative solutions that could enhance the user's wellbeing.
In this research, we are going to demonstrate the potential of a
system that enables users to monitor their own heart beat rate
in real time and use specialised software for personal health
coaching. In this paper we will explain and demonstrate how to
extract heart beat rate information from a user using the
camera of a commercially available mobile phone which will
enable us to supply the users of the system with vital
information and utilize interactive tools useful for personal
health coaching. Our industrial partner Health Smart Limited
have filed a patent [1] for this application, they retain the full
intellectual property of this project.
Index Terms- HBR, camera, mobile, cbt, health coaching
I. INTRODUCTION
T
HE rapid evolution of technology has given us with the
opportunity to gradually provide access of cost effective
sophisticated electronic devices to the public, currently most
people can buy a smart phone with a camera for a reasonable
cost. From an engineering point of view this creates
additional possibilities of application deployment and
service delivery: a modem mobile phone gives you enough
processing power in conjunction with mobility while it can
also be used as a media platform for increased user
interaction [3],[8].
Our industrial collaborator, Health-Smart Limited has
substantial experience in self-care ICT for prevention and
controlling Long Term Conditions (LTC) including
cardiovascular and psychological conditions, one of their
purpose is to empower patients to assess their state of body
and mind and train them to improve their health and prevent
Long Term Conditions (LTC), by using inexpensive friendly
consumer ICT and sensors [2]. We have developed a
demonstration based on an invention - patent they had filed
Manuscript submitted 21th August 2009.
K.Banitsas .P. Pelegris and K. Sidiropoulos are with the Electrical and
Computer Engineering Department, Brunei University, West London,
Uxbridge, Middlesex, UB8 3PH, UK (phone: +44(0) I895266886;(e-
mail:{konstantinos.banitsas,panagiotis.pelegris,konstantinos.sidiropoulos}
@brunel.ac.uk).
D. Cavouras and S. Kostopoulos are with the Department of Medical
Instruments Technology, Technological Educational Institute of Athens,
Ag. Spyridonos Street, Egaleo, 122 10, Greece. (email
{cavouras,skostopoulos }@teiath.gr).
978-1-4244-5379-5/09/$26.00 2009 IEEE
[1] to extract heart rate (HR) information from a user, using
mobile phone with a camera (without any other sensor).
The importance of this invention is that it enables anyone
to use a standard mobile phone with a camera as a heart rate
monitor and health coach. In this research the delivered
service is actually pre-emptive in character and aims in
enhancing the well-being of the users by providing them
with simple tools, methods, information regarding the state
of their heart and coach them on how to improve their
physical and emotional state [9]. In the proposed system we
are examining the usage of an embedded camera in a mobile
phone or PDA as means of measuring the heart rate (HR).
This will help us provide near real time feedback to the users
and will assist them in using self coaching software to
enhance their well-being.
Although there are numerous researchers active on
working with mobile phone and PDA's implementing
healthcare systems the vast majority of them use the mobile
phone and PDA as a processing and media interface unit (or
communication device) and never as a sensor by itself. In
each and every case the user has to wear some kind of sensor
unit that enables a device to monitor the user's physiological
signs. The less intrusive a solution is the more chances it has
to be accepted by people [8]. A touch screen mobile phone /
PDA with a simple interface that has no need for extra
wearable sensors might prove to be more successful.
The proposed system is compatible with most of the
commercially available modem mobile phones with a video
camera. Basically the camera is used to capture a video from
the users' fmger and also as a media interface that will
generate the feedback and provide them with useful self
coaching advice that can improve their well-being [5]. In
order to do that they just need to press their fingertip on the
camera while the PDA device captures a short length video.
When this is over the device will analyze their heart beat
information and provide feedback for the actual heart beat
rate. Following that personal health coaching software takes
over to utilize this information for an actual pre-emptive
health service. For example software can teach the users
how to breathe properly and help them regulate their heart
rate. The process can be repeated over a period of time so
T. Orbach is with Health-Smart Limited, 77b Fleet Road, Hampstead,
London, NW3 2QU, UK (e-mail: Tuvi@health-smart.co.uk, www.health-
smart.co.uk).
that users can check their progress and see immediate results
on heart beat self regulation. In the future it will be possible
to analyze the HR continuously and provide feedback and
coaching to the users in near real-time.
The key challenge here is to engage the user in a process
that will enhance his well-being providing him with
personalised health coaching, delivering effective pre-
emptive health services. In order to succeed in that you need
an easy to use interface, a system that requires no extra
effort like wearing sensor units and a way to provide quick
feedback for the users enabling them to check the effects of
their "treatment" right away [4]. The user friendliness of self
monitoring equipment is crucial for the adoption and success
of it. Familiarising users with such solutions will help them
adapt to more complex systems in the near future. The
ongoing standardisation on telecare and Telehealth will soon
start producing practical health monitoring systems based on
multiple sensor readings.
This research was designed to be carried out in two
stages: firstly take the input from a PDA and do the
calculations on a PC and finally implement the whole system
in real time in a mobile device. In this paper we demonstrate
the process in a desktop environment using video files taken
from a PDA Mobile phone (Eten Glofish X800) as input.
II . METHODOLOGY
The invention is based on two interesting phenomena:
I) Every heart beat creates a wave of blood that reach the
capillaries in the tip of the finger; when the capillaries are
full of blood less light can pass through them. So the
changes in the amount and the colours of light which is
passing through the finger can represent the changes in the
shape of the pulse and its timing (the HR).
2) Normally to take a photo or a video we need focus and
some distance between the lens and the object. However we
are interested only in the quantity of light, therefore there is
no need of focus, and it is possible to cover the object lens of
the camera with the finger tip as shown in Figure 1 . As long
as there is a source of light (which can be natural ambient
light or artificial light) which can pass through the finger, it
is possible to view the pulsation of the blood as changes in
the amount of light in the video.
The major advantage of this invention is that the user does
not need any sensors, does not require any external hardware
whatsoever, it does not require even to focus; one just
touches and cover the lens of the camera with their finger tip
and with the right software they can view their HR and learn
to improve their health and fitness. It was worth to examine
if one can discern any information about the pulse wave
when placing the finger directly on top of a camera with
strong flash light right next to it.
Figure 1: Pressing the finger on top ofthe camera while the flash is
turned on
The initial findings where very encouraging, as it is
illustrated in Figure 2 the red portion of the finger is clearly
visible and easily distinguishable from the rest of the picture,
unfortunately in this paper we cannot visually demonstrate
the fluctuation from the pulse wave as it only appears on
video but it is important to note that the resolution provides
enough information to estimate the heart rate as long as the
finger is kept right on top of the camera.
Figure 2: Sampleframesfromfinger images on PDA camera
arrows denote the pulsating regions.
In order to develop the system we used both desktop and
mobile environment applications for testing. The PDA
Mobile phone used was an Eten Glofish X800. The
resolution used was 320 x 240 pixels with an effective frame
capture rate of 25 frames / sec. Although these are not the
current highest available characteristics for a mobile phone
with a camera they proved to be adequate for our
experiments [6].
240 - -- -- --------' -- - - - - - - - - - -
500 600
Frames
400 300 200 100
20
0
'--- ----'.,,----- --,-'-:--- ....,-L,- - --'-- - -'-- --'
---_. _... ------- ---- ------- ----------
50 -----------t -----------t- --- ----- --------- -- \---- ---- ---\----------
40 ------- --- r---------r---- ----- ------ ------1 ---------- -1--- ---- -- -
30 ------ -----f--- --------f-- --------- ---- -------j--------- --j----------
, ,
, ,
, ,
, ,
, ,
, ,
Normal ised Amp
8Or-- ----r- --.------,--- --,-- - -,---,
70 -------- --- --- -------
200 250
Verti cal Position
150 100 50
180
0
'---- -::'::-- -----'.,,----- - --'-:- ------'-- - ----'
220 - - - - - - - - ~
200 - ~
Red Channel
2601 - -----;-----;= :::::;::::= = =:== = =:== =-1
Figure 3: Frame Profile with virtual vertical axis along the middle Figure 4: Crude pulse signal, illustrating unexpected dive
The system's behaviour was initially simulated on Matlab
using videos captured from the camera on the PDA as input.
For each video the frames were extracted and a profile was
created for every frame. We deliberately scanned the profile
of a virtual vertical axis right in the middle of the frame
calculating light variation along this axis. The same
procedure was repeated for every frame. However this is
only one method to calculate the changes in amount of light.
Figure 4 represents a crude preview of the pulse
variation, notice how the signal dives at approximately
frame 210 indicating a possible sudden movement of the
finger or an external source of noise, however this
unforeseen development does not considerably affect the
outcome of the calculation as the information we are looking
for is basically contained in the frequency of the signal
rather than on its amplitude.
500 600
Frames
400 300
-. _. _; - - - - -- -- t -
, .
, .
, .
, . ,
, . ,
-------_..-._---- ---- .-----------.----------
200 100
-3
0
'--- ----'.,----- --,-'-:--- ----'-- - --'-- - ---'---- ----'
, ,
, ,
-0.5 - - - -- --- -f- - --------f
, ,
, ,
, ,
, ,
-1 ------ - --- -; -- ---- - - -- - ;
,
,
, , , ,
-1.5 ~ ----------: - - - - - - - - - - - : - - - - - --- - - - : - - -- - -----
, I , ,
, I ,
I I ,
-2 -----------f------ ---- -f-- ---- -----i---------- -f--- --------f------- ---
I ,
I ,
, I ,
-25 - - - - - - - - - -- r--- -- -- -- --r -----------i-----------i---- --- -- -- [-- -- ------
1 -----------f------- ----f-
, ,
, ,
, ,
, ,
0.5 -- -- -- - ' --- ~
Normal ised Output
1.5 r-----,-----r- - -.------,-----,------,
The next step is to normalise the crude signal into
something more meaningful , which is easily achievable
using smooth differentiation. From this plot it is simple to
get the number of peaks, which divided by the running time
of the video will give us the estimation for the heart beat
rate. In the given example the result was 61.3 beats / min as
shown in Figure 5.
Figure 5: Normalized signal
Our technique is based on determining red channel
variation along a given area of the image. This represents a
good estimation on how much light is getting through. The
process is followed for all frames and amplitude is
normalised giving the crude pulse signal as shown in Figure
4.
An alternative technique would be to process data from all
channels in RGB mode, or take multiple measurements from
different areas of the image to reinforce our result accuracy.
However in this work we mostly aim in demonstrating the
efficiency of the algorithm in this application as the fine
tuning is a process that will be explored in the development
of the final product.
Another issue is that we can directly scan the red channel
without any other type of conversion ensuring that our input
data goes through a minimum of transformation and
compression stages giving us the maximum possible
efficient information.
Moreover since our target platform is a mobile device
with limited resources both in terms of processing power and
memory, it is reasonable to get the input algorithm fairly
simple as long as the results appear to be highly accurate.
III. RESULTS
After the simulation stage, we needed to evaluate
accuracy of the algorithm using real examples and think of
ways to improve performance. A custom software
application was built to process the frames from the video,
scan for colour variations, locate peaks and calculate beats
per minute.
It appears that videos of 20 seconds produce results with
around 91% average accuracy, with minimum 85% and
maximum 99% compared to the measured HBR values.
Samples of 14 - 16 and 18 seconds have an average
accuracy of 89%, the minimum was 81% at 14 second
samples and the maximum was 100% at 16 second samples.
The results are summarised on Table. 1.
Fnlmes 400 16 Sec ID FaslMode
Frames : 400
The samples of 12 and 10 seconds have only 87% and
86% accuracy respectively, that is more or less anticipated
since when calculations are based on data from 10 seconds,
any error would appear to be six times higher when you are
estimating beats per minute. A simple optimisation that we
will incorporate in future versions is the sharp definition of
start and end time based on the actual beginning or ending of
a pulse. This would directly result in an improvement of up
to 12 pulses for a 10 second video in some cases, this change
might make it possible for shorter streams to give one
enough data for reliable analysis.
The flexibility of the system lets the user decide on the
required resolution, for example a sample of 20 seconds
would produce results with a resolution of 3 beats. If a user
for example has 25 beats in 20 seconds based on that sample
he has 75 beats in 60 seconds. If the same user had 26 beats
in 20 seconds then the software would calculate 78 beats in
60 seconds. Similarly samples of 10 seconds would have a
resolution of 6 beats, being far less reliable since only the
integer portion of the beats is used in the calculations. The
sampling time effectively gives you the flexibility to define
the desired resolution based on your needs.
Figure 6 illustrates the demonstrating application in action
while calculating results based on a 16 second sample; this
process doesn't take more than 3 seconds on a 2.4 GHz
processor. The frames are extracted from video files into
PNG images using FFMPEG and are later processed by the
application to produce the actual results. During testing we
found out that PNG images give much finer detail when it
comes colour compression compared to the standard IPG
images. The sources files from the X800 are in 3GP format
and frames are extracted directly in order to avoid the
detrimental effects of subsequent recompression stages into
different compression schemes.
PlAses : 75
Figure 6: Sample desktop application to calculate pulses
In the above example the actual beats per minute where
76 and the application calculated 75 in a 16 second sample.
In this case the 16 second result was more accurate
compared to the 20 second result which was 78 but in both
cases the results are within the given resolution for 20
second samples which are 3 beats per minute.
Table. I
Measured and Calculated HBR for different sampling times
I s.mplelHBR I Me.sure<l ICOIC.1Osec COIC.1Zsec Ie. C.l4Sec COIC.l6Sec ICOIC.lesecICOIC.lOsec
69 60 65 68 67 63 83
78 60 60 64 63 66 66
3 78 66 65 68 67 70 72
4 75 72 70 68 67 70 69
5 74 60 60 60 60 63 69
6 79 72 75 77 78 76 78
7 72 60 60 60 63 63 63
8 75 66 65 64 63 63 66
9 75 66 70 77 75 73 72
10 78 66 65 68 67 66 69
Table.l summarizes the heart beat rate calculation results
for 10 random video samples. This is still an early stage on
the development of the algorithm and it appears that a few
minor adjustments will have significant impact.
Table.2
Relative HBR accuracy percentages
Sample/Accuracy 10sec 12sec 14sec 16sec 18sec 20sec
1 0.87 0.94 0.99 0.97 0.91 0.91
2 0.77 0.77 0.82 0.81 0.85 0.85
3 0.85 0.83 0.87 0.86 0.90 0.92
4 0.96 0.93 0.91 0.89 0.93 0.92
5 0.81 0.81 0.81 0.81 0.85 0.93
6 0.91 0.95 0.97 0.99 0.96 0.99
7 0.83 0.83 0.83 0.86 0.86 0.88
8 0.86 0.87 0.85 0.84 0.84 0.88
9 0.88 0.93 0.97 1.00 0.97 0.96
10 0.85 0.83 0.87 0.86 0.85 0.88
IV. DISCUSSION
As life expectancy increases over the last decades so does
the burden on the health system supporting people with
chronic conditions, the only way out of this is implementing
telecare solutions that will manage to increase the quality of
delivered health care while maintaining low installation and
running costs [7].
It is not only that health services delivery will change, but
also that the nature of the service itself will change shifting
from re-active treatment of conditions to pre-emptive health
care. Avoiding health risks would be more efficient than
sustaining patient's with chronic conditions that could have
been avoided [2]. This is where health monitoring and
cognitive therapy comes into play, providing the users with
information on how to avoid getting a health condition rather
than focusing on how to treat it.
Biomedical data acquisition will also playa major role on
applications where monitoring physical condition and
alertness will be critical in respect to safety minimizing
associated risks in sensitive and high risk areas like
aerospace and aviation. Cross disciplinary synergies in the
past have produced important advancements in the area of
unobtrusive multi sensorial data acquisition systems [10].
Such platforms can provide invaluable input to self coaching
health systems that will aim in enhancing well being.
The challenge remains both for researchers and the
industry to agree on standards, engineer innovative solutions
that will cover the needs while minimising the cost and take
telecare to the next level. This application is a subsystem of
a telecare platform that is being developed in collaboration
with our industrial partner Health-Smart Limited, as a stand-
alone system though it may serve as a media platform to
deliver cognitive therapy treatment and positive psychology
advice and practice.
Future work includes migrating the application and
algorithm on mobile devices and including it in a larger
software package that will deliver health-smart solutions.
This includes further optimisation of the code to improve
accuracy and increase tolerance to noise sources and
combine the end result with interactive "game-style"
applications that will run on a PDA in the context of
personal health coaching.
V. CONCLUSION
In this work demonstrated the efficiency of an innovative
algorithm to detect heart beats per minute using inexpensive
widely available hardware. The objective of this research is
to provide a simple and fairly accurate tool for pre-emptive
health services to be delivered on patients using mobile
devices.
Furthermore real time feedback information for a user will
greatly improve his general feeling about using new modem
technology in order to maintain well-being or improve
health through specialised self coaching.
The invention will be used in the near future as a
subsystem for a self help application that aims to help users
reduce their stress levels and improve their well-being.
ACKNOWLEDGMENT
We would like to express our appreciation to Health-
Smart Limited for its important contribution to this paper
and the general research that resulted in a patent application
[1].
REFERENCES
[1] PCT patent application No. PCT/GB2009/050989 Blood Analysis -
Health-Smart Ltd.
[2] T. Orbach and 1. Vasquez, "Self-care and the need for interactive
ICT", Journal of holistic healthcare, vol. 6, pp. 35-39, August 2009.
[3] R. Gururajan, S. Murugesan and 1. Soar, "Bringing Mobile
Technologies in Support of Healthcare Recommendations for a
Healthy Beginning and Growth," Cutter IT Journal Article, Aug.
2005.
[4] A. Duckworth, T. Steen and M. Seligman, "Positive Psychology in
Clinical Practice," Annual Review of Clinical Psychology, vol. 1, pp.
629 - 651, April 2005.
[5] M. Seligman, T. Steen, N. Park and C. Peterson, "Positive Psychology
Progress : Empirical Validation of Interventions", American
Psychologist, July 2005.
[6] M. Fischer, Y. Yang Lim, E. Lawrence and L. Ganguli, "ReMoteCare:
Health Monitoring with Streaming Video," in 7th International
Conference on Mobile Business, July 2008.
[7] V. Shnayder, B. Chen, K. Lorincz, T. Fulford-Jones and M.Welsh,
"Sensor Networks for Medical Care," Technical Report TR-08-05,
Division of Engineering and Applied Sciences, Harvard University,
2005.
[8] A. Pantelopoulos and N. Bourbakis, "A Survey on Wearable
Biosensor Systems for Health Monitoring," 30th Annual International
IEEE EMBS Conference, Vancouver, British Columbia, Canada,
August 20-24, 2008.
[9] H. Jang, S. Kim and C. Bae, "Personalized Healthcare through
Intelligent Gadgets," 30th Annual International IEEE EMBS
Conference, Vancouver, British Columbia, Canada, August 20-24,
2008.
[10] A. Astaras, PD. Bamidis, C. Kourtidou-Papadeli and N. Maglaveras,
"Biomedical real-time monitoring in restricted and safety-critical
environments", Hippocratia, vol. 12, suppl. 1, pp 10 - 14, August
2008.