Weather forecasting using BigData
Sandeep Kathula Rohit Surve
Indiana University Bloomington
F16-IG-3010
skathula@indiana.edu
ABSTRACT
Weather forecasting is the application of science and tech-
nology to predict the state of the atmosphere for a given
location. Weather forecasting is one of the most challenging
problems in the world. It has been one of the most inter-
esting and fascinating domain. Scientists have been trying
to forecast the weather using large set of methods, some of
them more accurate from others. For the weather prediction
different types of sensors are being employed in all the places
which collect massive amount of data. This data has to be
processed to forecast the weather. Even radars and satellites
which help in weather prediction produce enormous amount
of data. Different sensor values along with data from the
radars and satellites like temperature and humidity are used
to predict the rain fall etc. When the sensors increase, data
produced by them becomes huge in volume. The rate of pro-
duction (velocity) is very high. Thus, sensor data is a kind
of big data. Big data cannot be processed using traditional
approaches. General computers cannot handle such huge
data. So, we wanted to process such data using leveraging
map reduce with Hadoop [3]. Hadoop is open source frame-
work developed by Apache for processing massive amount of
data. Map Reduce programing helps to process huge data
using parallel and distributed techniques. Processing sen-
sor data with Map Reduce in Hadoop framework removes
scalability bottleneck [4]. Speed of processing sensor data
can increase many times when processed across cluster of
nodes in a network. The main aim of our project is to build
the code for processing huge volume and high velocity of
data from sensors using Map Reduce on Hadoop and an-
alyze that data using statistical programming language R
which will help us to identify the hidden patterns inside this
large dataset so as to transfer the retrieved information into
usable knowledge for classification and prediction of climate
condition.
1. MOTIVATION
Changing climatic conditions around the world are of ma-
jor concern for many countries and through our project we
Indiana University Bloomington
F16-IG-3019
rohsurve@indiana.edu
tend to analyze this quantitatively by predicting this change.
Moreover, predicting weather is useful for making informed
decisions. For example, weather forecast is used to deter-
mine when to schedule important outdoor activities, meet-
ings etc. Predicting the future weather requires us to ana-
lyze the weather patterns from the past many years which
requires huge amount of weather data from last 50 - 100
years. This data cannot be processed by traditional methods
and requires huge data processing techniques like Hadoop.
2. PROBLEM STATEMENT
We, the human beings are trying to predict the weather in-
formally from thousands of years and formally since 19th
century. Weather forecasting is generally done by collecting
huge amount of data about current state of atmosphere and
also from the past at a given place and by using scientific
processes to project how it is going to change. In ancient
days, weather forecast usually based on observed patterns
of events. For example, we used to believe that if the sunset
on a particular day was red, the following day would have
fair weather. This experience accumulated over the genera-
tions to produce weather myths. But not all the predictions
prove reliable, and majority of them have been found not
to succeed in rigorous statistical testing. In the 20th cen-
tury, advancement of atmospheric physics led to foundation
of modern numerical weather prediction. The first comput-
erized weather forecast was performed by a team led by the
mathematician John von Neumann; von Neumann publish-
ing the paper Numerical Integration of the Barotropic Vor-
ticity Equation in 1950. Practical use of numerical weather
prediction began in 1955, spurred by the development of
programmable electronic computers.
To predict the weather accurately we must first know about
the processes occurring in the atmosphere to produce the
present weather for the place which we are trying to fore-
cast. This is done by making certain observations like tem-
perature, air pressure, humidity, cloud cover, wind direction
and speed, precipitation, monsoon etc. As many factors
we take into account, as we do more complete observation
across the earths surface which effect the weather we ex-
perience, the better picture we have about the processes
producing the current weather we experience. By observ-
ing the changes to these factors over time and comparing
changing patterns with the historical patterns, we can fore-
cast future weather conditions. If we can understand how
the atmosphere changes over time in response to various fac-
tors; i.e., differences in warming across the earths surface
from solar radiation, radiational cooling at night, warming of
the atmosphere due to latent heat release during condensa-
tion, etc., and can write mathematical equations to express
these changes, then a useful tool becomes available to the
forecaster - computer models - which can be constructed to
express how the atmosphere is changing and will appear at
some future time. The output from these models can be used
as an aid to forecasters in preparing the forecasts. For ob-
serving changes to the factors like temperatures in predicting
the future weather conditions we need huge data produced
by the sensors from the past and present. This data needs to
be processed effectively to make future predictions. Process-
ing such huge data cannot be done my traditional methods
and by using normal computers.
3. OUR SOLUTION
Different sensors which are employed in all the places can be
years-2016) of same day of some particular station and y
values representing corresponding temperature readings of
that particular weather station. Based on these values will
build a best fitting line or Regression line.
The idea is to establish a below hypothesis function and
substitute the values of x (years) and y(temperatures) in
the established function which will give us a straight line on
the x-y plot.
y= h(x) = 0 + 1 x
Our goal is to determine the 1 and 2 values that forms a
best fitting line through the plot which will calculated by
computing the cost function
1
(h (x)(i) y (i) )2
J(1 , 2 ) =
2m

used to measure weather parameters. Weather forecast de-

This equation gives the mean of squared difference between
partment has begun collecting and analyzing massive amount
a point in the graph and line i.e., the vertical distance be-
of data like temperature. When the number of sensors in-
tween the two. The lesser the value the more accurate is
creases, the volume of the data will increase and the ve-
the hypothesis. We will iterate for range of values of 1 and
locity of data will be high. There is a need of a scalable
2 until the hypothesis that minimizes the cost function to
analytics tool to process massive amount of data. The tradi-
the least possible value is determined. This hypothesis will
tional approach of process the data is very slow. We propose
constitute for the Best fitting line or is called the Regression
leveraging MapReduce with Hadoop to process the massive
Equation.
amount of data. Hadoop is an open source framework suit-
able for large scale data processing. MapReduce program-
With Regression Equation available we will substitute for x
ming model helps to process large data sets in parallel, dis-
value in hypothesis representing year (past or future) and
tributed manner. Processing the sensor data with MapRe-
the curve will give the corresponding y value representing
duce in Hadoop framework removes the scalability bottle-
temperature of that particular curve, which represents a par-
neck. The speed of processing data can increase rapidly
ticular day of a particular station. This algorithm is simi-
when across multi cluster distributed network. This project
larly applied for 365 days of whole year giving 365 different
aims to build a data analytical engine for high velocity,
curves each capable of predicting temperature of the days
huge volume temperature data from sensors using MapRe-
they represent. In this project we will predict for 5 upcom-
duce on Hadoop for forecasting the weather. For predicting
ing years (2016-2020). We will using Map Reduce model for
the weather forecast, we will be experimenting various ap-
computation process to achieve parallel execution and fast
proaches for prediction and further compare the accuracy
processing of the input data.
level from each of the below methods:
3. Sliding Window for prediction: [2] To predict the futures
1. Mean weather calculation using Map Reduce
weather condition, the variation in the conditions in past
years must be utilized. The probability that the weather
2. Linear Regression Algorithm Using Map Reduce
condition of the day in consideration will match the same
day in previous year is very less. But the probability that it
3. Sliding Window for prediction
will match within the span of adjacent fortnight of previous
year is very high. So, for the fortnight considered for previ-
1. Mean weather calculation using Map Reduce: We would
ous year a sliding window is selected of size equivalent to a
predict the weather by calculating mean value of tempera-
week. Every week of sliding window is then matched with
ture from dataset having temperatures of around 100 years
that of current years week in consideration. The window
using Map Reduce and predicting the future temperature
best matched is made to participate in the process of pre-
by using the mean values. The input weather dataset con-
dicting weather conditions. The prediction is made based
tains the values of temperature, time, place etc. In the
on sliding window algorithm. The month-wise results will
proposed method, the Map process will create a series of
be being computed for three years to check the accuracy.
key-value pairs where the key will be the Plain Old Java
Object(POJO), comprising of data fields like place, date,
We will be implementing the sliding window algorithm using
etc., and the value will be the temperature. These key-value
the approaches suggested in the paper to predict weather
pairs will be then shuffled into lists by key type. The shuffled
forecast using sliding window algorithm. Implementation
lists will be fed to the Reduce tasks, which will reduce the
will be done using R.
dataset volume by the values. The Reduce output is then a
simple list of averaged key-value pairs.
4. SOURCE OF OUR DATA
2. Linear Regression Algorithm Using Map Reduce: By us- [1] National Climatic Data Center (NCDC) have provide
ing simple linear regression algorithm, we are planning to weather datasets. Daily Global Weather Measurements 1929-
create a training set with x values representing years (past 2009 (NCDC, GSOD) dataset is one of the biggest dataset
available for weather forecast. Its total size is around 20 GB.
The United States National Climatic Data Center (NCDC),
previously known as the National Weather Records Center
(NWRC), in Asheville, North Carolina is the worlds largest
active archive of weather data. The Center has more than
150 years of data on hand with 224 gigabytes of new infor-
mation added each day. NCDC archives 99 percent of all
NOAA data, including over 320 million paper records; 2.5
million microfiche records; over 1.2 petabytes of digital data
residing in a mass storage environment. NCDC has satellite
weather images back to 1960.

Proposed System used the temperature dataset of NCDC,

GHCN (Global Historical Climatology Network)-Daily. It
is an integrated database of daily climate summaries from
land surface stations across the globe.

5. REFERENCES
[1] National centers for environmental information.
[2] S. S. B. Piyush Kapoorm. Weather forecasting using
sliding window algorithm. 2013:5, 2013.
[3] S. M. V. Riyaz P.A. Leveraging map reduce with
hadoop for weather data analytics. 17, 2015.
[4] T. White. Hadoop The Definitive Guide. OReilly
Media Inc, 4 edition.