Probability Foundations for Electrical Engineers

Prof. Krishna Jagannathan

Department of Electrical Engineering
Indian Institute of Technology, Madras

Lecture - 1
Introduction

Welcome, this is Probability Foundations EE5110, Probability Foundation for Electrical

Engineering. So, this is a post graduate level course on probability theory. So, this course
will essentially, it is a somewhat triggers treatment of probability theory. I guess, you
could say it is a more regulars treatment of probability theory. Then, engineers are
normally use to. So, in this course we will take an axiomatic view and focus more on
deriving fundamental theorems and proving results.

Rather than, more under graduate like treatments, where you are probably the focuses is
more on solving problems. And computing expectations and computing probabilities and
so on. So, at that level, so you can think of these courses, a more conceptual course.
They would be more emphasis on proves, more emphasis on rigorously deriving things,
under greater degree of conceptual understanding.

So, this course, so who should take this course. So, if you think about it in that way, is it
right for you. So, these courses meant for people who essentially more mathematically
minded. So, people who will benefit from eight or the once, who will probability to more
mathematical research. So, if you are working on topics like ((Refer Time: 01:33))
computer networks or caustic control or machine learning, where you need a somewhat
strong foundation, conceptual foundation in probability.

This course will be very useful for you. So, this course EE5110 runs in two different
avatars. It is offered in both semesters, both odd and even semesters. It has the same
number, but the version that is offered in the even semester is more operational. It is
more computational, operational and there is more emphasis on problem solving. So, if
you are more practical person, who has greater use for problem solving and computing
things and so on and not so much.

If you are not so interested in really getting into the nuts and bolts of probability theory,
may be the even semester course may be more appropriate for you. So, this is a call that
you may want to take. Whether you want to take it in this semester or during the even
semester. It is call the same thing, but this version will be more theoretical, more
conceptual. So, for example, this semester we with, this is the measure theoretic version
of the course.

This course will have measure theory in it. And the even semester version, although it is
call the same thing, it does not have measure theory in it. It is a much more slightly more
elementary treatment, conceptually. Although there will be more emphasis on problems
and computational aspects of probability. So, you may want to decide which one you are,
which one is for you in terms sets. You can always take this call in a few weeks, I guess.

So, I am Krishna Jagannathan, Electrical Engineering Department. That is, where you
contact me. So, the course has a website. This is, if you go to my web page, you can find
the link to this web page. This link write here, has list the entire course contents. So, all
the topics we will cover are in fact given here. So, I suggest you visit this today.

(Refer Slide Time: 03:24)

So, reference material. Let me write down here. So, there is no one book out there, which
closely follows what we are going to do. So, there is a number of materials that I am
going to suggest. So, you do not have to buy all these books. So, what I would I think,
the book that you should probably consider seriously buying, is Grimmet and Stirzaker.
This is probably, most well matched with the kind of a... This is a very good book, it is a
very classic book.
So, this is something you may want to consider buying, Stirzaker 3rd addition,
Probability and Random Processors, Oxford university press. It is an expensive book and
it is about 2800 rupees and so. It is an excellent book though. So, if you ((Refer Time:
04:30)) this is be a standard reference. And there is another open source reference, which
I will follow quite a bit. It is a MIT OCW Open Course Where. There is a link on the
home page.

So, if you go to the web page there is a link. So, this is also very useful source. It is a, we
will follow certain lectures from there. It is a good source. So, these two are roughly at
the level at which, we will be doing this course. And the other text books, which are
either slightly at more elementary level or at a more advance level. So, I will list of
couple of them also. So, that is a book by Bertsimas and Tsitsiklis.

This is rightly more UG level. So, an excellent book Bertsimas and Tsitsiklis,
Introduction to Probability. And there is David Williams, Probability with Martingales.
This is a beautiful book, but it is more advanced than what we will need, what we will
cover. But, there are certain topics here, which are just beautiful. So, this one and the one
by Rosenthal, these two books are more advanced, David Williams and Rosenthal.

They both more advance than what we will cover. But, they may be a few results and
few thing, that I make refer to these books. But, many case both these books are
beautifully written very, very good books and more advance probability theory, there is
been. So, this is what, so roughly this is really what mostly you need focus on. These two
references are enough, I think. And these are occasionally, I think it will be occasionally
useful.

There is also an effort going on by students, from previous years, to actually latex the
notes from the previous year, so of this class. So, that I mean as I mentioned, there is no
one book that covers all the material in one place. So, we difficult there it may be good
idea to just put everything down on late take. So, students from previous years have
actually collaborated and formed a group to latexed the notes from previous years.

So, those notes are they, the first round of the, they editing is over. And so I will make
them available to you.
(Refer Slide Time: 07:30)

So, I will put them. So, class notes latxed. Let me write the spelling latexed class notes,
to be uploaded periodically on modal. So, these notes will be uploaded from time to
time. So, thing is these notes are I think they are all right. They not it fully polished. So,
you have to realize that, they are just coming out of the press. So, they may be some
minor errors and they may not be very polished.

So, I give them to you. But, just remember the Kavya. That, they may be an occasional
mistake or you know, they may be some errors or bugs or type version there. If you find
any bugs or errors, please let us know. So, that the idea has to improve these notes as we
go along. It is like a collaborate away effort to get it done. So, we will build up from
basic. So, there is nothing that I am going to really assume other than some basic real
analysis concepts.

So, we will really be starting from the very basic stuffs. I will start, you know the real
course material I will start on, from the next lecture. Today, I just want to get some sense
from you, on what you think, why we study probability. And what it is that, probability
theory does, why are you, why are we interested in it. So, can I get some of your feelers
form you? Why you think? What is probability theory and why people study?

Student: ((Refer Time: 09:00))

So, to study non deterministic events, this is that convincing answer for you?
Student: ((Refer Time: 09:13))

Mathematical tools is to study experiments with uncertain outcomes. So, it is basically

what it is. So, if you want a very concise way of looking at a probability theory is a.... It
essentially is the science behind randomness. It is the science of randomness. So, there
are in real life, we know that there are so many events that we encounter, which we do
not seem to have a perfect control over or perfect ability to predict or a perfect
knowledge of...

Things, such as toss of a coin or toss of a die or what the weather, it is going to be
tomorrow or whether a you know, whether a child is going to be male or female. These
are things that, we do not seem to have control over or we do not seem to have complete
understanding over. And so, these are... But, if think about these random events, so call
this random or non determined events, is that. There is a larger pattern to it. That is a
central point.

Although apparently the one time, I toss a coin have no idea whether, it going to be
heads or tails. Or I have no idea whether, I really can tell the temperature tomorrow.
There is a larger pattern to these things. So, you know for example, let although you do
not know the sex of the baby that is going to be born. On an average, half of them are
male and half of them are female. So, roughly if you toss a coin million times, roughly
half of them turn out to be heads.

So, these are certain larger large scale pattern or you know long term certain patterns or
what may be call. I guess, pattern is the right word. So, patterns that, one observes in this
seemingly random events. And probability theory tries to quantify these aspects. It is a
theory that helps us to mathematically capture the pattern behind the seemingly random
events. That is really, what probability theory does.

Now, this you know these games of chance have been played, even since in the ancient
civilizations. So, it is been around for a long time. And actually, even for a last several
centuries people have been computing odds and winning bets and so on. So, I guess for
few centuries, people have been computing probability at some level or the other.
Although, the mathematical theory of probability the more regulars foundation of it,
where laid only about a century ago.
So, the probability theory as we know it today, is only about a hundred years old. It is
primarily, the primary person the mathematician who primarily contributed to it, to the
modern theory of probability. What do you know?

Student: ((Refer Time: 11:59))

It is Laplace made contribution. In fact, Laplace was a yes. It did not make significant
contributions about. That is about three centuries ago, I guess. But, more I said modern
theory of probability is only 100 years old. So, very famous mathematician.

Student: ((Refer Time: 12:18))

As a great Russian mathematician by name Andrey Kolmogorov. So, he is the father of

modern probability theory. So, essentially he cast probability theory as a... He realize
there is basically as a special case of, what is known as measure theory, which was
developed by two French mathematicians primarily, Boral and Lay beg. And then of
course, last 100 years there will be an explosive development of this probability theory.

So, what we will do is, in fact this axiomatic modem probability, axiomatic approach.
So, that question arises, why do we need it. So, even very before Kolmogorov, people
have been computing probabilities. There may people have been computing there,
chance of winning bets. So, chance of winning bets and so on. So, why do even bother,
why do we need this more sophisticated theory or more rigorous mathematical theory. If
you can compute probability, you can.

That is all, we need. Practically, that is why we need. If I tell you that probability of a
certain event is, you think that. So, let us say if the probability of an event is 1 by 3, why
it occurs, roughly 1 over 3 times. And we know, this is all we really need in practice. See
the reason that there is this. The people have to bother to make this rigorous
mathematical theory, is because before Kolmogorov, people have been running into all
such of paradoxes and contradictions.

Now, because without have proper theory, you run into all such of problems. There are
because, if you all only go by a certain intuitive understanding in convenient to
difficulties. To illustrate this, those to illustrate the importance of an axiomatic theory,
there is a very many paradoxes. Let you can come up with, if you not careful about doing
this probability theory in a rigorous way. So, there are many paradoxes that you can run
into.

So, what I will do is, I will just describe one such paradox to you, very famous paradox.
And that will hopefully send the message across to you, on why need to be little more
careful than just a normal intuitive understanding of probability.

(Refer Slide Time: 14:51)

Let so this paradox, it is a well known paradox. So, you take a circle. You take a circle
and you inscribe a equilateral triangle in it. That see, this is the center. So, you have
circle. And then inscribe equilateral triangle. So, let say this is radius. Let say the radius
is just 1. It can be r or 1. And so, if this is 1 then this is also 1. And then this will be. So,
this side will be square root of 3. Side of the triangle would be square root of 3, get it.

So, the question is the following. So, take a circle and draw a chord uniformly at random.
What is the probability that, this random chord is longer than square root of 3, which is
the side of the inscribe equilateral triangle. So, the question is you draw. You basically
close your eyes and draw a chord. And the question is. So, you draw chord. So, this is the
length of the chord. The chord may like this or chord may like that.

So, in this case this will be the length of the chord. So, if you pick a chord uniformly at
random, what is the probability that the length of the chord in this case. This in this case,
that is longer than the side of the inscribe equilateral triangle or rigorous mathematical
theory or. What is the probability? That, this is larger than square root of 3, which is the
side of 3 equilateral triangle. So, this is a question.

So, it turns out that. This is the reason. This is a paradox, is depending on how you look
at it. You get different answers to this question. So, there is in fact, you will there are
three perfectly reasonable sounding arguments, which give you three completely
different answers. So, the first.... So, let me see if I remember this correctly. So, the first
way of seeing this. So, you want the chord to be longer than this side.

So, what happened if you draw the in circle of this triangle. So, this guy is an equilateral
triangle. Let us say, you draw the in circle of this triangle. And if it so turns out, that the
chord you draw, let us say you draw the chord. If it is so turns out, that the midpoint of
chord is inside the in circle. You can show that, it will be longer than square root of 3.
Let so, in this case. In fact, the center of the chord is inside this in circle.

And you can see that, this guy is longer than square root of 3, where as if is a chord like
that and the midpoint is a outside the in circle, it will be shorter than square root of 3. So,
you may argue here that. So, in this case, so this is a first possible construction. In this
case, the probability that your random chord is longer than square root of 3, is simply the
probability that, the center of the chord falls in side, this is a little circle, in circle.

So, essentially you looking at.... So, the question you really looking at is, where is the
center of the chord. If it is inside the in circle, you are longer than square root of 3. If it is
outside of in circle, it is shorter than square root of 3. So, essentially you looking at the
probability that, the center of the chord is inside, that in circle. Now, you know that the
radius of this in circle is how much. This is half, because this is the centroid. And so, this
divides into 2 it to 2 to 1.

So, this radius is half. So, the probability that your center of the chord falls into this
circle, is how much. If it is uniformly at random then it should be area of this little circle
divided by the total area. So, in this, so the midpoint of the chord, so if we take the r d m
about the midpoint of chord. So, the answer you get is 1 by 4. See why? Because, you
are looking at the point, the midpoint inside the radius of the circle of radius half
whereas, the whole radius is 1.
So, area of this is of course, one fourth the area of the bigger circle. So, this is 1 answer.
This is the perfectly reasonable argument. The trouble is that, I mean you probably say
this is the answer and that is it. The trouble is, that there are also other perfectly
reasonable sounding argument, which give you other answers, different answers. So, the
second let say the second argument goes is follows. Let me erase this.

(Refer Slide Time: 21:45)

So, let us say that you fix. Actually, get out this triangle as well. Another way to draw a
chord uniformly at random, it is to just fix one and half the chord, wherever you want.
Let us say, you fix it here. And consider, so let say this is the tangent at that point. This is
one and half the chord. And you can draw the chord like that or like that. Now, whether
or not this chord is longer than square root of 3 or shorter than square root of 3 will
depend on.

So, if the here is where. So, if I draw an equilateral triangle from that point, it is look like
that right. So, it depends on the angle that this guy makes with the tangent. So, if this
chord is making an angle of. So, this equilateral triangle makes an angle of 60 degrees pi
over 3. If this angle is less than pi on 3, I will be shorter. By equivalently, the angle is
greater than 2 pi on 3. If it is like that will be shorter, say the chord be shorter, whereas if
the angle made with this, a tangent is between 60 degrees and 120 degrees pi on 3 and 2
pi 3.
My chord will be longer than the side of that triangle, make sense. So, essentially what I
am doing is fixing one end of the chord. And then looking at what angle it makes with
the tangent. So, if from that point of view, it looks like. So, if this angle is uniform with a
random chord. So, it does not prefer prepare any particular direction. So, if the angle is
uniform then the probability that your longer than square root of 3, simply the probability
that this theta, is laying between 60 and 120 pi on 3 and 2 pi on 3.

Student: ((Refer Time: 23:38))

What is that probability equal to? 1 by 3, because well this angle is uniformly at random
chosen, if at random. So, if you make this angle with tangent, this argument you get
answer equal to 1 over 3. Already, you have two different answers to the, what seems
like a same question. I will just argue that, I am not made any mistake. I am not cheating
you with any some simple.

This is not any mistake that is going on. I have completed it correctly and getting two
different answers. Actually, there is one more way of getting different answer, all
together. And that the argument is as follows.

(Refer Slide Time: 24:35)

So, there is a triangle. So, you take one side of the equivalent triangle. And you draw that
perpendicular. So, what you doing is now. You are fixing the... So, you are fixing the
direction of the chord. And you are just going to move the chord, up and down. So,
whichever angle you want, you fix the angle of the chord. So and you are just going to
move it up and down. And you are going to draw the equivalent triangle parallel to it.

The side of the equivalent triangle be parallel to it. So, now if you see, if you are only
going to draw chords, which parallel to this guy. It does not have to be horizontal. It can
be any other direction. I will just flip up the, I will just rotate the equivalent triangle. So,
that the fact that, this is horizontal is not big deal. So, you can see that if you draw chord
like that, it will be longer than square root of 3.

And if you draw chord like that, it will be smaller than square root of 3. So, what we are
seeing is. So, if you take that radius, the probability of the chord being longer than square
root of 3 is simply the probability that is lying above this point. The center of the chord is
lying above that point. So, if you are above this point, you are longer and you have
below, your shorter. So, if you look at this radius and if you think, if you let say that the
center of the chord is uniformly distributed on this radius.

Then, you are looking at... So, this is midway. So, this is midway between this and this.
So, you would conclude that, the probability the required probability is half, 1 by 2.
Because, this is, in this length you are longer and this length you are shorter. So, in that
case you will get. So, if you look at, what should I call this? So, distance center of circle
and center of chord. So, that distance is uniformly distributed, because it is not.

It does not prefer any particular radius. You will get the answer is half. So, there are....
So, it is what was posed in English, as seemingly well post question. It has laid to three
perfectly reasonable sounding answers. There is nothing. They have not made any
mistakes here. So, there is no cheap error. It is actually a deeper problem going on. Now,
the question is, what is happening? So, there are no paradoxes.

There are all paradoxes, if at all you want to be consistent. You have to have a
resolution. There should not be this kind of paradoxes, in there any theory you create. So,
what is the resolution? Would you have any, you already know or do you have any
guesses. So, this is a same question. So, I am taking the same English question. And it
translating it to mathematical languages, I have derived it in three different ways with
three different answers, seemingly correct all three of them. So, that question is a same
question.
Student: ((Refer Time: 28:36))

So, I did quite understand what is said but I heard the word sample space. So, that is an
important concept. Any other, guesses?

Student: First method. Actually all the points are not equally likely even. Like the center
point can have n codes, infinity number of calls going to that.

Well all points can have infinity many points which is call going to that.

Student: ((Refer Time: 29:11)) May not be infinite number of course going. Infinite
number of course will be there, but as send a point as that point, it would not be there
infinite.

That is not true. Each point you will have infinitely many possible chords. That is not the
answer reason. So, there is something... So, this is not some cheap error. This is not some
little mistake that, I have made somewhere. So, it is actually in trigging question. Why
this is happening? It is actually a slightly non trivial explanation. So, the one word I
heard is this about sample phase, which is a very important concept that we will study.

So, to state it in plain English, we are getting three different answers, because you are
answering actually three different mathematical questions. So, what I post as a plain
English question, which I here wrote as random chord, which in chords. So, we are
getting three different answers, because we are actually they are answers to three
different mathematical questions. So, the questions are different, the answers can be
different.

It seems like they are the same English question. But, mathematically actually they are
answers to three different questions. In more formal terms, the sample phases of the
probability phases involved are in fact, very different in the three cases. I will of course,
define these terms more carefully as we go along. But, in this case the midpoint of the
chord being within the circle, there the sample phase.

We are looking at the center of the chord. And there is sample phase, we are looking at is
the whole circle. And we are looking at uniform distribution of the center of the chord,
within the circle. Similarly, in the second case we are looking at the uniform distribution
of the angle, that it makes. So, although it seems like there all uniformly drawn chord,
mathematically we have sold three different mathematical problems.

So, the question the mathematical questions under probability spaces, the sample spaces
behind each of them are in fact are different. They are not the same it is not the same
problem. We are getting three different answers, because we are answering three
different questions, in mathematical terms. Is that? So, which is why we are getting three
different answers? So, it is actually not what seems like the same English questions are
different mathematical questions, corresponding to three different sample spaces.

Therefore, you get three different answers. So, which is why now? So, this while I give
this example is just to warn you or give you Kavya that, if you are generally lose about
these things. So, if you generally say random chord without really, mentioning what the
sample space or what the underlining probability spaces. You can get into all such of
confusions. You can get any of these three answers.

May be there are more answers, you can get if you. For some other problems, you will
get different answers also. So, I hope this as kind of slightly open your eyes, at least to
the possibility that. You have to be a bit more careful in talking about thinks, like a
random point or a random chord or you know. These things you cannot be very lose
about. You have to specify in a more precise way.

Student: ((Refer Time: 32:20))

So, the three different questions are... So, we will do this, we can. Once we do all these
sample phases, it is tough. It will be become more clear. So, in this case as I said, you are
considering the distribution of a midpoint, your distribution of point inside the circle
uniformly. And you are looking at the probability, the midpoint of the chord lying inside
this circle. So, it is like, so the sample phase is the entire circle itself.

And is uniform distribution you say the circle. And you looking at the center of the point
being it is a smaller circle. Whereas for a second case for example. The sample space is
this theta. Day time in the set of the thing, what you are varying is theta. It is not the
center of the chord. So, it is finally. So, the sample phases between 0 1 2 pi. And you
have uniform distribution in that.
So, there all different mathematical questions, not all the same questions. It is seems like
a same question may English, but not in mathematical terms, in the second. So, in the
final case for example, your sample spaces 0 1, in terms of 0 1, where you are putting the
chord. So, they are actually different probabilities phases, different sample phases and
therefore, different mathematical questions. So, you can learn. So, this called Bertrand's
paradox.

So, actually Wikipedia has a good article on this. And book by ((Refer Time: 33:55))
also has this, I think. You can there even simulated all three of this and shown on, what
the chord is look like. So, this is pretty interesting article to the. And this is a very
famous paradox in probability. I will point out few more paradoxes as we go along.
There are some interesting paradoxes you can look up.

There is another paradox in probability theory. On why you need message theoretic,
measure theory view of probability, that also I will point out later. This is just saying that
we have to be careful defining sample phases, now defining what you underlying
probability spaces and so on.