Krdytkyu

I am reading a book on NLP which gives an introduction on Probability Theory. Here it talks about $sigma$-field. It says,

"The foundations of probability theory depend on the set of events $mathscr{F}$ forming a $sigma$-field".

I understood the definition of $sigma$-field.

My questions are:

• What are the foundations of probability theory?


• How are these foundations dependent upon a $sigma$-field?

I will try an answer less poetic than @Matthias - I hope not too technical.

Probability when there are only finitely many outcomes is a matter of counting. There are $36$ possible results from a roll of two dice and $6$ of them sum to $7$ so the probability of a sum of $7$ is $6/36$. You've measured the size of the set of outcomes that you are interested in.

It's harder to make rigorous sense of things when the set of possible results is infinite. What does it mean to choose two numbers at random in the interval $[1,6]$ and ask for their sum? Any particular pair, like $(1.3, pi)$, will have probability $0$.

You deal with this problem by replacing counting with integration. Unfortunately, the integration you learn in first year calculus ("Riemann integration") isn't powerful enough to derive all you need about probability. (It is enough to determine the probability that your two rolls total exactly $7$ is $0$, and to find the probability that it's at least $7$.)

For the definitions and theorems of rigorous probability theory (those are the "foundations" you ask about) you need "Lebesgue integration". That requires first carefully specifying the sets that you are going to ask for the probabilities of - and not every set is allowed, for technical reasons without which you can't make the mathematics work the way you want. It turns out that the set of sets whose probability you are going to ask about carries the name "$sigma$-field" or "sigma algebra". (It's not a field in the arithmetic sense.)
The essential point is that it's closed under countable set operations. That's what the "$sigma$" says. Your text may not provide a formal definition - you may not need it for NLP applications.

At the beginning, there was the sample space $Omega$, the set of all outcomes of a random experiment. And the Lord said, "Let there be events!", and Lo! certain subsets of the sample space became events, subject to their countable union and their complement being another event. And the Lord said, "Let us measure how often we can observe these events if we repeat the experiment many times!", and thus was born the probability measure: a number between 0 and 1 assigned to every event, subject to the same axioms as any other measure (such as length, cardinality, etc.). And on the third day, the Lord said, "Let us do math with the outcomes of random experiments!", and thus he created the random variable, which is a measurement made on the outcome of a random experiment, such that the pre-image of a Borel-set in the real numbers is an event...