Krdytkyu

Let $bar{X}$ be the distribution of sample mean for $n$ identical and independent distributed as Normal distributions $N(mu, sigma^2)$.

The random variable
$$ frac{bar{X} - mu}{frac{sigma}{sqrt{n}}} $$
has standard normal distribution. Now let
$$ S^2 =frac{1}{n-1} sum_{i=1}^{n} (X_{i} - bar{X})^2, $$
then random variable
$$ frac{bar{X} - mu}{frac{S}{sqrt{n}}} $$
has student-t distribution with $n-1$ degrees of freedom.
From this, we can conclude then when $n$ is large, the random variable above will converge to the standard normal distribution,
$$ lim_{n rightarrow infty} S^2 = lim_{n rightarrow infty} frac{1}{n-1} sum_{i=1}^{n} (X_{i} - bar{X})^2 = lim_{n rightarrow infty} frac{1}{n} sum_{i=1}^{n} (X_{i} - mu)^2 = sigma^{2} $$

But why we should choose student-t distribution when the sample size is small..? What is the mathematical explanation..? thanks.

The t-test is based on a Student's t-distribution which is sensitive to the number of observations. Furthermore, a t-statistic is calculated for small sample sizes where you do not know the population standard deviation. Even in the case of a large sample, we likely do not "know" the population standard deviation, but there are some nice results that help here.

The law of large numbers (LLN) gives us the result that sample averages converge in probability to the population average (formally, $bar{x} to mu$ as $n to infty$). That is, that as the sample size increases, the sample average gets closer and closer to the population average.

The central limit theorem (CLT) describes how the distribution of the difference in the sample and population averages ($bar{x}-mu$) changes with respect to the sample size $n$. Ultimately, what it tells us is that for sufficiently large $n$, this distribution approximates the normal distribution, $N(0,sigma^2 / n)$. Manipulating this expression, we can show the following
$$ bar{x}-mu sim N(0,sigma^2 / n) implies Z = frac{bar{x}-mu}{sigma/sqrt{n}} sim N(0,1) $$

So, in summary, the reason we can use a normal distribution for large samples is due to these results regarding convergence in distribution. This convergence happens fairly quickly (typically $nge30$ is sufficient), but for smaller samples these results do not hold and the Student's t-distribution is more appropriate.

The $t$ distribution is exact for all $n$. Therefore, we must use it or an adequate approximation. The Normal distribution is only a valid approximation for large $n$. The exact pdf is $frac{Gamma (frac{nu+1}{2})}{sqrt{nupi}Gamma(frac{nu}{2})}(1+frac{x^2}{nu})^{-frac{nu+1}{2}}$ with $nu=n-1$. For large $n$, this approximates $sqrt{frac{Gamma (frac{nu}{2}+1)}{Gamma (frac{nu}{2})nupi}}exp-frac{x^2}{2}=frac{1}{sqrt{2pi}}exp-frac{x^2}{2}$, the $N(0,,1)$ pdf.

I would like to post my view relating to @Chris answer also.

By the Central Limit Theorem, for large $m$, random var $X_{i}$ with mean $mu$ and st.dev $sigma$ from any distribution ($X_{i}$s have same dist) will have $$ sqrt{m} frac{ bar{X} - mu }{ sigma } $$ converge to standard normal. Also, the random.var $bar{X} - mu$ will be normal with mean $0$.

Now.., if $X_{i}$s are normal, then distribution
$$ T = sqrt{m} frac{bar{X} - mu}{S} $$ has student-t distribution with $m-1$ degrees of freedom. But we can only approximate the CLT using student-t when $X_{i}$s are normal.

Now I have an argument:
If $X_{i}$s are from the same distribution (any dist) with mean $mu$, then $bar{X} - mu$ will converge to normal with mean $0$ as $m$ gets large. So that
$$ sqrt{m}frac{frac{sum (bar{X}_{i} - mu)}{m} - 0}{S} $$
will have student-t distribution, and we can rewrite it as
$$ sqrt{m}frac{frac{sum (bar{X}_{i} )}{m} - mu}{S} $$
and for large $m$, $frac{sum (bar{X}_{i} )}{m}$ will converger to $bar{X}$. So we can still use $T$ as approximation even when $X_{i}$s are not normal.