Law of Large Numbers Let {f_{n}}, n = 1, 2 … be a sequence of orthogonal functions on a probability space (X, dν) and suppose E(f^{2}) = ∫ |f|^{2}dν ≤ 1. Then

\frac{1}{n}S_{n} = \frac{1}{n} ∑_{k=1}^{n} f_{k} → 0,

a.e. (with respect to ν) as n → ∞.

Proof. We begin with the simple observation that if {g_{n}} is a sequence of functions on a probability space (X, dν) such that

∑_{n} ∫ |g_{n}|^{2}dν < ∞,

then ∑_{n} |g_{n}|^{2} < ∞ a.e. (dν) and hence g_{n} → 0 a.e. (dν).

Using this, it is easy to verify the law of large numbers (LLN) for n → ∞ along the sequence of squares. Namely,

∫ (\frac{1}{n}S_{n})^{2}dν = \frac{1}{n^{2}} ∫ |S_{n}|^{2}dν = \frac{1}{n^{2}} ∑_{k=1}^{n} ∫ |f_{k}|^{2}dν ≤ \frac{1}{n}.

Therefore if we set g_{n} = \frac{1}{n^{2}} S_{n^{2}}, we have

∫ \frac{1}{n^{2}} |S_{n^{2}}|^{2}dν ≤ \frac{1}{n^{2}} .

Since the right hand side is summable, the observation above implies g_{n} → 0 a.e. (dν). This is the same as \frac{1}{n^{2}} S_{n^{2}} → 0, a.e..

To deal with limit over all the integers take m^{2} ≤ n < (m + 1)^{2} and set m(n) = ⌊\sqrt{n}⌋. Then

∫ |\frac{1}{m^{2}} S_{n} − \frac{1}{m^{2}} S_{m^{2}}|^{2}dν = \frac{1}{m^{4}} ∫ | ∑_{k=m^{2}+1}^{n} f_{k}|^{2}dν

= \frac{1}{m^{4}} ∫ ∑_{k=m^{2}+1}^{n} |f_{k}|^{2}dν
≤ \frac{2}{m^{3}}

since the sum has at most 2m terms, each of size at most 1. Put

g_{n} = \frac{S_{n}}{m(n)^{2}} − \frac{S_{m(n)^{2}}}{m(n)^{2}} .

Then since each m = m(n) is associated to at most 2m + 1 different n’s we get

∑_{n=1}^{∞} ∫ |g_{n}|^{2}dμ ≤ ∑_{n=1}^{∞} \frac{2}{m(n)^{3}} ≤ ∑_{m}(2m + 1)\frac{2}{m^{3}} < ∞,

so by the initial observation, g_{n} → 0 a.e. with respect to ν. This implies \frac{1}{m(n)^{2}} S_{n} → 0 a.e., which in turn implies \frac{1}{n}S_{n} → 0 a.e., which is what we wanted.

This version is sometimes called the strong law of large numbers because it gives a.e. convergence, as opposed to the weak version which only says that \frac{1}{n}S_{n} converges to 0 in L^{2}.

As a remark we should note that better estimates for the decay of S_{n} are possible if we assume that the functions {f_{n}} are independent with respect to the measure ν. This means that for any n and any collection of measurable sets {A_{1}, …, A_{n}} we have

ν(x ∈ X : f_{j}(x) ∈ A_{j}, j = 1, …, n}) = ∏_{j=1}^{n} ν({x ∈ X : f_{j}(x) ∈ A_{j}}).

Roughly, this says that knowing the values of any of the f_{j}’s at x does not give us any information about the values of the remaining functions there.

By 1915 Hausdorff had proved that if {f_{n}} ∈ L^{2}(ν) ∩ L^{1}(ν) are independent, orthonormal (orthogonal and have L^{2} norm 1) and satisfy ∫ f_{n}dν = 0 then

lim_{N→∞} \frac{1}{N^{\frac{1}{2} +ϵ}} ∑_{n=0}^{N} f_{n}(x) = 0, for a.e. x

and for every ϵ > 0. After that Hardy-Littlewood, and independently Khinchin, proved

lim_{N→∞} \frac{1}{\sqrt{N log N}} ∑_{n=0}^{N} f_{n}(x) = 0 for a.e. x.

The “final” result, found by Khinchin for a special case in 1928 and proved in general by Hartman-Wintree in 1941 says

lim sup_{N→∞} \frac{1}{\sqrt{2N log log N}} ∑_{n=0}^{N} f_{n}(x) = 1 for a.e. x.