The switchboard of Chapter 5 recorded \(3\) , \(5\) , \(2\) , \(4\) , \(6\) calls in five minutes, and the manager wants to know the probability \(e^{-\lambda}\) that a minute passes with no call at all. One honest estimator is the proportion of observed minutes with no call. It is unbiased, since each minute is empty with probability \(e^{-\lambda}\) . For these data it is \(0/5=0\) , and it would be \(0\) for almost any five minutes of a busy line. The maximum likelihood estimate is \(e^{-4}\approx0.018\) , which is at least positive, but Chapter 5 gave no assurance about its bias.
Chapter 4 proposed a way out of the impossibility of a uniformly best estimator: restrict attention to unbiased estimators, and seek the one with the smallest variance. This chapter carries the programme through. It answers three questions in order. How small can the variance of an unbiased estimator possibly be? Given an unbiased estimator that ignores part of the information, such as the proportion of empty minutes, how can it be improved? And when is the improved estimator the best one, with no further improvement possible? The answers use, respectively, the Fisher information of Chapter 5, the sufficiency of Chapter 3, and the completeness of Chapter 3.
Definition 6.1 (uniformly minimum variance unbiased estimator). An estimator \(W^\ast\) is a uniformly minimum variance unbiased estimator of \(g(\theta)\) , abbreviated UMVUE, if \(E_\theta W^\ast=g(\theta)\) for every \(\theta\) and, for every other unbiased estimator \(W\) of \(g(\theta)\) ,
The word uniformly is the demanding one: the same estimator must win at every parameter value. Proposition 4.6 showed that this is impossible among all estimators. Among unbiased ones it often is possible.
6.1 A floor under the variance
Theorem 6.2 (Cramér–Rao inequality). Let \(X_1,\dots,X_n\) be a random sample from a regular model with Fisher information \(I(\theta)\in(0,\infty)\) . Let \(W=W(X_1,\dots,X_n)\) have finite variance and expectation \(\psi(\theta)=E_\theta W\) , and suppose that \(\psi\) is differentiable and that its derivative may be computed by differentiating under the integral sign. Then
In particular every unbiased estimator of \(\theta\) satisfying these conditions has variance at least \(1/(nI(\theta))\) .
Proof. Write \(f_n(x;\theta)=\prod_if(x_i;\theta)\) for the joint density and \(\ell'(\theta)=\frac{\partial}{\partial\theta}\log f_n(X;\theta)\) for the score of the sample, which has mean \(0\) and variance \(nI(\theta)\) by Theorem 5.6 and Definition 5.7. Since \(\partial f_n/\partial\theta=\ell'\,f_n\) ,
the last step because \(\ell'\) has mean zero. The Cauchy–Schwarz inequality for covariances gives \(\operatorname{Cov}(W,\ell')^2\le\operatorname{Var}(W)\operatorname{Var}(\ell')\) , that is, \(\psi'(\theta)^2\le\operatorname{Var}_\theta(W)\cdot nI(\theta)\) .
The proof shows what the bound means. The derivative \(\psi'(\theta)\) measures how fast the mean of \(W\) must move when \(\theta\) moves, and it equals the covariance of \(W\) with the score. A statistic cannot track the parameter without being correlated with the score, and a given correlation with a variable of variance \(nI(\theta)\) forces a minimum variance. Information limits precision: the more the log-likelihood reacts to \(\theta\) , the smaller the floor.
Equality in the Cauchy–Schwarz inequality holds exactly when one variable is a linear function of the other. So the bound is attained at \(\theta\) if and only if
for some constant \(a(\theta)\) . Requiring this for all \(\theta\) and integrating the score shows that the density must be of one-parameter exponential-family form with \(W\) as natural statistic, up to a constant multiple. The bound is thus attained only in exponential families, and there only for linear functions of the mean of the natural statistic.
Definition 6.3 (efficiency). The efficiency of an unbiased estimator \(W\) of \(\psi(\theta)\) is the ratio of the Cramér–Rao bound to its variance, \(\psi'(\theta)^2/\bigl(nI(\theta)\operatorname{Var}_\theta W\bigr)\) . An estimator with efficiency \(1\) for every \(\theta\) is called efficient.
Example 6.1: Three comparisons with the bound
Poisson mean. Here \(I(\lambda)=1/\lambda\) , so every unbiased estimator of \(\lambda\) has variance at least \(\lambda/n\) . The sample mean has variance exactly \(\lambda/n\) . It is efficient: no unbiased estimator to which Theorem 6.2 applies can go below the floor. Section 6.3 will show, by a different argument that needs no regularity condition on the competitors, that it is the UMVUE. The equality condition can be seen directly, since \(\ell'(\lambda)=\sum x_i/\lambda-n=(n/\lambda)(\bar x-\lambda)\) is linear in \(\bar x\) .
Normal variance. For \(N(\mu_0,v)\) with known mean, \(I(v)=1/(2v^2)\) by Exercise 5.10, so the bound for unbiased estimators of \(\sigma^2\) is \(2\sigma^4/n\) . When \(\mu\) is unknown the sample variance has variance \(2\sigma^4/(n-1)\) by Example 4.1, which is above the bound; its efficiency is \((n-1)/n\) , or \(0.9\) for \(n=10\) . Section 6.3 will show that \(S^2\) is nevertheless the UMVUE. The bound is a floor, not a guarantee that something stands on it.
An estimator below the “bound”. For the uniform distribution on \((0,\theta)\) , Example 4.3 found the unbiased estimator \(\frac{n+1}{n}M\) with variance \(\theta^2/(n(n+2))\) . A formal calculation with \(\log f=-\log\theta\) gives “\(I(\theta)=1/\theta^2\) ” and a “bound” of \(\theta^2/n\) , which the estimator beats by a factor of \(n+2\) . There is no contradiction. The uniform model is not regular, Theorem 5.6 fails for it (Exercise 5.12), and Theorem 6.2 says nothing about it.
Common mistake. Applying the Cramér–Rao bound without checking regularity, or concluding that an unbiased estimator which misses the bound can be improved. The first error produces false lower bounds in models with parameter-dependent support. The second overlooks that the bound is often unattainable.
For the switchboard, the estimand \(\psi(\lambda)=e^{-\lambda}\) has \(\psi'(\lambda)^2=e^{-2\lambda}\) , so every unbiased estimator of the probability of an empty minute has variance at least \(\lambda e^{-2\lambda}/n\) . At \(\lambda=4\) and \(n=5\) that is about \(0.000268\) . The proportion of empty minutes has variance \(e^{-\lambda}(1-e^{-\lambda})/n\) , which is \(0.00360\) at the same values, thirteen times the floor.
6.2 Improvement by conditioning
The proportion of empty minutes wastes information: it records only whether each count is zero, while the sufficient statistic is the total. The next theorem turns any estimator that is not a function of a sufficient statistic into one that is, and never makes it worse.
Theorem 6.4 (Rao–Blackwell). Let \(T\) be a sufficient statistic and let \(U\) be an estimator of \(g(\theta)\) with finite variance. Define \(\varphi(T)=E[U\mid T]\) . Then
(a) \(\varphi(T)\) is a statistic: it does not depend on \(\theta\) ;
(b) \(E_\theta\,\varphi(T)=E_\theta\,U\) , so \(\varphi(T)\) has the same bias as \(U\) ;
(c) \(\operatorname{MSE}_\theta(\varphi(T))\le\operatorname{MSE}_\theta(U)\) for every \(\theta\) , with equality if and only if \(U=\varphi(T)\) with \(P_\theta\) -probability one.
Proof. (a) The conditional expectation is computed from the conditional distribution of the data given \(T\) , which is free of \(\theta\) by sufficiency.
(b) This is the tower property: \(E_\theta\bigl[E[U\mid T]\bigr]=E_\theta U\) .
(c) Write \(g=g(\theta)\) and split \(U-g=(U-\varphi(T))+(\varphi(T)-g)\) . Then
Conditioning the cross term on \(T\) , the factor \(\varphi(T)-g\) is fixed and \(E[U-\varphi(T)\mid T]=0\) , so the cross term vanishes. Hence
with equality exactly when \(E_\theta(U-\varphi(T))^2=0\) .
Part (a) is where sufficiency is used, and it is indispensable. Conditioning on a statistic that is not sufficient also reduces variance, but the result generally involves \(\theta\) and is not an estimator. The operation \(U\mapsto E[U\mid T]\) is called Rao–Blackwellization. The identity in the proof says that \(\varphi(T)\) is the function of \(T\) closest to \(U\) in mean square, and the part of \(U\) that it removes, \(U-\varphi(T)\) , is noise unrelated to the parameter.
Example 6.2: The probability of an empty minute
Let \(X_1,\dots,X_n\) be a random sample from the Poisson distribution with mean \(\lambda\) , with sufficient statistic \(T=\sum X_i\) . A crude unbiased estimator of \(e^{-\lambda}\) is \(U=\mathbf 1\{X_1=0\}\) , which looks at the first minute only. Then
Given \(T=t\) , each of the \(t\) events falls in the first interval with probability \(1/n\) , independently, so \(X_1\) is binomial with \(t\) trials and success probability \(1/n\) (Exercise 3.3 is the case \(n=2\) ). Therefore
For the switchboard, \(n=5\) and \(T=20\) , and the estimate is \(0.8^{20}\approx0.0115\) .
The variances can be computed exactly. Since \(T\sim\operatorname{Poisson}(n\lambda)\) and \(E\,a^T=e^{n\lambda(a-1)}\) , taking \(a=(1-1/n)^2\) gives \(E\,\varphi(T)^2=e^{-2\lambda+\lambda/n}\) , so
At \(\lambda=4\) , \(n=5\) :
| Estimator of \(e^{-\lambda}\) | Variance |
|---|---|
| \(\mathbf 1\{X_1=0\}\) | \(0.0180\) |
| proportion of empty minutes | \(0.00360\) |
| \((1-1/n)^T\) | \(0.000411\) |
| Cramér–Rao bound | \(0.000268\) |
Conditioning on the total reduces the variance of the one-minute indicator by a factor of about \(44\) , and it beats the five-minute proportion by a factor of nearly \(9\) . It does not reach the Cramér–Rao bound; its efficiency is \(0.000268/0.000411\approx0.65\) . Applying the same conditioning to the proportion of empty minutes gives the same \(\varphi(T)\) , by symmetry. Whether anything can do better is the next question.
6.3 When the improved estimator is the best one
Rao–Blackwellization produces an unbiased function of the sufficient statistic. If two different starting estimators could lead to two different such functions, there would be more work to do. Completeness is precisely the statement that they cannot.
Theorem 6.5 (Lehmann–Scheffé). Let \(T\) be a complete sufficient statistic, and let \(\varphi(T)\) be an unbiased estimator of \(g(\theta)\) with finite variance. Then \(\varphi(T)\) is a UMVUE of \(g(\theta)\) , and it is the only one: any other UMVUE equals \(\varphi(T)\) with probability one under every \(\theta\) .
Proof. Let \(U\) be any unbiased estimator of \(g(\theta)\) with finite variance and put \(\psi(T)=E[U\mid T]\) . By Theorem 6.4, \(\psi(T)\) is an unbiased estimator with \(\operatorname{Var}_\theta\psi(T)\le\operatorname{Var}_\theta U\) . Both \(\psi(T)\) and \(\varphi(T)\) are unbiased for \(g(\theta)\) , so
The difference is a function of \(T\) with expectation zero under every \(\theta\) , and completeness forces \(\psi(T)=\varphi(T)\) with probability one. Therefore \(\operatorname{Var}_\theta\varphi(T)=\operatorname{Var}_\theta\psi(T)\le\operatorname{Var}_\theta U\) , which is the UMVUE property.
For uniqueness, suppose \(U\) is also a UMVUE. Then \(\operatorname{Var}_\theta U=\operatorname{Var}_\theta\varphi(T)=\operatorname{Var}_\theta\psi(T)\) , so equality holds in part (c) of Theorem 6.4, which requires \(U=\psi(T)=\varphi(T)\) with probability one.
The theorem yields two practical methods. Method one: find a complete sufficient statistic \(T\) and solve the equation \(E_\theta\,\varphi(T)=g(\theta)\) for the function \(\varphi\) , by guessing and correcting a bias or by matching series. Method two: take any unbiased estimator, however crude, and condition it on \(T\) . Both arrive at the same place, because there is only one unbiased function of \(T\) .
In Example 6.2, \(T\) is complete (Exercise 3.11) and \((1-1/n)^T\) is an unbiased function of it. By the Lehmann–Scheffé theorem it is the UMVUE of \(e^{-\lambda}\) . No unbiased estimator has smaller variance at any \(\lambda\) , so the gap between \(0.000411\) and the Cramér–Rao value \(0.000268\) cannot be closed: the bound is simply not attainable for this estimand.
Example 6.3: Uniform endpoint, normal parameters, Bernoulli variance
Uniform on \((0,\theta)\) . The maximum \(M\) is sufficient (Example 3.2) and complete (Section 3.5), and \(\frac{n+1}{n}M\) is unbiased (Example 4.3). It is the UMVUE of \(\theta\) . This is method one, and it settles a model to which the Cramér–Rao theory does not apply at all.
Normal with both parameters unknown. The statistic \((\bar X,S^2)\) is complete and sufficient by Theorem 3.9. Since \(\bar X\) is unbiased for \(\mu\) and \(S^2\) for \(\sigma^2\) , they are the UMVUEs of \(\mu\) and \(\sigma^2\) . By Exercise 4.8, \(\bar X^2-S^2/n\) is an unbiased function of the complete sufficient statistic, so it is the UMVUE of \(\mu^2\) , even though it can be negative. And Example 4.1 showed that the UMVUE \(S^2\) is dominated in mean squared error by a biased estimator. Being best among unbiased estimators is not the same as being best.
Bernoulli variance. For \(n\) Bernoulli trials the total \(T\) is complete and sufficient. To estimate \(p(1-p)\) start from the crude unbiased estimator \(U=X_1(1-X_2)\) , which equals one when the first trial succeeds and the second fails. Given \(T=t\) , the \(t\) successes are equally likely to occupy any \(t\) of the \(n\) positions, so
The UMVUE of \(p(1-p)\) is \(T(n-T)/(n(n-1))\) . With \(3\) successes in \(20\) trials it equals \(51/380\approx0.1342\) , while the MLE \(\hat p(1-\hat p)=0.15\times0.85=0.1275\) is smaller by the factor \((n-1)/n\) , in agreement with Exercise 4.16.
Example 6.4: An unbiased estimate of a tail probability
A process produces measurements that are \(N(\mu,\sigma^2)\) with \(\sigma=0.3\) known, and a part is rejected when its measurement exceeds \(10.5\) . From a random sample of \(n=9\) with \(\bar x=10.1\) , estimate the rejection probability \(g(\mu)=P_\mu(X>10.5)\) .
With \(\sigma\) known, \(\bar X\) is complete and sufficient. Use method two with \(U=\mathbf 1\{X_1>10.5\}\) . The pair \((X_1,\bar X)\) is bivariate normal, with \(\operatorname{Cov}(X_1,\bar X)=\sigma^2/n=\operatorname{Var}\bar X\) , so the regression of \(X_1\) on \(\bar X\) has slope one, and
The conditional distribution is free of \(\mu\) , as sufficiency requires. Hence the UMVUE is
and for the data
The MLE, by invariance, is \(1-\Phi(0.4/0.3)=1-\Phi(1.333)\approx0.0912\) . The two differ only in the factor \(\sqrt{1-1/n}\) in the denominator, which disappears as \(n\) grows. The unbiased estimator uses a slightly smaller standard deviation because, once \(\bar X\) is fixed, a single observation is less variable about \(\bar X\) than about \(\mu\) .
6.4 What the theory leaves open
The chain of results is complete: the score gives a floor, sufficiency gives an improvement, completeness gives uniqueness. It is worth being exact about what has and has not been achieved.
A UMVUE need not exist. If no unbiased estimator exists, as for \(1/p\) in the binomial model, there is nothing to optimise. If the minimal sufficient statistic is not complete, as in the uniform model on \((\theta,\theta+1)\) or the family \(N(\theta,\theta^2)\) , the Lehmann–Scheffé theorem does not apply, different unbiased functions of the sufficient statistic exist, and typically none is best for all \(\theta\) .
A UMVUE need not be a good estimator. It may take impossible values, like \(\bar X^2-S^2/n<0\) or the estimator \((-1)^X\) of Example 4.5, which is the UMVUE of \(e^{-2\lambda}\) since it is the only unbiased estimator. It may be inadmissible under squared error, like \(S^2\) .
Unbiasedness is tied to a parametrization. The UMVUE of \(\sigma^2\) is \(S^2\) , but the UMVUE of \(\sigma\) is not \(S\) (Exercise 10). Maximum likelihood, by contrast, is invariant.
In practice the UMVUE and the MLE are usually close. In Example 6.2 they are \(0.8^{20}\approx0.0115\) and \(e^{-4}\approx0.0183\) with five observations, and in general \((1-1/n)^{n\bar x}\to e^{-\bar x}\) as \(n\) grows. In Example 6.4 they differ by a factor \(\sqrt{1-1/n}\) inside the normal distribution function. The exact finite-sample theory of this chapter and the large-sample theory of Chapter 9 single out essentially the same estimators in regular models; the first explains what is special about them for every \(n\) , and the second explains why the MLE is a safe default when exact optimality is unavailable.
A point estimate, however well chosen, is a single number, and a single number says nothing of its own reliability. The next chapter replaces it by a set.
Exercises
-
State the Cramér–Rao inequality with its hypotheses. Compute the bound for unbiased estimators of \(p\) from \(n\) Bernoulli trials and verify that the sample proportion attains it.
Answer
\(I(p)=1/(p(1-p))\) , so every unbiased estimator of \(p\) satisfying the hypotheses has variance at least \(p(1-p)/n\) . The sample proportion has variance exactly \(p(1-p)/n\) ; its score \(\ell'(p)=n(\hat p-p)/(p(1-p))\) is linear in \(\hat p\) , which is the equality condition.
-
For a random sample from the Poisson distribution with total \(T\) , show that \(T(T-1)/n^2\) is the UMVUE of \(\lambda^2\) , compute its variance using \(\operatorname{Var}\bigl(T(T-1)\bigr)=4m^3+2m^2\) for \(T\sim\operatorname{Poisson}(m)\) , and compare it with the Cramér–Rao bound for unbiased estimators of \(\lambda^2\) .
Solution
\(T\sim\operatorname{Poisson}(m)\) with \(m=n\lambda\) , and the second factorial moment of a Poisson variable is \(E\,T(T-1)=m^2\) . Hence
\[ E\Bigl[\frac{T(T-1)}{n^2}\Bigr]=\frac{n^2\lambda^2}{n^2}=\lambda^2 , \]so the estimator is unbiased. It is a function of \(T\) , which is complete and sufficient (Exercise 3.11), and by the Lehmann–Scheffé theorem it is the UMVUE of \(\lambda^2\) .
Variance: with the given formula,
\[ \operatorname{Var}\Bigl[\frac{T(T-1)}{n^2}\Bigr]=\frac{4n^3\lambda^3+2n^2\lambda^2}{n^4}=\frac{4\lambda^3}{n}+\frac{2\lambda^2}{n^2}. \]Cramér–Rao bound: for \(\psi(\lambda)=\lambda^2\) , \(\psi'(\lambda)=2\lambda\) and \(I(\lambda)=1/\lambda\) , so the bound is \((2\lambda)^2\lambda/n=4\lambda^3/n\) .
The UMVUE exceeds the bound by \(2\lambda^2/n^2\) . No unbiased estimator attains the bound, because the UMVUE is the best there is. The excess is of smaller order than the bound itself: the ratio of the two is \(1+1/(2n\lambda)\) , which tends to \(1\) . The bound is attained only for linear functions of \(\lambda\) in this model, and \(\lambda^2\) is not one; asymptotically the distinction disappears.
-
For a random sample of size \(n\ge3\) from \(\operatorname{Exp}(\lambda)\) : (a) show that \(\bar X\) is an efficient estimator of the mean \(1/\lambda\) ; (b) show that \((n-1)/\sum X_i\) is the UMVUE of the rate \(\lambda\) , find its variance, and show that it does not attain the Cramér–Rao bound.
Solution
(a) Parametrize by the mean \(\theta=1/\lambda\) . By Exercise 5.11, the information about \(\theta\) is \(I(\lambda)/g'(\lambda)^2\) with \(g(\lambda)=1/\lambda\) , that is, \(\lambda^{-2}/\lambda^{-4}=\lambda^2=1/\theta^2\) . The bound for unbiased estimators of \(\theta\) is \(\theta^2/n\) . Since \(\operatorname{Var}X=1/\lambda^2=\theta^2\) , the sample mean has variance exactly \(\theta^2/n\) and is efficient.
(b) \(T=\sum X_i\sim\operatorname{Gamma}(n,\lambda)\) , and for a gamma variable \(E\,T^{-1}=\lambda/(n-1)\) and \(E\,T^{-2}=\lambda^2/\bigl((n-1)(n-2)\bigr)\) when \(n\ge3\) . Therefore \(E[(n-1)/T]=\lambda\) : the estimator is unbiased, and being a function of the complete sufficient statistic \(T\) it is the UMVUE. Its variance is
\[ (n-1)^2\cdot\frac{\lambda^2}{(n-1)(n-2)}-\lambda^2=\lambda^2\Bigl(\frac{n-1}{n-2}-1\Bigr)=\frac{\lambda^2}{n-2}. \]The Cramér–Rao bound for unbiased estimators of \(\lambda\) is \(1/(nI(\lambda))=\lambda^2/n\) , which is strictly smaller. The efficiency is \((n-2)/n\) .
The same model, then, has an efficient estimator for one parametrization and none for another. The bound is attained for the mean \(1/\lambda\) , a linear function of the expectation of the natural statistic, and not for the rate. The MLE of the rate, \(n/T\) , is biased upward by the factor \(n/(n-1)\) .
-
Suppose an unbiased estimator \(W\) of \(\theta\) satisfies \(W-\theta=a(\theta)\ell'(\theta)\) for all \(\theta\) in a regular model. By integrating the score, show that the joint density has exponential-family form with \(W\) as natural statistic.
Hint
Write \(\ell'(\theta)=(W-\theta)/a(\theta)\) and integrate with respect to \(\theta\) : \(\log f_n(x;\theta)=W(x)\,c(\theta)-d(\theta)+k(x)\) , where \(c'=1/a\) and \(d'=\theta/a\) , and \(k\) is the constant of integration, which may depend on \(x\) . Exponentiate and compare with Definition 3.5.
-
For \(n\ge2\) Bernoulli trials with total \(T\) , Rao–Blackwellize the estimator \(U=X_1X_2\) of \(p^2\) and identify the UMVUE of \(p^2\) .
Solution
\(U=X_1X_2\) equals \(1\) when the first two trials both succeed, so \(E_pU=p^2\) by independence: it is unbiased for \(p^2\) .
Given \(T=t\) , the \(t\) successes are equally likely to occupy any of the \(\binom nt\) subsets of positions (Example 3.1). The number of subsets containing positions \(1\) and \(2\) is \(\binom{n-2}{t-2}\) , so for \(t\ge2\)
\[ E[U\mid T=t]=P(X_1=1,X_2=1\mid T=t)=\frac{\binom{n-2}{t-2}}{\binom nt}=\frac{t(t-1)}{n(n-1)}, \]and the formula also gives the correct value \(0\) for \(t=0\) and \(t=1\) . So the Rao–Blackwellized estimator is
\[ \varphi(T)=\frac{T(T-1)}{n(n-1)}. \]It is unbiased by Theorem 6.4(b), which can be confirmed from \(E\,T(T-1)=n(n-1)p^2\) . Since \(T\) is complete and sufficient for the Bernoulli model, the Lehmann–Scheffé theorem identifies \(\varphi(T)\) as the unique UMVUE of \(p^2\) . It is not the square of the UMVUE of \(p\) : \((T/n)^2\) overestimates \(p^2\) on average, and \(\varphi(T)\) is slightly smaller to compensate.
-
For a random sample from the uniform distribution on \((0,\theta)\) , show that \(U=2X_1\) is unbiased for \(\theta\) and compute \(E[U\mid M]\) directly, where \(M\) is the maximum, using the fact that given \(M=m\) the observation \(X_1\) equals \(m\) with probability \(1/n\) and is otherwise uniform on \((0,m)\) .
Solution
Unbiasedness: \(E\,2X_1=2\cdot\theta/2=\theta\) .
Conditional expectation: given \(M=m\) , with probability \(1/n\) the first observation is the maximum, in which case \(X_1=m\) ; with probability \((n-1)/n\) it is one of the others, in which case it is uniform on \((0,m)\) with mean \(m/2\) . Hence
\[ E[X_1\mid M=m]=\frac1n\,m+\frac{n-1}{n}\cdot\frac m2=\frac{2m+(n-1)m}{2n}=\frac{(n+1)m}{2n}, \]and
\[ E[2X_1\mid M]=\frac{n+1}{n}M . \]Conditioning the crude estimator \(2X_1\) on the sufficient statistic produces the bias-corrected maximum of Example 4.3, which is the UMVUE because \(M\) is complete. The same estimator results from conditioning \(2\bar X\) , since \(E[2\bar X\mid M]\) is the average of \(n\) copies of \(E[2X_1\mid M]\) . The variance drops from \(\theta^2/3\) for \(2X_1\) , and \(\theta^2/(3n)\) for \(2\bar X\) , to \(\theta^2/(n(n+2))\) .
-
For the switchboard data of Example 6.2, find the UMVUE of the probability of exactly one call in a minute, \(\lambda e^{-\lambda}\) , by Rao–Blackwellizing \(\mathbf 1\{X_1=1\}\) , and compare its value with the MLE.
Answer
Given \(T=t\) , \(X_1\sim\operatorname{Binomial}(t,1/n)\) , so the UMVUE is \(P(X_1=1\mid T)=\frac Tn\bigl(1-\frac1n\bigr)^{T-1}\) . With \(n=5\) , \(T=20\) : \(4\times0.8^{19}\approx0.0576\) . The MLE is \(4e^{-4}\approx0.0733\) .
-
A student computes that \(\frac{n+1}{n}M\) has variance \(\theta^2/(n(n+2))\) in the uniform model on \((0,\theta)\) , compares it with \(1/(nI(\theta))=\theta^2/n\) , and concludes that the Cramér–Rao inequality is false. Explain precisely which hypothesis fails and which step of the proof of Theorem 6.2 is unavailable.
Solution
Theorem 6.2 is stated for regular models, and the uniform model on \((0,\theta)\) is not regular: its support \((0,\theta)\) changes with \(\theta\) , violating the second condition of Definition 5.5.
The failure enters the proof at two points. First, the proof uses \(E_\theta\ell'(\theta)=0\) and \(\operatorname{Var}_\theta\ell'(\theta)=nI(\theta)\) from Theorem 5.6. Here \(\ell'(\theta)=-n/\theta\) is a nonzero constant on the support, so its mean is \(-n/\theta\) and its variance is \(0\) . The student’s “information” \(1/\theta^2\) is the second moment of the score of one observation, which coincides with the variance only when the mean of the score is zero; it is not the variance of anything here. Second, the proof needs \(\psi'(\theta)=\int W\,\partial f_n/\partial\theta\,dx\) , obtained by differentiating under the integral sign. With limits of integration that depend on \(\theta\) , the derivative of \(\int W f_n\,dx\) has an additional boundary term, exactly as in Exercise 5.12, and the identity \(\psi'(\theta)=\operatorname{Cov}(W,\ell')\) is false. Indeed here \(\operatorname{Cov}(W,\ell')=0\) since \(\ell'\) is constant, while \(\psi'(\theta)=1\) .
So the inequality has not been contradicted; it does not apply. The estimator’s variance of order \(1/n^2\) is a genuine feature of models with a moving boundary.
-
For a random sample from \(N(\mu,\sigma^2)\) with \(\sigma^2\) known, show that \(\bar X^2-\sigma^2/n\) is the UMVUE of \(\mu^2\) , compute its variance, and compare it with the Cramér–Rao bound for unbiased estimators of \(\mu^2\) .
Solution
\(\bar X\sim N(\mu,\tau^2)\) with \(\tau^2=\sigma^2/n\) , so \(E\,\bar X^2=\mu^2+\sigma^2/n\) and \(\bar X^2-\sigma^2/n\) is unbiased for \(\mu^2\) . With \(\sigma^2\) known, \(\bar X\) is complete and sufficient (Example 3.6), and the estimator is a function of it, so it is the UMVUE by the Lehmann–Scheffé theorem.
Variance: write \(\bar X=\mu+\tau Z\) with \(Z\) standard normal. Then \(\bar X^2=\mu^2+2\mu\tau Z+\tau^2Z^2\) , and since \(\operatorname{Var}Z=1\) , \(\operatorname{Var}Z^2=2\) , and \(\operatorname{Cov}(Z,Z^2)=E\,Z^3=0\) ,
\[ \operatorname{Var}\bar X^2=4\mu^2\tau^2+2\tau^4=\frac{4\mu^2\sigma^2}{n}+\frac{2\sigma^4}{n^2}. \]Cramér–Rao bound: \(\psi(\mu)=\mu^2\) , \(\psi'(\mu)=2\mu\) , \(I(\mu)=1/\sigma^2\) , so the bound is \(4\mu^2\sigma^2/n\) .
The UMVUE exceeds the bound by \(2\sigma^4/n^2\) , as in Exercise 6.2. At \(\mu=0\) the bound is zero, which no estimator based on noisy data can achieve, while the UMVUE has variance \(2\sigma^4/n^2\) ; the bound is uninformative exactly where \(\psi'\) vanishes. The estimator is negative whenever \(\lvert\bar X\rvert<\sigma/\sqrt n\) , an event of probability about \(0.68\) when \(\mu=0\) .
-
For a random sample from \(N(\mu,\sigma^2)\) with both parameters unknown, find the UMVUE of \(\sigma\) in terms of \(S\) and the constant \(c_n=E\,S/\sigma\) , and explain why \(S\) itself is not the UMVUE.
Hint
\(E\,S=c_n\sigma\) with \(c_n<1\) (Exercise 2.5), so \(S/c_n\) is unbiased and is a function of the complete sufficient statistic. \(S\) is biased, and a UMVUE must be unbiased. For normal samples \(c_n=\sqrt{2/(n-1)}\,\Gamma(n/2)/\Gamma((n-1)/2)\) .
-
Prove, without assuming a complete sufficient statistic, that a UMVUE is unique: if \(W_1\) and \(W_2\) are both UMVUEs of \(g(\theta)\) , consider \((W_1+W_2)/2\) and show that \(W_1=W_2\) with probability one.
Solution
Let \(W_1\) and \(W_2\) be UMVUEs of \(g(\theta)\) , with common variance \(v=v(\theta)\) , the minimal variance. Their average \(\bar W=(W_1+W_2)/2\) is unbiased, so \(\operatorname{Var}\bar W\ge v\) . On the other hand
\[ \operatorname{Var}\bar W=\tfrac14\bigl(v+v+2\operatorname{Cov}(W_1,W_2)\bigr)=\tfrac12\bigl(v+\operatorname{Cov}(W_1,W_2)\bigr). \]By the Cauchy–Schwarz inequality \(\operatorname{Cov}(W_1,W_2)\le\sqrt{v\cdot v}=v\) , so \(\operatorname{Var}\bar W\le v\) . Combining the two inequalities, \(\operatorname{Var}\bar W=v\) and \(\operatorname{Cov}(W_1,W_2)=v\) .
Now compute the variance of the difference:
\[ \operatorname{Var}(W_1-W_2)=v+v-2\operatorname{Cov}(W_1,W_2)=0 . \]A variable with variance zero equals its mean with probability one, and \(E_\theta(W_1-W_2)=g(\theta)-g(\theta)=0\) . Hence \(W_1=W_2\) with \(P_\theta\) -probability one, for every \(\theta\) . The argument used only unbiasedness and minimal variance; no sufficient statistic was needed.
-
Let \(W\) be a UMVUE of \(g(\theta)\) and let \(U\) be any statistic with finite variance and \(E_\theta U=0\) for all \(\theta\) . Prove that \(\operatorname{Cov}_\theta(W,U)=0\) for all \(\theta\) , by considering the unbiased estimators \(W+aU\) .
Hint
For every real \(a\) , \(W+aU\) is unbiased for \(g(\theta)\) , so \(\operatorname{Var}(W+aU)=\operatorname{Var}W+2a\operatorname{Cov}(W,U)+a^2\operatorname{Var}U\ge\operatorname{Var}W\) . A quadratic in \(a\) that is nonnegative for all \(a\) and vanishes at \(a=0\) must have zero linear coefficient.
-
In the uniform model on \((\theta,\theta+1)\) with \(n=5\) , both \(\bar X-\tfrac12\) and \(\tfrac12(X_{(1)}+X_{(n)})-\tfrac12\) are unbiased for \(\theta\) . Compute both variances, using the fact that the midrange of \(n\) uniform variables on \((0,1)\) has variance \(1/\bigl(2(n+1)(n+2)\bigr)\) , and explain why the Lehmann–Scheffé theorem does not name either as the UMVUE.
Answer
\(\operatorname{Var}(\bar X-\tfrac12)=\frac{1}{12n}=\frac1{60}\approx0.0167\) and the variance of the centred midrange is \(\frac{1}{2\cdot6\cdot7}=\frac1{84}\approx0.0119\) . The midrange is better here, but the minimal sufficient statistic \((X_{(1)},X_{(n)})\) is not complete (Example 3.5), so the Lehmann–Scheffé theorem gives no UMVUE.
-
Under the sampling scheme of Exercise 1.11, in which items are drawn until the \(r\) -th defective and \(N\) is the number drawn, \(N\) is complete and sufficient. Identify the UMVUE of \(p\) and the UMVUE of \(1/p\) , and contrast the second with the binomial design, in which \(1/p\) has no unbiased estimator.
Answer
By Exercise 1.11, \((r-1)/(N-1)\) is unbiased for \(p\) ; as a function of the complete sufficient \(N\) it is the UMVUE. Since \(E\,N=r/p\) , the UMVUE of \(1/p\) is \(N/r\) . Whether an unbiased estimator of \(1/p\) exists depends on the sampling design, not only on the data record.
-
Compute the efficiency of \(S^2\) as an estimator of \(\sigma^2\) for normal samples of size \(5\) , \(10\) , and \(50\) , and the efficiency of the UMVUE of the exponential rate in Exercise 3 for the same sample sizes.
Answer
\(S^2\) : \((n-1)/n=0.8\) , \(0.9\) , \(0.98\) . Exponential rate: \((n-2)/n=0.6\) , \(0.8\) , \(0.96\) .
-
For a random sample from \(N(\mu,\sigma^2)\) with both parameters unknown, find the UMVUE of the \(95\) th percentile \(\mu+1.645\,\sigma\) , justifying each step with a result of this chapter or of Chapter 3.
Hint
\(\bar X\) is unbiased for \(\mu\) and \(S/c_n\) for \(\sigma\) (Exercise 6.10), so \(\bar X+1.645\,S/c_n\) is unbiased for the percentile by linearity. It is a function of \((\bar X,S^2)\) , which is complete and sufficient by Theorem 3.9. Apply Theorem 6.5.
-
In the notation of Theorem 6.4, prove that \(E_\theta\bigl[(U-\varphi(T))\,h(T)\bigr]=0\) for every function \(h\) with finite variance, and deduce that \(E_\theta(U-h(T))^2\ge E_\theta(U-\varphi(T))^2\) : among functions of \(T\) , the conditional expectation is the best mean-square approximation to \(U\) .
Solution
Fix \(h\) with finite variance. Conditioning on \(T\) and using the tower property,
\[ E_\theta\bigl[(U-\varphi(T))\,h(T)\bigr]=E_\theta\Bigl[h(T)\,E\bigl[U-\varphi(T)\mid T\bigr]\Bigr]=E_\theta\bigl[h(T)\cdot0\bigr]=0, \]because \(h(T)\) is fixed once \(T\) is given and \(E[U\mid T]=\varphi(T)\) by definition. So the residual \(U-\varphi(T)\) is uncorrelated with every function of \(T\) .
Now split \(U-h(T)=(U-\varphi(T))+(\varphi(T)-h(T))\) . The second bracket is itself a function of \(T\) , so by what was just shown the cross term in the expansion of the square vanishes:
\[ E_\theta(U-h(T))^2=E_\theta(U-\varphi(T))^2+E_\theta(\varphi(T)-h(T))^2\ge E_\theta(U-\varphi(T))^2 . \]Equality holds only when \(h(T)=\varphi(T)\) with probability one. This is the statement that \(\varphi(T)\) is the orthogonal projection of \(U\) onto the space of functions of \(T\) , with mean-square distance playing the role of length. The Rao–Blackwell inequality is the Pythagorean identity for that projection, with the constant \(g(\theta)\) as the third vertex.
-
For the Poisson model with \(n=5\) and \(\lambda=4\) , compute the bias and the mean squared error of the MLE \(e^{-\bar X}\) of \(e^{-\lambda}\) , using \(E\,a^T=e^{n\lambda(a-1)}\) , and compare the mean squared error with the variance of the UMVUE found in Example 6.2.
Solution
Here \(T\sim\operatorname{Poisson}(20)\) and the MLE is \(e^{-T/5}=a^T\) with \(a=e^{-0.2}\) . Using \(E\,a^T=e^{20(a-1)}\) :
\[ E\,e^{-T/5}=\exp\bigl(20(e^{-0.2}-1)\bigr)\approx0.02664,\qquad E\,e^{-2T/5}=\exp\bigl(20(e^{-0.4}-1)\bigr)\approx0.001369 . \]The target is \(e^{-4}\approx0.01832\) , so the bias is \(0.02664-0.01832\approx0.0083\) : the MLE overestimates the probability of an empty minute by about \(45\%\) of its true value on average. The mean squared error is
\[ E\,e^{-2T/5}-2e^{-4}E\,e^{-T/5}+e^{-8}\approx0.001369-0.000976+0.000335\approx0.00073 . \]The UMVUE of Example 6.2 has variance, and hence mean squared error, \(0.000411\) . At these values the unbiased estimator is the more accurate of the two, by a factor of about \(1.8\) in mean squared error. The comparison is not uniform in \(\lambda\) and it fades as \(n\) grows, since both estimators are functions of \(\bar X\) that converge to each other; with \(n=5\) and a convex target, the bias of the plug-in estimate is large enough to matter.
-
Let \(W\) be an unbiased estimator of \(\theta\) in a regular model whose variance equals the Cramér–Rao bound for every \(\theta\) . Show that no unbiased estimator to which Theorem 6.2 applies has smaller variance than \(W\) at any \(\theta\) , and give an example from this chapter of a UMVUE whose variance exceeds the bound.
Answer
Any unbiased estimator to which Theorem 6.2 applies has variance at least the bound, and \(W\) equals it, so \(W\) has minimal variance among such estimators at every \(\theta\) . Examples of UMVUEs above the bound: \(S^2\) for \(\sigma^2\) (efficiency \((n-1)/n\) ), and \((1-1/n)^T\) for \(e^{-\lambda}\) (Example 6.2).