A polling firm interviews 1,000 voters and records, in order, whether each supports a proposal. The raw data are a string of 1,000 zeros and ones. Everyone who analyses such data begins by counting the ones and never looks at the string again. Is anything lost?
Under one model, nothing is lost, and this can be proved. If the responses are independent with a common probability \(p\) of support, then once the count is known, the order in which the ones and zeros occurred is pure noise: its conditional distribution is the same for every \(p\) , so it cannot help to distinguish one value of \(p\) from another. Under a different model, for instance one in which opinion drifted during the day of interviewing, the order would carry information and discarding it would be an error.
This chapter makes the idea precise. A statistic is sufficient for a model when the data, given the statistic, have a distribution free of the parameter. The factorization theorem turns that definition into a test that takes one line to apply. Minimal sufficiency identifies the greatest reduction that loses nothing. Completeness, a sharper property, is what later makes best unbiased estimators unique. The chapter contains no estimation at all; it is about deciding what to carry forward before estimation begins.
3.1 What can be thrown away
Definition 3.1 (sufficient statistic). A statistic \(T=T(X)\) is sufficient for the model \(\{P_\theta:\theta\in\Theta\}\) if the conditional distribution of the data \(X\) given \(T=t\) does not depend on \(\theta\) , for every value \(t\) at which the conditional distribution is defined.
The definition can be read as a two-stage description of how the data arise. First nature draws \(T\) from its sampling distribution, which depends on \(\theta\) . Then the full data are drawn from the conditional distribution given \(T\) , which does not. Someone who is told only \(T\) could carry out the second stage personally, with a random number generator, and produce a data set with exactly the same distribution as the original. Whatever a statistician could do with the real data could be done, with the same sampling distribution for every \(\theta\) , with the regenerated data. In that sense the second stage contains no information about \(\theta\) .
Example 3.1: The order of four Bernoulli trials
Let \(X_1,\dots,X_4\) be independent Bernoulli variables with success probability \(p\) , and let \(T=X_1+X_2+X_3+X_4\) . The probability of any particular sequence with two ones is \(p^2(1-p)^2\) , for example
There are \(\binom42=6\) such sequences, so \(P_p(T=2)=6p^2(1-p)^2\) . The conditional probability of the particular sequence given the total is
and the same holds for each of the six sequences. The parameter has cancelled. Given \(T=2\) , the arrangement is uniformly distributed over the six possibilities whether \(p\) is \(0.1\) or \(0.9\) . The same computation with \(n\) trials and \(T=t\) gives conditional probability \(1/\binom nt\) for each arrangement, so \(T\) is sufficient.
Now consider instead the statistic \(U=X_1\) , the first trial alone. Given \(U=1\) , the sequence \((1,1,1,1)\) has conditional probability \(p^3\) , which depends on \(p\) . So \(U\) is not sufficient: the three trials that \(U\) ignores have a distribution that still involves the parameter.
3.2 The factorization theorem
Checking the definition directly requires a conditional distribution, which can be laborious. The following criterion needs only the joint density.
Theorem 3.2 (Fisher–Neyman factorization). Let the data \(X\) have density or mass function \(f(x;\theta)\) . A statistic \(T\) is sufficient if and only if there are nonnegative functions \(g\) and \(h\) such that, for all \(x\) and all \(\theta\in\Theta\) ,
The content is that the parameter and the data interact only through \(T(x)\) ; whatever else of \(x\) appears in the density is collected in a factor \(h(x)\) that does not involve \(\theta\) .
Proof. The proof is given for discrete data, where \(f(x;\theta)=P_\theta(X=x)\) .
Suppose \(T\) is sufficient. For any \(x\) , the event \(X=x\) is contained in the event \(T=T(x)\) , so
whenever \(P_\theta(T=T(x))>0\) . Put \(g(t,\theta)=P_\theta(T=t)\) , and let \(h(x)\) be the conditional probability, which by sufficiency is the same for every \(\theta\) at which it is defined; if it is defined for no \(\theta\) , put \(h(x)=0\) . When \(P_\theta(T=T(x))=0\) both sides are zero. This is the factorization.
Conversely, suppose the factorization holds, and fix \(t\) and \(\theta\) with \(P_\theta(T=t)>0\) . Summing over all \(y\) with \(T(y)=t\) ,
For \(x\) with \(T(x)=t\) ,
and for \(x\) with \(T(x)\ne t\) the conditional probability is zero. Neither expression involves \(\theta\) , so \(T\) is sufficient.
For continuous data the statement is the same and the idea of the proof is the same, but conditioning on an event of probability zero requires the measure-theoretic definition of conditional distributions, and the full argument belongs to a course on that subject. The theorem will be used freely in both cases.
Example 3.2: Reading sufficient statistics off three densities
Poisson. For a random sample from the Poisson distribution with mean \(\lambda\) ,
so \(T=\sum X_i\) is sufficient. As a check of the definition, with \(n=3\) the conditional probability of the sample \((2,1,1)\) given \(T=4\) is the multinomial probability \(\frac{4!}{2!\,1!\,1!}\bigl(\tfrac13\bigr)^4=\tfrac{4}{27}\) , in which \(\lambda\) does not appear.
Normal, both parameters unknown. Expanding the square in the exponent,
The data enter only through \(\sum x_i\) and \(\sum x_i^2\) , so \(T=(\sum X_i,\sum X_i^2)\) is sufficient, with \(h(x)=1\) . The pair \((\bar X,S^2)\) is a one-to-one function of \(T\) and is therefore sufficient as well.
Uniform on \((0,\theta)\) . Each observation has density \(\theta^{-1}\mathbf 1\{0<x_i<\theta\}\) , and the product of the indicators equals one exactly when the largest observation is below \(\theta\) and the smallest is above zero:
The maximum \(X_{(n)}\) is sufficient. The indicator is part of the density; a factorization that forgets it gives the absurd conclusion that no data are needed at all.
Common mistake. Dropping the support indicator when the support depends on the parameter. For the uniform, shifted exponential, and Pareto families the indicator carries all or part of the dependence on the data, and the sufficient statistic comes from it.
Sufficiency is preserved by one-to-one transformations, because if \(T=\psi(T')\) with \(\psi\) invertible, a factorization through \(T\) is also a factorization through \(T'\) . More generally, if \(T\) is sufficient and \(T\) is a function of another statistic \(T'\) , then \(T'\) is sufficient too: the whole sample is always sufficient, and so is the vector of order statistics for any random sample. Sufficiency by itself does not say that a statistic is a good reduction, only a lossless one.
3.3 The greatest reduction that loses nothing
Definition 3.3 (minimal sufficient statistic). A sufficient statistic \(T\) is minimal sufficient if it is a function of every other sufficient statistic: for each sufficient \(T'\) there is a function \(\psi\) with \(T=\psi(T')\) .
A function of a statistic cannot distinguish more data sets than the statistic does, so a minimal sufficient statistic lumps together as many data sets as any sufficient statistic can. It is the coarsest summary that is still lossless. The next theorem characterises it by looking at pairs of data sets.
Theorem 3.4 (ratio criterion for minimal sufficiency). Let \(f(x;\theta)\) be the density or mass function of the data, and let \(\mathcal X_0\) be the set of \(x\) with \(f(x;\theta)>0\) for at least one \(\theta\) . Suppose a statistic \(T\) has the following property for all \(x,y\in\mathcal X_0\) : there is a constant \(c(x,y)>0\) with \(f(x;\theta)=c(x,y)f(y;\theta)\) for every \(\theta\in\Theta\) if and only if \(T(x)=T(y)\) . Then \(T\) is minimal sufficient.
Proof. Sufficiency. For each value \(t\) in the range of \(T\) on \(\mathcal X_0\) choose one representative point \(x_t\in\mathcal X_0\) with \(T(x_t)=t\) . Any \(x\in\mathcal X_0\) has the same value of \(T\) as its representative \(x_{T(x)}\) , so by hypothesis \(f(x;\theta)=c(x,x_{T(x)})\,f(x_{T(x)};\theta)\) for all \(\theta\) . Setting \(h(x)=c(x,x_{T(x)})\) and \(g(t,\theta)=f(x_t;\theta)\) gives a factorization, and Theorem 3.2 applies. Points outside \(\mathcal X_0\) have density zero for all \(\theta\) and are covered by \(h(x)=0\) .
Minimality. Let \(T'\) be any sufficient statistic, with factorization \(f(x;\theta)=g'(T'(x),\theta)h'(x)\) . Take \(x,y\in\mathcal X_0\) with \(T'(x)=T'(y)\) . Since each has positive density for some \(\theta\) , both \(h'(x)\) and \(h'(y)\) are positive, and
By hypothesis this proportionality forces \(T(x)=T(y)\) . So \(T\) takes equal values wherever \(T'\) does, which means \(T\) is a function of \(T'\) .
In practice one forms the ratio \(f(x;\theta)/f(y;\theta)\) and asks for which pairs it is free of \(\theta\) .
Example 3.3: A uniform window of unit width
Let \(X_1,\dots,X_n\) be a random sample from the uniform distribution on \((\theta,\theta+1)\) , with \(\theta\in\mathbb R\) . The joint density is \(1\) when every observation lies in the window and \(0\) otherwise:
By the factorization theorem the pair \(T=(X_{(1)},X_{(n)})\) is sufficient. To see that it is minimal, note that as a function of \(\theta\) the density is the indicator of the interval \((x_{(n)}-1,\,x_{(1)})\) . Two data sets \(x\) and \(y\) have proportional densities for all \(\theta\) exactly when these intervals coincide, that is, when \(x_{(1)}=y_{(1)}\) and \(x_{(n)}=y_{(n)}\) . Theorem 3.4 gives minimal sufficiency.
For the observations \(2.3\) , \(2.9\) , \(2.5\) the density is positive exactly for \(1.9<\theta<2.3\) . The middle observation \(2.5\) plays no role, and neither would any number of further observations between \(2.3\) and \(2.9\) . This is a one-parameter model whose minimal sufficient statistic is two-dimensional: the dimension of the parameter does not determine how far the data can be reduced.
For the normal model with both parameters unknown, the same criterion shows that \((\bar X,S^2)\) is minimal sufficient (Exercise 7). For the Bernoulli model the ratio \(p^{\sum x_i-\sum y_i}(1-p)^{\sum y_i-\sum x_i}\) is free of \(p\) exactly when the two sums agree, so the count is minimal sufficient, confirming the polling firm’s practice.
3.4 Exponential families
Most of the models met so far share an algebraic form in which sufficiency is visible at a glance and in which the sufficient statistic has the same dimension for every sample size.
Definition 3.5 (exponential family). A family of densities or mass functions is a \(k\) -parameter exponential family if it can be written
where \(h\ge 0\) and \(T_1,\dots,T_k\) do not depend on \(\theta\) , and \(\eta_1,\dots,\eta_k\) and \(B\) do not depend on \(x\) . The functions \(\eta_j\) are the natural parameters and \((T_1,\dots,T_k)\) is the natural statistic. When the family is indexed directly by \(\eta\) , so that \(f(x;\eta)=h(x)\exp(\sum_j\eta_jT_j(x)-A(\eta))\) , it is in natural form, and the set of \(\eta\) for which the density can be normalised is the natural parameter space.
A necessary feature is that the set where \(f(x;\theta)>0\) is the set where \(h(x)>0\) , which does not depend on \(\theta\) . The uniform families of Examples 3.2 and 3.3 are therefore not exponential families.
| Family | Natural parameter | Natural statistic | \(B(\theta)\) |
|---|---|---|---|
| Bernoulli, success probability \(p\) | \(\log\frac{p}{1-p}\) | \(x\) | \(-\log(1-p)\) |
| Poisson, mean \(\lambda\) | \(\log\lambda\) | \(x\) | \(\lambda\) |
| Exponential, rate \(\lambda\) | \(-\lambda\) | \(x\) | \(-\log\lambda\) |
| Normal, mean \(\mu\) , known variance \(\sigma^2\) | \(\mu/\sigma^2\) | \(x\) | \(\mu^2/(2\sigma^2)\) |
| Normal, both unknown | \(\bigl(\mu/\sigma^2,\ -1/(2\sigma^2)\bigr)\) | \((x,\ x^2)\) | \(\mu^2/(2\sigma^2)+\log\sigma\) |
Proposition 3.6 (sufficiency in exponential families). Let \(X_1,\dots,X_n\) be a random sample from the exponential family of Definition 3.5. Then the joint density is again of exponential-family form, with the same natural parameters and natural statistic \(\bigl(\sum_i T_1(X_i),\dots,\sum_i T_k(X_i)\bigr)\) , which is sufficient.
Proof. Multiply the \(n\) densities. The exponents add:
This has the form required by the factorization theorem with the stated statistic.
The proposition explains why counts, sums, and sums of squares appear so often: they are the natural statistics of the common families. It also shows that a sample of any size is reduced to \(k\) numbers. Outside exponential families such a fixed-dimensional reduction is unusual when the support is the same for all \(\theta\) . In the Cauchy location family, for example, no reduction beyond the order statistics is possible.
In natural form the normalising function \(A\) generates the moments of the natural statistic.
Proposition 3.7 (moments from the normaliser). Let \(f(x;\eta)=h(x)\exp(\eta T(x)-A(\eta))\) be a one-parameter exponential family in natural form, and let \(\eta\) be an interior point of the natural parameter space. Then
Proof. Normalisation says \(e^{A(\eta)}=\int h(x)e^{\eta T(x)}\,dx\)
, with a sum in place of the integral for discrete data. At interior points of the natural parameter space this integral may be differentiated under the integral sign any number of times; the justification is the Dominated Convergence Theorem of Measure Theory (math.measure-theory.dominated-convergence-theorem), applied to difference quotients of \(e^{\eta T}\)
. Differentiating once,
and dividing by \(e^{A(\eta)}\) gives \(A'(\eta)=E_\eta T\) . Differentiating again,
so \(A''(\eta)+A'(\eta)^2=E_\eta T^2\) and \(A''(\eta)=E_\eta T^2-(E_\eta T)^2=\operatorname{Var}_\eta T\) .
Example 3.4: Means and variances without summation
Poisson. With \(\eta=\log\lambda\) the mass function is \(\frac{1}{x!}\exp(\eta x-e^{\eta})\) , so \(A(\eta)=e^{\eta}\) . Then \(A'(\eta)=e^\eta=\lambda\) and \(A''(\eta)=e^\eta=\lambda\) : the mean and variance of a Poisson variable are both \(\lambda\) , obtained by differentiation with no series to sum.
Bernoulli. With \(\eta=\log\frac{p}{1-p}\) , the mass function is \(\exp(\eta x-\log(1+e^\eta))\) , so \(A(\eta)=\log(1+e^\eta)\) . Then
Normal with mean zero. The \(N(0,\sigma^2)\) density is \((2\pi)^{-1/2}\exp(\eta x^2+\tfrac12\log(-2\eta))\) with \(\eta=-1/(2\sigma^2)<0\) and \(T(x)=x^2\) , so \(A(\eta)=-\tfrac12\log(-2\eta)\) . Then \(A'(\eta)=-1/(2\eta)=\sigma^2\) and \(A''(\eta)=1/(2\eta^2)=2\sigma^4\) , which are the mean and variance of \(X^2\) , in agreement with the chi-square moments of Section 2.2.
Since a variance is nonnegative, \(A''\ge 0\) : the function \(A\) is convex, and strictly convex when \(T\) is not constant, since then \(\operatorname{Var}_\eta T>0\) . That strict convexity is the reason a likelihood equation in an exponential family has at most one solution, a fact Chapter 5 relies on.
3.5 Completeness, ancillarity, and Basu’s theorem
A sufficient statistic can still contain components whose distribution is free of \(\theta\) , and such components are dead weight. Completeness is the property that rules this out.
Definition 3.8 (complete statistic). A statistic \(T\) is complete for the model \(\{P_\theta:\theta\in\Theta\}\) if, for every real function \(g\) for which the expectations exist,
Completeness is a statement about the family of sampling distributions of \(T\) : it is rich enough that no nonzero function of \(T\) averages to zero under every member. An equivalent reading is that two functions of \(T\) with the same expectation for every \(\theta\) must coincide. That uniqueness is how completeness will be used in Chapter 6.
Binomial counts are complete. Let \(T\sim\operatorname{Binomial}(n,p)\) with \(p\in(0,1)\) and suppose \(E_p\,g(T)=0\) for all \(p\) . Dividing by \((1-p)^n\) and writing \(r=p/(1-p)\) ,
A polynomial in \(r\) that vanishes on an interval has all coefficients zero, so \(g(t)\binom nt=0\) and hence \(g(t)=0\) for \(t=0,\dots,n\) .
The uniform maximum is complete. Let \(T=X_{(n)}\) for a random sample from the uniform distribution on \((0,\theta)\) ; by Example 2.3 it has density \(nt^{n-1}/\theta^n\) on \((0,\theta)\) . If \(E_\theta\,g(T)=0\) for all \(\theta>0\) then \(\int_0^\theta g(t)t^{n-1}\,dt=0\) for all \(\theta>0\) . For continuous \(g\) , differentiating with respect to \(\theta\) gives \(g(\theta)\theta^{n-1}=0\) , so \(g\) vanishes identically; for general integrable \(g\) the same differentiation is valid at almost every \(\theta\) , which is enough.
For exponential families completeness comes with a simple sufficient condition.
Theorem 3.9 (completeness in exponential families). In a \(k\) -parameter exponential family in natural form, if the set of natural parameter values contains an open subset of \(\mathbb R^k\) , then the natural statistic is complete, and it is minimal sufficient.
The proof rests on the uniqueness of Laplace transforms and is not given here. The condition is easy to check. For a random sample from \(N(\mu,\sigma^2)\) with both parameters free, the natural parameter \((\mu/\sigma^2,-1/(2\sigma^2))\) ranges over the open half-plane \(\mathbb R\times(-\infty,0)\) , so \((\sum X_i,\sum X_i^2)\) is complete. For the family \(N(\theta,\theta^2)\) with \(\theta>0\) , the natural parameter \((1/\theta,-1/(2\theta^2))\) traces a curve in the plane, the condition fails, and Exercise 8 shows that completeness fails with it.
Example 3.5: A sufficient statistic that is not complete
Return to the uniform distribution on \((\theta,\theta+1)\) of Example 3.3, with minimal sufficient statistic \((X_{(1)},X_{(n)})\) . Write \(X_i=\theta+U_i\) with \(U_i\) uniform on \((0,1)\) . The range
does not involve \(\theta\) at all. From Proposition 2.9, \(E\,U_{(n)}=n/(n+1)\) and \(E\,U_{(1)}=1/(n+1)\) , so \(E_\theta R=(n-1)/(n+1)\) for every \(\theta\) ; for \(n=5\) this is \(2/3\) . The function
has expectation zero for every \(\theta\) and is not the zero function. So the minimal sufficient statistic is not complete. Minimal sufficiency and completeness are different properties; a complete sufficient statistic is always minimal, and the converse fails.
The range in Example 3.5 is an instance of a general notion.
Definition 3.10 (ancillary statistic). A statistic \(V\) is ancillary if its distribution is the same under every \(P_\theta\) .
An ancillary statistic, taken alone, carries no information about \(\theta\) . In a location family every function of the differences \(X_i-X_j\) is ancillary, and in a scale family every function of the ratios \(X_i/X_j\) is. One might guess that an ancillary statistic is independent of a sufficient one. That is false in general, as the next paragraph will show, and true when the sufficient statistic is complete.
Theorem 3.11 (Basu). If \(T\) is a complete sufficient statistic and \(V\) is ancillary, then \(T\) and \(V\) are independent under every \(P_\theta\) .
Proof. Fix an event \(B\)
in the range of \(V\)
. Since \(V\)
is ancillary, \(c=P_\theta(V\in B)\)
does not depend on \(\theta\)
. Since \(T\)
is sufficient, the conditional probability \(g(t)=P(V\in B\mid T=t)\)
does not depend on \(\theta\)
either, because \(V\)
is a function of the data. By the tower property of conditional expectation (math.probability-theory.tower-property),
Thus \(E_\theta[g(T)-c]=0\) for all \(\theta\) , and completeness gives \(g(T)=c\) with probability one. That is, \(P(V\in B\mid T)=P(V\in B)\) for every \(B\) , which is independence.
In Example 3.5, \(R\) is ancillary and is a function of the minimal sufficient statistic, so it is certainly not independent of it; there is no contradiction, because that statistic is not complete.
Example 3.6: Two uses of Basu’s theorem
Mean and variance of a normal sample. Fix \(\sigma^2\) and consider the model \(\{N(\mu,\sigma^2):\mu\in\mathbb R\}\) . It is a one-parameter exponential family with natural parameter \(\mu/\sigma^2\) ranging over all of \(\mathbb R\) , so \(\bar X\) is complete and sufficient. The sample variance \(S^2\) depends on the data only through the differences \(X_i-\bar X\) , which are unchanged when a constant is added to every observation, so \(S^2\) is ancillary in this location model. By Basu’s theorem \(\bar X\) and \(S^2\) are independent. Since \(\sigma^2\) was arbitrary, this is a second proof of the independence statement in Theorem 2.5, with no matrices.
A ratio in an exponential sample. Let \(X_1,\dots,X_{10}\) be a random sample from \(\operatorname{Exp}(\lambda)\) . The total \(T=\sum X_i\) is complete and sufficient, since the natural parameter \(-\lambda\) ranges over an open interval. The ratio \(V=X_{(1)}/\bar X\) is unchanged when every observation is multiplied by a constant, so it is ancillary in this scale family. By Basu’s theorem \(V\) and \(\bar X\) are independent, and therefore
By Exercise 2.10 the minimum is \(\operatorname{Exp}(10\lambda)\) with mean \(1/(10\lambda)\) , and \(E\,\bar X=1/\lambda\) . Hence \(E\,V=\tfrac{1}{10\lambda}\big/\tfrac1\lambda=0.1\) . The expectation of a ratio has been obtained from two easy expectations, a step that is legitimate only because the independence was proved first.
The chapter has produced a hierarchy. Any sufficient statistic may replace the data. A minimal sufficient statistic is the smallest such replacement. A complete sufficient statistic, when one exists, is minimal and has no ancillary part. What this buys for estimation appears in Chapter 6, where an estimator that is not a function of a sufficient statistic is shown to be improvable, and where completeness singles out one unbiased estimator as best. First, the next two chapters ask what “best” should mean and where estimators come from.
Exercises
-
State the definition of a sufficient statistic. For three independent Bernoulli trials with success probability \(p\) , list the conditional distribution of \((X_1,X_2,X_3)\) given \(X_1+X_2+X_3=1\) and confirm that it is free of \(p\) .
Answer
Given the total is \(1\) , the three sequences \((1,0,0)\) , \((0,1,0)\) , \((0,0,1)\) each have conditional probability \(p(1-p)^2/\bigl(3p(1-p)^2\bigr)=1/3\) , and every other sequence has conditional probability \(0\) . No \(p\) appears.
-
Use the factorization theorem to find a sufficient statistic for a random sample of size \(n\) from each family: (a) the geometric mass function \(p(1-p)^{x}\) on \(x=0,1,2,\dots\) ; (b) the gamma density with both shape and rate unknown; (c) the density \(\theta x^{\theta-1}\) on \((0,1)\) with \(\theta>0\) ; (d) the shifted exponential density \(e^{-(x-\theta)}\) on \(x>\theta\) .
Answer
(a) \(\sum X_i\) . (b) \(\bigl(\sum X_i,\ \sum\log X_i\bigr)\) , or equivalently \(\bigl(\sum X_i,\ \prod X_i\bigr)\) . (c) \(\sum\log X_i\) , or equivalently \(\prod X_i\) . (d) The minimum \(X_{(1)}\) : the joint density is \(e^{n\theta}\mathbf 1\{x_{(1)}>\theta\}\cdot e^{-\sum x_i}\) .
-
For two independent Poisson observations with mean \(\lambda\) , compute the conditional distribution of \(X_1\) given \(X_1+X_2=t\) directly from the definition, identify it by name, and conclude that \(X_1+X_2\) is sufficient.
Solution
Let \(T=X_1+X_2\) . A sum of independent Poisson variables with means \(\lambda\) and \(\lambda\) is Poisson with mean \(2\lambda\) , so \(P_\lambda(T=t)=e^{-2\lambda}(2\lambda)^t/t!\) . For \(0\le k\le t\) the event \(\{X_1=k,\ T=t\}\) is the event \(\{X_1=k,\ X_2=t-k\}\) , with probability
\[ \frac{e^{-\lambda}\lambda^k}{k!}\cdot\frac{e^{-\lambda}\lambda^{t-k}}{(t-k)!}=\frac{e^{-2\lambda}\lambda^t}{k!\,(t-k)!}. \]Dividing,
\[ P_\lambda(X_1=k\mid T=t)=\frac{e^{-2\lambda}\lambda^t}{k!\,(t-k)!}\cdot\frac{t!}{e^{-2\lambda}(2\lambda)^t}=\binom tk\Bigl(\frac12\Bigr)^t . \]This is the binomial distribution with \(t\) trials and success probability \(1/2\) . It contains no \(\lambda\) . Since \(X_2=t-X_1\) on the conditioning event, the conditional distribution of the whole data \((X_1,X_2)\) given \(T=t\) is also free of \(\lambda\) , and by Definition 3.1 the statistic \(T\) is sufficient. Intuitively, once the total is known, each of the \(t\) events is equally likely to have come from either observation.
-
For three independent Bernoulli trials with success probability \(p\) , show that \(U=X_1+2X_2+3X_3\) is not sufficient by computing \(P_p\bigl(X=(1,1,0)\mid U=3\bigr)\) .
Solution
The value \(U=3\) arises from exactly two outcomes: \((1,1,0)\) , since \(1+2=3\) , and \((0,0,1)\) , since the third trial alone contributes \(3\) . Their probabilities are \(p^2(1-p)\) and \(p(1-p)^2\) . Hence
\[ P_p\bigl(X=(1,1,0)\mid U=3\bigr)=\frac{p^2(1-p)}{p^2(1-p)+p(1-p)^2}=\frac{p}{p+(1-p)}=p . \]The conditional probability equals \(p\) itself, so the conditional distribution of the data given \(U\) depends on the parameter and \(U\) is not sufficient. The reason is visible in the two outcomes: they share a value of \(U\) but have different numbers of successes, two against one, and the number of successes is what the likelihood depends on. A sufficient statistic may not merge data sets whose likelihood functions are not proportional, and \(U\) does.
-
Find a one-dimensional sufficient statistic for a random sample from the uniform distribution on \((-\theta,\theta)\) , \(\theta>0\) , and show that it is minimal sufficient.
Hint
The density is \((2\theta)^{-n}\mathbf 1\{\max_i\lvert x_i\rvert<\theta\}\) . For minimality apply Theorem 3.4: as a function of \(\theta\) the density of \(x\) is a multiple of the indicator of \(\theta>\max_i\lvert x_i\rvert\) , and two such functions are proportional exactly when the thresholds coincide.
-
Let \(T\) be sufficient. Prove from the factorization theorem that (a) \(\psi(T)\) is sufficient for every one-to-one \(\psi\) , and (b) the pair \((T,U)\) is sufficient for every statistic \(U\) . Explain what (b) shows about sufficiency as a measure of data reduction.
Hint
(a) If \(f=g(T,\theta)h\) and \(T'=\psi(T)\) , then \(f=g(\psi^{-1}(T'),\theta)h\) . (b) Define \(\tilde g((t,u),\theta)=g(t,\theta)\) . Part (b) shows that adding irrelevant components never destroys sufficiency, so sufficiency says only that nothing has been lost, not that anything has been discarded; minimal sufficiency is what measures reduction.
-
Use Theorem 3.4 to prove that \((\bar X,S^2)\) is minimal sufficient for a random sample from \(N(\mu,\sigma^2)\) with both parameters unknown.
Solution
From Example 3.2 the joint density is
\[ f(x;\mu,\sigma^2)=(2\pi\sigma^2)^{-n/2}\exp\Bigl(-\frac{1}{2\sigma^2}\sum x_i^2+\frac{\mu}{\sigma^2}\sum x_i-\frac{n\mu^2}{2\sigma^2}\Bigr), \]positive for all \(x\) , so \(\mathcal X_0=\mathbb R^n\) . For two data sets \(x\) and \(y\) ,
\[ \frac{f(x;\mu,\sigma^2)}{f(y;\mu,\sigma^2)}=\exp\Bigl(-\frac{1}{2\sigma^2}\Bigl(\sum x_i^2-\sum y_i^2\Bigr)+\frac{\mu}{\sigma^2}\Bigl(\sum x_i-\sum y_i\Bigr)\Bigr). \]If \(\sum x_i=\sum y_i\) and \(\sum x_i^2=\sum y_i^2\) the ratio is \(1\) for every parameter value. Conversely, suppose the ratio is a constant \(c\) for all \((\mu,\sigma^2)\) . Taking \(\mu=0\) shows that \(-\frac{1}{2\sigma^2}(\sum x_i^2-\sum y_i^2)\) does not depend on \(\sigma^2\) , which forces \(\sum x_i^2=\sum y_i^2\) . Then \(\frac{\mu}{\sigma^2}(\sum x_i-\sum y_i)\) must not depend on \(\mu\) , which forces \(\sum x_i=\sum y_i\) . So proportionality holds exactly when \(T(x)=T(y)\) for \(T=(\sum X_i,\sum X_i^2)\) , and Theorem 3.4 shows \(T\) is minimal sufficient. Finally \(\bar X=\sum X_i/n\) and \(S^2=(\sum X_i^2-n\bar X^2)/(n-1)\) determine and are determined by \(T\) , and a one-to-one function of a minimal sufficient statistic is minimal sufficient.
-
Let \(X_1,\dots,X_n\) be a random sample from \(N(\theta,\theta^2)\) with \(\theta>0\) and \(n\ge 2\) . Show that \((\bar X,S^2)\) is still minimal sufficient, compute \(E_\theta\bar X^2\) and \(E_\theta S^2\) , and use them to construct a nonzero function of \((\bar X,S^2)\) whose expectation is zero for every \(\theta\) .
Solution
With \(\mu=\theta\) and \(\sigma^2=\theta^2\) the density ratio of Exercise 3.7 becomes
\[ \exp\Bigl(-\frac{a}{2\theta^2}+\frac{b}{\theta}\Bigr),\qquad a=\sum x_i^2-\sum y_i^2,\quad b=\sum x_i-\sum y_i . \]If this is constant in \(\theta>0\) then \(-a/(2\theta^2)+b/\theta\) is constant; letting \(\theta\to\infty\) the constant is \(0\) , and then \(-a/2+b\theta=0\) for all \(\theta>0\) gives \(b=0\) and \(a=0\) . So \((\sum X_i,\sum X_i^2)\) , equivalently \((\bar X,S^2)\) , is minimal sufficient by Theorem 3.4, even though there is only one parameter.
By Theorem 2.2, \(E_\theta\bar X=\theta\) and \(\operatorname{Var}_\theta\bar X=\theta^2/n\) , so
\[ E_\theta\bar X^2=\theta^2+\frac{\theta^2}{n}=\frac{n+1}{n}\theta^2,\qquad E_\theta S^2=\theta^2 . \]Therefore
\[ g(\bar X,S^2)=\frac{n}{n+1}\bar X^2-S^2 \]satisfies \(E_\theta\,g=0\) for every \(\theta>0\) . It is not zero with probability one, since \(\bar X^2\) and \(S^2\) are independent continuous variables. Hence the minimal sufficient statistic is not complete. The model has two different unbiased estimators of \(\theta^2\) that are both functions of the minimal sufficient statistic, and the theory of Chapter 6 will not single out a best one.
-
Write the \(\operatorname{Gamma}(a,b)\) family with known shape \(a\) and unknown rate \(b\) as a one-parameter exponential family in natural form, and use Proposition 3.7 to obtain the mean and variance of \(X\) .
Hint
The density is \(\frac{x^{a-1}}{\Gamma(a)}\exp(-bx+a\log b)\) . Take \(\eta=-b\) , \(T(x)=x\) , and \(A(\eta)=-a\log(-\eta)\) . Differentiate twice; you should recover the mean \(a/b\) and the variance \(a/b^2\) .
-
Decide which of the following are exponential families and give the reason in each case: (a) binomial with known number of trials; (b) uniform on \((0,\theta)\) ; (c) the Weibull density \(\tfrac{2x}{\theta}e^{-x^2/\theta}\) on \(x>0\) ; (d) the Cauchy location family.
Answer
(a) Yes: natural parameter \(\log\frac{p}{1-p}\) , natural statistic \(x\) . (b) No: the support \((0,\theta)\) depends on \(\theta\) . (c) Yes: natural parameter \(-1/\theta\) , natural statistic \(x^2\) , with \(h(x)=2x\) . (d) No: \(\log f\) is \(-\log(1+(x-\theta)^2)\) up to a constant, which cannot be written as a finite sum of products \(\eta_j(\theta)T_j(x)\) .
-
Prove that for a random sample from the Poisson distribution with mean \(\lambda\in(0,\infty)\) the statistic \(T=\sum X_i\) is complete, working directly from the definition and the fact that \(T\) is Poisson with mean \(n\lambda\) .
Solution
Since \(T\sim\operatorname{Poisson}(n\lambda)\) , the condition \(E_\lambda\,g(T)=0\) for all \(\lambda>0\) reads
\[ \sum_{t=0}^{\infty}g(t)\,\frac{e^{-n\lambda}(n\lambda)^t}{t!}=0\qquad\text{for all }\lambda>0 . \]Multiply by \(e^{n\lambda}\) , which is never zero:
\[ \sum_{t=0}^{\infty}\frac{g(t)\,n^t}{t!}\,\lambda^t=0\qquad\text{for all }\lambda>0 . \]The left side is a power series in \(\lambda\) that converges for every \(\lambda>0\) , because the expectation exists. A power series that vanishes on an interval has all its coefficients equal to zero; one sees this by letting \(\lambda\to 0\) to get the constant term, dividing by \(\lambda\) , and repeating. Hence \(g(t)n^t/t!=0\) for every \(t\) , and since \(n^t/t!>0\) , \(g(t)=0\) for all \(t=0,1,2,\dots\) . So \(g(T)=0\) with probability one under every \(\lambda\) , which is completeness. The argument used that \(\lambda\) ranges over an interval; if the model allowed only two values of \(\lambda\) , the two linear conditions on infinitely many unknowns \(g(t)\) would have nonzero solutions and completeness would fail.
-
A student claims: “For a random sample from the uniform distribution on \((0,\theta)\) the sample mean is sufficient, because \(E_\theta\bar X=\theta/2\) determines \(\theta\) .” Explain what property the student has actually verified, and show that \(\bar X\) is not sufficient when \(n=2\) by exhibiting two data sets with the same mean whose density ratio depends on \(\theta\) .
Solution
The student has verified that \(\theta\) is identifiable through the mean of \(\bar X\) , in the sense of Proposition 1.7. That is a property of the parametrization. Sufficiency is a different property: it concerns whether the data given the statistic have a distribution free of \(\theta\) .
Take \(n=2\) and the two data sets \(x=(1,1)\) and \(y=(0.5,1.5)\) , both with mean \(1\) . The joint density is \(\theta^{-2}\mathbf 1\{\max(x_1,x_2)<\theta\}\) for positive observations. For \(\theta=1.2\) it equals \(1/1.44\) at \(x\) and \(0\) at \(y\) , because \(1.5>1.2\) . For \(\theta=2\) it equals \(1/4\) at both. So the ratio \(f(x;\theta)/f(y;\theta)\) is not constant in \(\theta\) . If \(\bar X\) were sufficient, the factorization theorem would give \(f(x;\theta)/f(y;\theta)=h(x)/h(y)\) for any two data sets with the same mean, a constant. Hence \(\bar X\) is not sufficient. The data set \(y\) rules out \(\theta\le 1.5\) and the data set \(x\) does not, and that distinction, carried by the maximum, is lost when only the mean is kept.
-
Show that in a location family every statistic of the form \(\psi(X_1-X_n,\dots,X_{n-1}-X_n)\) is ancillary, and that in a scale family every statistic of the form \(\psi(X_1/X_n,\dots,X_{n-1}/X_n)\) is ancillary.
Hint
Write \(X_i=\mu+Z_i\) in the location case; then \(X_i-X_n=Z_i-Z_n\) . Write \(X_i=\sigma Z_i\) in the scale case; then \(X_i/X_n=Z_i/Z_n\) . In both cases the statistic is a function of the \(Z_i\) alone.
-
Let \(X_1,\dots,X_n\) be a random sample from \(N(\mu,\sigma^2)\) . Use Basu’s theorem to prove that \(\bar X\) is independent of the range \(X_{(n)}-X_{(1)}\) , stating carefully the model in which the theorem is applied and why the conclusion holds when \(\sigma^2\) is unknown.
Solution
Fix a value \(\sigma_0^2>0\) and consider the submodel \(\{N(\mu,\sigma_0^2):\mu\in\mathbb R\}\) . Its joint density is an exponential family with natural parameter \(\mu/\sigma_0^2\) , which ranges over all of \(\mathbb R\) , an open set, and natural statistic \(\sum X_i\) . By Theorem 3.9, \(\bar X\) is complete and sufficient in this submodel.
Write \(X_i=\mu+\sigma_0Z_i\) with \(Z_i\) standard normal. Then \(X_{(n)}-X_{(1)}=\sigma_0(Z_{(n)}-Z_{(1)})\) , whose distribution involves \(\sigma_0\) , which is fixed, and not \(\mu\) . So the range is ancillary in the submodel. Basu’s theorem gives independence of \(\bar X\) and the range under \(N(\mu,\sigma_0^2)\) for every \(\mu\) .
Independence of two statistics is a property of their joint distribution under one particular law. Every law \(N(\mu,\sigma^2)\) in the full two-parameter model belongs to the submodel with \(\sigma_0^2=\sigma^2\) , and the argument above applies to each such submodel. Therefore \(\bar X\) and the range are independent under every member of the full model. Note that in the full model with \(\sigma^2\) unknown the range is not ancillary and \(\bar X\) alone is not sufficient; Basu’s theorem could not have been applied there directly.
-
For a random sample of size \(n\) from \(\operatorname{Exp}(\lambda)\) let \(W=X_1/\sum_i X_i\) . Use Basu’s theorem to find \(E\,W\) and \(E\,W^2\) without integrating a joint density.
Answer
\(T=\sum X_i\) is complete sufficient and \(W\) is ancillary (scale-invariant), so they are independent and \(E\,X_1^k=E\,W^k\,E\,T^k\) . With \(T\sim\operatorname{Gamma}(n,\lambda)\) : \(E\,W=\frac{1/\lambda}{n/\lambda}=\frac1n\) and \(E\,W^2=\frac{2/\lambda^2}{n(n+1)/\lambda^2}=\frac{2}{n(n+1)}\) .
-
Two Poisson samples of size three are \((1,4,1)\) and \((2,2,2)\) . Show that their likelihood functions \(\lambda\mapsto f(x;\lambda)\) are proportional, find the constant of proportionality, and relate the result to minimal sufficiency.
Answer
Both samples have total \(6\) . The likelihoods are \(e^{-3\lambda}\lambda^6/(1!\,4!\,1!)\) and \(e^{-3\lambda}\lambda^6/(2!\,2!\,2!)\) , so the first is \(8/24=1/3\) of the second for every \(\lambda\) . Proportional likelihoods are exactly the condition under which a minimal sufficient statistic takes equal values, here \(\sum x_i=6\) .
-
Explain, using the factorization theorem, why the vector of order statistics is sufficient for every model in which the observations are a random sample from a density, and why the number of defectives is sufficient for \(D\) in the finite-lot model of Example 1.5 when the draws are recorded in order.
Hint
The joint density \(\prod f(x_i;\theta)\) is a symmetric function of \(x_1,\dots,x_n\) , so it can be written as a function of the sorted values alone. For the lot, write the probability of a particular ordered sequence of defectives and non-defectives and check that it depends on the sequence only through the number of defectives.
-
Let \(X\sim N(0,\sigma^2)\) be a single observation with \(\sigma^2>0\) unknown. Show that \(X\) is sufficient but not complete, that \(\lvert X\rvert\) is sufficient, and that \(X^2\) is complete; then explain why these three facts are consistent.
Solution
Sufficient, not complete. The whole data set is always sufficient, and here the data are \(X\) . Take \(g(x)=x\) . Then \(E_\sigma\,g(X)=0\) for every \(\sigma^2\) , yet \(P(X=0)=0\) , so \(g(X)\) is not almost surely zero. Hence \(X\) is not complete.
Sufficiency of the absolute value. The density \((2\pi\sigma^2)^{-1/2}\exp(-x^2/(2\sigma^2))\) depends on \(x\) only through \(x^2=\lvert x\rvert^2\) , so the factorization theorem applies with \(T=\lvert X\rvert\) and \(h=1\) .
Completeness of the square. By Example 3.4 the model is a one-parameter exponential family with natural statistic \(X^2\) and natural parameter \(-1/(2\sigma^2)\) , which ranges over the open interval \((-\infty,0)\) . Theorem 3.9 gives completeness of \(X^2\) , and \(\lvert X\rvert\) , being a one-to-one function of \(X^2\) , is complete as well.
Consistency. Completeness is a property of a particular statistic. \(X\) can be written as the pair (sign of \(X\) , \(\lvert X\rvert\) ). The sign is \(\pm1\) with probability \(1/2\) each for every \(\sigma^2\) : it is ancillary. A statistic that contains an ancillary component cannot be complete, because a centred function of that component has mean zero under every parameter value. Stripping away the sign leaves \(\lvert X\rvert\) , which is both sufficient and complete, and by Basu’s theorem the sign and \(\lvert X\rvert\) are independent.
-
For \(n\) independent Bernoulli trials with total \(T\) , compute \(E[X_1\mid T=t]\) and observe that it does not depend on \(p\) . What general property of sufficient statistics does this illustrate?
Answer
By symmetry \(E[X_i\mid T=t]\) is the same for each \(i\) , and the conditional expectations sum to \(t\) , so \(E[X_1\mid T=t]=t/n\) . No \(p\) appears: the conditional expectation of any statistic given a sufficient statistic can be computed without knowing the parameter, so it is itself a statistic. Chapter 6 builds the Rao–Blackwell theorem on this.