Chapter 4, Asymptotics and Connections to non-Bayesian Approaches Video Solutions, Bayesian Data Analysis

Problem 1

Normal approximation: suppose that $y_1, \ldots, y_5$ are independent samples from a Cauchy distribution with unknown center $\theta$ and known scale 1: $p\left(y_i \mid \theta\right) \propto 1 /\left(1+\left(y_i-\theta\right)^2\right)$. Assume that the prior distribution for $\theta$ is uniform on $[0,1]$. Given the observations $\left(y_1, \ldots, y_5\right)=(-2,-1,0,1.5,2.5)$ :
(a) Determine the derivative and the second derivative of the log posterior density.
(b) Find the posterior mode of $\theta$ by iteratively solving the equation determined by setting the derivative of the log-likelihood to zero.
(c) Construct the normal approximation based on the second derivative of the log posterior density at the mode. Plot the approximate normal density and compare to the exact density computed in Exercise 2.11.

Abhirup Pal

Numerade Educator

02:27

Problem 2

Normal approximation: derive the analytic form of the information matrix and the normal approximation variance for the bioassay example.

Adriano Chikande

Numerade Educator

Problem 3

Normal approximation to the marginal posterior distribution of an estimand: in the bioassay example, the normal approximation to the joint posterior distribution of $(\alpha, \beta)$ is obtained. The posterior distribution of any estimand, such as the LD50, can be approximated by a normal distribution fit to its marginal posterior mode and the curvature of the marginal posterior density about the mode. This is sometimes called the 'delta method.' Expand the posterior distribution of the LD50, $-\alpha / \beta$, as a Taylor series around the posterior mode and thereby derive the asymptotic posterior median and standard deviation. Compare to the histogram in Figure 4.2.

Check back soon!

01:21

Problem 4

Asymptotic normality: assuming the regularity conditions hold, we know that $p(\theta \mid y)$ approaches normality as $n \rightarrow \infty$. In addition, if $\phi=f(\theta)$ is any one-to-one continuous transformation of $\theta$, we can express the Bayesian inference in terms of $\phi$ and find that $p(\phi \mid y)$ also approaches normality. But a nonlinear transformation of a normal distribution is no longer normal. How can both limiting normal distributions be valid?

Dominador Tan

Numerade Educator

Problem 5

Approximate mean and variance:
- Suppose $x$ and $y$ are independent normally distributed random variables, where $x$ has mean 4 and standard deviation 1 , and $y$ has mean 3 and standard deviation 2 . What are the mean and standard deviation of $y / x$ ? Compute this using simulation.
- Suppose $x$ and $y$ are independent random variables, where $x$ has mean 4 and standard deviation 1 , and $y$ has mean 3 and standard deviation 2 . What are the approximate mean and standard deviation of $y / x$ ? Determine this without using simulation.
- What assumptions are required for the approximation in (b) to be reasonable?

Check back soon!

Problem 6

Statistical decision theory: a decision-theoretic approach to the estimation of an unknown parameter $\theta$ introduces the loss function $L(\theta, a)$ which, loosely speaking, gives the cost of deciding that the parameter has the value $a$, when it is in fact equal to $\theta$. The estimate $a$ can be chosen to minimize the posterior expected loss,

$$
\mathrm{E}(L(a \mid y))=\int L(\theta, a) p(\theta \mid y) d \theta
$$

This optimal choice of $a$ is called a Bayes estimate for the loss function $L$. Show that:
(a) If $L(\theta, a)=(\theta-a)^2$ (squared error loss), then the posterior mean, $\mathrm{E}(\theta \mid y)$, if it exists, is the unique Bayes estimate of $\theta$.
(b) If $L(\theta, a)=|\theta-a|$, then any posterior median of $\theta$ is a Bayes estimate of $\theta$.
(c) If $k_0$ and $k_1$ are nonnegative numbers, not both zero, and

$$
L(\theta, a)= \begin{cases}k_0(\theta-a) & \text { if } \quad \theta \geq a \\ k_1(a-\theta) & \text { if } \quad \theta<a\end{cases}
$$

then any $\frac{k_0}{k_0+k_1}$ quantile of the posterior distribution $p(\theta \mid y)$ is a Bayes estimate of $\theta$.

Check back soon!

01:05

Problem 7

Unbiasedness: prove that the Bayesian posterior mean, based on a proper prior distribution, cannot be an unbiased estimator except in degenerate problems (see Bickel and Blackwell, 1967, and Lehmann, 1983, p. 244).

Victor Salazar

Numerade Educator

03:04

Problem 8

Regression to the mean: work through the details of the example of mother's and daughter's heights on page 94 , illustrating with a sketch of the joint distribution and relevant conditional distributions.

Kyle Ukes

Numerade Educator

Problem 9

Point estimation: suppose a measurement $y$ is recorded with a $\mathrm{N}\left(\theta, \sigma^2\right)$ sampling distribution, with $\sigma$ known exactly and $\theta$ known to lie in the interval $[0,1]$. Consider two point estimates of $\theta$ : (1) the maximum likelihood estimate, restricted to the range $[0,1]$, and (2) the posterior mean based on the assumption of a uniform prior distribution on $\theta$. Show that if $\sigma$ is large enough, estimate (1) has a higher mean squared error than (2) for any value of $\theta$ in $[0,1]$. (The unrestricted maximum likelihood estimate has even higher mean squared error.)

Check back soon!

Problem 10

Non-Bayesian inference: replicate the analysis of the bioassay example in Section 3.7 using non-Bayesian inference. This problem does not have a unique answer, so be clear on what methods you are using.
(a) Construct an 'estimator' of ( $\alpha, \beta$ ); that is, a function whose input is a dataset, ( $x, n, y$ ), and whose output is a point estimate $(\hat{\alpha}, \hat{\beta})$. Compute the value of the estimate for the data given in Table 5.2.
(b) The bias and variance of this estimate are functions of the true values of the parameters ( $\alpha, \beta$ ) and also of the sampling distribution of the data, given $\alpha, \beta$. Assuming the binomial model, estimate the bias and variance of your estimator.
(c) Create approximate $95 \%$ confidence intervals for $\alpha, \beta$, and the LD50 based on asymptotic theory and the estimated bias and variance.
(d) Does the inaccuracy of the normal approximation for the posterior distribution (compare Figures 3.3 and 4.1) cast doubt on the coverage properties of your confidence intervals in (c)? If so, why?
(e) Create approximate $95 \%$ confidence intervals for $\alpha, \beta$, and the LD50 using the jackknife or bootstrap (see Efron and Tibshirani, 1993).
(f) Compare your $95 \%$ intervals for the LD50 in (c) and (e) to the posterior distribution displayed in Figure 3.4 and the posterior distribution based on the normal approximation, displayed in 4.2b. Comment on the similarities and differences among the four intervals. Which do you prefer as an inferential summary about the LD50? Why?

Check back soon!

Problem 11

Bayesian interpretation of non-Bayesian estimates: consider the following estimation procedure, which is based on classical hypothesis testing. A matched pairs experiment is done, and the differences $y_1, \ldots, y_n$ are recorded and modeled as independent draws from $\mathrm{N}\left(\theta, \sigma^2\right)$. For simplicity, assume $\sigma^2$ is known. The parameter $\theta$ is estimated as the average observed difference if it is 'statistically significant' and zero otherwise:

$$
\hat{\theta}= \begin{cases}\bar{y} & \text { if } \bar{y} \geq 1.96 \sigma / \sqrt{n} \\ 0 & \text { otherwise }\end{cases}
$$

Can this be interpreted, in some sense, as an approximate summary (for example, a posterior mean or mode) of a Bayesian inference under some prior distribution on $\theta$ ?

Check back soon!

Problem 12

Bayesian interpretation of non-Bayesian estimates: repeat the above problem but with $\sigma$ replaced by $s$, the sample standard deviation of $y_1, \ldots, y_n$.

Check back soon!

Problem 13

Objections to Bayesian inference: discuss the criticism, 'Bayesianism assumes: (a) Either a weak or uniform prior [distribution], in which case why bother?, (b) Or a strong prior [distribution], in which case why collect new data?, (c) Or more realistically, something in between, in which case Bayesianism always seems to duck the issue' (Ehrenberg, 1986). Feel free to use any of the examples covered so far to illustrate your points.

Check back soon!

00:53

Problem 14

Objectivity and subjectivity: discuss the statement, 'People tend to believe results that support their preconceptions and disbelieve results that surprise them. Bayesian methods encourage this undisciplined mode of thinking.'

Prashant Bana

Numerade Educator

Problem 15

Coverage of posterior intervals:
(a) Consider a model with scalar parameter $\theta$. Prove that, if you draw $\theta$ from the prior, draw $y \mid \theta$ from the data model, then perform Bayesian inference for $\theta$ given $y$, that there is a $50 \%$ probability that your $50 \%$ interval for $\theta$ contains the true value.
(b) Suppose $\theta \sim \mathrm{N}\left(0,2^2\right)$ and $y \mid \theta \sim \mathrm{N}(\theta, 1)$. Suppose the true value of $\theta$ is 1 . What is the coverage of the posterior $50 \%$ interval for $\theta$ ? (You have to work this one out; it's not $50 \%$ or any other number you could just guess.)
(c) Suppose $\theta \sim \mathrm{N}\left(0,2^2\right)$ and $y \mid \theta \sim \mathrm{N}(\theta, 1)$. Suppose the true value of $\theta$ is $\theta_0$. Make a plot showing the coverage of the posterior $50 \%$ interval for $\theta$, as a function of $\theta_0$.

Check back soon!