Chapter A07: Accept–Reject Sampling for gaussian Regression models with independent normal-gamma priors8 days ago
Envelope construction with independent Normal–Gamma prior (Gaussian regression) | 1. Model Setup (Gaussian Regression with Independent Normal–Gamma Prior) | 1.1 Weighted log‑likelihood | [\mathrm{RSS}(\beta) | [\ell(\beta,\phi) | [\ell(\beta,\tau) | 1.2 Independent Normal–Gamma prior | [\log p(\beta,\tau) | [p_{\text{trunc}}(\tau) | \frac{\tau^{a_0 - 1} e^{-b_0\tau}}{F_{\Gamma}(\tau_{\max};a_0,b_0) | 2. Full Joint Log‑Posterior (Gaussian Case, Weighted) | [\log p(\beta,\tau \mid y) | 3. General Sampling Process | 4. Main Proposition and Supporting claims | | Constant | Definition | Description ||----------|------------|-------------|| Posterior Gamma parameters | $\text{shape}2 = \text{Shape} + \tfrac{n{w}}{2}$ $\text{rate}2 = \text{Rate} + \mathrm{RSS}{\text{post}}/2$ | Posterior shape and rate parameters. || Envelope dispersion anchor | $d^{}_{1} = \dfrac{\text{rate}2}{\text{shape}2 - 1}$ | Dispersion used for baseline envelope. || Dispersion bounds | $\begin{aligned}& \text{low} \[4pt]& \text{upp}\end{aligned}$ | Lower and upper bounds for dispersion. || Tangency offset vector | $B(d) = P\mu + \dfrac{1}{d},X^\top W(\alpha - y)$ | Offset vector used in the inverse map $c^{-1}(\bar c, d)$. || Tangency slope components | $\begin{aligned} Q &= X^\top X \ A(d) &= Q + dP \ r &= X^\top(y-\alpha) \ V(d) &= \bar c_j - P,A(d)^{-1}(r + d,\bar c_j) \end{aligned}$ | Components for quad–linear slope term. || Tangency inverse map | $c^{-1}(\bar c, d) = A(d)^{-1}(\bar c - B(d))$ | Maps gradient vector $\bar c$ and dispersion $d$ to tangency point $\theta$. || Tangency map | $\theta(d) = c^{-1}(\bar c_j, d)$ | Tangency point in coefficient space at dispersion $d$. || Tangency face energy | (g{1j}(d) = -\tfrac12,\theta(d)^\top P,\theta(d) + \bar c_j^\top \theta(d)) | Quadratic–linear face energy at dispersion (d). || Baseline face constant | (g{1j}(d^{}{1})) | Face‑specific constant obtained by evaluating (g{1j}(d)) at the anchor (d^{}{1}). || Derivative of face energy | $\begin{aligned} & g'{1j}(d) = V(d)^\top A(d)^{-1},\bar c_j \[6pt] & \qquad -, \Big( V(d)^\top A(d)^{-1} P,A(d)^{-1} \[2pt] & \qquad\qquad\qquad;;\times \big(r + d,\bar c_j\big) \Big) \end{aligned}$ | Derivative of $g_{1j}(d)$ with respect to dispersion. || Derivative at the anchor | $\begin{aligned}& g'_{1j}(d^{}{1}) = V(d^{*}{1})^\top A(d^{}_{1})^{-1},\bar c_j \[6pt]& \qquad -, \Big( V(d^{}{1})^\top A(d^{*}{1})^{-1} P,A(d^{}_{1})^{-1} \[2pt]& \qquad\qquad\qquad;;\times \big(r + d^{}{1},\bar c_j\big) \Big)\end{aligned}$ | Value of the derivative at the envelope anchor $d^{*}{1}$. || Tangency face energy | (g_{1j}(d) = -\tfrac12,\theta(d)^\top P,\theta(d) + \bar c_j^\top \theta(d)) | Quadratic–linear face energy at dispersion (d). || Mean quad–linear slope | $\mathrm{m}{g'{1}} = \displaystyle \operatorname*{mean}\limits_{j}!\big(g'{1j}(d^{*}{1})\big)$ | Average derivative $g'{1j}(d^{*}{1})$ across faces. || Supporting line for face (j) | (g_{2j}(d) = g_{1j}(d^{}_{1}) + (d - d^{}{1}),g'{1j}(d^{}{1})) | Linear supporting line of (g{1j}(d)) at the anchor (d^{}{1}). || Extrapolated face constants | $\begin{aligned}& g{2j}(\text{upp}) = g_{1j}(d^{}_{1}) \[2pt]& \qquad\qquad;; +, (\text{upp}-d^{}{1}), g'{1j}(d^{}{1}) \[6pt]& g{2j}(\text{low}) = g_{1j}(d^{}{1}) \[2pt]& \qquad\qquad;; +, (\text{low}-d^{*}{1}), g'{1j}(d^{*}{1})\end{aligned}$ | Linear extrapolations to dispersion bounds. || Endpoint maxima | $\begin{aligned}\mathrm{max_upp} & = \max_j!\big(g_{2j}(\text{upp})\big) \[8pt]\mathrm{max_low} & = \max_j!\big(g_{2j}(\text{low})\big)\end{aligned}$ | Maxima at upper and lower dispersion bounds. || Mean lower-bound maximum | $\begin{aligned}&\mathrm{max_low_mean} \[4pt]&= \mathrm{max_upp} \[4pt]&\quad - \mathrm{m}{g'{1}},(\text{upp}-\text{low})\end{aligned}$ | Linearized lower-bound maximum. || Global line parameters | $\text{lmc}2 = \dfrac{\mathrm{max_upp} - \mathrm{max_low_mean}}{\text{upp}-\text{low}}$ $\text{lmc}1 = \mathrm{max_low_mean} - \text{lmc}2,\text{low}$ | Slope and intercept of global affine bound. || Log–linear anchor | $d^{*}{2} = \dfrac{\text{upp}-\text{low}}{\log(\text{upp}/\text{low})}$ | Anchor point for log–tilt. || Log–tilt coefficients | $\begin{aligned} & \mathrm{lm_log2} = \[4pt]& \text{lmc}2,d^{*}{2} \[4pt]& \mathrm{lm_log1} = \[4pt]& \text{lmc}1 + \text{lmc}2 d^{*}{2} - \text{lmc}2\log(d^{*}{2}) \[4pt]& \mathrm{max_LL_log_disp} =\[4pt]& \mathrm{lm_log1} + \mathrm{lm_log2}\log(\text{upp}) \end{aligned}$ | Coefficients for log–tilt bounding function. || Face-specific RSS | $\mathrm{RSS}j(d) = \sum{i=1}^n w_i,(y_i - x_i^\top c^{-1}(\bar c_j,d))^2$ | Residual sum of squares for face $j$ at dispersion $d$. || Global minimum RSS | $\mathrm{RSS_Min} = \min{j};\min{d\in[\text{low},,\text{upp}]};\mathrm{RSS}j(d)$ | Global minimum RSS across all faces and dispersion values. || UB2 term | $\mathrm{UB2}j(d) = \dfrac{1}{2d}\big(\mathrm{RSS}j(d) - \mathrm{RSS_Min}\big)$ | Nonnegative UB2 bound for face $j$. || Per-face UB2 minimum | $\mathrm{UB2_Min}j = \min{d\in[\text{low},,\text{upp}]}\mathrm{UB2}j(d)$ | Minimum UB2 value for face $j$. || Per‑face shift (UB3A) | $\begin{aligned}& \mathrm{lg_prob_factor1}{j} = \[4pt]& \max\Big{, g{2j}(\text{upp}) - \mathrm{max_upp}, \[-2pt]& \qquad;; g{2j}(\text{low}) - \mathrm{max_low} \Big}\end{aligned}$ | Raw per‑face shift used in UB3A construction. || Per‑face shift (PLSD) | $\begin{aligned}& \mathrm{lg_prob_factor2}{j} = \[4pt]& \mathrm{lg_prob_factor1}{j} ;-; \mathrm{UB2_Min}{j}\end{aligned}$ | UB2‑adjusted shift used in PLSD mixture weights. || Global affine bound (g3_j) | $\displaystyleg3{j}(d) | From the definitions of the global affine bound,[\mathrm{lmc}_2 | \frac{\mathrm{max_upp} - \mathrm{max_low}}{\text{upp} - \text{low}},\qquad\mathrm{lmc}_1 | At (d = \text | 4.2 Proposal distributions | Gamma proposal in dispersion (d) (with truncation) | Mixture of truncated normals for (\beta) (two-step sampling) | 4.3 Correction terms | 4.4 Proposition: Log‑posterior decomposition in dispersion form | Explanation | Proof of Proposition | 5. Supporting Claims | 5.1 Claim 1 | (1a) | (1b) | (1c) | (2a) | (2b) | (3a) | (3b) | (4a) | (4b) | (4c) | (4d) | (5a) | F_{\Gamma}!\Big(\tfrac{1}{\mathrm{disp}{\mathrm{upper}}};\text{Shape},\text{Rate}\Big)\Bigg) \[6pt]&\quad + \log!\Bigg(F{\Gamma}!\Big(\tfrac{1}{\mathrm{disp}{\mathrm{lower}}};\text{Shape} + \tfrac{n{2}}{2} - \mathrm{lm_log2},\text{Rate} + \tfrac{\mathrm{RSS}_{\mathrm{Min}}}{2}\Big)\[-2pt]&\qquad\qquad\qquad | (5b) | (5c) | (5d) | F_{\Gamma}(\tfrac{1}{\mathrm{disp}{\mathrm{upper}}};\text{Shape},\text{Rate})\Big) \[4pt]&\quad - \log!\Big(F{\Gamma}(\tfrac{1}{\mathrm{disp}{\mathrm{lower}}};\text{Shape} + n{2}/2 - \mathrm{lm_log2},\text{Rate} + \mathrm{RSS}_{\mathrm{Min}}/2)\[-2pt]&\qquad\qquad\qquad | 5.2 Claim 2 | 5.3 Claim 3 | 5.4 Claims 4 and 5 | [g3_{j}(d) = \mathrm{lg_prob_factor1_j} + \text{lmc}_1 + \text{lmc}_2,d.] | By definition of the per‑face UB3A shift,[\mathrm{lg_prob_factor1}_j | 5.5 Claims 6 and 7 | 5.5.1 Notation and setup | 5.5.2 Remarks for Claim 6 | Proof. From the definition of the tangency point in implementationnotation,[\beta_j(d) | A(d)^{-1},\bigl(\bar{c}_j - B_0(d)\bigr),]with(A(d) = P + \mathrm{base_A}/d)and(B_0(d) = \mathrm{base_B0}/d + P\mu.)Rewriting in terms of (t = 1/d) and using (A(t) = P + tQ),(B_0(t) = t,\mathrm{base_B0} + P\mu),we obtain[\tilde{\beta}_j(t) | A(t)^ | A(t)^{-1},\bigl(\bar{c}_j - P\mu - P\hat{\beta}\bigr) | A(t)^{-1} r^{*}_j.]Substituting this into the exact decomposition of Remark 5.5.1 with(\beta = \beta_j(d)) and (t = 1/d) gives[\mathrm{RSS}_j(d) | \mathrm{RSS}_{ML}+(\beta_j(d) - \hat{\beta})^{T} Q (\beta_j(d) - \hat{\beta}) | Proof. For (t > 0), we have(A(t) = P + tQ \succ tQ) in the Loewner order, so(A(t)^{-1} \prec (tQ)^{-1}). Therefore[\tilde{M}(t) | A(t)^ | 5.5.3 Claim 6: RSS decomposition and lower bound | 5.5.4 Remarks for Claim 7 | Proof. From Remark 5.5.2 and the definition of (\mathrm{UB2}_j(d)),[\mathrm{UB2}_j(d) | \frac{1}{2d}\Bigl(\mathrm{RSS}_{ML}+ (r^{}_j)^{T} \tilde{M}(1/d), r^{}j- \mathrm{RSS}{\mathrm{Min}}\Bigr).]Writing (t = 1/d) and using (\tilde{M}(t) = Q^{-1/2}(K + tI)^{-2}Q^{-1/2}),we obtain[(r^{}_j)^{T} \tilde{M}(t), r^{}_j | v_j^{T}(K + tI)^{-2}v_j = g_j(t),]and hence[\tilde{\mathrm{UB2}}_j(t) | Proof. In the eigenbasis of (K+tI), write (v_j) in coordinates(v_j = \sum_i \sqrt | \frac | \frac | Remark 5.5.7 (Critical points of (\tilde | Proof. From Remark 5.5.4 and Remark 5.5.5,[\tilde | \frac | By Remark 5.5.6, we can write[\frac | Remark 5.5.8 (Inflection points of (\tilde | Proof. From Remark 5.5.5,[\tilde{\mathrm{UB2}}_j''(t)= g_j'(t) + \frac{t}{2},g_j''(t)= -2, v_j^{T}(K+tI)^{-3} v_j+ 3t, v_j^{T}(K+tI)^{-4} v_j.]Writing this in the eigenbasis of (K+tI), let(\mu_i = \lambda_i + t) denote the eigenvalues of (K+tI) and write[v_j^{T}(K+tI)^{-3} v_j= \sum_i w_i \mu_i^{-3},\qquadv_j^{T}(K+tI)^{-4} v_j= \sum_i w_i \mu_i^{-4},]for some weights (w_i \ge 0) (not all zero). Then[\tilde{\mathrm{UB2}}_j''(t)= \sum_i w_i\Bigl(-2,\mu_i^{-3} + 3t,\mu_i^{-4}\Bigr)= \sum_i w_i \mu_i^{-4},\bigl(-2\mu_i + 3t\bigr).]At an inflection point (t^{}), we have (\tilde{\mathrm{UB2}}_j''(t^{}) = 0),so[0= \sum_i w_i \mu_i^{-4},\bigl(-2\mu_i + 3t^{*}\bigr).]Define normalized weights[\alpha_i | 5.5.5 Claim 7: UB2 derivatives and endpoint minimization | Any critical point (t^ | 5.5.5(') Corrected minimization for anisotropic (K)
