3
$\begingroup$

Given a standard Gaussian matrix $X\in\mathbb{R}^{n\times d}$, $d<n$, with entries sampled i.i.d. from $\mathcal{N}(0,1)$, is the corresponding orthogonal projection $X X^+ = X (X^\top X)^{-1} X^\top$ a random projector, i.e., a projection in $\mathbb{R}^n$ onto a random $d$-dimensional subspace uniformly distributed in the Grassmann manifold $G_{n,d}$?

Same question if now $X\in\mathbb{R}^{n\times d}$, $d<n$, has rows sampled i.i.d. from a multivariate Gaussian distribution $\mathcal{N}(0,\Sigma)$, where $\Sigma\in\mathbb{R}^{d\times d}$ is a covariance matrix.

If not, can we say anything about concentration bounds on the quantity $\|X X^+ y\|_2$, where $y\in\mathbb{R}^n$ is a given fixed vector?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

$\newcommand\R{\Bbb R}\newcommand\Si{\Sigma}$The answer to your both questions is yes, regardless of what the covariance matrix $\Sigma$ is.

Indeed, let $P_X:=X(X^\top X)^{-1}X^\top$, the orthoprojector onto $V_X:=X\R^d$, the column space of the random matrix $X$, so that $P_X\R^n=V_X$. Then for any orthogonal matrix $Q\in\R^{n\times n}$ we have $P_{QX}=QX(X^\top X)^{-1}X^\top Q^\top$ and hence $$V_{QX}=QX(X^\top X)^{-1}X^\top Q^\top\R^n =QX(X^\top X)^{-1}X^\top\R^n=QV_X.$$ Therefore and because $QX$ equals $X$ in distribution (see the details on this below), it follows that $V_X$ equals $QV_X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$. So, $V_X$ is uniformly distributed (over $G_{n,d}$ if $\Si$ is nonsingular, as it is usually assumed, or, more generally, over $G_{n,r}$ where $r$ is the rank of $\Si$). $\quad\Box$

Details on why $QX$ equals $X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$: Write $X=[X_{ij}]_{i\in[n],j\in[d]}$ and $Q=[Q_{ij}]_{i\in[n],j\in[n]}$, where $[n]:=\{1,\dots,n\}$. Then the entries $(QX)_{ij}=\sum_{k\in[n]}Q_{ik}X_{k,j}$ of the matrix $QX$ are zero-mean jointly normal random variables, and for all $i,i'$ in $[n]$ and $j,j'$ in $[d]$ we have $$E(QX)_{ij}(QX)_{i'j'} =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}EX_{kj}X_{k'j'} \\ =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}1(k=k')\Si_{jj'} =\sum_{k\in[n]}Q_{ik}Q_{i'k}\Si_{jj'} =1(i=i')\Si_{jj'},$$ so that the covariances of the $(QX)_{ij}$'s do not depend on $Q$.

Improve this answer
$\endgroup$
7
  • $\begingroup$ What if $\det\Sigma=0 \text{?} \qquad$ $\endgroup$ Commented Jun 6, 2024 at 20:42
  • 1
    $\begingroup$ @MichaelHardy : I have added a detail on this. $\endgroup$ Commented Jun 6, 2024 at 21:29
  • $\begingroup$ Noted. One should also note that the case $\det \Sigma = 0$ arises very naturally in some contexts, perhaps the simplest of which is that if $X_1,\ldots,X_n\sim\text{i.i.d.} \operatorname N_1(\mu,\sigma^2)$ so that $\overline X = (X_1 + \cdots + X_n)/n \sim \operatorname N(\mu, \sigma^2/n),$ then the vector whose $i$th component is $X_i - \overline X$ for $i=1,\ldots,n$ has a singular covariance matrix, and one uses that to show that $\sum_{i=1}^n (X_i-\overline X)^2 \sim \sigma^2 \chi_{n-1}^2,$ and if$\,\ldots\qquad$ $\endgroup$ Commented Jun 6, 2024 at 21:47
  • $\begingroup$ $\ldots X_i \sim \operatorname N(a + bw_i, \sigma^2)$ and $\widehat a,\widehat {b\,}$ are the least-squares estimators of $a,b$ then $\sum_{i=1}^n \left(X_i - (\widehat a + \widehat {b\,} w_i)\right)^2 \sim \sigma^2 \chi_{n-2}^2$ and that that sum is probabilistically independent of the least-squares estimators $\widehat a,\widehat{b\,}.$ $\endgroup$ Commented Jun 6, 2024 at 21:49
  • $\begingroup$ @MichaelHardy : I agree with your latter comments. $\endgroup$ Commented Jun 6, 2024 at 21:59

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.