\documentclass{article}

\usepackage{hyperref}

\usepackage{amsmath}

\usepackage{amssymb}

\newtheorem{theorem}{Theorem}

\begin{document}

\title{The 2 by 2 Real Pseudo Inverse}

\author{John Mount}

\date{2019-01-07}

This is a brief note on the math behind the direct

PRESS statistic calculation\footnote{\url{http://www.win-vector.com/blog/2014/09/estimating-generalization-error-with-the-press-statistic/}}

found in the RcppDynProg package\footnote{\url{https://github.com/WinVector/RcppDynProg}}.

The actual `C++` code\footnote{\url{https://github.com/WinVector/RcppDynProg/blob/master/src/xlin_fits.cpp}} is a bit ugly and intimidating. That is because we are using a verbose scalar notation to represent matrix concepts. In matrix notation we are solving a linear system by inverting a two by two matrix.\footnote{Yes, there are the usual admonitions that one should not invert a matrix to solve a linear system, but for systems this small they do not apply.} We are in turn inverting the two by two matrix by exploiting the following well know rule of how to invert a two by two matrix.

If $a d - b c$ is not zero then:

Throughout $a$, $b$, $c$, and $d$ all real scalars.

This can be re-written as the following general relation.

The above can be directly checked just by applying the rules for multiplying matrices to the left two matrices. This has a particularly pleasant presentation if we recognize that $a d - b c$ is the determinant of the left matrix and use the traditional vertical bar determinant notation.

Now there is an issue of what to do when $a d - b c$ is zero. For our implementation we apply Tikhonov regularization\footnote{\url{https://en.wikipedia.org/wiki/Tikhonov_regularization}} which (barring the exact numeric coincidence of minus two times the expected value of the independent variable equaling our regularization constant) is going to be non-singular. For our actual application, we could simply switch degenerate situations to the out-of sample mean implementation\footnote{\url{https://github.com/WinVector/RcppDynProg/blob/master/src/const_costs.cpp}}.

But, for fun, let's play with the math a bit.

There is an additional lesser known algebraic relation for two by two matrices.

Or (using transpose, matrix squared Frobenius norm, and determinant notation):

\begin{theorem}

For any real 2 by 2 matrix

$\begin{pmatrix} a & b \\ c & d \end{pmatrix}$ we have:

\].

The superscript "top" denoting the transpose operation, the $||.||^2_2$ denoting sum of squares norm, and the single $|.|$ denoting determinant.

\\ $\square$

\label{thm:fmla}

\end{theorem}

This means, if $a d - b c$ is zero then:

Once we [confirm the above relation\footnote{\url{https://github.com/WinVector/RcppDynProg/blob/master/extras/PseudoInverse.ipynb}} we can also confirm that if $a^2 + b^2 + c^2 + d^2$ is not zero, then the following matrix:

satisfies all of the conditions for being the Moore-Penrose inverse\footnote{\url{https://en.wikipedia.org/wiki/Moore–Penrose_inverse}} (or pseudo-inverse) of our original matrix. The superscript-plus denoting the Moore-Penrose inverse operation.

Or in transpose notation:

This is called the pseudo-inverse because it acts like an inverse, even for non-invertible matrices. For $A^{+}$ to be

a More-Penrose inverse we must confirm it obeys the following relations:

\begin{align*}

\end{align*}

Theorem \ref{thm:fmla} lets us check the first relation, the other follow quickly as our $A^{+}$ is a simple scalar multiple of the transpose.\footnote{With the same scaling for both $A^{+}$ and $(A^{\top})^{+}$.}

All of the above check relations would be true for a classic inverse. We can think of $A^{+}$ as almost canceling a single $A$ to the left or the $A$ to the right.

\begin{theorem}

For any real 2 by 2 matrix $\begin{pmatrix} a & b \\ c & d \end{pmatrix}$

The Moore-Penrose inverse is:

\begin{itemize}

\item When $a d - b c$ is not zero:

\item When $a d - b c$ is zero and $a^2 + b^2 + c^2 + d^2$ is not zero:

\item Otherwise:

\end{itemize}

.

\\ $\square$

\end{theorem}

For general matrices the situation is much more complicated.

The wealth of symmetries and relations is really kind of neat.

\end{document}