These are some interesting matrices. I did not find that $B^n=I_{2d}$ always, though this was true when $n$ was a multiple of $d$.
The case of $r=1$ seemed different from the more general case $r>1$ and I did not consider it in detail since it may not be of interest.
Preliminaries
The matrices $R(\theta ),~\theta \in \mathbb{R}$ comprise a family of
mutually commuting orthogonal matrices with common eigenvectors. The unitary
matrix $U$ with columns of the normalized eigenvectors can be used to
diagonalize any $R(\theta )$, $
R(\theta )=U\Lambda U^{\ast },$ $
\Lambda =U^{\ast }R(\theta )U$, where
$$
\Lambda =
\begin{bmatrix}
\exp (+2\pi i\theta ) & 0 \\
0 & \exp (-2\pi i\theta )
\end{bmatrix}
,~U=\frac{1}{\sqrt{2}}
\begin{bmatrix}
-i & i \\
1 & 1
\end{bmatrix}
\text{.}
$$
Therefore any product of the $2\times 2$ blocks here can be easily
diagonalized,
$$U^{\ast }R(\theta )R(\theta ^{\prime })U=U^{\ast }R(\theta
)UU^{\ast }R(\theta ^{\prime })U=\Lambda \Lambda ^{\prime }$$, which
significantly simplifies the problem.
Let $R_{m}=R(1/m)$, then the powers $R_{m}^{k}=R(k/m)$ have eigenvalues $
\exp (\pm 2\pi ik/m)$. $R_{m}^{k}$ is equal to $I_{2}$ when $k=m$, i.e., the
sequence $R_{m}^{k},~k=0,1,\ldots $has period $m$. More generally the
sequence $\left( R_{m}^{j}\right) ^{k},~k=0,1,\ldots $ has period $\operatorname{lcm}
(j,m)/j\leq m$.
Eigenvalues of $A^{a}$
The eigenvalues of $A$ are the union of the eigenvalues of its blocks, and
therefore are $\exp (\pm 2\pi ir^{k}/m),k=0,\ldots ,d-1$. The eigenvalues of
$A^{a}$ are $\exp (\pm 2\pi ir^{k}a/m),k=0,\ldots ,d-1,$ and these are all
unity for $a=m$, i.e., $A^{m}=I_{2d}$ and the sequence of the $A^{a}$ has
period $m$. Since the first block of $A$ is $R_{m}$ with period $m$, there
cannot be a shorter period for the $A^{a}$.
Period of $B^{b}$
The situation for $B$ is more complicated. The block $R_{n}^{d}$ has period $
p:=\operatorname{lcm}(d,n)/d\leq n$, but the block structure is not diagonal and has
its own period. It is instructive to write the first few matrices in the
sequence $B^{b},b=0,1,\ldots $ for the case $d=4$. ($R_{n}^{d}$ is here
abbrieviated as $R$ and $I_{2}$ as $I$.)
$$
\begin{bmatrix}
I & 0 & 0 & 0 \\
0 & I & 0 & 0 \\
0 & 0 & I & 0 \\
0 & 0 & 0 & I%
\end{bmatrix}%
,~
\begin{bmatrix}
0 & I & 0 & 0 \\
0 & 0 & I & 0 \\
0 & 0 & 0 & I \\
R & 0 & 0 & 0
\end{bmatrix}
,~
\begin{bmatrix}
0 & 0 & I & 0 \\
0 & 0 & 0 & I \\
R & 0 & 0 & 0 \\
0 & R & 0 & 0
\end{bmatrix}
,~
\begin{bmatrix}
0 & 0 & 0 & I \\
R & 0 & 0 & 0 \\
0 & R & 0 & 0 \\
0 & 0 & R & 0
\end{bmatrix}
,
$$
$$
\begin{bmatrix}
R & 0 & 0 & 0 \\
0 & R & 0 & 0 \\
0 & 0 & R & 0 \\
0 & 0 & 0 & R
\end{bmatrix}
,~%
\begin{bmatrix}
0 & R & 0 & 0 \\
0 & 0 & R & 0 \\
0 & 0 & 0 & R \\
R^{2} & 0 & 0 & 0
\end{bmatrix}
,~%
\begin{bmatrix}
0 & 0 & R & 0 \\
0 & 0 & 0 & R \\
R^{2} & 0 & 0 & 0 \\
0 & R^{2} & 0 & 0
\end{bmatrix}
,\ldots
$$
The zero/non-zero block structure has periodicity $d$: $B$ becomes block
diagonal when $b$ is a multiple of $d$. Each such quasi-period gains an
additional factor of $D_{R}=\operatorname{diag}(R,R,\ldots )$ and the true
period occurs when $D_{R}^{p}=I_{2d}$ and $R^{p}=I$. That is, the period of $
B^{b}$ will be $\operatorname{lcm}(p,d)$. If $n$ is a multiple of $d$ then $
B^{n}=I_{2d}$
Eigenvalues of $B^{b}$
The matrix $B$ is in block companion matrix form, with the $2\times 2$
blocks along the last row corresponding to the negated coefficients of the
characteristic matrix polynomial $I_{2}x^{d}-R_{n}^{d}$ .Applying the
unitary similarity with $U$ gives $I_{2}x^{d}-\operatorname{diag}(\rho ,\overline{
\rho })$, where $\rho $ and $\overline{\rho }$ are the eigenvalues of $
R_{n}^{d}$. Taking the determinant gives $(x^{d}-\rho )(x^{d}-\overline{\rho
})=0$ and solving for $x$ gives the $2d$ eigenvalues for $B,$ namely $\omega
^{k}\rho ^{1/d}$ and $\omega ^{k}\overline{\rho }^{1/d}$, $k=0,\ldots ,d-1$
where $\omega $ is the principal $d$th root of unity. Explicitly, these are $
\exp (2\pi ik/d)\exp (\pm 2\pi i/n),$ $k=0,\ldots ,d-1$. It follows that the
eigenvalues of $B^{b}$ are $\exp (2\pi ikb/d)\exp (\pm 2\pi ib/n),$ $
k=0,\ldots ,d-1$. For the case $b\operatorname{mod}d=0,$ the matrices $B^{b}$ are
block diagonal and the first factor simplifies to $1$; the eigenvalues are
then $d$ copies of a complex-conjugate pair.
Eigenvalues of $A^{a}B^{b}$
We first show that the $2d\times 2d$ problem splits into two $d\times d$
problems. Note that $B^{b}$ has one block in each row and column, and
premultiplying by the block diagonal $A^{a}$ multiplies the block in the $i$%
th pair of rows of $B^{b}$ by the $i$th block of $A^{a}$, preserving the
block structure. Performing the similarity $U_{2d}^{\ast }A^{a}B^{b}U_{2d}$
with $U_{2d}=\operatorname{diag}(U,U,\ldots )$ diagonalizes all the blocks, with
the diagonal entries of each block now containing the products of the
respective eigenvalues of the matrix factors in the block.
A permutation
similarity that moves odd rows and columns to the beginning and even rows
and columns to the end generates a block diagonal form with the first $%
d\times d$ block, $W$, having the same structure as $A^{a}B^{b}$ but having
the first eigenvalues from each $2\times 2$ block, and with the second $%
d\times d$ block having the second eigenvalues. Therefore the eigenvalues
of $A^{a}B^{b}$ come in two sets of $d$, with the second being complex
conjugates of the first.
We deal now only with the first $d\times d$ block.
We write $\alpha _{1},\ldots ,\alpha _{d}$ for the first eigenvalues of each
$2\times 2$ block of $A,$ $\lambda _{i}=\alpha _{i}^{a}$ for the first
eigenvalues of the blocks of $A^{a}$, and (as above) $\rho $ for the first
eigenvalue of $R_{n}^{d}$. For example, with $d=4$ and $b=5$ we have
$$
W=
\begin{bmatrix}
0 & -\alpha _{1}^{a}\rho & 0 & 0 \\
0 & 0 & -\alpha _{2}^{a}\rho & 0 \\
0 & 0 & 0 & -\alpha _{3}^{a}\rho \\
-\alpha _{4}^{a}\rho ^{2} & 0 & 0 & 0
\end{bmatrix}
$$
$$
=\operatorname{diag}\left( \lambda _{1},\ldots \lambda _{4}\right) \operatorname{diag}
\left( \rho ,\rho ,\rho ,\rho ^{2}\right)
\begin{bmatrix}
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0
\end{bmatrix}
$$
where the last matrix is $C_{4}^{5}=C_{4}$, the basic circulant permutation
matrix of order $4$, corresponding to the permutation $\sigma =(1234)$. In
the common case, as here, where the permutation has only one cycle, the
characteristic polynomial $\left\vert xI_{d}-W\right\vert $ has only one
term other than $x^{d}$, which arises from the product of all the nonzero
entries, with sign $(-1)^{d}\operatorname{sgn}\sigma =-1$. The characteristic polynomial is
$$
x^{d}-\rho ^{b}\lambda _{1}\lambda _{2},...,\lambda _{d}
$$
so its roots are the eigenvalues
$$
\left( \rho ^{b}\prod_{i}\lambda _{i}\right) ^{1/d}\omega
^{k},~k=0,1,\ldots ,d-1.
$$
where the $\omega ^{k}\ $are the $d$th roots of unity. Using the general
theory below, this case occurs when $d$ and $b$ are coprime.
The product $\prod_{i}\lambda _{i}$ of first eigenvalues of the
blocks of $A^{a}$ can be evaluated using the formula above as
$$
\prod_{k=0}^{d-1}\exp (+2\pi ir^{k}a/m)=\exp \left( +2\pi
i(a/m)\sum_{k=0}^{d-1}r^{k}\right) =\exp \left( +2\pi i\frac{a\left(
r^{d}-1\right) }{m\left( r-1\right) }\right) =1
$$
where the last equality uses $r^{p}\operatorname{mod}m=1$ or $m|r^{p}-1$. For the
second eigenvalues replace the explicit $+$ with $-$; the result is the
same. (For $r=1,$ $\sum_{k=0}^{d-1}r^{k}=d$ and the relationship may not
hold.) That is, the eigenvalues of $A^{a}B^{b}$ are the same as those of $%
B^{b}$ for the common case of a single-cycle permutation.
Another simple case is when $b\operatorname{mod}d=0$. This is the case where $W$ is
diagonal, $C_{d}=I_{d}$, and $B^{b}$ is in block diagonal form. The
permutation has only cycles of length 1 (fixed points). The blocks of $B^{b}$
are all $\left( R_{n}^{d}\right) ^{b/d}=R_{n}^{b}$ with eigenvalues $\exp
(\pm 2\pi ib/n)$ and combine with the respective block sof $A^{a}$ so the
eigenvalues of $A^{a}B^{b}$ are the respective products
$$
\exp (\pm 2\pi ir^{k}a/m)\exp (\pm 2\pi ib/n),k=0..d-1
$$
Here both first and second eigenvalues are included, with upper and lower
signs respectively.
In the general case where the permutation has multiple cycles, the
characteristic polynomial factors in a pattern related to the cycle lengths.
Then some eigenvalues have products of only a proper subset of $\lambda _{i}$
and the eigenvalues of $A^{a}B^{b}$ generally will not be the same as those
of $B^{b}$. The general form is $W=D_{A^{a}}D_{\rho ,b}C_{d}^{b}$, where $%
D_{A^{a}}=\operatorname{diag}\left( \lambda _{1},\ldots \lambda _{d}\right) $ and
$$
D_{\rho ,b}=\operatorname{diag}\left( \rho ^{\lfloor b/d\rfloor },\ldots ,[d-(d
\operatorname{mod}b)\,\mathrm{times}],\rho ^{\lfloor b/d\rfloor +1},\ldots ,[d\operatorname{
mod}b~\mathrm{times}]\right) .
$$
The eigenvalues for the product of a diagonal matrix and a permutation
matrix (a PD or monomial matrix) are discussed in Ref. 1. Working this
through for the case here leads to the following prescription. Find the
cycles in the permutation $(1,2,\ldots ,d)^{b}$. There are $s=\gcd (d,b)$
cycles each of length $l=d/s$. For each cycle of length $l$, there is a set
of $l$ first eigenvalues given by
$$
\left( \rho ^{b/s}\prod_{cycle}\lambda _{i}\right) ^{1/l}\omega
_{l}^{k},~k=0,1,\ldots ,l-1.
$$
where the $\prod_{cycle}\lambda _{i}$ means the product of $
\lambda _{i}$ with subscripts that are the $l$\ numbers in the cycle, and $
\omega _{l}=\exp (2\pi i/l)$. As before the second eigenvalues are the
complex conjugates of these.
- P.J. Davis, Circulant Matrices, Wiley, 1979, Sec 5.3, p. 166.