Isotropy groups of the action of orthogonal similarity on skew-symmetric and on complex orthogonal matrices

We compute and analyze isotropy subgroups of the complex orthogonal group with respect to the similarity transformation on itself and on skew-symmetric matrices. Their group structure is related to a group of certain nonsingular block matrices whose blocks are rectangular block Toeplitz.

💡 Research Summary

The paper investigates the isotropy (stabiliser) subgroups of the complex orthogonal group (O_n(\mathbb C)) under two natural actions: (i) orthogonal similarity on the space of skew‑symmetric matrices (\mathrm{Skew}n(\mathbb C)) and (ii) conjugation on the group itself. For a matrix (M) (either skew‑symmetric or orthogonal) the isotropy group is defined as (\Sigma_M={Q\in O_n(\mathbb C)\mid Q^TMQ=M}). The authors first recall the canonical Jordan‑type forms for skew‑symmetric and orthogonal matrices under orthogonal similarity, noting that in the generic situation where all eigenvalues are simple the isotropy groups are straightforward products of copies of the two‑dimensional complex rotation group (SO_2(\mathbb C)) (Proposition 2.1). Specifically, for even (n) one obtains (\Sigma_M\cong\bigoplus{j=1}^{n/2}SO_2(\mathbb C)); for odd (n) an extra factor ({\pm1}) appears.

The main contribution concerns the non‑generic case, i.e. when eigenvalues have multiplicities or zero eigenvalues occur. To describe the resulting stabilisers the authors introduce a new class of block matrices (T_{\alpha,\mu}). Here (\alpha=(\alpha_1,\dots,\alpha_N)) is a strictly decreasing sequence of positive integers describing block sizes, and (\mu=(m_1,\dots,m_N)) records how many copies of each size appear. An element (X\in T_{\alpha,\mu}) is an (N\times N) block matrix whose ((r,s))‑block is a rectangular upper‑triangular Toeplitz matrix of size (\alpha_r\times\alpha_s) (or its transpose when (\alpha_r>\alpha_s)). The diagonal blocks are themselves Toeplitz matrices built from a sequence of matrices (A_{rs}^{(j)}). Proposition 2.2 shows that (T_{\alpha,\mu}) is a group, in fact a semidirect product (D\ltimes U) where (D) consists of block‑diagonal matrices and (U) is a unipotent normal subgroup of unitriangular Toeplitz blocks. Additional constraints (I)–(III) enforce pseudo‑orthogonal relations with respect to certain symmetric or skew‑symmetric matrices (B_r).

Theorem 2.5 is the central result. Part (1) treats non‑zero eigenvalues (\lambda\neq0). For each such (\lambda) the isotropy groups of the corresponding skew‑symmetric block (K_\lambda) and orthogonal block (O_\lambda) coincide and are conjugate to the subgroup \